BPP master dark expectations (flats overcorrecting)

[Greg Schwimer]

I've found a few threads that loosely indicate what I think I've found. This thread is in part for ease of future discovery by others, and in part for my own knowledge. If others would please confirm what I'm seeing it would be greatly appreciated.

Regarding BPP - it seems that it expects a non-calibrated master dark (i.e. not bias subtracted).

I've been manually preprocessing my data for a while. I have a master library that I reference frequently.  For the first time in a while I ran BPP. I fed it my master frames and lights and let it rip. The result was I suppose you might call an overapplicaton of the flat master to the lights. That is - my master flat (which was created by BPP in the first place) grossly overcorrected the light frames. I ran through the process manually using the BPP generated master flat - no overcorrection.

I was able to reproduce the overcorrection by checking the "calibrate" box for the master dark in ImageCalibration. I also found BPP outputs the following further confirming what I'm seeing:

Code: [Select]

Applying bias correction: master dark frame ...
The result of subtracting bias twice from the master dark seems to push the flats to overcorrect.

So it seems that if you wish to use your master frame library for both BPP and manual preprocessing you should at the very least not calibrate your master darks. I see no log indication of similar behavior of BPP with master flats so I'm not sure if it's doing the same to those. Anyone know?

[pfile]

that's correct re: darks. on the other hand, master flats get calibrated before being written out. i think that's to support DSLRs where you might have made your flats at a different ISO than the lights. so you can do a run where you just create a master flat with the appropriate ISO darks/bias and then do another run against the lights - in that run the master flat is not touched.

rob

[Greg Schwimer]

Thanks for confirming Rob!

It would be great if BPP had a way of knowing what the state of the master dark is and acting accordingly. The appropriate fits headers would probably need to be in place to indicate this. I don't think ImageIntegration does that.

Of course, an alternative would be to let the user decide via a switch in BPP.

[msmythers]

Just to bring up one thing that Juan has said at times in the past is that BPP is not and should not be considered for final production work. It is here has a way to quickly inspect an image. All final processing should be done through the standard tools where you have full control.



Mike

[Greg Schwimer]

Just to bring up one thing that Juan has said at times in the past is that BPP is not and should not be considered for final production work. It is here has a way to quickly inspect an image. All final processing should be done through the standard tools where you have full control.

Mike

Understood and agreed. I rarely use BPP. It does seem it is useful to speed up the calibration process at the least, but that implies that the libraries all work well together.

[pfile]

Just to bring up one thing that Juan has said at times in the past is that BPP is not and should not be considered for final production work. It is here has a way to quickly inspect an image. All final processing should be done through the standard tools where you have full control.



Mike

well i think he was talking about the ImageIntegration at the end of BPP, not BPP itself. BPP for calibration and registration is totally fine for production work...

rob

[andreasmax]

found this thread too late for me... had the same issues:

because of a weak battery i was unable to do the darks at my last session and decided to use the masterdard from a session some weeks ago take at the same temp.
after using the BPP script i got overcorrecting master lights. used a masterbias (out of 200 files), masterbias, flats and lights.
took some time for me to come to the same conclusion than you, schwim, that BPP does a processing to the masterdark.
after that i did a stack with the original dark files and the result was as expected and good.

would be great to have the ability to uncheck the processing of masterflats inside the BPP script.

cheers from austria,
andreas

[tsaban]

I can also confirm this (I beleive recently introduced) bug in BPP. I lost several hours wondering what went wrong. Actually I had processed my frames manually and wanted to try the BPP script for the benefit of easy bayer drizzle application. The manually calibrated frames were correct, but those calibrated by BPP had wrongly applied flat fielding.

Swimm, thanks for the Information, I will check if that helps in my case.

Clear skies!
Tahir

[vicent_peris]

Hi,

If you're using master calibration frames to run the BPP script, the state of these frames should be as shown below:

- Master bias: A simple integration.
- Master dark: Also a simple integration. We don't remove the bias from the dark. This is very convenient since you preserve the bias signal in the master dark. That allows you to run BPP without biases, if you're not scaling your darks.
- Master flat: Completely calibrated.

If you manually generate your masters and you calibrate the dark frames before integrating them, then you'll have problems when running BPP.

On the other hand, I think BPP is a powerful tool in its current state. I still remember when I gave my first US workshop at the Adler Planetarium, many years ago. I was teaching how to manually calibrate everything. I was deeply wrong (although at that time we didn't have the BPP script!). Right now, if you care about the critical step in the automation (the integration of the master light), BPP will give you very good results with a big benefit: it will optimize your working time! This is really important. Our time is always limited. A combination of automation and manual control of the critical parts is almost always the best option IMO. Think about this: you can have really bad results even by running a completely manual procedure if you don't know what to look at the rejection maps.


Best regards,
Vicent.

[tsaban]

Thanks for the explanation Vincent.
In my case I made an error during taking flats and used the master flat of an older session which was not calibrated.

Regards
Tahir