32 bit BatchFormatConversiion of raw Bias subs

[Cleon_Wells]

I created 40 new 32 bit Bias .fit sub with BatchFormatConversion set to (32bit IEEE 754 floating point) yesterday and integrated a new MasterBias, I also recalibrated my Dark and Flat subs with this new MasterBias and integrated a new Master, Dark and Flat file. The one measurable difference I see in IC , see attached image,  is the amount of the (master Flat scaling factor ) is slightly different and I also see a small improvement in HotPixel reduction in my Lights after Light Calibration. Is this a better way to do the BFC of the Bias sub or have I fooled myself with a error in my process?
      The attached image also points out the difference that temperature makes during IC in the Dark scaling factors. I used these Dark subs for this test because of the wide temperature change, ( C13 to C26 over 28 minutes), in my incorrect  acquiring of these 120 sec Darks. I assume the perfect Dark scaling factor is 1.000, I deduced my integrated MasterDark temperature from the IC Dark scaling factor numbers.

I still can’t use any Rejection during integration of the Dark_c subs and I even tried different Dark_c subs and still get large Dark scaling factors when calibrating with WS 4_3 rejection on. I have had success with Dark_c integration with WS 4-3 rejection in the past.
Cleon

[astropixel]

Cleon. I have to say that I have not encountered problems with calibration for some time now. Careful to acquire calibration frames under similar conditions to the lights (dithering is also a factor - without dithering I have found that bad pixel rejection is not too successful).

Recently, I have taken Craig Stark's advice and sealed the device I use for taking flats and darks and bias frames with aluminium/aluminum cooking foil. And while it may be my imagination the results are better. Lens caps just don't cut it. Craig Stark recommends covering the lens cap with foil. I can't wait for the replies on that one...

Scaling factors are around 0.8 - 0.75 for darks. As I recall about 0.4 for flats.

Cheers

Rowland

[Cleon_Wells]

Rowland, I’ve spent a couple of hours talking to Craig over dinners we've had after the last two Chicago, Mid West Astro-Imaging Conferences and have listened  to his presentations (very impressive), so if he has said foil helps with the calibration files, I say --- where’s the FOIL…. LOL
I’m down in Nashville TN right now visiting are two granddaughters so I’ll continue with the 32bit bias BFC idea on Monday.
Cleon

[Cleon_Wells]

Rowland, I took a set of new raw Bias subs (41) the other day and did the 32, 16 bit BFC test and did not see a difference in the MasterBias, so I was fooling myself.
I still can't use the WS 4_3 & NoNorm settings when integrating my calibrated dark subs (11) for the MasterDark file, the integration process takes 10 times longer with large rejection numbers (like 10%) and creates a MasterDark that doesn't work, if I use a No Rejection setting the MasterDark integration works, if I integrate the same  dark subs uncalibrated, WS 4_3 & NoNorm settings, the MasterDark, with Dark Calibrate checked works.
If I use Sigma 4_3  & NoNorm setting, it works normally, if I use WS 4_3 and scale+zero offset Normalization setting  the MasterDark integration also works normally.   
Cleon

[astropixel]

Hi Cleon. Looking at all of that, it appears that there are different means for processing darks depending on the number. Perhaps with a small number of darks pixel rejection is detrimental, and the reason that the process icon tutorial (using only 3 darks) recommends no pixel rejection - there is no explanation as to the rationale.

From what I understand of your experiments, 'No normalisation' is not the issue, it's the use of pixel rejection that has an effect depending on whether the darks have been calibrated - If the darks are not calibrated prior to integration, then it is possible to use pixel rejection successfully.

Is that because the bias signal is still there, perhaps that makes a difference during pixel rejection. I suspect that would work with any number of frames, but you need lots more to use pixel rejection on calibrated subs.

Not sure it this is helpful?

Rowland

[Cleon_Wells]

Rowland, well I finally figured out my DSLR Dark calibrated sub integration problem  when pixel rejection is set to WS 4_3 and No Normalization.
You should edit your post on creating Calibration files, showing how to change the Output Pedestal to eliminate negative numbers during pixel rejection with the Dark calibrated subs.These subs have very low signal level after the Bias is removed.
Cleon

[pfile]

what are the symptoms when you should have had a pedestal and did not?

[astropixel]

You should edit your post on creating Calibration files

I wish it were my post. Don't shoot the messenger.

Hi Cleon. I'm glad you have it sorted. Very useful. I'm not sure that anyone else has noticed or raised the issue.

As pfile is asking. It will be helpful to understand how you identified the issue. Perhaps provide guidelines as to the causes. Is it related to particular conditions - exposure time of the darks, or the number of darks - how is that related to bias subtraction and integration. Just so it's not an orphan statement.

Rowland.

[Cleon_Wells]

In this image I'm   calibrating  15-120sec_iso400 Dark subs for integration, making the MasterDark. The calibration Output pedestal (DN) is set to zero, this is the default setting and I have produced MasterDarks from calibrated Dark subs with this (DN) setting in the past. The pixel rejection counts are high and the integrating row process takes about 10 times longer then normal, I assume this is caused by negative numbers in the pixel rejection math.

[Cleon_Wells]

In this image I'm calibrating a light with the defective MasterDark, note the large Dark scaling factors and what the DeBayered  calibrated light looks like.

[Cleon_Wells]

Here's a MasterDark created with the same Dark subs calibrated with the Output pedestal (DN) set to 15 and the resulting MasterDark15DN calibrating the same light. Note the reasonable dark scaling factors. I have wondered how you could subtract a MasterBias created from 40+ subs from one Dark sub and not have some negative results in some of the pixels.
pfile ask

what are the symptoms when you should have had a pedestal and did not?

I found out that the Output pedestal (DN) would fix my problem  (a defective MasterDark), after hours of experimenting trying everything I could think of, like taking my Canon T2i, used for family pictures and creating 40+Bias subs and 15-120sec Dark subs to see if there was something wrong with my Hap mod T1i. It was after the T2i created the same calibrated Dark sub integration problem I had with the T1i that I started to look at the negative pixel math problem and then held my mouse pointer over the Output pedestal (DN), see extra attached image. The main reason I was looking for the negative math problem was that I could get WS 4_3 to work if I set normalization to scale="zero offset".
By the way; does anybody know what (DN) stands for?

Cleon
PS from now on when calibrating my Dark subs I'll always set the Output pedestal (DN) to 15, I tried a (DN) of 5 but it didn't fix my problem

[astropixel]

Thanks Cleon, great work  8) I will try the same and experiment with some short exposure darks.

I have left the issue open, in the comments added to the DSLR_RAW workflow, to let users decide, because I'm not convinced that increasing the pedestal is always necessary. I do think however, that short exposure darks are most affected by bias subtraction - that has been my experience - though I never identified the issue, as you have clearly done.

I remain convinced for the time being that longer and more exposures, with DSLR cameras that are not cooled, is necessary for producing quality master calibration frames.

[Cleon_Wells]

Rowland, my light frames for ngc7635 were taken at 120sec iso400 with the Hap mod T1i on the 10" RC, the date was 10/2/11 with a waxing crescent Moon and the lights of the Chicago suburbs to the North, an almost idyllic evening, of course the seeing was bad and is never good . 8)
I took these 50-120sec subs with a 10sec separation, after Registration with a 0.7 distortion setting 46,  I took the 15-Dark subs, the same duration as the lights,  about 10 minutes after the last light frame, (my bad) , note I don't say science frames, remember I'm entertaining my 69 earth/orbit old self, while trying to come up with something to impress  my wife. .
I have just purchased a HA clip on for the camera and hope to get more of the surrounding nebula and with the help of some of you , process and add the HA component  to what I've got , creating more/better wife impressing stuff......
Getting back to the Calibration Output pedestal (DN). from what I can tell ( which isn't much ) a (DN) of 15 ( remember this is 15 out of 65,535 ) reduces the possibility of the WS 4_3/no normalization settings from negatively effecting the integration of the Dark DSLR calibrated subs.
Cleon

[pfile]

i'm still trying to understand what the effect on the calibrated light would be. does it just fail to calibrate, or are there artifacts in the calibrated light, or? what is defective about the master dark? i want to check if i've been having this problem and did not know about it. if it only manifests as increased noise in the calibrated lights, it would be hard to notice this.

"DN" just means "data number" and i guess people use this to refer to the output of an astronomical CCD. raw CCDs will give you 16-bit integers as output. so yes, 15 is 15 out of 65535 and is pretty much black. DN is used synonymously with "ADU" or Analog (to) Digital (converter) Units.

[Cleon_Wells]

pfile, here's an example of one light calibrated with the Defective MasterDark and then with the MasterDark created with the OutPut pedestal set to 15. I don't understand what the WS and no normalization rejection is doing, but it seems to me that  DSLR Dark subs may work better with some small amount of Output pedestal used when Calibrating the Dark subs.
Cleon