How does Image Calibration scale darks if Bias is 0.000 second?

[topboxman]

How does Image Calibration scale darks if Bias is 0.000 second?

That's how long my bias subs are: 0.000 second. Would it be better if each bias sub was something like 10 msec so that it's easier to scale darks?

Peter

[NGC7789]

It is my understanding that darks are scaled based on their noise analysis and not their metadata (ie exposure, temperature, etc). But even if that's not the case it's the dark that is scaled not the bias. In order for a dark to be scalable it must have the bias subtracted first.

Bias frames should be the shortest possible exposure.

[topboxman]

Thanks for the clarification.

Lately I've been using Nebulosity for capturing images and pre-processing (calibration only) and all other processings are done with PixInsight. My cameras have been Sony based CCDs and don't require dark subtraction so I use Nebulosity for Bad Pixel Mapping and calibrate with bias and flats also with Nebulosity.

My new camera will be QSI 683wsg which contains Kodak KAF-8300 CCD. This will be my first Kodak chip and I understand that Kodak CCDs require dark subtraction. I am trying to learn PixInsight Image Calibration but I am not successful calibrating flats. I followed PixInsight's tutorial:

http://www.pixinsight.com/tutorials/master-frames/en.html

Using procedures described from Master Flat Generation (I) section, it always removes too much data from flat subs using Master Dark and Master Bias. It's almost as if it's calibrating with 10 minute darks instead of scaled darks. I have been practicing Image Calibration using images taken with Atik 460EX (Sony ICX694) camera. I am not sure what I am doing wrong. I have no issues running Image Integration of Bias and Dark subs to create Master Bias and Master Dark. I also have Warren Keller's video tutorial but my Image Calibration of flat subs are still unsuccessful.

Image Calibration of flat subs seems to work if I put Master Bias image in Master Dark section of IC and checked "Calibration" box and uncheck "Optimize" box. Master Bias section of IC was disabled. I am at work right now so I am not 100% positive this is what i did to make IC of flat subs successful.

I appreciate any help and suggestions.

Thanks,
Peter

[pfile]

the sony ccds have really small dark current. that's why people don't use darks, there's just not that much dark signal to remove.

imho in that situation there's no reason to calibrate the flat subs with any darks, short or long. just calibrate the flats with a master bias frame and you should be good. with that CCD PI may be having a problem getting a handle on the dark signal in order to scale it properly.

even with my STT-8300M which has the same kodak sensor as your new camera, i don't do any flat darks. i just calibrate the flats with bias frames and the flat scaling is always perfect.

rob

[topboxman]

Hi pfile,

I tried calibrating flat subs with Master Bias and it still removed too much data from flat subs. Did you use Master Bias image in Master Bias or Master Dark section of Image Calibration? Can you give me step by step procedure of how you got flat calibration to work with Image Calibration?

What about calibrating with lights? Do you take darks for dark calibration with lights or use bad pixel mapping (or cosmetic correction) with KAF-8300?

FYI, I never take or use flat darks. I was using 10 minute Master Dark (because my light subs were also 10 minutes) along with Master Bias to calibrate with flat subs and enabled "Optimize" so the Master Dark will be scaled. Is this wrong?

Thanks,
Peter

[Juan Conejero]

That's how long my bias subs are: 0.000 second. Would it be better if each bias sub was something like 10 msec so that it's easier to scale darks?

Not at all. The bias signal pedestal can only be characterized by an exposure of insignificant (ideally, zero) duration. If you expose longer, you'll get some thermal noise.

Anyway, our dark scaling routine knows nothing at all about exposure times, temperatures, and other acquisition conditions. They are simply irrelevant because our solution is purely numerical. If you are interested in a detailed description of our dark frame optimization algorithm, here is one I wrote one year ago:

http://pixinsight.com/forum/index.php?topic=4161.msg29156#msg29156

[topboxman]

Thank you Juan. Now I need to figure out what I am doing wrong with calibrating flats using Image Calibration tool.

Peter

[pfile]

Hi pfile,

I tried calibrating flat subs with Master Bias and it still removed too much data from flat subs. Did you use Master Bias image in Master Bias or Master Dark section of Image Calibration? Can you give me step by step procedure of how you got flat calibration to work with Image Calibration?

What about calibrating with lights? Do you take darks for dark calibration with lights or use bad pixel mapping (or cosmetic correction) with KAF-8300?

FYI, I never take or use flat darks. I was using 10 minute Master Dark (because my light subs were also 10 minutes) along with Master Bias to calibrate with flat subs and enabled "Optimize" so the Master Dark will be scaled. Is this wrong?

Thanks,
Peter

'

i use the master bias section. i can post a screenshot when i'm back to my other machine.

it's not wrong to scale a master dark, but sometimes i wonder if it can be scaled properly for such short exposures. usually you find people making 1800s darks and then scaling them down to 600s or thereabouts for their lights. flats are so short though, that the amount of dark signal present is already pretty low. i trust that PI does it right, but i've found that calibrating with a bias is "good enough".

for my lights, i usually just use a master dark. dithering helps me reject any residual hot pixels. however, with my widefield setup i can't dither, so i usually inspect the calibrated frames and if i see hot pixels around i'll use CosmeticCorrection to remove them.

rob

[topboxman]

I am finally home from work now.

The only time I can successfully calibrate Flat subs with Master Bias if I use Master Bias image in Master Dark section of Image Calibration AND check BOTH "Calibrate" and "Optimize" boxes.

Exactly the same thing is happening when calibrating Dark subs with Master Bias.

It's almost as if there is a bug in Master Bias section of Image Calibration.

Calibrating Flat and Dark subs using same Master Bias (stacked by PixInsight Image Integration) works perfectly with Nebulosity software.

If you are using the latest PixInsight 1.8 with latest upgrade, could you please try it with your images. Just calibrate one flat sub with Master Bias only using Image Calibration tool.

Thanks,
Peter

[topboxman]

Another thing I noticed.

When using Master Bias section of Image Calibration for calibrating flat subs, it resulted slightly different size output files. For example all original (uncalibrated) flat subs sizes are 23,617KB and the calibrated file sizes are 23,620KB.

But if I use Master Dark section of Image Calibration, calibrated files are exactly the same size as uncalibrated files.

Likewise when calibrating with Nebulosity, both uncalibrated and calibrated file sizes are the exactly the same.

Shouldn't file sizes be the same for uncalibrated and calibrated files?

I uploaded sample files for you to evaluate:

Master Bias: https://www.dropbox.com/s/spakin2az4nnwta/Master_Bias.fit
Single Flat sub: https://www.dropbox.com/s/1wekbsdnw0hmf8g/Bubble_Flat_L_001.fit

Notice both files are exactly the same size.

Peter

[pfile]

the master bias frame does not look like a bias frame to me. how was it constructed? the levels are just way too high. a bias frame is usually completely black until an STF has been applied.

for comparison here is my 200 frame master bias from my 8300M:

https://www.dropbox.com/s/u1z7jcgndddudf6/ori200.SuperMasterBias%280.0S-30X1%29_200f.fit

so i think we have to debug the master bias construction. that's why your flat is getting hammered.

rob

[topboxman]

Looks like you are correct. When I build a Master Bias with Nebulosity, it builds a perfect Master Bias. I need to look at what I did wrong with PixInsight Image Integration of Bias subs to create a master Bias.

Peter

[pfile]

were the bias subs somehow touched by another application? what format are they in? they need to be i16 (and not 16-bit integer data represented as floating point numbers) in order for PI to read them with no help. otherwise you have to go into the fits reader and make sure the data ranges are set up right.

rob

[topboxman]

I found the problem but I am not sure how to fix it.

Take a look at a single sub of Bias taken with Atik 460EX and Nebulosity:

https://www.dropbox.com/s/0qmbh9xyfdxg6oy/Atik460EX_Bias_001.fit

According to Nebulosity histogram, it looks perfect with a ADU range of 232 to 943.

But PixInsight seem to hate this file and the midpoint is around 32,000!!!!!!

This was just one night. Other nights, bias looks perfect with both Nebulosity and PixInsight. I usually take Bias and Darks after every imaging session probably because I have OCD. It may not be necessary but I always make sure I take flats for every single imaging session.

Bottom line I found the source of the problem. Something about the way Nebulosity saved the bias subs that PixInsight did not like for just one imaging session.

Thanks for your help.

Peter

[topboxman]

I figured out how to fix it. Originally I captured and save Bias subs in 32 bit floating point with Nebulosity and that cause PixInsight to hate the files.

Then I saved the same original files as 16 bit unsigned integer with a different file names using Nebulosity and PixInsight was finally happy!!!!!

From now on I will make sure to save all captured files as 16 bit unsigned integer with Nebulosity. I always save PixInsight processed files (including calibrated files) as 32 bit floating point. There is no need to save captured files in 32 bits anyway.

Whew!!!!!

Peter