Pre-Processing HELP! PLEEEEASE!!

[cdavid]

HI Everyone....I am utterly confused and frustrated.

I use a DSLR Canon 1000D which has been modified.

I have created (as described by Vincent Peris) a Master Bias for both my ISO 800 and ISO 1600 shots, both Master Bias' look right.

I take my darks immediately after my imaging session (sometimes the temperature is slightly higher due to coming daylight)

So...

When I preprocess my lights using my Library MasterBias, the new darks (Bias subtracted), and Flats etc...

I at times get the no correlation between master dark and target.....or it seems to proceed normally but then the image is horribly noisy.

I have read more posts on this than I can stand and hence I am now totally confused....so questions that I hope someone can answer:

1. When saving the master bias- should it be saved as 32 bit floating, 16 bit Integer, signed, fixed? (I read a post where the raw dslr images were 16 bit and the MasterBias 32 bit Floating and this led to issues)

2. Similarly...the darks and flats....16 bit Integer, 32 bit floating???

3. When using the Masterdark should the optimized box be unchecked or checked?  I tend not to get the correlation warning when unchecked.

Any response would be greatly appreciated.....thanks in advance!

Carlos

[Don]

Hi Carlos,

When I preprocess my lights using my Library MasterBias, the new darks (Bias subtracted), and Flats etc...

There are different ways to do the calibration, but the "normal" way it's done in PixInsight is to NOT subtract the bias from the darks when creating the master dark - just integrate the dark frames with no prior calibration.  That way the master dark includes the bias signal.  This is how it's done if you use the BatchPreProcessing script, for example.  If you do subtract the master bias from the darks before or after integrating them, you will need to subtract the master bias from the lights during the light calibration step too. Otherwise you just subtract the non-bias corrected master dark from the lights, and the bias signal is subtracted automatically.  Either way, the master flat MUST be bias corrected, and I believe this is done in BatchPreProcessing before the individual flats are integrated.

I at times get the no correlation between master dark and target ...

It is pretty normal to see this message when the master dark is subtracted from the flats, assuming your flats are much shorter exposures than your darks.  It indicates that the noise (hot pixels, etc.) found in the master dark was not found in the flat.  If this message occurs when the master dark is subtracted from the lights, it indicates something is not right ...

When saving the master bias- should it be saved as 32 bit floating, 16 bit Integer, signed, fixed? (I read a post where the raw dslr images were 16 bit and the MasterBias 32 bit Floating and this led to issues)

The master bias should be saved as 32 bit floating point.  It's OK if your raw DSLR images are in a different format (I think the 1000D uses 14 bit integer).  Be sure to select 32-bit floating point in the Sample format field of the Output Files section in the ImageCalibration tool.  You can also tell ImageCalibration that the lights are raw files but typing "raw cfa" (without the quotes) in the Input hints field of the Format Hints section of ImageCalibration, but this doesn't seem to be strictly necessary.  The raw DSLR files will be calibrated properly and from that point on the intermediate files will all be 32-bit floating point.

Similarly...the darks and flats....16 bit Integer, 32 bit floating???

All calibration files should be saved as 32-bit floating point, but again it's OK that your raw DSLR image files are not.

When using the Masterdark should the optimized box be unchecked or checked?  I tend not to get the correlation warning when unchecked.

Dark optimization is sort of an advanced subject, probably best to leave the box unchecked for now.  If you check it, leave the Optimization Threshold set at zero for now.  Using a non-zero Optimization Threshold requires that you first determine the correct non-zero value, as described by Juan in this thread:  http://pixinsight.com/forum/index.php?topic=7365.msg49400#msg49400

Also, if you haven't already, make sure your DSLR_RAW preferences are set as in this screenshot:  http://pixinsight.com/forum/index.php?action=dlattach;topic=2570.0;attach=6941;image.  To set the DSLR_RAW preferences, go to the Format Explorer, click on DSLR_RAW in the format list, and click the Edit Preferences button at the bottom of the panel.

Don

[cdavid]

Thank you!!!

So then let me make sure I have this.....if I use a "non bias subtracted Master Dark" and a "bias subtracted Master Flat" to calibrate my light frames....I DO NOT need to also calibrate the light frames with the "Master Bias"?

.....so then you only really need the master bias to calibrate the flats?

Thanks for the other info..much clearer now.

Carlos

[Don]

Carlos, before someone else does it, let me correct myself.

I just ran BatchPreprocessing and examined the log.  The dark calibration does not appear to use the master bias, i.e., the master dark is not bias corrected.  And the flat calibration does use the master bias, i.e., the master flat IS bias corrected.

But the master bias is also used in the light calibration - so BatchPreprocessing seems to be subtracting the master bias from the master dark and from each individual light frame during light calibration.  I confirmed this by running BatchPreprocessing again with previously prepared master flat and master bias files, using an intentionally wrong master bias (wrong geometry, produced with frames made on a different camera).  BatchPreprocessing proceeded through the dark calibration without any errors (i.e. it doesn't use the master bias for dark calibration), then failed with a "wrong geometry" message at the beginning of light calibration.

Like I said (and as I'm sure you know), there's more than one way to do it, but this seems to be the way PI does it if you use the BatchPreprocessing script.  By the way, I recommend that you DO use BatchPreprocessing, as it saves time and simplifies the process, and greatly simplifies the Bayer Drizzle process if you want to use it.  Just feed BatchPreprocessing all of your raw DSLR files, pick a registration reference image and an output folder, and let it go.

Don

[cdavid]

Thanks Don....

I guess thats what i'm running into....I used batch preprocess with a Master Bias and Master Flat and fed it the raw darks and lights and the finished output was awful....noisy, signal lost....

When Ive run it manually using the individual processes (calibrate, integrate, etc) according to Vincent tutorials I've gotten some lousy results and one good one on the same imaging data set...ive run it so many times with different parameters that i'm now all confused.  I did get one great result but stupid me forgot to write down what I did so I am back to the drawing board.

I am currently running several instances with and without the Master Bias, Master dark or raw darks, etc....Trying to determine which method works....

[Don]

Well, good luck with it Carlos.  Perhaps if you have a way to upload your raw data someone here can help you sort it out.

I think most of us stumble a bit at first with PixInsight.  I'm trying to hone my skills after two years, and I appreciate the chance to work with someone else's data occasionally, especially DSLR data since that's what I use.

Don

[cdavid]

Thanks....actually Ive  been using pixinsight for close to three years.  I've had on and off trouble with the calibration.  Each time I think I have the process consistent I'm wrong.  I am starting to get suspicious that the problem is my "Darks"

I'll report back my results using the various permutations Ive been running all afternoon.

Carlos

[cdavid]

Ok....12 hours later.....

So I ran every permutation I could think of in both BPP and using the module manually as I have in the past.  I just couldn't figure what was happening....  Ive had success with PI in the past with just occasional glitches like this...but this was different.

I suddenly stumbled on the fact that this is a new version on my computer.  I used to run the 32 bit version....now the 64.  So was I assuming that the preferences, etc. were all the same.  What clued me in was one image that I had processed earlier this morning that seemed right (processed as a raw) but it was upside down compared to all the other failures which were converted to Fits first.

So I looked at the FITS preferences.....it had the "bottom up" setting chosen.  Now I am not sure why this makes such a big difference, but when I changed it to "up bottom" it fixed everything!  See images below:  What I was getting all day...weak signal coming through after calibration/integration vs. nice strong signal in the image after changing that preference.

Just in case you are wondering why they are monochrome.  These are Ha images that I debayered, extracted Red channel and aligned and integrated, after the calibration.

Thanks for your help earlier. 

[pfile]

it matters because the bayer matrix is not symmetrical (RGGB) and so the direction you read the file matters. unfortunately there's no good way for the software to figure out on it's own that the reader direction is backwards from the writer direction, so this problem crops up with DSLRs/OSCs from time to time.

rob

[cdavid]

LOL!  Just wish I had recognized this earlier in the day.....12 hours of utter frustration were almost too much!