Image calibration with flats changed behaviour

[mxpwr]

I have used my combination of bias, dark and flat frame to calibrate hundreds of raw images (canon 550d).
While occasionally the image calibration seemed to randomly over compensate the flattening, I could usually just clear the process or restart pi to get the expected behaviour.
But now I can't get it to work anymore (no change in optics).
Even with images that I could flatten 2 weeks ago, I get strange overflattened images after the calibration now (edges have higher intensity than center).
I noticed some updates the last time I started pi and was wondering if something has been changed, like a default setting or something?
Any other pointers towards fixing this?

[pfile]

it could have something to do with this if you have recently upgraded PI and are using calibration masters made with an older version of PI

https://pixinsight.com/forum/index.php?topic=14219.0

rob

[mxpwr]

Hi Rob,
That's not the issue. I did indeed encounter the problem late last year that my calibration frames had the wrong size suddenly, but I used a more pragmatic way to solve this; I cropped the two top lines.
Yes, dirty, but it does the trick.

Anyway, the flat frame in question (I actually tried different ones) have been created afterwards and have the proper dimension.

On a side note, I tried the new weighted preprocessing script a while ago and had the same issue that flats over-flat the image and stopped using it, but now the calibration does the same...

[pfile]

i suppose if you describe how you calibrated the flat subs and also maybe post the master dark, master bias, master flat and a light sub someone can take a look.

rob

[mxpwr]

I uploaded the calibration frames, the raw image, the previously correctly calibrated version of that image and the now over-flattened image of it here

https://www.dropbox.com/sh/7qyb0no2srv7xv9/AACOUxDwl0O_uoVAet84xgXFa?dl=0

I created the masters following the instructions on vortex

https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html

This has yielded very good results in terms of correcting the vignetting in the past with dozens of different image sets.

[dld]

After a quick look at your files, here are my observations:

Your dark is zero-clipped. It contains a lot of zero values probably because you probably pre-calibrated your darks. Please read Bernd's guide: https://pixinsight.com/forum/index.php?topic=11968 and also take a look at Vincent's tutorial: https://www.pixinsight.com/tutorials/master-frames/.
 
Why are you cropping your frames (by 1 row of pixels)? This scares me a lot

[mxpwr]

I clipped the top 2 lines after the dcraw change.
I don't have the raw data for the master dark/ bias anymore and I'm lazy.
I still have the uncropped masters, but I don't think that this is causing the issue I have.

I'm planning on redoing the masters based on the new instructions, but haven't gotten around doing it yet.

[dld]

Only one row is missing. I cropped the top row in order to calibrate your light frame. I manually calibrated your (cropped) light frame with none of the tick boxes checked (no "calibrate" and no "optimize" in the Master Bias, Dark and Flat section of the ImageIntegration process) and the resulted image seems flat. You have probably accidentally ticked one of these tick boxes, thus the over-correction. I debayered the calibrated light using the GBRG Bayer pattern (as a consequence of the cropping). See the attached image (heavily compressed to comply with forum restrictions).

In summary:

• The latest version of PI will properly handle camera raw files and no messing with cropping and compensating with a different Bayer Matrix setting is needed.
• I am sorry to say that but proper acquisition of calibration data is a must. Everything else is a workaround, only used when no access to useful data is possible.
  

Cheers and clear skies!

[mxpwr]

Thanks for the analysis.
I'll double check that I don't have any boxes ticked.
Though I'm a bit confused.
You said the light frame is cropped?
I haven't actively cropped the light frame, only the bias and dark.
It's the raw, untouched cr2 from my camera.

[dld]

The raw file has an extra row (5202 x 3465) while your dark, flat and bias is  5202 x 3464. With the extra 1 row, ImageCalibration fails with an "Incompatible image geometry" error. That's why I cropped the first row from your raw CR2 file, then saved as xisf and proceeded calibrating the cropped light frame using the provided dark, flat and bias.

Edit:
I'm under an older version of PI (1.08.05.1353) which uses dcraw for handling DSLR data.

[mxpwr]

Only one row is missing. I cropped the top row in order to calibrate your light frame. I manually calibrated your (cropped) light frame with none of the tick boxes checked (no "calibrate" and no "optimize" in the Master Bias, Dark and Flat section of the ImageIntegration process) and the resulted image seems flat. You have probably accidentally ticked one of these tick boxes, thus the over-correction. I debayered the calibrated light using the GBRG Bayer pattern (as a consequence of the cropping). See the attached image (heavily compressed to comply with forum restrictions).

In summary:

• The latest version of PI will properly handle camera raw files and no messing with cropping and compensating with a different Bayer Matrix setting is needed.
• I am sorry to say that but proper acquisition of calibration data is a must. Everything else is a workaround, only used when no access to useful data is possible.
  

Cheers and clear skies!

I tried this, even generated a new master dark. Regardless of checking calibration/optimization, it always comes out over-flattened...

[mxpwr]

If I run the same files on my laptop with an older version of pixinsight, all works ok.
So my conclusion is that something has changed under the hood that makes the result of the flat calibration change...

[dld]

My guess is that you have mixed master calibration files from a version of PI using dcraw for DSLR raw file handling and a latest version which uses libraw. Remember that our DSLRs use proprietary file formats and the PI developers can't reinvent the wheel and reverse-engineer every proprietary raw file format existing out there. They have to use an external library for the task. I don't think the PI developers have changed a fairly stable and robust process like the ImageCalibration process.

Please read https://pixinsight.com/forum/index.php?topic=14219.0 and Bernd's guide I posted earlier. He's much better than me on explaining the calibration procedure. If you are using a version of PI which uses libraw, you probably should redo your master calibration files.

[mxpwr]

The master flat was created with the new version of pi.
It works ok with the old version of pi, even when mixing old and new masters.
With the new version of pi, using newly created master, it suddenly stopped (it worked fine 2 weeks ago).
Maybe it's some sort of cached data or something that's messsing up stuff, though I actually reinstalled pi and still had the same issue.

[dld]

As a sanity check, pick a single raw (.CR2) light, dark, flat and bias frame. Calibrate the raw light frame using the calibration frames as master frames with ImageIntegration, having the "Calibrate" option ticked only in the Master Dark and Master Flat section. Ensure that the Master Bias, Dark and Flat tickboxes on the left are all checked. No need to Evaluate Noise. Examine the calibrated light for vignetting and debayer it with the "Auto" Bayer Pattern setting. If things behave as expected then the problem may be somewhere in the generation of your master calibration frames. I've just tried this procedure using data obtained with my 550d and an f/2 lens (with considerable vignetting). I encourage you to read Bernd's guide. I see that I'm repeating what others have said earlier so I can't see how I can further assist you.