Flatfield calibration fails in image calibration but not in pixel math

[kupezi]

Hello,
my name ins Kurt-Peter and I am new to this forum. I’ve been using Pixinsight for quite a few years now and updated recently after a longer postprocessing pause to version 1.8.9.

While calibrating a large dataset I ran yesterday into a severe problem when applying flatfield frames.
Using the standard image calibration process including flatfields I get results with even more vignetting (see below: middle picture) than using no flatfield at all (left picture).
First of all I was considering my flatfields were not matching but that does not seem to be the point.

When I use the same calibration but only with a bias & darkframe reduction and then manually apply the flatfield to that result using pixel math I get a proper image (right picture).
So what is the difference between these two processes? Mathematically they should be the same and I would like to use the standard calibration process.

I have included the dataset as download link for inspection and a screenshot of the calibration process.

What am I doing wrong? Any ideas are highly welcome.

Kurt-Peter

Left = only bias and darkframe reduction with image calibration process
Middle = bias, darkframe and flatfield reduction with image calibration process
Right = bias and darkframe reduction with image calibration, then flatfield with pixel math

settings image calibration process:

 

[robyx]

The problem is the master dark optimization. The optimization introduces an additive component that is the cause of this global vignetting residual.
Disable the optimization and use dark frames with the proper matching exposure: you'll get correct results - assuming that your data has no other issues, of course (like lights infiltrations or the flats not being properly calibrated) -

 

[kupezi]

Thanks for your reply and your proposal to solve the issue.
Unfortunately, in my case, there is no difference checking the "optimization" checkbox or not.
I have also tried these version and the results are exactly the same.
The length of the masterdark is also 600s and matches exactly the light frame and the master flat is also properly calibrated and ok in my opinion.

As I wrote earlier: Applying the flat MANUALLY within pixelmath gives perfect results but NOT within the standard image calibration routine.
This makes me wonder and I don't understand why this happens.

 

[bulrichl]

The MasterBias, MasterDark and MasterFlat (in XISF format) look OK to me, judging from the metadata, but what are the 3 files:

NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot.xisf,
NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot_c.xisf,
NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot_c_1.xisf?

The first one is definitely not a light frame which is matched by the MasterBias an MasterDark. In order to know what happened, one would need to have an unprocessed light frame.

Please also provide the PixelMath expression, together with the frames that were processed with PixelMath.

Bernd

 

[kupezi]

Bernd,
thanks for your time trying to help me.
Here comes the requested additional information:

...
The first one is definitely not a light frame which is matched by the MasterBias an MasterDark. In order to know what happened, one would need to have an unprocessed light frame.
....

#1) NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot.xisf
= This is the unprocessed RAW file directly out of MaximDL, rotated by 180° (for calibration) and converted to 32BIT IEEE XISF-format.
The original Fits file is available here.
Why do you think this is not a light frame?

#2) NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot_c.xisf
= This is file #1 calibrated (bias, dark and flat field) within the image calibration process. For processing refer to screenshot here:

#3) NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot_c_1.xisf
= This is file #1 calibrated (only bias and dark) within the image calibration process. No flat field was applied.

#4) new download, this picture was so far not in the provided dataset:
NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot_c_flat_pm.xisf
= This is file #3 divided by the Masterflat processed in pixel math.
Formula: NGC7635_20160815_033229_S001_R001_Ha_W_Ha_600sec_B1_IEEE_rot_c/masterFlat_BIN_1_4096x4096_FILTER_Ha_mono_integration

This is the result I'd like to have when using the "standard" image calibration process (compared to image #2 which I get!)

Do you have any ideas?
I am looking forward to receiving your comments.

Kurt-Peter

 

[pfile]

= This is the unprocessed RAW file directly out of MaximDL, rotated by 180° (for calibration) and converted to 32BIT IEEE XISF-format.

just out of curiosity, why would you rotate the image 180 degrees before calibration?

 

[bulrichl]

I didn't say that 'NGC7635-20160815-033229-S001-R001-Ha-W-Ha-600sec-B1_IEEE_rot.xisf' is not a light frame. What I said is: light frame and MasterDark don't match.

Comparing the statistics of light, MasterDark and MasterBias:

            light       MB          MD
count (%)   99.99902    100.00000   99.99997
count (px)  16777051    16777216    16777211
mean        0.0014891   0.0150201   0.0151112
median      0.0013014   0.0150156   0.0150778
variance    0.0000164   0.0000000   0.0000024
stdDev      0.0040530   0.0002189   0.0015631
avgDev      0.0005127   0.0000229   0.0000667
MAD         0.0002986   0.0000169   0.0000254
minimum     0.0000155   0.0148650   0.0148513
maximum     0.9988535   0.2780790   0.9542397

Mean and median of both MB and MD are more than a magnitude larger than mean and median of the light frame.

There are several additional points that I don't understand:

1. Why do you have to rotate the light frames by 180° (pfile asked the same question already)?
2. Why do you convert the raw light frame (in Maxim DL) to 32-bit IEEE 754 floating point format instead of leaving 16-bit integer format?

Apparently the bias, dark and flat frames are captured in a different way than the light frames. What are the differences?


Bernd

 

[bulrichl]

I applied ImageCalibration with the settings you showed. The following warning was issued in the process console:

** Warning: negative or insignificant pixel sample values detected after calibration.
Applying automatic pedestal: 1000 DN (huge!)

The statistics of the calibration result revealed that the problem - as expected - already arises in the bias-subtraction:

            light_c
count (%)   83.18163
count (px)  13955561
mean        0.0020050
median      0.0020596
variance    0.0000234
stdDev      0.0048372
avgDev      0.0010750
MAD         0.0009234
minimum     0.0000000
maximum     0.9989216

83 17 % of all pixels are clipped. The result is completely unusable.
(Sorry, my fault: 83 % of all pixels are unclipped, 17 % are clipped)

When I applied image calibration using only MB and MD, the following warning is issued in the process console:

** Warning: negative or insignificant pixel sample values detected after calibration.
Applying automatic pedestal: 970 DN

and the statistics of the result are:

            light_c1
count (%)   99.98952
count (px)  16775457
mean        0.0011865
median      0.0010226
variance    0.0000233
stdDev      0.0048281
avgDev      0.0004843
MAD         0.0003027
minimum     0.0000000
maximum     0.9997979

The applied huge pedestal (970 DN = 0.0148) is only just sufficient to compensate the (for whatever reason) mismatched zero points of light frames versus MB and MD. The difference between applying an image calibration using MB, MD and MF (case A) and your approach of applying an image calibration using only MB and MD, then performing flat field correction with PixelMath (case B) is:

A: After bias-subtraction and application of the dark frame optimzation, a large fraction of the total pixels have negative intensities. Now the flat field correction is executed with a catastrophic result, because of the mismatched zero points of light frame and MB. Subsequently the pedestal of 1000 DN is applied, but of course that won't save anything.

B: After bias-subtraction and application of the dark frame optimzation, a large fraction of the total pixels have negative intensities. Subsequently, the pedestal of 970 DN is applied. Now you perform the flat field correction in PixelMath which doesn't care about a pedestal. It is only by chance that the applied pedestal of 970 DN just compensates the mismatch of the zero points. Please note: 1000 DN is the maximum value for a pedestal that is usable in image calibration!

Bottom line: you will have to check why the zero points of light frames and MB / MD are different. This is the root cause of the issues you observe. Probably this is caused by using different acquisition software or different camera drivers for the acquisition of light frames and bias / dark frames.

Bernd

 

[kupezi]

just out of curiosity, why would you rotate the image 180 degrees before calibration?

All my westbound pictures are automatically rotated by 180° during acquisition. It's an enabled feature in ACP so all pictures look "upright" no matter if they are take in the East or West.
The corresponding keyword FLIPSTAT is not read in Pixinsight so all westbound images have to be rotated for calibration.

 

[kupezi]

...
There are several additional points that I don't understand:
1. Why do you have to rotate the light frames by 180° (pfile asked the same question already)?
2. Why do you convert the raw light frame (in Maxim DL) to 32-bit IEEE 754 floating point format instead of leaving 16-bit integer format?
Apparently the bias, dark and flat frames are captured in a different way than the light frames. What are the differences?
...

Bernd,
thank you for your ongoing analysis.

Regarding your questions:
1) 180° Rotation: see my earlier post to pfile.
2) At the beginning of post processing I convert all my files into 32bit floating format in order not to loose any information during processing.

Kurt-Peter

 

[kupezi]

Hello Bernd,
I am glad you could reproduce my issue.
Indeed I got exactly the same warnings as you but I don't have any idea about the cause of all that.

The statistics of the calibration result revealed that the problem - as expected - already arises in the bias-subtraction:

            light_c
count (%)   83.18163
count (px)  13955561
mean        0.0020050
median      0.0020596
variance    0.0000234
stdDev      0.0048372
avgDev      0.0010750
MAD         0.0009234
minimum     0.0000000
maximum     0.9989216

83 % of all pixels are clipped. The result is completely unusable.

I have to confess that I am not familiar with statistics in Pixinsight. Where do you see the clipped data are 83%?
Isn't it only the pixel count that shows 83%?

When I applied image calibration using only MB and MD, the following warning is issued in the process console:

** Warning: negative or insignificant pixel sample values detected after calibration.
Applying automatic pedestal: 970 DN
and the statistics of the result are:

            light_c1
count (%)   99.98952
count (px)  16775457
mean        0.0011865
median      0.0010226
variance    0.0000233
stdDev      0.0048281
avgDev      0.0004843
MAD         0.0003027
minimum     0.0000000
maximum     0.9997979

The applied huge pedestal (970 DN = 0.0148) is only just sufficient to compensate the (for whatever reason) mismatched zero points of light frames versus MB and MD.

Same question here: is the pedestal seen anywhere in the statistics?

Bottom line: you will have to check why the zero points of light frames and MB / MD are different. This is the root cause of the issues you observe. Probably this is caused by using different acquisition software or different camera drivers for the acquisition of light frames and bias / dark frames.

I have a permanent setup and settings normally don't change for many months/years. Camera, software, drivers are all the same - nothing has been altered in between.
Calibration frames were taken a few days after the light frames and I have also checked the fits headers for differences - nothing.
I don't know what else I could try.

What I will test now is not to use Master calibration files but a single bias, dark, flat to calibrate the light frame. Perhaps then a difference will be visible. I'll keep you posted.

Thank you so much so far.

Kurt-Peter

 

[bulrichl]

Thank you for your feedback. I am happy to answer your questions, and I hope that we will be able clarify the cause of your issue and put things right.

1. Clipped data
Sorry, I made a mistake: 83 % of the pixels are NOT clipped, but 17 % are indeed clipped (still a much too high value). Open the frame that was fully calibrated in PixInsight (with MB, MD and MF) and open the Statistics tool. Click on the wrench icon ('Statistics options') and read the tooltip for 'Count', moving the mouse pointer over the item 'Count':

Total number of pixel samples within the estimation range ]0,1[ (clipped) or [0,1] (unclipped).

This means: if 'Unclipped' is disabled, the pixel count relates to the pixels that have intensities 0 < I < 1 (thus 0 and 1 are excluded), if 'Unclipped' is enabled, it relates to pixels that have intensities 0 <= I <= 1 (thus 0 and 1 are included). So (100 % - count %) is the sum of pixels having intensity of either 0 or 1.

You can determine the number of pixels having intensity of 0 directly from the histogram: open the histogram tool, set horizontal zoom to 400 and move the mouse pointer within the graph to the left until the intensity value 0 is displayed below the graph. The full line reads now: "x= 0.0000(0-0) 2838251, %16.9173", i.e. 2838251 pixels = 16.9173 % have an intensity of 0. Pushing the scrollbar to the right and pointing within the graph to the right reveals that 61 pixels = 0.0004 % have an intensity of 65535.


2. Pedestal
The pedestal is applied because you set in ImageCalibration, section 'Output Files', the options 'output pedestal mode' to 'Automatic' and 'Auto pedestal limit' to 0.00010. Please view the corresponding tooltips.

The pedestal value that is applied is outputted to the process console during image calibration. It is also written to the FITS header (keyword PEDESTAL) of the calibrated frame.


3. Conversion to 32-bit files
Assuming that you are using a camera with KAF-16803 or KAF-16801, the ADC has a bit depth of 16 bit. There is no point in converting unprocessed files to a number format of 32-bit IEEE 754 floating point, this will only double the file size. A number format of 16-bit integer is sufficient, and no loss of information is to be suspected using it. PixInsight's ImageCalibration process defaults to a number format of 32-bit IEEE 754 floating point for output files. This can be changed in the 'Output Files' section, option 'Sample format'.


According to the tests I made, the zero point difference between light frame and MB is just 1000 DN. Some cameras provide the possibility of changing the offset (the zero point of the analog voltage that is converted by the ADC). However, this is rather a feature of CMOS cameras; I don't know whether there are CCD cameras that support changing the offset. Anyway different offset settings for the acquisition of light and calibration frames possibly could be another explanation of the issue, if different acquisition software or camera drivers are definitely excluded.

I would like to view one original light and one original bias frame in FITS format, if you saved them.

Bernd

 

[kupezi]

Hello Bernd,
thank you for your ongoing help and support which I really appreciate a lot.

After reading your last post I have analyzed my fits raw files using your hints especially on statistics.
Especially examining the median values were very helpful and they provide additional information to histograms...

1. Clipped data
Sorry, I made a mistake: 83 % of the pixels are NOT clipped, but 17 % are indeed clipped (still a much too high value). ....

OK - now I agree with you ;-)

This means: if 'Unclipped' is disabled, the pixel count relates to the pixels that have intensities 0 < I < 1 (thus 0 and 1 are excluded), if 'Unclipped' is enabled, it relates to pixels that have intensities 0 <= I <= 1 (thus 0 and 1 are included). So (100 % - count %) is the sum of pixels having intensity of either 0 or 1.

Transferring 16 or 32bit values into a range between 0 and 1 is still a bit strange for me - even after years.

You can determine the number of pixels having intensity of 0 directly from the histogram: ....

I am familiar with histograms. They are a great way to analyze and alter images. Another way to show the amount of a certain intensity value could be to display them graphically directly in the image using pixel math.

2. Pedestal
The pedestal value that is applied is outputted to the process console during image calibration. It is also written to the FITS header (keyword PEDESTAL) of the calibrated frame.

Exactly. MaximDL adds a pedestal of 100 to its images to be on the "safe side" especially for signals with very low values (Narrowband).
Some people even tend to subtract this pedestal for more accurate post processing which I don't do.

3. Conversion to 32-bit files
Assuming that you are using a camera with KAF-16803 or KAF-16801, the ADC has a bit depth of 16 bit. There is no point in converting unprocessed files to a number format of 32-bit IEEE 754 floating point, this will only double the file size. A number format of 16-bit integer is sufficient, and no loss of information is to be suspected using it. PixInsight's ImageCalibration process defaults to a number format of 32-bit IEEE 754 floating point for output files. This can be changed in the 'Output Files' section, option 'Sample format'.

My camera is a FLI Proline 16803 - a great and reliable workhorse. The 16bit files it produces are stored in 16bit FITS format. Everything else would be a waste of memory. At the beginning of the post processing I used to convert them into 32 bit IEEE floating format for further processing. These files are only temporarily stored.

Some cameras provide the possibility of changing the offset (the zero point of the analog voltage that is converted by the ADC). However, this is rather a feature of CMOS cameras; I don't know whether there are CCD cameras that support changing the offset.

Indeed a variable offset setting is a feature of CMOS (and it can be dangerous). I am not aware of any CCD-based camera where you can adjust the offset - at least my CCD-camera(s) don't have it.

Anyway different offset settings for the acquisition of light and calibration frames possibly could be another explanation of the issue, if different acquisition software or camera drivers are definitely excluded.

I guess I found the issue and it was not an offset setting.
In the meantime I examined my original FITS raw data statistically and found out that the median is approx. the same in all lights, darks and flats - even the same as in the master calibration files. And it was NOT the same in the XISF raw file as you had found out.

So I saved the 16bit FITS file again as 32bit IEEE floating XISF file and now the median was within the expected range!!!
What had happened? First I had exported the 16bit FITS files to 32bit IEEE floating FITS files in MaximDL and then used these files in Pixinsight.
Now I had used the 16bit FITS files directly in Pixinsight.
There must be the difference. File formats in both software applications seem not to be 100% compatible.
So far I didn't reproduce the export from MaximDL as it runs on another system. I'll do that tomorrow to be sure.

Here you see the statistical numbers:

And the calibrated result even with the flat field looks perfect )
No more warnings etc....

My final question is why the median of the calibrated file is again more than 10 times less than the input frames?

If you are interested I have attached the the original FITS files and the calibrated file in the new dataset.
I am looking forward to receiving your input.

Kurt-Peter

 

[pfile]

yes, that is the difference. PI uses the numerical range [0.0-1.0] for 32-bit floating point samples. maxim probably does not. PI tries to convert any "non-conforming" samples (you can see the configuration for this in the file explorer - double click the FITS entry in the explorer). PI probably ends up converting the darks/bias/lights/flats differently and so the calibration goes wrong. but that's just a guess.

this is why 16-bit integer files are preferred for moving between software. there's just no ambiguity as 16 bit integers are pretty much the same on all systems (except for endianness of course, but i think FITS files have a set endianness so there's still no ambiguity.)

there's really no reason to convert the 16-bit files to 32-bit before calibration. as bernd mentioned, PI is going to save the calibrated files as 32-b float files anyway.

by the way, i would turn off ACP's rotation. the less software that touches your raw data the better, and it would remove the need to rotate the files before calibration. i've used ACP and didn't even know this feature existed. PI's not going to have any problem registering the "upside-down" images so i don't see any clear reason to have ACP rotate the images.

 

[bulrichl]

I am glad that you were able to elucidate the issue.

What pfile wrote is spot-on:
I would not convert the raw frames from 16-bit integer to 32-bit floating point, neither with PixInsight - it is no use. I would not rotate the images in the acquisition software if this the rotation is done not only for displaying purpose but also for the saved files. pfile is right when he writes: "the less software that touches your raw data the better". So the input frames for calibration in PixInsight should be FITS files in number format 16-bit integer, unrotated.

I will take a look at the new dataset tomorrow.

Bernd

 

[bulrichl]

For the sake of completeness, please upload the light frame that was converted to number format 32-bit floating point in Maxim DL (FITS file). I would like to investigate what is happening when it is imported into PixInsight.

My final question is why the median of the calibrated file is again more than 10 times less than the input frames?

In image calibration, the following calculation is performed:

Case 1, with dark frame optimization:

Cal1 = (L - MB - k0*(MD - MB)) / MF * mean(MF)

Case 2, no dark frame optimization:

Cal2 = (L - MD) / MF * mean(MF)

For simplicity, let's examine case 2 (no dark frame optimization). Term (L - MD) is the dark calibrated light frame, and the subsequent operation is the flat field correction. The flat field correction has little impact on the median of the calibrated frame, the main effect is produced in the subtraction of MD from L. The following median values are read from the light and the dark frame (original FITS files in dataset2):

        median
L       0.01642
dark    0.01513
---------------
diff    0.00129

The median of the dark frame contains the bias offset (median = 0.01500) as main component. The bias offset is added to the analog voltage before analog-to-digital conversion of pixel values. This is necessary in order to ensure that no negative values occur. Negative values cannot be processed by the ADC and would be truncated without application of the bias offset. In preprocessing, the contributions of bias offset and dark current have to be subtracted since they are not signal that is generated by light.

Bernd

 

[kupezi]

Bernd,
here comes the FITS-raw file in 32bit floating IEEE format which was generated by Maxim.

Later on I will have a look at your calculations...
Thanks so far,

Kurt-Peter

 

[bulrichl]

These are the median, min and max values of the light frames NGC7635_20160815_033229_S001_R001_Ha_W_Ha_600sec_B1_RAW.fts (= RAW) and NGC7635_20160815_033229_S001_R001_Ha_W_Ha_600sec_B1_IEEE.fit (= IEEE):

        RAW            IEEE
Median    0.01642        0.0013014
min        0.01514        0.0000155
max        0.99887        0.9988535

This reveals that Maxim DL performed the conversion from number format 16-bit integer to 32-bit floating point with rescaling, i.e. [min,max] -> [0.0,65535.0] by calculating for each pixel:

v' = (v - min)/(max - min)

However, rescaling of raw data impedes a correct image calibration - this is the issue which caused the wrong flat field correction.

Bernd

 

[kupezi]

Bernd,
now I have worked through your various calculations and explantations
I am very happy that my raw data are consistent and no acquisition problem has occurred.
This dataset consists of more than 800 light files and was made over 4 years with different telescopes. It would have been too bad if smth. went wrong...

...
The median of the dark frame contains the bias offset (median = 0.01500) as main component. The bias offset is added to the analog voltage before analog-to-digital conversion of pixel values. This is necessary in order to ensure that no negative values occur. Negative values cannot be processed by the ADC and would be truncated without application of the bias offset. In preprocessing, the contributions of bias offset and dark current have to be subtracted since they are not signal that is generated by light.

I understand - the electrical process of collecting photons on a chip were clear to me but now the maths part is also added - great!

This reveals that Maxim DL performed the conversion from number format 16-bit integer to 32-bit floating point with rescaling, i.e. [min,max] -> [0.0,65535.0] by calculating for each pixel:

For the future I have learned not to do anything else with MaximDL rather than taking images ;-)

I am very glad you have pointed out where to look at because I still have a few other datasets with the same problem.
In 2017 I attended an official PI onsite workshop over a few days but the trainer couldn't find the problem...

So I am VERY grateful about your help and the time you have invested into my problem.
THANK YOU SO MUCH!!!
Are you a professional astronomer?

Off topic one last question which might sound stupid: I use PI simultaneously to other software like file explorer or Excel.
As soon as I click on a NON-PI window all windows in PI disappear. Thus parallel working gets very difficult. Is there a way to change that behavior?

Kurt-Peter

 

[bulrichl]

I am glad I could help.

Are you a professional astronomer?

No, astrophotography is a hobby for me.

Off topic one last question which might sound stupid: I use PI simultaneously to other software like file explorer or Excel.
As soon as I click on a NON-PI window all windows in PI disappear. Thus parallel working gets very difficult. Is there a way to change that behavior?

I am not a programmer, so I don't know. I guess, a larger display definitely helps (mine is too small as well).

Bernd