Author Topic: ImageIntegration Reference Frame Anomaly  (Read 341 times)

Offline jdupton

  • Newcomer
  • Posts: 19
    • View Profile
ImageIntegration Reference Frame Anomaly
« on: 2019 July 08 11:04:21 »
Hey folks,

   I have encountered a puzzle and hope someone can explain what might be going on.

   I am using ImageIntegration to integrate 50 (uncalibrated) Flat Frames. I find that if I do the integration and allow the first frame to be the reference, ImageIntegration gives totally inappropriate / incorrect results. After some experimentation, I find that if I set the reference frame to be the second in the list, the results of the ImageIntegration Process are as expected.

   Specifically, what I am seeing is that when it gives bad results, the output integration has a mean ADU value of ~6474 and has a great amount of clipping in the output frame. By changing to Frame #2 as the reference, the output is what one would expect. The full set of pixels in the set of 50 frames have a minimum pixel value of 20280 and a maximum pixel value of 65532 (saturated). Having the integrated flat so much darker and clipped given the input frames is surprising.

   Here are the integrated frame statistics for a run with Frame #1 as the reference:
integration1
            K
count (%)   100.00000
count (px)  11694368
mean        6473.985
median      6544.484
stdDev      1940.732
avgDev      1558.552
MAD         1306.004
minimum     0.000
maximum     65535.000



   And here are the integrated frame statistics for a run with Frame #2 as the reference:
integration
            K
count (%)   100.00000
count (px)  11694368
mean        26201.517
median      26248.410
stdDev      1290.915
avgDev      1036.700
MAD         868.713
minimum     21895.221
maximum     65487.079



   For reference, here are the statistics for Frame #1 (the one that causes the failure):
Calibration_g200_o30_0c_9sec_1x1_0001
            K
count (%)   100.00000
count (px)  11694368
mean        26225.6
median      26248.0
stdDev      1388.8
avgDev      1114.1
MAD         944.0
minimum     20676.0
maximum     65532.0



   For reference, here are the statistics for Frame #2 (the one that works correctly):
Calibration_g200_o30_0c_9sec_1x1_0002
            K
count (%)   100.00000
count (px)  11694368
mean        26201.8
median      26228.0
stdDev      1383.2
avgDev      1109.7
MAD         940.0
minimum     20740.0
maximum     65532.0


   So, my main question concerns what is it in the two potential reference frames that seem so similar that can cause the ImageIntegration Process to give such anomalous results? This is the first time in 10 years of using PixInsight that I have seen anything like this. Have others encountered it as well?


John

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6644
    • View Profile
    • http://pixinsight.com/
Re: ImageIntegration Reference Frame Anomaly
« Reply #1 on: 2019 July 09 05:38:59 »
Hi John,

What you are observing here is the result of output normalization (multiplicative in this case). If you disable normalization, you should get identical results irrespective of the reference frame you choose. For master flat generation, output values after integration are irrelevant because the master flat is always normalized for calibration.

However, the numbers you are showing here are indeed weird. The minimum pixel value should not be zero under normal conditions. So something strange happens with these flat frames. Can you please upload the entire set (dropbox, etc)?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline jdupton

  • Newcomer
  • Posts: 19
    • View Profile
Re: ImageIntegration Reference Frame Anomaly
« Reply #2 on: 2019 July 09 07:39:18 »
Juan,

   Thanks for looking into this.

   I had found / verified later yesterday that the multiplicative normalization was involved. If I choose no normalization, I get expected results. If it weren't for the clipping in the integrated flat, I suspect that the use of the result might be OK. However, application of the very low oddball flat frame in target image calibration was so far off that it was not usable -- normalization during flat calibration aside.

   I have reduced the number of frames and found that I can still reproduce the problem. If I choose the first five frames of the set of 50, I get the odd frame result when using #1 or #2 as a reference but I get normal results using any of the other three as the reference. (Before, when using the full set of 50 frames, the second frame gave a correct result but the first frame failed. Now, with only five frames, both the first and second frames cause the integration to fail with a bad result. I did not try using all other frames in the 50 frame set as references but expect some will fail when used as reference while others will be OK.)

   I haven't used DropBox before but have created a folder there with the first five frames of my original set of 50. I also added  a Process Icon for the ImageIntegration settings I was using. Hopefully, you will also be able to recreate this with the five frame subset.

   Let me know if you cannot access the files. I am new to DropBox. The DropBox folder link is:
https://www.dropbox.com/sh/q9o1hbf0zu764cx/AABvgKQ1ghWzoylymlp1y3Sua?dl=0

   Any insights as to what attributes of the frames causes the very odd results in ImageIntegration is much appreciated.

   My fear / apprehension is that something similar might happen from time to time with flats and if the results weren't so wildly obvious, I might not recognize it was happening. ("Averaging" 5 files with means of about 26000 giving a result of 6500 is easy to spot. If the result of averaging the same five files came out as 20000, I might not notice and the clipping I see could still cause me problems in later processing stages.) I'd like to learn to recognize when my input files might lead to such odd results.


John

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6644
    • View Profile
    • http://pixinsight.com/
Re: ImageIntegration Reference Frame Anomaly
« Reply #3 on: 2019 July 10 04:49:21 »
Hi John,

Thank you for uploading the images. As expected, there is no problem at all, neither with the flat frames (other than the fact that they are uncalibrated frames, but that's a different matter), nor with the integrated results, irrespective of the frame selected as reference for integration.

Multiplicative normalization is extremely sensitive to small differences among reference images, so you can also expect relatively large differences in the corresponding integrated images. However, all of them are equally valid and there is no truncation of pixel values.

To demonstrate this, let's count the number of pixels with a value of zero in the integrated result using different reference frames. This can be done very easily with the PixelMath tool. Define a PixelMath instance with the following parameters (the rest with default values):

- RGB/K Expression:

      n += $T <= 1.19e-07

- Symbols:

      n = global(+)

- Generate output (Destination section): disabled.

To perform a more rigorous test, we'll count the number of pixels with values less than or equal to the machine epsilon for 32-bit floating point (1.19x10-7), instead of zero. After executing the PixelMath instance on each image, the results are written to the console:

Code: [Select]
PixelMath: Processing view: integration_with_reference_0003
Executing PixelMath expression: combined RGB/K channels: n += $T <= 1.19e-07: done

* Global variables:
n(+) = { 0, 0, 0 }

35.835 ms

PixelMath: Processing view: integration_with_reference_0001
Executing PixelMath expression: combined RGB/K channels: n += $T <= 1.19e-07: done

* Global variables:
n(+) = { 1, 1, 1 }

42.657 ms

As you can see, the image integrated with the first frame as reference has a single pixel with an insignificant value. This has no statistical significance, and is irrelevant for all practical purposes.

Let's perform a second test. Define the following PixelMath instance (other parameters with default values):

- RGB/K Expression:

      $T/mean($T)

- Rescale result (Destination section): enabled.

This instance will normalize the master flat frames, basically in the same way ImageCalibration does before applying a master flat to calibrate light frames. After applying the instance, these are the normalized/unclipped statistics:

integration_with_reference_0003
            K
count (%)   100.00000
count (px)  11694368
mean        1.034921e-01
median      1.044943e-01
avgDev      2.986219e-02
MAD         2.971942e-02
minimum     0.000000e+00
maximum     1.000000e+00

integration_with_reference_0001
            K
count (%)   100.00000
count (px)  11694368
mean        1.034922e-01
median      1.044944e-01
avgDev      2.986219e-02
MAD         2.971948e-02
minimum     0.000000e+00
maximum     1.000000e+00


The differences are insignificant (beyond 32-bit floating point resolution). You can also compare the histograms for both integrated frames to verify that they are indeed identical.

You can be sure that, irrespective of the reference frame selected for integration, the resulting master flat will work perfectly for image calibration. If the flat frames are valid, that is.

Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline jdupton

  • Newcomer
  • Posts: 19
    • View Profile
Re: ImageIntegration Reference Frame Anomaly
« Reply #4 on: 2019 July 10 08:17:29 »
Juan,

   Thank you again for the time you have put into this. I will need a little more time to digest all you have written and duplicate the checks myself (and understand their significance).

   However, I don't understand the following part of your comments.

Quote
The differences are insignificant (beyond 32-bit floating point resolution). You can also compare the histograms for both integrated frames to verify that they are indeed identical.

You can be sure that, irrespective of the reference frame selected for integration, the resulting master flat will work perfectly for image calibration. If the flat frames are valid, that is.

   I may be interpreting these comments incorrectly but I am think I am seeing differences that do matter in the use of the two integrations (using Frame #1 vs Frame #3 as references). When I tried to use the "oddball" integration as a flat, it was not usable for correcting lights. (I was actually using a calibrated set of flats for that and only included raw flat results here because they show the same problem and remove one complication from the discussion.)

   I tried a quick experiment to attempt to show the differing results I see when using the two integrations for calibration. Let me know if this is not a valid check.
  • I first integrated the five frame set I posted using Frame #1 as reference and called the result "integration_Ref_1"

  • Next, did the same integration using Frame #3 as reference and called the result "integration_Ref_3"

  • Finally, I used each of these to correct a hypothetical 25% gray image. I used PixelMath to compare the results of the two flat corrections as follows:
     
    abs((0.25 / (integration_Ref_1 / mean(integration_Ref_1))) - (0.25 / (integration_Ref_3 / mean(integration_Ref_3))))

   If I did that correctly and it is a valid test, the expected result would be zeros if the two integrated flats were equivalent after normalization. Instead, I see significant differences as shown below. This difference matches what I saw when I attempted to apply the "oddball" flat to a light.

   I will review your detailed response some more to understand better what you think is happening. I still need to reconcile your comments regarding the application of either integrated flat to a light as being good for calibration in light of my results.


John

PS: I have added an XISF copy of the PixelMath flat correction difference image result experiment to the same DropBox folder as the other files. (https://www.dropbox.com/sh/q9o1hbf0zu764cx/AABvgKQ1ghWzoylymlp1y3Sua?dl=0)
« Last Edit: 2019 July 10 08:30:37 by jdupton »

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6644
    • View Profile
    • http://pixinsight.com/
Re: ImageIntegration Reference Frame Anomaly
« Reply #5 on: 2019 July 11 02:09:30 »
John,

Yes, it seems we have a problem here, and it is very strange. I haven't seen anything like this under normal conditions before.

Although both integrated results (with 0001 and 0003 as integration references) are valid models of the same illumination distribution numerically (to prove this, enable the Rescale result parameter when you apply your PixelMath expression, to prevent truncation of values > 1), the result with 0001 as reference does not work for image calibration. Let me some time to analyze what is happening here. Either there is something anomalous in the data, or we have a bug. In the latter case, it would be of the maximum priority. At any rate, thank you so much for alerting us about this.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6644
    • View Profile
    • http://pixinsight.com/
Re: ImageIntegration Reference Frame Anomaly
« Reply #6 on: 2019 July 11 02:54:13 »
John,

Can you please upload the master bias and dark frames for these flats, or even better, the individual bias and dark frames, so we can generate the masters? We think the problem is the additive components in the uncalibrated flat frames. With non-negligible additive components in the individual flat frames, generation of a valid master flat is impossible.
« Last Edit: 2019 July 11 03:27:38 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline jdupton

  • Newcomer
  • Posts: 19
    • View Profile
Re: ImageIntegration Reference Frame Anomaly
« Reply #7 on: 2019 July 11 06:14:20 »
Juan,

   Thanks yet again for doing a deep dive on this strange anomaly.

   I will upload a partial set of calibration frames today. The full set of data is over 8 GB and I have only the personal DropBox account limited to 2 GB. I am also on a slow internet connection and it may take many hours to upload even a subset of the data.  (The full date set includes 185 Bias Frames (not normally used), 50 Flat Frames, and 50 Flat-Dark Frames.)

   First, I want to verify that the odd issue can still be reproduced using only a subset of calibrated flat frames. I discovered the problem using calibrated frames but uploaded the small subset of uncalibrated flats once I found they could reproduce what I was seeing. I am not positive at this time whether I can reproduce the issue with a partial set of calibrated flat frames. I don't want to waste your time chasing phantoms if only a partial set of flats / flat-darks or bias frames don't show what I saw with the full set. I can upload my Master Frames also in addition to the subsets.

   I will also note that I do not usually use Bias Frames with the camera I have (ASI294MC-Pro color CMOS). Bias Frames from this camera are a bit odd -- they are greater that the y-intercept of a series of dark frames of differing exposures. In other words, for this camera, a Bias Frame is not the same as you would expect from an extrapolated zero exposure length Dark Frame. For that reason, I normally only calibrate my flats with Flat-Dark Frames.

   I will post again once I can verify the issue with a subset of frames and have them uploaded to DropBox for your use in tracking this down.


John

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6644
    • View Profile
    • http://pixinsight.com/
Re: ImageIntegration Reference Frame Anomaly
« Reply #8 on: 2019 July 11 07:20:02 »
Hi John,

I think we have identified the bug that is causing this anomaly. It can only happen when the integrated image has very high pixel values, 1.0 or very close. In practice, this only happens when the data set contains saturated pixels at the same location in all of the frames. Such pixels cannot be rejected. When this happens and the reference image leads to a scaling factor greater than one, the output image is rescaled automatically. For integration of lights, this rescaling operation is the correct choice to preserve all of the integrated data, but it is not for flats, since any rescaling operation invalidates the illumination profile.

We haven't seen this problem before because ImageIntegration is intended to be used with well calibrated data, including calibrated flat frames, where there should be no saturated hot pixels. The solution to prevent this issue is to replace the rescaling operation with a truncation to the [0,1] range only for integration of flat frames. This will be implemented as an additional parameter of the ImageIntegration process. In addition, the tool will issue a warning message when the output image is rescaled or truncated, so you can know exactly when this happens.

Thank you for discovering this problem. I still have to make more tests to implement these modifications. Once completed, I'll release an update with a new version of the ImageIntegration module.

So for now, there is no need to upload new data if you prefer. I'll let you know if I need them.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline jdupton

  • Newcomer
  • Posts: 19
    • View Profile
Re: ImageIntegration Reference Frame Anomaly
« Reply #9 on: 2019 July 11 07:41:49 »
Juan,

   I am glad you found something here.

   One more quick question comes to mind. Should it be considered best practice to do CosmeticCorrection on the calibrated integrated Flat Frame before using it?

   Since flats are usually reasonably bight anyway, they are much more likely to contain saturated pixels in the same location for all sets of the raw flats. It seems to me that calibration of the flats with equal duration short darks may not always work to help fully saturated pixels if their problem results from higher than normal sensitivity. Should we then run CosmeticCorrection (with CFA Awareness when using color cameras) just in case?


John

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 984
    • View Profile
    • http://www.astrofoto.es/
Re: ImageIntegration Reference Frame Anomaly
« Reply #10 on: 2019 July 11 08:20:25 »
Juan,

   I am glad you found something here.

   One more quick question comes to mind. Should it be considered best practice to do CosmeticCorrection on the calibrated integrated Flat Frame before using it?

   Since flats are usually reasonably bight anyway, they are much more likely to contain saturated pixels in the same location for all sets of the raw flats. It seems to me that calibration of the flats with equal duration short darks may not always work to help fully saturated pixels if their problem results from higher than normal sensitivity. Should we then run CosmeticCorrection (with CFA Awareness when using color cameras) just in case?


John

Hi John,

If you have a saturated pixel in the flat frame, it means that pixel has probably 30-40k ADUs in a few seconds. That pixel will be completely saturated in a 5-minute light exposure. That means that you won't be able to recover in any way the light signal in that pixel. So, in that case, it's not worth to apply CosmeticCorrection to the master flat; simply apply it to the light frames and, of course, use dithering in your imaging session.

Very important as well: always calibrate your flat frames with bias and dark to remove those additive components. Ideally, any signal in the master flat should have strictly a multiplicative component, since the master flat should only show the relative light sensitivity of each pixel in the image.


Regards,
Vicent.