No convergence in MRS noise evaluation routine

[mschuster]

In ImageIntegration I am getting the warning 'No convergence in MRS noise evaluation routine' on the resulting integration. However when I run the NoiseEvaluation script on the result I don't get this warning.

Thanks, Mike

[Juan Conejero]

Hi Mike,

This shouldn't happen. I'd need some of these images to investigate what happens. Could you upload them to Endor? Thanks!

[mschuster]

Hi Juan,

Data uploaded as MRSNoiseWarning.zip in Forum Shared Files mschuster.

18x2400s dithered H-a subs at 4.2"/pixel calibrated in PI. sub09 is the SA master (not resampled).

Thanks,
Mike

[mschuster]

What is the status of this problem? I am seeing this error repeatedly on all of my integrations since July, and sometimes also on individual registered frames.
Thanks,
Mike

[Juan Conejero]

Hi Mike,

This bug is now fixed in version 1.8. Sorry for not reporting it here. MRS noise evaluation should never fail in the next version, except in very strange cases (basically, it can only fail when there is virtually no signal in the image).

Along with this bug, I have added the noise evaluation functionality to the ImageCalibration tool. This fixes the problem that we had in previous versions with noise evaluation on smoothed data due to registration interpolation. IC now evaluates the noise just after calibration, and stores the noise estimates in special FITS header keywords. ImageIntegration detects these keywords and uses them if they are present.

[mschuster]

Thank you Juan. All of this will be really helpful!
Mike

[Simon Großlercher]

Hello Juan,

I'm still getting this message when integrating Bias in 1.8

Cheers,
Simon

[Philippe B.]

Simon

You should analyse your images.
Sometimes I get the same message but when I look at the image, I see lot of values at 0 (zero). It happens also when I remove masterBias from the dark frames, then generate a MasterDark, then calibrate the light images. If for any reasons the k0 coeficient is near ZERO, you will get this message.

But that's right, it happens depending of the image. Not really seen it for only BIAS integration.

Cheers

[AstroScience]

I got the same warning (3 times for each file) when loading DSLR raw Bias files with Integration tool.
Something to worry about?

[mschuster]

I have seen three kinds of warnings. 'No convergence on a sub', 'No convergence on an integration', 'Near zero dark scaling'. The first risks a compromised integration because the sub could not be weighted relative to the others. The second makes it more difficult to find proper rejection clipping points. The last compromises the calibration results, but note, this is a calibration process warning not an integration warning.

My workaround for the first two is to use different noise estimation and weighting methods that work better on my undersampled Ha subs. My fix for the third is to use darks whose exposure is more equal to that of the lights. More recently I match exposures exactly and no longer dark scale.

Mike

[mschuster]

This image shows which pixels the MRS noise method used to measure the noise in one of my subs. Since noise varies with intensity due to Poisson statistics (by about a factor of 3 on this sub), the output number is some sort of blend of noises since pixels of various intensities are used. Just what that blend is and how it might vary between subs I don't know.



This image shows the mask for the other method I am using. It concentrates on the dimmest percentile range of intensities (in this case the dimmest 2%) and uses median to robustly select amongst many local standard deviation measurements. The method does not iterate so convergence is not an issue.



Mike

[ruediger]

MRS noise evaluation should never fail in the next version, except in very strange cases (basically, it can only fail when there is virtually no signal in the image).

Hello Juan,

I like the new concept that noise is now evaluated during debayering and this value is later used in ImageIntegration.
However, I'm still facing the same problems like in 1.7, namely that in the DSLR workflow for some pictures of a stack the noise is evaluated with MRS, and with k-sigma in others. This leads to very different weights during ImageIntegration and less overall quality compared to using the "Don't care (weights=1)" weighting scheme.


open "file:///Z:/PixInsight/Orion_50mm/RGB_1/IMG_8065_c.fit"
Reading 1 file(s):
Z:/PixInsight/Orion_50mm/RGB_1/IMG_8065_c.fit
Reading FITS: 32-bit floating point, 1 channel(s), 4770x3178 pixels: done

Debayer: Processing view: IMG_8065_c
Bilinear debayering: done
MRS noise evaluation: done
Gaussian noise estimates:
s0 = 6.055e-004, n0 = 0.308 (MRS)
s1 = 5.529e-004, n1 = 0.017 (MRS)
s2 = 2.468e-004, n2 = 0.315 (MRS)
29.99 s

open "file:///Z:/PixInsight/Orion_50mm/RGB_1/IMG_8066_c.fit"
Reading 1 file(s):
Z:/PixInsight/Orion_50mm/RGB_1/IMG_8066_c.fit
Reading FITS: 32-bit floating point, 1 channel(s), 4770x3178 pixels: done

Debayer: Processing view: IMG_8066_c
Bilinear debayering: done
MRS noise evaluation: done
Gaussian noise estimates:
s0 = 9.230e-004, n0 = 1.000 (K-Sigma)
s1 = 5.810e-004, n1 = 0.014 (MRS)
s2 = 3.727e-004, n2 = 1.000 (K-Sigma)
30.17 s

Along with this bug, I have added the noise evaluation functionality to the ImageCalibration tool. This fixes the problem that we had in previous versions with noise evaluation on smoothed data due to registration interpolation. IC now evaluates the noise just after calibration, and stores the noise estimates in special FITS header keywords. ImageIntegration detects these keywords and uses them if they are present.

In my DSLR workflow, when using wide angle lenses, I extract the R/G/B channels after debayering and align them separately against the reference frame. This allows compensation of chromatic abberation without the need for first aligning and stacking RGB-frames, than extracting R and B-channels and aligning them against G in a separate step.

But the noise estimates inside the RGB-fits after the debayering stage are lost in the channel extraction process (and the BatchChannelExtraction utliity script), so I can't take advantage from the early noise evaluation. Maybe this is easy to fix.


Besides these minor problems, I'm very happy with the new 1.8 RC3 (installed it yesterday). Fantastic work!

Regards,

Rüdiger

[Juan Conejero]

Hi Simon,

I'm still getting this message when integrating Bias in 1.8

You should not be using noise evaluation when integrating bias and dark frames. Please refer to Vicent Peris' tutorial on master frame generation (Bias and Dark Integration section). The BatchPreprocessing script disables noise evaluation for master bias and dark frames.

[Juan Conejero]

Hi Rüdiger,

I'm still facing the same problems like in 1.7, namely that in the DSLR workflow for some pictures of a stack the noise is evaluated with MRS, and with k-sigma in others.

This should not happen in version 1.8, unless the images are basically lacking any signal. So we perhaps have a bug here that we have not found with any of our test datasets during the stress tests we have conducted before the RC3 release.

So I definitely need some of those images to investigate the cause of the problem. Could you please upload a reduced dataset where this happens (you have an Endor account, don't you?).

[Juan Conejero]

But the noise estimates inside the RGB-fits after the debayering stage are lost in the channel extraction process (and the BatchChannelExtraction utliity script), so I can't take advantage from the early noise evaluation. Maybe this is easy to fix.

Yes, it should be rather easy. I'll try to release an update today or early tomorrow. Sorry for the inconvenience.