NormalizeScaleGradient: Bookmark website now!

[Tromat]

I struggled with normalization on an image of M78 I captured this winter over several nights, and this script solved the problem easily and efficiently. That's a great addition for Pixinsight, thanks a lot

 

[robyx]

@jmurphy thanks for the extensive response, I think that I still an open point to fully solve the puzzle (that probably exists only in my mind )

Using noiseMRS on its own is not sufficient. The image (brightness) scale factor is crucial. If an image is scaled by a factor of 2, the measured noise will also increase by the same factor. Since the image weight is proportional to 1 / noise^2, an invalid scale factor will also have a very strong impact on the calculated weight. In NSG I have already applied the scale factor to the normalized images, so I don't need to include the scale factor in the equation (that would apply it twice).

This is exactly what I mean by using ImageIntegration's noise evaluation on the nsg-corrected images. Let me use a bit of simple math to clarify without ambiguity.

Let's call the initial frame noise s, during the processing you subtract a 2D-spline background and, after the subtraction, you scale the entire image by a single scale factor of k. This makes the noise of your image become k*s, but to be more precise you don't use this scaling formula, you measure directly the noise after the correction, which we call s'. Now, since subtracting the 2d-spline to the image could have an impact on the noise measurement (depending on the scale of the spine and the noise) let's assume that in general the identity k*s = s' is only an approximation (maybe a very good one or a bad one depending on the case) and the correct noise is the one measured on the corrected/scaled image s'.

The weights are then computed as

w = sr / s'

where w is the FWEIGHT value as the ratio between the reference image noise sr (which is constant for all frames) and the nsg-corrected frame noise s'. (please confirm that I am not wrong here )

Now, let's get to the core point of my question: when Image Integration measures the noise of a nsg-corrected frames it measures s', which is the noise of the image "already scaled"; being already scaled we don't scale it anymore and the noise-based weight becomes

w' = 1 / s'

where w' is the weight of the ImageIntegration noise-based method.

So, if I didn't commit any conceptual error and I didn't misinterpret the JS code and how the script works, I would conclude that

w = w' * sr

proving that the weights used by image integration and the weights stored in FWEIGHT should be proportional by a constant factor.

So, since using as weights [1 2 3 4 5] or any other weights given by n * [1 2 3 4 5] gives the same result, would conclude that the two weighting strategies should produce the same master file because the two weights are proportional each other (n=sr).


I hope I was clear enough and sorry for maybe being tedious with such formalities but I am really interested in understanding this weighting strategy which looks great!

Robyx

 

[johnpane]

I think what @robyx is saying is that if ImageIntegration is set to calculate weights based on noise evaluation of the NSG-corrected subframes (rather than using FITS keyword weights that were previously calculated on uncorrected subframes), it will essentially weight the subframes the same as if it uses the NSG-generated weights.

 

[robyx]

I think what @robyx is saying is that if ImageIntegration is set to calculate weights based on noise evaluation of the NSG-corrected subframes (rather than using FITS keyword weights that were previously calculated on uncorrected subframes), it will essentially weight the subframes the same as if it uses the NSG-generated weights.

yes! as far as I understood, this is not the case!
I've just tried to be more precise in my reasons to better understand why

 

[rab722]

It's not apparent when the script finishes that it will only launch ImageIntegration when you exit. Maybe a note at the bottom of the script page or an Integrate button to that is displayed after the script finishes.

 

[jmurphy]

OK, I think I understand!
So I just ran NSG on some images:

Summary
[1] NGC6946_L_0001_c_r, 300s, NWEIGHT 0.58349
[2] NGC6946_L_0002-1_c_r, 300s, NWEIGHT 0.92081
[3] NGC6946_L_0002_c_1_r, 300s, NWEIGHT 0.86473
[4] NGC6946_L_0002_c_r, 300s, NWEIGHT 0.78653
[5] NGC6946_L_0003_c_1_r, 300s, NWEIGHT 0.92694
[6] NGC6946_L_0003_c_r, 300s, NWEIGHT 0.72743
[7] NGC6946_L_0004_c_r, 300s, NWEIGHT 0.95345
[8] NGC6946_L_0005_c_r, 300s, NWEIGHT 0.95708
[9] NGC6946_L_0006_c_1_r, 300s, NWEIGHT 1.0

If I now run ImageIntegration on the NSG normalised images, specifying noise evaluation instead of relying on the NWEIGHT keyword, this is the output we get for image [1]

(Image [9] was used as the reference frame for both NSG and ImageIntegration)

ImageIntegration console output for image [1]
F:/00-TMP/NormalizeTests/FireworksGalaxy/tmp/NGC6946_L_0001_c_r_nsg.fit
Scale factors : (7.862349e-01,8.366488e-01)
Zero offset : -3.035806e-05
Noise estimates : 8.6055e-04
Weight : 0.94396

ImageIntegration calculated the Weight as 0.94396
NSG calculated the Weight as 0.58349

Why is there a large discrepancy here? Well, nobody told ImageIntegration that the input files were already normalized. So it has tried to determine the scale factor for each image. We can see that for image [1] it incorrectly thought that the scaling factor was 0.7862349, instead of the correct 1.0

If we square the ImageIntegration scale factor, and multiply it by the ImageIntegration Weight, we get
0.7862349^2 * 0.94396 = 0.58352
Which gives us the NSG Weight!

So we can see that the discrepancy is entirely due to the invalid scale factor.

Hope this helps!
John Murphy

 

[johnpane]

Why is there a large discrepancy here? Well, nobody told ImageIntegration that the input files were already normalized. So it has tried to determine the scale factor for each image. We can see that for image [1] it incorrectly thought that the scaling factor was 0.7862349, instead of the correct 1.0

What would happen if you set Normalization to "No normalization"?

 

[jmurphy]

What would happen if you set Normalization to "No normalization"?

It makes no difference. If Weights is set to Noise evaluation within ImageIntegration, to make the noise evaluation meaningful, the scale factor is always calculated and used within the weight calculation. Any errors in that scale factor will result in the square of that error in the calculated weight.

 

[jlodge]

This looks great, I am excited to try it. I use weights from subframe selection though during Image Integration, will this no longer be possible if using NSG?

 

[jmurphy]

This looks great, I am excited to try it. I use weights from subframe selection though during Image Integration, will this no longer be possible if using NSG?

NSG writes the weights into the FITSHeader NWEIGHT. The user can then decide to use this FITSHeader for the weight, or set ImageIntegration to use a different FITSHeader weight keyword.

Personally I would recommend using the NSG NWEIGHT for galaxies and nebula, and SubFrameSelector for star clusters and globular clusters. The reason for this is that for faint objects, resolution is mainly determined by the signal to noise ratio. I believe that NSG NWEIGHT will be optimum for this.

Regards, John Murphy

 

[rockenrog]

Hi John,
When I use the script, the Altitude column is not filled in.
In a typical FITS header I find:
OBJCTALT '39.9289' Nominal altitude of center of image
I am using Maxim DL6. Any ideas to get it to show up?
You have the 7 files I dropboxed to you when you were at Ver 0.8.
Remember I had the focal length rejection becuase it was not an integer...

Thanks, Roger

 

[jmurphy]

Hi John,
When I use the script, the Altitude column is not filled in.
In a typical FITS header I find:
OBJCTALT '39.9289' Nominal altitude of center of image
I am using Maxim DL6. Any ideas to get it to show up?
You have the 7 files I dropboxed to you when you were at Ver 0.8.
Remember I had the focal length rejection becuase it was not an integer...

Thanks, Roger

Thanks for reporting it.
It appears that MaximDL is storing the altitude as a string instead of a number.
I will try and handle this in the NormalizeScaleGradient version 1.1

 

[niccoc1603]

It is not currently compatible with drizzle.

Will it be possible in future releases?

 

[jmurphy]

Will it be possible in future releases?

I first need to port the code to C++.
It should then be possible to create a drizzle option.
Since this is all done in my spare time, there is likely to be a significant wait...

Regards, John Murphy

 

[nico1038]

Hello @jmurphy and thanks for this great script.

Concerning drizzling I did try the following :

1) I used WBPP to process and registered my images and to generate drizzle data.
2) I used your script to normalize the registered files.
3) I renamed the normalized files, removing the "_nsg" suffix
4) I used ImageIntegration with these renamed files and with the drizzles files generated in the first step (and with the option "Generate drizzle data")
5) Finally I did a drizzle integration with the resulting drizzle files

I have a poor understanding of either your script or drizzling so maybe it makes no sense at all but it seems to work: the resulting image seems ok to me and the drizzling did the job.


What do you think?

 

[jmurphy]

When I use the script, the Altitude column is not filled in.
In a typical FITS header I find:
OBJCTALT '39.9289' Nominal altitude of center of image
I am using Maxim DL6. Any ideas to get it to show up?

I have updated the code to cope with the altitude being stored as either a number or a string. In your case, MaximDl is storing it as a string.

 

[jlodge]

NSG writes the weights into the FITSHeader NWEIGHT. The user can then decide to use this FITSHeader for the weight, or set ImageIntegration to use a different FITSHeader weight keyword.

Personally I would recommend using the NSG NWEIGHT for galaxies and nebula, and SubFrameSelector for star clusters and globular clusters. The reason for this is that for faint objects, resolution is mainly determined by the signal to noise ratio. I believe that NSG NWEIGHT will be optimum for this.

Regards, John Murphy

Thank you John, I was not aware of that, also thank you for taking the time to make NSG.

 

[Larry Phillips]

Your change regarding the altitude worked for me.
Larry

 

[jmurphy]

It's not apparent when the script finishes that it will only launch ImageIntegration when you exit. Maybe a note at the bottom of the script page or an Integrate button to that is displayed after the script finishes.

Version 1.1 will be available though the update system very soon. In the new version, the tooltip explains that ImageIntegration launches on NSG's exit. This is also described in the updated help file (quick start section).

The reason I don't display ImageIntegration straight away after normalization has finished is that the JavaScript processes are modal. While a script is running, you cannot interact with any other process.

Regards, John Murphy

 

[tdgm]

I am using 1.1 Beta 2 and every time I try ti use the script I getthe following error message: I am using 1.1 Beta 2

*** Error [022]: /Applications/PixInsight/src/scripts/JohnMurphy/NormalizeScaleGradient/NormalizeScaleGradient.js, line 74: TypeError: (intermediate value).getExposure is not a function

With original 1.0 I am able to run program correctly.

I am using this on a Mac Book Pro