NormalizeScaleGradient 1.4.3

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
John,

My images are captured using INDI. With a GEM it reports RA and DEC but not altitude. It does report Airmass. Could NSG make use of Airmass to rank the subframes by altitude? I don't notice any other FITS keyword that would work, but maybe I'm missing something.

Thanks,
John

Code:
File Type                       : FITS
File Type Extension             : fits
MIME Type                       : image/fits
Bitpix                          : 16
Naxis                           : 2
Naxis 1                         : 9576
Naxis 2                         : 6388
Extend                          : T
Bzero                           : 32768
Bscale                          : 1
Roworder                        : TOP-DOWN
Instrument                      : ZWO CCD ASI6200MC Pro
Telescope                       : LX850
Observer                        : Unknown
Object                          : M81_M82
Exptime                         : 1.200000e+02
Ccd-temp                        : -1.00e+01
Pixsize 1                       : 3.760000e+00
Pixsize 2                       : 3.760000e+00
Xbinning                        : 1
Ybinning                        : 1
Xpixsz                          : 3.760000e+00
Ypixsz                          : 3.760000e+00
Frame                           : Light
Imagetyp                        : Light Frame
Xbayroff                        : 0
Ybayroff                        : 0
Bayerpat                        : RGGB
Focallen                        : 6.75e+02
Aptdia                          : 1.30e+02
Scale                           : 1.149218e+00
Sitelat                         : 4.064333e+01
Sitelong                        : -8.011583e+01
Airmass                         : 1.139418e+00
Objctra                         :  9 55 28.35
Objctdec                        : 69 18 38.92
Ra                              : 1.488681e+02
Dec                             : 6.931081e+01
Equinox                         : 2000
Crval 1                         : 1.4886813095e+02
Crval 2                         : 6.9310812380e+01
Radecsys                        : FK5
Ctype 1                         : RA---TAN
Ctype 2                         : DEC--TAN
Crpix 1                         : 4.7880000000e+03
Crpix 2                         : 3.1940000000e+03
Secpix 1                        : 1.1492184809e+00
Secpix 2                        : 1.1492184809e+00
Cdelt 1                         : 3.1922735580e-04
Cdelt 2                         : 3.1922735580e-04
Crota 1                         : 8.9256800000e+01
Crota 2                         : 8.9256800000e+01
Observation Date                : 2021-03-20T03:43:24.821
Gain                            : 100.
Offset                          : 20.
Fratio                          : 5.19230769230769
Great idea!
OK, in version 1.2.2, if no altitude header is found, it uses the "AIRMASS" header.
The last target image read sets the column header to either "Airmass" or "Altitude".
The C++ version (in development) will include "Airmass" as an extra column.
Regards, John Murphy

[Edit: deleted NormalizeScaleGradient (new version attached to next post)]
 
Last edited:

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
In this version I have increased the displayed "Airmass" precision to 4 decimal places.

[Edit: Version 1.3 now attached to message #193]
 
Last edited:

grsotnas

New member
Nov 20, 2019
2
0
John, first off I would like to thank you for developing such an awesome tool in our integration arsenal. I have used NSG successfully on two previous images, and it did a fabulous job.

However, with a large dataset (243 images on M83, multiple nights, imperfect frame-matching/rotation, 2 different camera configurations (but properly registered)), I keep getting a similar error message: NWEIGHT is being evaluated at infinity.

NSG_InfinityError.JPG


I initially tried to run NSG in the entire dataset, which resulted in errors as this one in several of the files. I then started debugging, rechecking file types and formats, registration errors, tried different reference frames...

After a long time I managed to isolate a small reproductible simple sample: 2 files, 1 to be selected as the reference, and one target which always returns the error. I send attatched screenshots. Both images were shot in the same night, with a difference of about 1h between then - so, it shouldn't have had problems.

Setup: OSC debayered and registered images from DSLR cameras (2 different cameras, T5 and 6D) using a 6" newtonian telescope. Settings on FL/pixelsize/imagescale are correct. Unfortunately, my DSLR FITS files have no altitude/airmass or RA/DEC headers.

Workflow: all images fully calibrated (bias/flat/dark) and registered in AstroPixelProcessor. The resulting calibrated frames were saved as FITS. Then I opened the files in NSG and ran the script with the settings below (mostly defaults).

NSG_Settings.JPG

Finally, here are the Photometry and Gradient graphs for the TARGET image, which look OK to me.

NSG_PhotometryGraph.JPG NSG_GradientGraph.JPG

I am not sure I am missing something or have an incorrect setting.

If possible, I would appreciate some light into the problem.
Please say if I can do anything to better diagnose it or help in finding a solution.

Thank you very much!

Best regards,
Gabriel
 

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
John, first off I would like to thank you for developing such an awesome tool in our integration arsenal. I have used NSG successfully on two previous images, and it did a fabulous job.

However, with a large dataset (243 images on M83, multiple nights, imperfect frame-matching/rotation, 2 different camera configurations (but properly registered)), I keep getting a similar error message: NWEIGHT is being evaluated at infinity.

View attachment 11576


I initially tried to run NSG in the entire dataset, which resulted in errors as this one in several of the files. I then started debugging, rechecking file types and formats, registration errors, tried different reference frames...

After a long time I managed to isolate a small reproductible simple sample: 2 files, 1 to be selected as the reference, and one target which always returns the error. I send attatched screenshots. Both images were shot in the same night, with a difference of about 1h between then - so, it shouldn't have had problems.

Setup: OSC debayered and registered images from DSLR cameras (2 different cameras, T5 and 6D) using a 6" newtonian telescope. Settings on FL/pixelsize/imagescale are correct. Unfortunately, my DSLR FITS files have no altitude/airmass or RA/DEC headers.

Workflow: all images fully calibrated (bias/flat/dark) and registered in AstroPixelProcessor. The resulting calibrated frames were saved as FITS. Then I opened the files in NSG and ran the script with the settings below (mostly defaults).

View attachment 11577

Finally, here are the Photometry and Gradient graphs for the TARGET image, which look OK to me.

View attachment 11578 View attachment 11579

I am not sure I am missing something or have an incorrect setting.

If possible, I would appreciate some light into the problem.
Please say if I can do anything to better diagnose it or help in finding a solution.

Thank you very much!

Best regards,
Gabriel
Hi Gabriel,
Could you send me a link to the two images? I will look into this straight away.
Regards, John
 

rockenrog

Well-known member
Apr 7, 2018
87
9
After a long time I managed to isolate a small reproductible simple sample: 2 files, 1 to be selected as the reference, and one target which always returns the error. I send attatched screenshots. Both images were shot in the same night, with a difference of about 1h between then - so, it shouldn't have had problems.
AND
(2 different cameras, T5 and 6D)
Gabriel/John
Just FYI,
I successfully used NSG V1.1 on a set of 100 calibrated/registered images over 7 nights throughout April/May. My data though was all from one cooled QHY camera with only one camera setting, and processed entirely in Pixinsight.
Now I am looking to buy more data storage as .nsg file are BIG.
Roger
 

grsotnas

New member
Nov 20, 2019
2
0
Hi Gabriel,
Could you send me a link to the two images? I will look into this straight away.
Regards, John
Thank you very much for the support John.

I uploaded 4 images to GDrive, one reference (prefix REF) and three targets. Link here

Two of them return similar errors (prefix TARGET1_ and TARGET2_), one of them worked nicely (prefix TARGET_OK_).

The large black border is due to the complex registration with several rotation/camera combinations. Hopefully when integrating it should work OK (or I will then have to crop the integration to the intersection of all images).

Some metadata:
150mm f/5 newtonian with coma corrector, Canon EOS T5 (modded). 180" ISO 1600light frames. Calibrated with dark, flat and bias frames. Calibration and registration in APP.
Pixel size is 4.3um, FL 714mm. [1.24"/px]
Object is M83. Approx coordinates of center: 13h 37m 1s -29° 51' 56"

Best regards, Gabriel
 

johnpane

Well-known member
Jan 13, 2015
168
8
57
Wexford, PA, USA
Hi John,

I have certain "personal default" settings for ImageIntegration. Previously, I used a process icon to set these for me. Now, I would like to use the ImageIntegration settings that are pre-loaded by NSG. This means I have to manually re-set my personal defaults each time. Do you think there might be a way for the user to request that NSG start with a custom configuration for ImageIntegration then make the NSG-specific changes onto that config? Maybe something like, allow the user to specify a .xpsm file containing the desired defaults?

John
 

jiberjaber

Member
Feb 2, 2014
15
3
Hi John,

I have certain "personal default" settings for ImageIntegration. Previously, I used a process icon to set these for me. Now, I would like to use the ImageIntegration settings that are pre-loaded by NSG. This means I have to manually re-set my personal defaults each time. Do you think there might be a way for the user to request that NSG start with a custom configuration for ImageIntegration then make the NSG-specific changes onto that config? Maybe something like, allow the user to specify a .xpsm file containing the desired defaults?

John
....or a process icon and use the settings in that ? or you might be able to customise the sketch if you look at lines 529 on wards I think (v1.2.1)
 
  • Like
Reactions: jmurphy

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
you might be able to customise the script if you look at lines 529 on wards I think (v1.2.1)
This is probably the best way forward.
Right click on an ImageIntegration processIcon, and select "Edit Instance Source Code".
From the information displayed, it should be straight forward to customize the script.

Coding a more generic solution is quite a lot of work due to the large number of parameters. Maybe some time in the future...

Regards, John Murphy
 

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
John, first off I would like to thank you for developing such an awesome tool in our integration arsenal. I have used NSG successfully on two previous images, and it did a fabulous job.

However, with a large dataset (243 images on M83, multiple nights, imperfect frame-matching/rotation, 2 different camera configurations (but properly registered)), I keep getting a similar error message: NWEIGHT is being evaluated at infinity.

View attachment 11576


I initially tried to run NSG in the entire dataset, which resulted in errors as this one in several of the files. I then started debugging, rechecking file types and formats, registration errors, tried different reference frames...

After a long time I managed to isolate a small reproductible simple sample: 2 files, 1 to be selected as the reference, and one target which always returns the error. I send attatched screenshots. Both images were shot in the same night, with a difference of about 1h between then - so, it shouldn't have had problems.

Setup: OSC debayered and registered images from DSLR cameras (2 different cameras, T5 and 6D) using a 6" newtonian telescope. Settings on FL/pixelsize/imagescale are correct. Unfortunately, my DSLR FITS files have no altitude/airmass or RA/DEC headers.

Workflow: all images fully calibrated (bias/flat/dark) and registered in AstroPixelProcessor. The resulting calibrated frames were saved as FITS. Then I opened the files in NSG and ran the script with the settings below (mostly defaults).

View attachment 11577

Finally, here are the Photometry and Gradient graphs for the TARGET image, which look OK to me.

View attachment 11578 View attachment 11579

I am not sure I am missing something or have an incorrect setting.

If possible, I would appreciate some light into the problem.
Please say if I can do anything to better diagnose it or help in finding a solution.

Thank you very much!

Best regards,
Gabriel
The images have large areas of black due to registration. NSG requires these regions to be black in order to work correctly (it knows to ignore black areas).
Unfortunately, the noiseMRS() function I am using is measuring the noise for ALL pixels. On a few images, this causes noiseMRS() to return a noise level of zero. Not good!

The C++ version I am writing will be able to cope with this. I can then access the full API for noiseMRS() and tell it to ignore black pixels.

I can't do this for the JavaScript version, so in the attached version 1.3, when this situation arises, I calculate the weight from 1/scale instead. This will be an underestimate because it relies entirely on transmission and ignores the effect of light pollution. But it is much better than a divide by zero! I expect this condition to be extremely rare.
[Edit: Version 1.3 now attached to message #193]
Let me know if this works OK
John Murphy
 
Last edited:

johnpane

Well-known member
Jan 13, 2015
168
8
57
Wexford, PA, USA
Coding a more generic solution is quite a lot of work due to the large number of parameters. Maybe some time in the future...
John,

I think the following code would look for a process icon "NSG_ImageIntegration_Defaults" and use that to initialize ImageIntegration; otherwise it will create a new default instance as you do now. This would be inserted in your script where you currently have "let P = new ImageIntegration;"

Code:
let P = ProcessInstance.fromIcon("NSG_ImageIntegration_Defaults");
if (P===null || P.processId() != "ImageIntegration") {
   P = new ImageIntegration;
   // insert NSG's recommended settings here
}
// insert NSG's overriding required settings here
 

johnpane

Well-known member
Jan 13, 2015
168
8
57
Wexford, PA, USA
Some specific suggestions for placement of recommendations versus overriding requirements.

JavaScript:
let P = ProcessInstance.fromIcon("NSG_ImageIntegration_Defaults");
if (P===null || P.processId() != "ImageIntegration") {
   P = new ImageIntegration;
   // insert NSG's recommended settings here
    if (data.autoRejection){
        if ( nImages < 8 ){
            P.rejection = ImageIntegration.prototype.PercentileClip;
            P.pcClipLow = 0.2;
            P.pcClipHigh = 0.1;
        } else if ( nImages <= 10 ){
            P.rejection =  ImageIntegration.prototype.SigmaClip;
            P.sigmaLow = 3.5;
            P.sigmaHigh = 2.5;
        } else if ( nImages < 20 ){
            P.rejection =  ImageIntegration.prototype.WinsorizedSigmaClip;
            P.sigmaLow = 3.5;
            P.sigmaHigh = 2.2;
            P.winsorizationCutoff = 3.5;
        } else {
            P.rejection = ImageIntegration.prototype.Rejection_ESD;
            P.esdOutliersFraction = 3.2 / nImages;
            P.esdAlpha = 0.05;
            P.esdLowRelaxation = 1.50;
        }
}
// insert NSG's overriding required settings here
P.images = images;
P.combination = ImageIntegration.prototype.Average;
P.weightMode = ImageIntegration.prototype.KeywordWeight;
P.weightKeyword = data.noiseWeightKeyword;
P.normalization = ImageIntegration.prototype.NoNormalization;
P.rejectionNormalization = ImageIntegration.prototype.NoRejectionNormalization;
P.generateDrizzleData = false;
P.launchInterface();
 

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
Some specific suggestions for placement of recommendations versus overriding requirements.

JavaScript:
let P = ProcessInstance.fromIcon("NSG_ImageIntegration_Defaults");
if (P===null || P.processId() != "ImageIntegration") {
   P = new ImageIntegration;
   // insert NSG's recommended settings here
    if (data.autoRejection){
        if ( nImages < 8 ){
            P.rejection = ImageIntegration.prototype.PercentileClip;
            P.pcClipLow = 0.2;
            P.pcClipHigh = 0.1;
        } else if ( nImages <= 10 ){
            P.rejection =  ImageIntegration.prototype.SigmaClip;
            P.sigmaLow = 3.5;
            P.sigmaHigh = 2.5;
        } else if ( nImages < 20 ){
            P.rejection =  ImageIntegration.prototype.WinsorizedSigmaClip;
            P.sigmaLow = 3.5;
            P.sigmaHigh = 2.2;
            P.winsorizationCutoff = 3.5;
        } else {
            P.rejection = ImageIntegration.prototype.Rejection_ESD;
            P.esdOutliersFraction = 3.2 / nImages;
            P.esdAlpha = 0.05;
            P.esdLowRelaxation = 1.50;
        }
}
// insert NSG's overriding required settings here
P.images = images;
P.combination = ImageIntegration.prototype.Average;
P.weightMode = ImageIntegration.prototype.KeywordWeight;
P.weightKeyword = data.noiseWeightKeyword;
P.normalization = ImageIntegration.prototype.NoNormalization;
P.rejectionNormalization = ImageIntegration.prototype.NoRejectionNormalization;
P.generateDrizzleData = false;
P.launchInterface();
Great idea!

I have updated version 1.3 to include a 'Template icon':
1626164893126.png

Let me know if this works, John Murphy
 
Last edited:
  • Like
Reactions: jiberjaber

johnpane

Well-known member
Jan 13, 2015
168
8
57
Wexford, PA, USA
Hi John,

I am running NSG on monochrome subframes extracted from RGB using BatchChannelExtraction. BatchChannelExtraction inserts metadata for filter like this:
RGB --> R

When NSG checks the target frame's filter for consistency with the reference frame (which has the same filter identifier) it produces this warning:

Screen Shot 2021-07-14 at 05.49.28.png
 

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
I am running NSG on monochrome subframes extracted from RGB using BatchChannelExtraction. BatchChannelExtraction inserts metadata for filter like this:
RGB --> R

When NSG checks the target frame's filter for consistency with the reference frame (which has the same filter identifier) it produces this warning:
View attachment 11596
Hi John,
The code reads the FITS header in two different ways. If the image has been read, it can use PixInsight methods to read them. This works correctly. However, I also use code that can quickly read the FITS header without reading the whole file. Alas, this gets confused by the '>' in the filter name.
I have modified the code so that during the 'Run', the FITS headers are only read with the PixInsight methods. There is no performance penalty at this point because the images have to be fully read here anyway. This will hopefully fix the problem.
Regards,
John Murphy
 
Last edited:
  • Like
Reactions: johnpane

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
After having a conversation with Juan Conejero, I realized I could improve the calculated weights by using the NOISExx header entries created by ImageCalibration. Using noise estimations that were created before registration can significantly reduce scatter in the results. See graph:
1626553097220.png
Notice how the blue line has much less scatter.

If the NOISExx header does not exist, the code falls back to calculating the weight from the noise detected in the registered images.

So, how do you make sure that your images contain the NOISExx header(s)?
If you use WBPP, the headers will have been created.
If you use ImageCalibration directly, make sure that the 'Evaluate noise' check box is ticked.

Color images: All color channels are used to calculate the weight. NWEIGHT is set to the average weight. If you wish to use a weight from a specific channel (for example, using a narrow band filter with a color camera), you can set the ImageIntegration 'Weights: FITS keyword' to NWEIGHT0 (red), NWEIGHT1 (green), or NWEIGHT2 (blue).

The console 'Summary' text now shows both the average weight and the weights for all channels (provided NOISExx headers were available):
1626554179986.png

Enjoy! John Murphy
 
Last edited:
  • Like
Reactions: rbotero

johnpane

Well-known member
Jan 13, 2015
168
8
57
Wexford, PA, USA
Hi John,

I am curious about this. First, what are the xx in NOISExx?

I just calibrated a CFA RAW image and got keywords NOISE00, NOISEF00, and NOISEAOO (gaussian noise estimate, fraction of noise pixels, and noise algorithm). These are all labeled channel #0. So, apparently, there is only one noise estimate here, NOISE00. Why the notation NOISExx?

This is a color image and I would have expected if there would be multiple noise estimates it would be in a color image, one for each of three channels (and only one for monochrome images). If there could be more than one, the xx notation makes more sense.

Does DeBayer change the noise characteristics, and if so would it be better to use noise estimates generated by that process when applicable?

Finally, in your chart it seems curious that the two series do not even move in the same direction from frame to frame. For example, the red line decreases from [5] to [6] but the blue line increases. This seems surprising if the cause of the difference is simply registration.

Thanks,
John
 

rockenrog

Well-known member
Apr 7, 2018
87
9
JMurphy said:
"... I could improve the calculated weights by using the NOISExx header entries created by ImageCalibration. Using noise estimations that were created before registration can significantly reduce scatter in the results. See graph:..."
------------------------
I do not understand the innerworking of the script, just want to understand enough to use it correctly. So please bear with me.
My Questions:
-- Is the 'calculated weights' meaning keyword NWEIGHT?
-- Do you expect the new way will improve NWEIGHT variability?
In my previously discussed 100 images test of the NSG script I had the impression the resulting NWEIGHT was very sensitive with a range from .3 to about 1.7, I took that as a good thing!

JMurphy said:
"So, how do you make sure that your images contain the NOISExx header(s)?
If you use WBPP, the headers will have been created.
If you use ImageCalibration directly, make sure that the 'Evaluate noise' check box is ticked."
------------------------
Question: Then in WBPP 2.1.2, do I need to check generate subframe weights, generating the keyword WBPPWGHT?

Roger
 

jmurphy

PTeam Member
Jun 13, 2010
306
141
Basingstoke, England
-- Is the 'calculated weights' meaning keyword NWEIGHT?
-- Do you expect the new way will improve NWEIGHT variability?
In my previously discussed 100 images test of the NSG script I had the impression the resulting NWEIGHT was very sensitive with a range from .3 to about 1.7, I took that as a good thing!
Yes, NWEIGHT is the weight NSG calculates from the image noise estimate. The photometrically determined scale factor is used to scale the noise estimate. NWEIGHT is very sensitive to both the noise estimate and the scale.

When an image is registered, pixels get moved. Suppose an image is shifted by half a pixel. The new image pixels have to be calculated from their neighboring pixels. This process has an averaging effect on the noise, making the images look less noisy than they really are. The registration reference frame is not moved at all, so this looks much more noisy in comparison. By getting noise estimates from the images before registration, this false variation is removed. We want the script to be very sensitive to sky conditions, but not sensitive to the registration process.
Question: Then in WBPP 2.1.2, do I need to check generate subframe weights, generating the keyword WBPPWGHT?
NSG does not use the WBPPWGHT keyword. The color M101 image you sent me contained the NOISE00, NOISE01, and NOISE02 FITS headers, which is all that is needed.

John Murphy