NormalizeScaleGradient: Bookmark website now!

Status
Not open for further replies.
Hi John - cracking script, I wish I had the skills to do just some simple scripting!

One thing that occurred to me is that it would be useful to be able to read the existing nsg files back in and just use the image integration section to see the impact of integrating at certain NWEIGHT minimums.... at the moment I am using the fits data view script to create a CSV of the files and weights and then manually create an integration which with 170 subs takes a long time to pick the right files! (unless I am missing a simple way of doing this already? perhaps I can do some spreadsheet formula bashing and paste in to a process icon source code..... )
 
jiberjaber said:
Hi John - cracking script, I wish I had the skills to do just some simple scripting!

One thing that occurred to me is that it would be useful to be able to read the existing nsg files back in and just use the image integration section to see the impact of integrating at certain NWEIGHT minimums.... at the moment I am using the fits data view script to create a CSV of the files and weights and then manually create an integration which with 170 subs takes a long time to pick the right files! (unless I am missing a simple way of doing this already? perhaps I can do some spreadsheet formula bashing and paste in to a process icon source code..... )
Click to expand...
I am currently porting the script to C++. It is slow going, because I am learning C++ as I go...
Once I have successfully completed the port, I will see what I can do.
 
rockenrog said:
Hi John,
I went back and processed my( M101), 100x600sec, @659mm, cooled CMOS color images using Ver1.1 of the script. I will break my results down into a couple of messages. My work may not be correct for those imagers that are located in suburban or dark sites. Although clouds can occur anywhere.

As I said before, my sky background is urban light polluted and some images with those pesky light clouds which may deceive weighting by normal integration's noise evaluation or using Subframe Selector. I think your script is a perfect match for high median images.
I found:
  1. The Ver1.1 script ran almost 2X faster than the old Ver0.8. Very important when setting the Gradient Correction Smoothness to 0.0 (vs default of 2). Now the run time was reduced to 47minutes on my Dell XPS15 -9570, apple to apple.
  2. It is very useful, and thus important, to study the Gradient Graph and then adjust the smoothness to reasonably follow the data of reference image. Look at each color one at a time on the graph to really see the gradient in detail. Move the graphed position up and down and left and right.
  3. The calculation of NWEIGHT is very sensitive, and varied from 0.31 to 1.6 for my chosen reference image.
  4. My reference image really did not have a simple gradient, so I expected DBE after integration would not be so easy. It was not easy! I think it is important for users to carefully choose the image with the best gradient-- the one that will be easiest for DBE. Don't just pick the image with the best looking signal.
  5. The script has an slider for Image Integration to set the minimum weight percentage to be used in Image Integration. Default is 50%. With this setting 44 of 100 images showed up selected in Image Integration. I integrated these 44 images, then 76 best NWEIGHT images, and then all 100 images.
  6. After integration, 76images had clearly less noisy background, and a bit more visibility in the galaxy arms. Integrating all 100 keyword NWEIGHT images was very, very slightly better. It certainly did not hurt the result to include the worst images. Their weight contribution was small also.
  7. The script generates a huge amount of data. My 3GB of 100 registered images became 18GB of .nsg files, ready for further processing. Does a 31MB image really need to become 191MB image?
  8. It is satisfying to me (my personal) to see the gradient graphs and photometry graphs (3 colors) and understand what the script is doing, and how the different settings will work on the data.
  9. The "zooming" in the Photometry Stars graph, and Detected Stars graph is very nice because the zooming in is centered on the cursor. I posted a request to have Pixinsight zooming to use this method. I always seem to get lost when zooming an image in PI.

Further Goals:
  1. One of my additional goals in using the NSG script is to quantitatively evaluate two, or more, integrated images to determine which is best to proceed for further integration. As I have written here before, there is no method to quantitatively measure if an image is better than another. Different PI book authors have written -- good signal to noise, and different processes can improve signal to noise. But none of the books has a method to quantify it. Thus my interest in using NWEIGHT to judge which image is better.
  2. I also want to confirm NWEIGHT is more reliable than normal Image Integration 'noise weight', and if perhaps a different parameter is good, such as Stars or Median in Subframe Selector.
  3. I want to confirm NWEIGHT results in a better looking image (sorry, no way to measure it) than the traditional Image Integration.
I will speak to these in a follow on post. Now I will say it again John....

Thank you for writing a great script, and making it very functional to use.

Roger
Click to expand...
+++++++++++++++++++
To continue, I made several comparison integrations....
  1. After using NSG script Ver1.1, I integrated the 100 .nsg files using the same reference image and NWEIGHT, as per above.
  2. Next I did normal integration on the original 100 calibrated, debayered, and registered images using noise evaluation for Weights, and Scale + zero offset in pixel rejection.
  3. I then ran Subframe Selector twice. Once outputting with STARSWEIGHT imbedded in the FITS headers, and another with MEDIANWEIGHT factor imbedded in the FITS headers.
  4. I integrated the 100 images using STARSWEIGHT weighting. Reference image same one as the NWEIGHT integration.
  5. I integrated the 100 images using MEDIANWEIGHT weighting. Reference image same one as the NWEIGHT integration.
  6. I don't know if valid to do so, but I then ran the NSG script on the 4 different integrated images, with the NSG integrated image as reference. I wanted to know if the NSG script could tell me which of the 4 images was best.
  7. Again, I don't know if valid, but I integrated the 4 integrations just to see what I would get.
Results:
  1. Visually I inspected the 4 integrations...
    • ***NWEIGHT image is slightly brighter in galaxy arms, and slightly darker between arms than others. (That elusive SNR?)
    • ***NWEIGHT image is slightly noisier than others, or maybe the noise is more defined/less blurred.
    • ***MEDIANWEIGHT image is very slightly brighter in galaxy arms then NoiseEvaluation integration. Backgrounds are identical.
    • ***STARSWEIGHT looked very slightly noisier than NoiseEvaluation integration. Stars measured very slightly brighter on STARSWEIGHT.
    • ***Visually, I would pick NWEIGHT to proceed post processing, but Item 5 below disagrees with me.
  2. I made several plots of NWEIGHT vs the other weights, and also my visual rating of each of the 100 images. See attached which has some conclusions.
  3. When I integrated the 4 integrated images (it worked!), I found these weights:
    1. Integrated NWeight = 1.00 as required.
    2. Normal Integration NWeight = .9599
    3. MEDIANWEIGHT from Subframe Selector = 1.083
    4. STARSWEIGHT from Subframe Selector =1.078
    5. According to NSG Script, both MEDIANWEIGHT AND STARSWEIGHT are better images! Is 1.083 visually better than 1.000? My descriptions in Item 1 do not say it is better. Maybe integrating integrated images is not valid.
  4. Interestingly, the Photometry Graphs of the 4 integrated-integrated images was extremely tight. Any comment John?
Please advise any comments/questions/data plots, or any other comparisons you would like to see.
Soon I want to delete all this 200GB of additional data created for this study. My SSD is stuffed.

Roger
 

Attachments

  • Comparison of NSG Ver1.1 with standard integration.png
    Comparison of NSG Ver1.1 with standard integration.png
    106.5 KB · Views: 88
  • Photometry graph of integrated images.png
    Photometry graph of integrated images.png
    69 KB · Views: 82
rockenrog said:
+++++++++++++++++++
To continue, I made several comparison integrations....
  1. After using NSG script Ver1.1, I integrated the 100 .nsg files using the same reference image and NWEIGHT, as per above.
  2. Next I did normal integration on the original 100 calibrated, debayered, and registered images using noise evaluation for Weights, and Scale + zero offset in pixel rejection.
  3. I then ran Subframe Selector twice. Once outputting with STARSWEIGHT imbedded in the FITS headers, and another with MEDIANWEIGHT factor imbedded in the FITS headers.
  4. I integrated the 100 images using STARSWEIGHT weighting. Reference image same one as the NWEIGHT integration.
  5. I integrated the 100 images using MEDIANWEIGHT weighting. Reference image same one as the NWEIGHT integration.
  6. I don't know if valid to do so, but I then ran the NSG script on the 4 different integrated images, with the NSG integrated image as reference. I wanted to know if the NSG script could tell me which of the 4 images was best.
  7. Again, I don't know if valid, but I integrated the 4 integrations just to see what I would get.
Results:
  1. Visually I inspected the 4 integrations...
    • ***NWEIGHT image is slightly brighter in galaxy arms, and slightly darker between arms than others. (That elusive SNR?)
    • ***NWEIGHT image is slightly noisier than others, or maybe the noise is more defined/less blurred.
    • ***MEDIANWEIGHT image is very slightly brighter in galaxy arms then NoiseEvaluation integration. Backgrounds are identical.
    • ***STARSWEIGHT looked very slightly noisier than NoiseEvaluation integration. Stars measured very slightly brighter on STARSWEIGHT.
    • ***Visually, I would pick NWEIGHT to proceed post processing, but Item 5 below disagrees with me.
  2. I made several plots of NWEIGHT vs the other weights, and also my visual rating of each of the 100 images. See attached which has some conclusions.
  3. When I integrated the 4 integrated images (it worked!), I found these weights:
    1. Integrated NWeight = 1.00 as required.
    2. Normal Integration NWeight = .9599
    3. MEDIANWEIGHT from Subframe Selector = 1.083
    4. STARSWEIGHT from Subframe Selector =1.078
    5. According to NSG Script, both MEDIANWEIGHT AND STARSWEIGHT are better images! Is 1.083 visually better than 1.000? My descriptions in Item 1 do not say it is better. Maybe integrating integrated images is not valid.
  4. Interestingly, the Photometry Graphs of the 4 integrated-integrated images was extremely tight. Any comment John?
Please advise any comments/questions/data plots, or any other comparisons you would like to see.
Soon I want to delete all this 200GB of additional data created for this study. My SSD is stuffed.

Roger
Click to expand...
Hi Roger,

Was the image with the invalid NWEIGHT the registration reference image?

I was expecting the registration reference frame to behave differently because it is the only frame that was not interpolated. This will affect the MRS noise evaluation. If the interpolation applies sharpening, this may also affect the photometry.

It was interesting to actually see it in a graph. In your case, the NWEIGHT error looks big enough to make a measurable difference. To have to manually adjust the FITS header NWEIGHT for this image would be a bit of a pain!

I am currently porting the script to C++. Once I have completed this, it should be possible to make the weight relate to the unregistered images, which would both fix this problem, and make the other weights more accurate.

Regards, John Murphy
 
John,
"Was the image with the invalid NWEIGHT the registration reference image?"
--- No, I did not have an invalid NWEIGHT. The point I was making is the question to you: Is it valid (correct usage) to use the NSG script on a series of already integrated images?
--- I think your next two comments were related to above. If not, then I don't understand them.

Your C++ comment.
---Not sure what problem you are referring to. But I do look forward to any updates.

Roger
 
rockenrog said:
Is it valid (correct usage) to use the NSG script on a series of already integrated images?
Click to expand...
Yes, this could be a valid use of NSG. The NWEIGHT produced should be a good guide to the (relative) signal to noise ratio between the integrated images. It is, of course, a relative measurement, so it is only valid for images from a single NSG run.
 
Code: 
*** Error: *** Error: /Applications/PixInsight/src/scripts/JohnMurphy/NormalizeScaleGradient/NormalizeScaleGradient.js, line 35: include file not found: lib/FitsHeaderReader.js

Previous mentions of this error link back to a now-defunct copy of 1.1a for a fix. Where can we download 1.2 to fix missing files?
 
neverfox said:
Code: 
*** Error: *** Error: /Applications/PixInsight/src/scripts/JohnMurphy/NormalizeScaleGradient/NormalizeScaleGradient.js, line 35: include file not found: lib/FitsHeaderReader.js

Previous mentions of this error link back to a now-defunct copy of 1.1a for a fix. Where can we download 1.2 to fix missing files?
Click to expand...
Sometimes windows AV programs intercept some subset of the javascript files, which comprises an update and break things, so check if you are running AV software and if it's quarantined that file.
 
jmurphy said:
Sometimes windows AV programs intercept some subset of the javascript files, which comprises an update and break things, so check if you are running AV software and if it's quarantined that file.
Click to expand...
I'm on a Mac and I'm not running any AV software.
 
Here is what I see in the lib folder after the 1.2 update:

/Applications/PixInsight/src/scripts/JohnMurphy/NormalizeScaleGradient/lib Permissions Size User Date Modified Name .rw-rw-r-- 6.6k root 7 Jul 09:13 FitsHeader.js .rw-rw-r-- 26k root 7 Jul 09:13 NsgData.js .rw-rw-r-- 64k root 7 Jul 09:13 NsgDialog.js
 
I'm on a Mac and just did the upgrade from 1.1 to 1.2. The list of files in that directory did not change:

Code: 
-rw-rw-r--  1 root  admin  19225 Jul  7 10:13 Cache.js
-rw-rw-r--  1 root  admin   9059 Jul  7 10:13 DetectedStarsDialog.js
-rw-rw-r--  1 root  admin  21504 Jul  7 10:13 DialogControls.js
-rw-rw-r--  1 root  admin   6562 Jul  7 10:13 FitsHeader.js
-rw-rw-r--  1 root  admin   8842 Jul  7 10:13 FitsHeaderReader.js
-rw-rw-r--  1 root  admin   8166 Jul  7 10:13 Gradient.js
-rw-rw-r--  1 root  admin  17231 Jul  7 10:13 GradientGraph.js
-rw-rw-r--  1 root  admin  15795 Jul  7 10:13 GradientGraphDialog.js
-rw-rw-r--  1 root  admin  22000 Jul  7 10:13 Graph.js
-rw-rw-r--  1 root  admin   4936 Jul  7 10:13 ImageScaleDialog.js
-rw-rw-r--  1 root  admin   5424 Jul  7 10:13 LeastSquareFit.js
-rw-rw-r--  1 root  admin  26407 Jul  7 10:13 NsgData.js
-rw-rw-r--  1 root  admin  64223 Jul  7 10:13 NsgDialog.js
-rw-rw-r--  1 root  admin  20733 Jul  7 10:13 PhotometryGraphDialog.js
-rw-rw-r--  1 root  admin  19944 Jul  7 10:13 PhotometryStarsDialog.js
-rw-rw-r--  1 root  admin  17817 Jul  7 10:13 PreviewControl.js
-rw-rw-r--  1 root  admin   4762 Jul  7 10:13 STFAutoStretch.js
-rw-rw-r--  1 root  admin  24261 Jul  7 10:13 SampleGrid.js
-rw-rw-r--  1 root  admin  21857 Jul  7 10:13 SampleGridDialog.js
-rw-rw-r--  1 root  admin  26254 Jul  7 10:13 StarDetector.jsh
-rw-rw-r--  1 root  admin  39868 Jul  7 10:13 StarLib.js
 
Last edited:
Wow, so the update definitely failed (despite being successful as far as the update system was concerned). I was running fine on v1.1. A reinstall of PI, followed by an update, fixed it.
 
Last edited:
NormalizeScaleGradient v1.2.1
Update history:

v1.2 (Released)
Added the FITS header "ALTITUDE" to the list of headers read to determine the image altitude.
The ImageIntegration rejection algorithm is now determined from the number of images. You should still check the rejection settings though!

v1.2.1 (very minor update!)
Improved the FITS header entries written to the NSG files.

Install
Unzip the script to a folder of your choice.
In the PixInsight SCRIPTS menu, select 'Feature Scripts...'
Select 'Add' and navigate to the folder.
Select 'Done'
The script will now appear in 'SCRIPTS > Batch Processing > NormalizeScaleGradient'
 

Attachments

  • NormalizeScaleGradient.zip
    102.8 KB · Views: 84
Last edited:
When I load the images and run the script (pretty much all default values), I get a a "Too few samples to create a surface spline" error. What am I doing wrong? I am using 600s mono subs.
 
Post an image of the 'Sample generation' dialog. The red grid of squares shows the auto generated samples. The error occurs if there are less than 3 samples.
 
window.JPG


No samples were generated with "Auto" selected. I reduced the sample size below 20 and now I see the grid of squares appearing.
 
If you could post a link to the reference frame and one target frame I will have a look at why the 'Auto' settings failed.

My guess is that:
(1) That 'Limit stars %' is 0.000 probably means that hot pixels are been detected as stars (check the 'Detected stars' dialog). Increase the Star detection setting to be less sensitive.

(2) Cold pixels are creating black holes (zeros) in the image. Version 1.2 is more tolerant of black holes, but it will still reject samples if they contain too many. If a sample has an area of N x N, it will be rejected if it contains more than N black pixels. Version 1.0 would not tolerate any black pixels.

If I am right (hard to be sure without seeing the data) CosmeticCorrection may be the best way forwards.
Regards, John Murphy
 
John,

My images are captured using INDI. With a GEM it reports RA and DEC but not altitude. It does report Airmass. Could NSG make use of Airmass to rank the subframes by altitude? I don't notice any other FITS keyword that would work, but maybe I'm missing something.

Thanks,
John

Code: 
File Type                       : FITS
File Type Extension             : fits
MIME Type                       : image/fits
Bitpix                          : 16
Naxis                           : 2
Naxis 1                         : 9576
Naxis 2                         : 6388
Extend                          : T
Bzero                           : 32768
Bscale                          : 1
Roworder                        : TOP-DOWN
Instrument                      : ZWO CCD ASI6200MC Pro
Telescope                       : LX850
Observer                        : Unknown
Object                          : M81_M82
Exptime                         : 1.200000e+02
Ccd-temp                        : -1.00e+01
Pixsize 1                       : 3.760000e+00
Pixsize 2                       : 3.760000e+00
Xbinning                        : 1
Ybinning                        : 1
Xpixsz                          : 3.760000e+00
Ypixsz                          : 3.760000e+00
Frame                           : Light
Imagetyp                        : Light Frame
Xbayroff                        : 0
Ybayroff                        : 0
Bayerpat                        : RGGB
Focallen                        : 6.75e+02
Aptdia                          : 1.30e+02
Scale                           : 1.149218e+00
Sitelat                         : 4.064333e+01
Sitelong                        : -8.011583e+01
Airmass                         : 1.139418e+00
Objctra                         :  9 55 28.35
Objctdec                        : 69 18 38.92
Ra                              : 1.488681e+02
Dec                             : 6.931081e+01
Equinox                         : 2000
Crval 1                         : 1.4886813095e+02
Crval 2                         : 6.9310812380e+01
Radecsys                        : FK5
Ctype 1                         : RA---TAN
Ctype 2                         : DEC--TAN
Crpix 1                         : 4.7880000000e+03
Crpix 2                         : 3.1940000000e+03
Secpix 1                        : 1.1492184809e+00
Secpix 2                        : 1.1492184809e+00
Cdelt 1                         : 3.1922735580e-04
Cdelt 2                         : 3.1922735580e-04
Crota 1                         : 8.9256800000e+01
Crota 2                         : 8.9256800000e+01
Observation Date                : 2021-03-20T03:43:24.821
Gain                            : 100.
Offset                          : 20.
Fratio                          : 5.19230769230769
 
Status
Not open for further replies.
Back
Top