NormalizeScaleGradient: Bookmark website now!

Status
Not open for further replies.
Minusman said:
Hi Jim, thanks for pointing that out. I have tried it with the correction factor, now the weighting looks better. How did you arrive at the value of 0.2?
Click to expand...
The binned images had a weight of about 200%, so I calculated a scale factor to reduce them to about 7%
So, we want to reduce the binned weights by 7 / 200
NWEIGHT is the signal to NOISExx ratio squared.
By dividing the signal by a correction factor , we reduce the calculated signal to NOISExx ratio.
Hence we want the signal^2 to be reduced by 7 / 200
Hence signal reduction needs to be the square root of (7 / 200), which is 0.19

Regards, John
 
Hi Jim, I think the problem was true the images I had calibrated and registered in another software. The NOISExx headers were missing. I have calibrated and registered everything again with the WBPP script. Now the weighting values look different and are somewhat lower on average. I think it should be correct now.
 
NormalizeScaleGradient 1.6.1 (Bug fix)
  • Improved gradient correction.
  • Images opened by double clicking on a target list image now only have an STF applied. Previously it was stretched, which may have been confusing.
  • If focal length or pixel size is missing from the FITS header, an attempt is now made to read them from the view's properties.
  • Gradient images are now full size.
  • If there is no filter name, this is now displayed as "" instead of "NONE"
  • FITS Headers are now read from the PixInsight 'FileFormatInstance' instead of a JavaScrpt method. In rare cases, the JavaScript method could fail to find all headers.
Install
Unzip the script to a folder of your choice.
In the PixInsight SCRIPTS menu, select 'Feature Scripts...'
Select 'Add' and navigate to the folder.
Select 'Done'
The script will now appear in 'SCRIPTS > Batch Processing > NormalizeScaleGradient 1.6'

I will be submitting this version to PixInsight.
Regards, John Murphy
 

Attachments

  • NormalizeScaleGradient.zip
    118.3 KB · Views: 107
Last edited:
Dear John,

first of all - thanks for this fantastic script, it is a fix part of my image processing workflow (for DSLR as well as for mono) and made a major impact.

For my mono data I also like to use MURE denoise, and encounter an issue: when NSG rescales (either when I do it by hand (before NSG1.5 - your earlier recommendation in case of truncations) or let NSG do the rescaling, MURE results in very unsatisfactory results for the NSG stack when I have considerable scaling factors (say a factor 4). It appears as if the values delivered by the detector script are no appropriate for the NSG scaled noise pattern of the stack?! Maybe one needs to appropriately scale the gain/Gaussian noise settings for MURE, but how? Any suggestion?

Matthias
 
GalacticRAVE said:
first of all - thanks for this fantastic script, it is a fix part of my image processing workflow (for DSLR as well as for mono) and made a major impact.
Click to expand...
Good to hear! :)

GalacticRAVE said:
For my mono data I also like to use MURE denoise, and encounter an issue: when NSG rescales (either when I do it by hand (before NSG1.5 - your earlier recommendation in case of truncations) or let NSG do the rescaling, MURE results in very unsatisfactory results for the NSG stack when I have considerable scaling factors (say a factor 4). It appears as if the values delivered by the detector script are no appropriate for the NSG scaled noise pattern of the stack?! Maybe one needs to appropriately scale the gain/Gaussian noise settings for MURE, but how? Any suggestion?
Click to expand...
My earlier advice to rescale the images before using NSG to provide more headroom was wrong :( . Doing so would reduce the accuracy of NWEIGHT.

Asking NSG to rescale the data fixes the weighting problems, but with very large scaling factors of x4, I can see that this could effect Mure denoise.

I have almost finished everything that is needed for NSG v2.0 In this new version, it will no longer be necessary to rescale the images, and NSG will finally be fully compatible with DrizzleIntegration. I have been working long hours and late into the nights to complete this. It should solve the main problem you are experiencing. Until then (probably some time next week), it may be necessary to turn the optional scaling off if you have very large scale differences and you want to use MURE denoise on the final stacked image.

Regards, John Murphy
 
Dear John,

excellent news, can't wait so play with NSG v2.0.

Regarding 1.6.1: as far as I understand the procedure the penalty for not rescaling (and thus truncating) would be that I clip the brightest stars (i.e. those pixels with values > 1/scale), do I see that correctly ?

Matthias
 
GalacticRAVE said:
Dear John,

excellent news, can't wait so play with NSG v2.0.

Regarding 1.6.1: as far as I understand the procedure the penalty for not rescaling (and thus truncating) would be that I clip the brightest stars (i.e. those pixels with values > 1/scale), do I see that correctly ?

Matthias
Click to expand...
Yes, in most situations, you only risk truncating the cores of the brightest stars.

Even if you rescale, if the number of images with saturated stars out number the images with a higher dynamic range, ImageIntegration's rejection methods will tend to reject the star peaks anyway.

Regards, John
 
Hi jmurphy,

I just updated to the script you've got for download up there in hopes of resolving a bug(?) where sometimes NSG would hang for extended periods of time when opening, loading files, etc. And indeed, it's running quickly now.

But I've also noticed a little more hand-holding. It requires me to use a postfix. It won't allow me to specify a reference image which isn't part of the subset. I've actually found some really neat uses for this script and have enjoyed flexibility in these areas. Are those changes necessary? Or am I an outlier and there is a practical reason for those restrictions.

Thank you for all the lovely work on this very useful tool.

James
 
Thank you for this script, I did my first real image with it (chosen because I did not want to drizzle anyway and had bad gradients), it worked well.

I hope it is not rude to ask, but is there a timeline for this becoming compatible with drizzle integration? Or is that just not in the cards anytime soon?
 
Linwood said:
Thank you for this script, I did my first real image with it (chosen because I did not want to drizzle anyway and had bad gradients), it worked well.

I hope it is not rude to ask, but is there a timeline for this becoming compatible with drizzle integration? Or is that just not in the cards anytime soon?
Click to expand...
I hope to release a version next week that is compatible with DrizzleIntegration.
The code is now complete, but I have a bit more testing to do.

Regards, John
 
jpeirce said:
... I've also noticed a little more hand-holding. It requires me to use a postfix. It won't allow me to specify a reference image which isn't part of the subset. I've actually found some really neat uses for this script and have enjoyed flexibility in these areas. Are those changes necessary? Or am I an outlier and there is a practical reason for those restrictions.
Click to expand...
OK, I will downgrade the 'overwrite with no postfix' error to a warning dialog.
I will add back the ability to specify a reference image that is not in the target list. This will be useful if it is necessary to split a very large set of images into smaller subsets.

Regards, John
 
Same (Windows), I'm building time on M82 and doing too much imaging with the moon, so have some good (i.e. bad) examples to test with, that won't matter if it breaks as I am far from finished accumulating data.
 
NormalizeScaleGradient 2.0 is now available. This version is finally compatible with DrizzleIntegration! Juan Conejero is not currently planning to include NSG in the next release of PixInsight, so you will need to download NSG, and its reference documentation, from my website:

sites.google.com

NormalizeScaleGradient

I have created a new website to host both NormalizeScaleGradient (NSG) and PhotometricMosaic (PMM): https://www.astroprocessing.com The NSG PixInsight Repository URL has also changed (see my new website for more information) Thank you!
sites.google.com sites.google.com

I would like to thank the 50 users that donated money, making this 2.0 release possible.
Thanks, John Murphy
 
Excellent for having this release but disappointed Juan doesn’t want to include this as part of the normal releases. It just complicates the life of Pixinsight users in the long run for no objective good reason. In any case keep the good work.
 
Is that because NSG will be an alternative to (as opposed to a complement to) the new LN stuff and weighting?
 
I think I'm doing something stupid, but I cannot get this to work.

I had several days of subs from M82 I had just done a drizzle integration for. I did WBPP through cosmetic correction, then did star alignment into the same folders as the CC files with drizzle generation, then integration, drizzle, etc. Worked.

So when this came out I installed it (good instructions no issue) and wanted to run it again. I usually just search (Agent Ransack) for all the files and drag and drop into integration, but NSG won't let me, so I successively loaded files from multiple folders for luminance into NSG. Checked normalization data and... had to pick an output directly, it won't write the xnml file to the same folder as the source, apparently, so I wrote them to yet another folder.

It finished, integration came up. Checked the drizzle box, and dragged all the xdrz files into integration (it properly showed <d>) and integrated. No issues.

Then I ran drizzle integration, and dragged all the xdrz files into it, then dragged all the xnml files in and ... it won't accept them. Says "No local normalization data files have been assigned to drizzle integration images". Did it all again from image integration checking "Static data targets" ... same result.

What am I missing? Don't I need the xnml files loaded in the drizzle integration screen?
 
Hmmm... I looked in the xnml file and see data like this:

Code: 
<xnml version="1.0" xmlns="http://www.pixinsight.com/xnml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com/xnml http://pixinsight.com/xnml/xnml-1.0.xsd">
   <CreationTime>2022-02-15T23:26:26.579Z</CreationTime>
   <ReferenceImage>T:/Stage/M82/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C_MORNING-20220204/LIGHT_M 82(1)_2022-02-04_01-09-54_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_2800m</ReferenceImage>
   <TargetImage>T:/Stage/M82/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C_MORNING-20220204/LIGHT_M 82(1)_2022-02-04_01-40-15_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_2800m</TargetImage>

The file names are badly truncated

The file name (picking up with Offset) looks like this:

Offset_50_bin_1x1_2800mm_-10.00C_0003_c_cc.xisf

The bolded part is not present in the file. NSG must have some kind of file name length limitation? Or file path, not sure which?

But I suspect that is the issue? Though I'm not at all sure why the integration worked with normalization if so.

Update: I don't know this is the root cause as NSG has a few more hours to run, but I did a star align again to a top level folder, then started NSG, and it's got the whole file name now, so it appears to be NSG has some path (not necessarily file) name length limit and is truncating the file names.

Will pick up in the morning after it runs, but if used with WBPP which can build really long paths, this can be a problem (assuming these file names are the actual problem I am having)

Update #2: With shorter file paths it worked fine. My NSG drizzle looked very like the regular drizzle, to the eye blinking no difference, but the regular had little gradient; I now need to do the other filters. But it's looking good. I suggest that NSG fail if it can't handle the file path lengths.
 
Last edited:
I haven’t used NSG so far because of the non compatibility with drizzle, but I’d be very keen now to try it.
 
Last edited:
John, the first test with 2.0 was successful and the xnml files work well. Therefore I want to thank you for this version explicitly! As NSG is getting a de facto standard in most workflows I know I can't understand why it wouldn't be part of the PI distribution.

Using xnml saves a lot of disk space, especially with large datasets, I just use the folder with the registered images as output folder, as it already contains the drizzle information. Therefore, a nice to have update would be a checkbox like "Use drizzle in integration" which would add the drizzle files and checks "Generate drizzle data" in the integration process where you could use an error catch if xdrz data exists on either source file location or target folder.
 
Status
Not open for further replies.
Back
Top