NormalizeScaleGradient: Bookmark website now!

Status
Not open for further replies.
Linwood said:
Checked normalization data and... had to pick an output directly, it won't write the xnml file to the same folder as the source, apparently, so I wrote them to yet another folder.
Click to expand...
Wow, those are really long filenames! This makes an excellent test :)
Windows 10 has a limit on the maximum file path length of 260 characters. NSG can add a weight prefix and an '_nsg' postfix, and I think this was pushing the full filename over the limit. I believe it was this 260 limit that was preventing you from saving the files to the same folder as the source.

If you need filenames this long, I think you can change a setting in Windows 10 to allow long names:
docs.microsoft.com

Maximum Path Length Limitation - Win32 apps

Maximum path length limitation.
docs.microsoft.com

Linwood said:
Hmmm... I looked in the xnml file and see data like this:

Code: 
<xnml version="1.0" xmlns="http://www.pixinsight.com/xnml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com/xnml http://pixinsight.com/xnml/xnml-1.0.xsd">
   <CreationTime>2022-02-15T23:26:26.579Z</CreationTime>
   <ReferenceImage>T:/Stage/M82/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C_MORNING-20220204/LIGHT_M 82(1)_2022-02-04_01-09-54_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_2800m</ReferenceImage>
   <TargetImage>T:/Stage/M82/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C_MORNING-20220204/LIGHT_M 82(1)_2022-02-04_01-40-15_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_2800m</TargetImage>

The file names are badly truncated

The file name (picking up with Offset) looks like this:

Offset_50_bin_1x1_2800mm_-10.00C_0003_c_cc.xisf

The bolded part is not present in the file. NSG must have some kind of file name length limitation? Or file path, not sure which?

But I suspect that is the issue? Though I'm not at all sure why the integration worked with normalization if so.

Update: I don't know this is the root cause as NSG has a few more hours to run, but I did a star align again to a top level folder, then started NSG, and it's got the whole file name now, so it appears to be NSG has some path (not necessarily file) name length limit and is truncating the file names.

Will pick up in the morning after it runs, but if used with WBPP which can build really long paths, this can be a problem (assuming these file names are the actual problem I am having)
Click to expand...
Bug confirmed. Even if there is a OS limit to the maximum filename length, NSG should still be writing the full filename to the .xnml data files. Currently it is truncating them at 225 characters. A fix is on its way!

Thanks for reporting this
John Murphy
 
jmurphy said:
NormalizeScaleGradient 2.0 is now available. This version is finally compatible with DrizzleIntegration! Juan Conejero is not currently planning to include NSG in the next release of PixInsight, so you will need to download NSG, and its reference documentation, from my website:
Click to expand...

First of all thank you for the hard work. Had 2 projects already done with the normalization data and it worked flawlessly.
Second, the updates of the script will be still visible only on this thread or is it possible to be notified via email?

Thank you
 
Last edited:
jmurphy said:
OK, I will downgrade the 'overwrite with no postfix' error to a warning dialog.
I will add back the ability to specify a reference image that is not in the target list. This will be useful if it is necessary to split a very large set of images into smaller subsets.

Regards, John
Click to expand...
Thank you kindly, John. Very much appreciated!

I'm very excited to try NormalizeScaleGradient 2.0 with the new drizzle integration compatibility as soon as support comes to Macs. I've been looking forward to this since I first saw you mentioned it was coming.
 
Grim408 said:
First of all thank you for the hard work. Had 2 projects already done with the normalization data and it worked flawlessly.
Click to expand...
Thanks for the feedback. It's good to know it's working properly. :)

Grim408 said:
Second, the updates of the script will be still visible only on this thread or is it possible to be notified via email?

Thank you
Click to expand...
Updates will mentioned here. You can also try following me on https://ko-fi.com/jmurphy I will post a message whenever there is an update available.

Thanks, John Murphy
 
NSG 2.0.1 Release
For install instructions and downloads:
sites.google.com

NormalizeScaleGradient

I have created a new website to host both NormalizeScaleGradient (NSG) and PhotometricMosaic (PMM): https://www.astroprocessing.com The NSG PixInsight Repository URL has also changed (see my new website for more information) Thank you!
sites.google.com sites.google.com

NSG Script update:
  • Drizzle data files are now added to ImageIntegration.
  • Improved file I/O error handling.
Reference documentation
Version 2.0.1 is now available.

NSGXnml
(optional PixInsight C++ module, required for DrizzleIntegration compatibility)
  • Updated to remove the limit on file path length of 225 characters to the maximum allowed by the operating system
1645183300748.png


To install the script:
  • Unzip the script to a folder of your choice (but don't use a PixInsight folder).
  • In the PixInsight SCRIPTS menu, select 'Feature Scripts...'
  • Select 'Add' and navigate to the folder.
  • Select 'Done'
The script will now appear in 'SCRIPTS > Batch Processing > NormalizeScaleGradient 2.0'

Regards, John Murphy
 

Attachments

  • NormalizeScaleGradient_2.0.1.zip
    127 KB · Views: 120
Hi Jim, congratulations on the new version of NSG Script. The drizzle support through NSGxnml works perfectly.
The only thing I noticed is that since version 2.0 adding images to the script takes an unusually long time.
Depending on the number of images, it takes several minutes until they are displayed in the list. in previous versions, it usually took only seconds, even with hundreds of images.

With kind regards Henry
 
Minusman said:
Hi Jim, congratulations on the new version of NSG Script. The drizzle support through NSGxnml works perfectly.
The only thing I noticed is that since version 2.0 adding images to the script takes an unusually long time.
Depending on the number of images, it takes several minutes until they are displayed in the list. in previous versions, it usually took only seconds, even with hundreds of images.

With kind regards Henry
Click to expand...
Hmm, I used to use a JavaScript method to read the FITS headers. It does make a few assumptions, and this could result in some headers not being read. I am now reading them using the PixInsight FileFormatInstance, which as you would expect, produce solid results. However, there does seem to be a time penalty for this robustness.

Looks like I need to look into this, and also see if I can improve the performance of the actual normalization.

Regards, John Murphy
 
John, I must be misunderstanding something. When you stated "Drizzle data files are now added to ImageIntegration", I assumed I would see the drizzle files (<d> prefix) show up in the ImageIntegration Input Images window along with the normalized files (<n> prefix). Instead, I only see the <n> files. Also, the default values for Normalization and Pixel Rejection are set to Local normalization instead of No Normalization. Rejection algorithm, also, does not default to No rejection. Mind you, when I manually input the drizzle files into ImageIntegration and check for No normalization and No rejection (and perform DrizzleIntegration afterwards) the script works perfectly in both versions 2.0 and 2.0.1. I assume its operator error. What am I missing. Thanks

Scott
 
Went back and reviewed the video I used to learn how to use your script (VisibleDark site) and apparently drizzle files were not required for ImageIntegration and the Rejection algorithim should not be set to No rejection. Only issue then is why doesn't No normalization become the default settings and what do you mean by "Drizzle data files are now added to ImageIntegration" .Thanks.
 
dgaman said:
John, I must be misunderstanding something. When you stated "Drizzle data files are now added to ImageIntegration", I assumed I would see the drizzle files (<d> prefix) show up in the ImageIntegration Input Images window along with the normalized files (<n> prefix). Instead, I only see the <n> files. Also, the default values for Normalization and Pixel Rejection are set to Local normalization instead of No Normalization. Rejection algorithm, also, does not default to No rejection. Mind you, when I manually input the drizzle files into ImageIntegration and check for No normalization and No rejection (and perform DrizzleIntegration afterwards) the script works perfectly in both versions 2.0 and 2.0.1. I assume its operator error. What am I missing. Thanks

Scott
Click to expand...
When 'Normalization data' is checked, NSG creates '.xnml' data files. These data files specify the scale and gradient that needs to be applied to make the target images match the reference frame. After NSG finishes, it starts ImageIntegration, and gives it the original uncorrected files along with the '.xnml' data files. It also sets Normalization to 'Local normalization' which tells ImageIntegration to apply the corrections specified in the '.xnml' files. These '.xnml' files also need to be given to DrizzleIntegration so that it also knows how to correct it's unregistered and uncorrected input files.

If 'Drizzle data' is checked, NSG will look for '.xdrz' files with the same full path name as the NSG input files (the registered images). If these are found, they are added to ImageIntegration. If they are not found, warning messages are written to the console when ImageIntegration is invoked. These '.xdrz' files are updated by ImageIntegration, and should then be given to DrizzleIntegration.

If you are not using DrizzleIntegration, 'Drizzle data' should be left unchecked.

I hope this makes it a little bit clearer.
Regards, John Murphy
 
Last edited:
Thank you, John. This makes it perfectly clear. Just needed that tad bit more of information to make things click for me. Keep up the good work.
Scott
 
Hello John, regarding the loading time of images. I have reprocessed my dataset again and this time debayed it as RGB data. Now when I load the images into the NSG script, the list rebuilds in seconds. But as separate channels it takes a long time. Although the number of images is the same.
Strange.

I just read in the forum that the NSG script is no longer an official part of Pixinsight.
What does that mean now? It worries me now.
 
Minusman said:
I just read in the forum that the NSG script is no longer an official part of Pixinsight.
What does that mean now? It worries me now.
Click to expand...
Juan Conejero has decided not to include the NSG script in the official build.
I am still actively developing and supporting it, so you will still be able to install it and continue to use it.

Regards, John Murphy
 
Hi John,
You are working too hard, and I am going around in circles with my today problem: I have defined 10% bands at top/bottom without any detected stars. There are many stars visible in the 10% areas. My images are calibrated, CC, debayered (to separate channels), and registered. There are thousands of stars in the subs. I am using NSG V2.0.1.

At first I believed this is due to registration of images from different nights imaging, because one night of images had about a (huge) 10% downward shift, corresponding to a 10% band vs non shifted. But upon further checking the reference image and the ones with the 10% bands (same night) are only different by a few dithering pixels.

So now I am asking your advice on these bands. For example...
I select the last 4 images which are from the same night and with only small dithering distances. These are higher PSF images. Any combination of reference and selected image have the approx 10% no detected stars bands. I also tried NSG1.6 with same result. The star detection slider shifted to -3.0 has same bands. Does NSG already know I have 10% registration difference on other subs?

Some other combinations of different (lower PSF) images have smaller bands.

Thanks,
Roger
 
Status
Not open for further replies.
Back
Top