NormalizeScaleGradient: Bookmark website now!

[fbitto]

Hi John,

Thanks a lot for the new update. NSG has become an essential part of my processing. I just have one question:

I always use an output pedestal in WBPP. It seems that auto integration does not take it into account. So I have to run ImageIntegration a second time.

I believe that you can look at the pedestal keyword and then set the option automatically in ImageIntegration. Or maybe show the option in the ImageIntegration tab in NSG.

Regards, Francisco

 

[jmurphy]

Hi John,

Thanks a lot for the new update. NSG has become an essential part of my processing. I just have one question:

I always use an output pedestal in WBPP. It seems that auto integration does not take it into account. So I have to run ImageIntegration a second time.

I believe that you can look at the pedestal keyword and then set the option automatically in ImageIntegration. Or maybe show the option in the ImageIntegration tab in NSG.

Regards, Francisco

The ImageIntegration Pedestal option can be specified by using an ImageIntegration 'Template' process icon. NSG will still set all the fields that it normally sets except for the rejection algorithm. All other fields will be read from the process icon. Hence you will also need to specify the rejection algorithm and its parameters within the Template icon.

• Start ImageIngegration. Set all the 'non NSG' fields as needed. Save as a process icon.
• Within NSG's ImageIntegration section, set the Template icon to the saved process icon

Regards, John Murphy

 

[Linwood]

NSG is very single streamed, and I have a lot of cores. I went to do a LRGB-HA set and decided to just keep hitting "PI - New Instance" and running it for a different filter, so I could set all 5 running at once.

Well, just in case someone tries to do that -- it was a fiasco. Windows! Despite begin an SSD-only storage space, PI in multiple instances completely overran what Windows 10 could do, the disk delay timed out NTSF parameters, and the drive eventually disappeared (a reboot brought it back), so 4 of the 5 PI instances failed. Not PI or NSG's fault of course.

What kind of worthless operating system fails a disk just due to workload; never seen that in windows server even with huge queues. Guess that's why they give windows 10 away free. One day... maybe soon... it's time to switch to Linux. Things like Adobe are keeping me on Windows. ?

So I guess I start over today and it will run all day.

 

[scotty38]

Well, just in case someone tries to do that -- it was a fiasco. Windows! Despite begin an SSD-only storage space, PI in multiple instances completely overran what Windows 10 could do, the disk delay timed out NTSF parameters, and the drive eventually disappeared (a reboot brought it back), so 4 of the 5 PI instances failed. Not PI or NSG's fault of course.

What kind of worthless operating system fails a disk just due to workload; never seen that in windows server even with huge queues. Guess that's why they give windows 10 away free. One day... maybe soon... it's time to switch to Linux. Things like Adobe are keeping me on Windows. ?

So I guess I start over today and it will run all day.

How about a dual boot setup, that way you keep windows for what you need it for plus you get to boot into Linux for the times you need it to run over night as above? Just a thought......

 

[Linwood]

How about a dual boot setup, that way you keep windows for what you need it for plus you get to boot into Linux for the times you need it to run over night as above? Just a thought......

Yeah, it's on my list. Or a VM. I've converted almost everything to linux except my main desktop, and a security NVR (which only runs on windows).

But we are off subject... I had just given people an idea above that ended up being a good idea for performance, but a bad idea for actually running, wanted to mention the downside result.

 

[jmurphy]

NSGXnml is now available for Linux

https://sites.google.com/view/normalizescalegradient

Regards, John Murphy

 

[Linwood]

Am watching a drizzle integration roll buy after NSG did the regular integration.

Some, not all, but some frames are doing this during the Drizzle run after NSG's integration. I used NSG's integration process (initialized with the icon I usually use), and did not notice any errors in that, but see infrequent but not an insignificant number of these red errors rolling by (I'm capturing the log so once it finishes I can look more closely).

Any suggestions what it means and where I look if it may be something I did?

And does it imply a bad run that I need to start over in some fashion?

I did SFS (with approval in case I decided later not to use NSG), then star align with drizzle, then filter by filter did NSG using a reference from SFS (that I also looked at manually) based on mostly star count and FWHM, then integration (auto-run) and then drizzle integration (load up the drz/nml files and run). I saw this in at least a couple filters then turned on logging so I do not have logging for all 5 (LRGB+Ha).

Ah... it just finished the drizzle integration, and it does show 19 failures (174 success), so it is flagging these subs as failures (see below):

So I dredged out a failure from the log, there is little detail, or more precisely the log shows no additional details:

[2022-03-05 18:17:13] * Parsing drizzle data file 183 of 193:
[2022-03-05 18:17:13] T:/Stage/M82/Registered/LIGHT_M 82(1)_2022-02-28_05-48-22_Ha-5nm_300.00s_SET-TEMP_-10.00C_Gain_100_Offset_50_Bin_1x1_2800mm_-10.00C_0059_c_cc_a_r.xdrz
[2022-03-05 18:17:13] *** Error: Missing required LocationEstimates element.
[2022-03-05 18:17:13]
[2022-03-05 18:17:13] * Applying error policy: Continue on error.
[2022-03-05 18:17:13]

The file is present in the DrizzleIntegration input data pain, properly shows the n for normalization data present, and all three files (xsif, xnml and xdrz) are present in the folder as expected, and the original input files have not moved either.

What does this mean, and is it a sign I did something wrong in NSG? Or a bug?

 

[jmurphy]

Am watching a drizzle integration roll buy after NSG did the regular integration.

Some, not all, but some frames are doing this during the Drizzle run after NSG's integration. I used NSG's integration process (initialized with the icon I usually use), and did not notice any errors in that, but see infrequent but not an insignificant number of these red errors rolling by (I'm capturing the log so once it finishes I can look more closely).

Any suggestions what it means and where I look if it may be something I did?

And does it imply a bad run that I need to start over in some fashion?

I did SFS (with approval in case I decided later not to use NSG), then star align with drizzle, then filter by filter did NSG using a reference from SFS (that I also looked at manually) based on mostly star count and FWHM, then integration (auto-run) and then drizzle integration (load up the drz/nml files and run). I saw this in at least a couple filters then turned on logging so I do not have logging for all 5 (LRGB+Ha).

Ah... it just finished the drizzle integration, and it does show 19 failures (174 success), so it is flagging these subs as failures (see below):


View attachment 13848

So I dredged out a failure from the log, there is little detail, or more precisely the log shows no additional details:

[2022-03-05 18:17:13] * Parsing drizzle data file 183 of 193:
[2022-03-05 18:17:13] T:/Stage/M82/Registered/LIGHT_M 82(1)_2022-02-28_05-48-22_Ha-5nm_300.00s_SET-TEMP_-10.00C_Gain_100_Offset_50_Bin_1x1_2800mm_-10.00C_0059_c_cc_a_r.xdrz
[2022-03-05 18:17:13] *** Error: Missing required LocationEstimates element.
[2022-03-05 18:17:13]
[2022-03-05 18:17:13] * Applying error policy: Continue on error.
[2022-03-05 18:17:13]

The file is present in the DrizzleIntegration input data pain, properly shows the n for normalization data present, and all three files (xsif, xnml and xdrz) are present in the folder as expected, and the original input files have not moved either.

What does this mean, and is it a sign I did something wrong in NSG? Or a bug?

The first step is to try to narrow down the cause of the error:

• Is it repeatable? Is it always the same images that fail?
• Does it still fail if the '.xnml' normalization data files are not added to DrizzleIntegration?
• What version of PixInsight are you running?
• Which OS ?

Thanks, John Murphy

 

[Linwood]

Thanks, I am running 1.8.8-12 on Windows 10x64 (fully patched). I will need to try the other things a bit later, I decided to do a conventional drizzle integration without NSG (starting from registration again) to see if I got any errors, it's still doing the (last) initial integration, so I probably cannot try redoing this until tomorrow at the rate it's going.

I may have found a clue though, I take it there is some process that disables some images.

[2022-03-05 16:34:32] ImageIntegration: Enabling images with weights > 1.0099 x 25% = 0.2525

This was one of those:

[2022-03-05 16:34:32] * Skipping disabled image: T:/Stage/M82/Registered/LIGHT_M 82(1)_2022-02-28_05-48-22_Ha-5nm_300.00s_SET-TEMP_-10.00C_Gain_100_Offset_50_Bin_1x1_2800mm_-10.00C_0059_c_cc_a_r.xisf

That's probably documented somewhere and I missed it? And results in this sort of error perhaps?

 

[jmurphy]

I may have found a clue though, I take it there is some process that disables some images.

[2022-03-05 16:34:32] ImageIntegration: Enabling images with weights > 1.0099 x 25% = 0.2525
This was one of those:

[2022-03-05 16:34:32] * Skipping disabled image: T:/Stage/M82/Registered/LIGHT_M 82(1)_2022-02-28_05-48-22_Ha-5nm_300.00s_SET-TEMP_-10.00C_Gain_100_Offset_50_Bin_1x1_2800mm_-10.00C_0059_c_cc_a_r.xisf

That's probably documented somewhere and I missed it? And results in this sort of error perhaps?

I think you have found the cause.

NSG can disable images that have low weights (See its ImageIntegration section). This defaults to 25%, which is a very arbitrary value. In some situations, image with much lower weights can still have real value.

I wonder if I should get rid of this option in the next NSG release. Now that the images are added to ImageIntegration in order of weight, and the weight can be displayed as either a prefix, or as a tooltip (when using '.xnml' normalization files), it is quite easy for the user to disable the images themselves. I think this option is no longer necessary, and can clearly cause problems...

Regards, John Murphy

 

[Linwood]

Well, now that I understand, easily handled. I do not really have a perspective if those 10% of images (in this filter) being omitted helped or harmed.

I really need to dig more before asking questions. Apologies, and thank you for the quick follow-up.

 

[jmurphy]

Juan Conejero feels that the NSG script is competing with PixInsight's standard image preprocessing tools in the image weighting and normalization tasks. He has therefore decided not to include NSG in the official PixInsight distribution, and to block the NSG thread from this forum within the next few days.

You need to take action now, and bookmark my website:

NormalizeScaleGradient

I have created a new website to host both NormalizeScaleGradient (NSG) and PhotometricMosaic (PMM): https://www.astroprocessing.com The NSG PixInsight Repository URL has also changed (see my new website for more information) Thank you!

sites.google.com

You can also contact me via email:
johnastro.info[at]gmail.com
(replace [at] with @, I did this in an attempt to avoid automated spam)

Thanks for your support
John Murphy

 

[Minusman]

Hello John, sad to hear. The NSG script was responsible for me finding interest in Pixinsight again. Is this now the reward for your hard work, that such big stones are thrown in the way. Actually you should be happy to have such a great script in Pixinsight. It enhances the software.
Will there also be a forum on your website?

 

[jmurphy]

Hello John, sad to hear. The NSG script was responsible for me finding interest in Pixinsight again. Is this now the reward for your hard work, that such big stones are thrown in the way. Actually you should be happy to have such a great script in Pixinsight. It enhances the software.
Will there also be a forum on your website?

It caught me completely by surprise, and I don't understand it.

I don't understand why I am considered as a competitor; a user of my software has to purchase PixInsight. I felt that I was simply offering an alternative to other processes within PixInsight. I have spent several years on the code, in the hope that end users might find it useful. The script has remained completely free, and I only charge a small amount for the optional NSGXnml C++ module. From the PixInsight frequently asked questions (General 1.7), I had understood that it was completely OK to do this. I left work in 2016 and have been living off savings, so a small amount of income has been useful.

With regret, John Murphy

NormalizeScaleGradient

I have created a new website to host both NormalizeScaleGradient (NSG) and PhotometricMosaic (PMM): https://www.astroprocessing.com The NSG PixInsight Repository URL has also changed (see my new website for more information) Thank you!

sites.google.com

 

[Linwood]

I don't understand why I am considered as a competitor; a user of my software has to purchase PixInsight.

It's a bit deeper still, several cases you can find (and at least one aimed at me) when asking for a change, it was said "PI is a development environment, if you do need something it doesn't provide, feel free to add it".

If it's the $30 or so you asked for the C++ module that made this happen -- gesh... for a hobby where people spend hundreds on cables and adapters without a thought, that hardly seems to worthy of note in terms of competing with PI.

Maybe they didn't like the praise Adam Block gave it in some videos. Maybe it's that sort of competition.

But now makes me wonder what happens next, will the EZ Suite be on the chopping block? Star de-emphasis?

 

[dghent]

Juan Conejero feels that the NSG script is competing with PixInsight's standard image preprocessing tools in the image weighting and normalization tasks. He has therefore decided not to include NSG in the official PixInsight distribution, and to block the NSG thread from this forum within the next few days.

This is not cool and would serve as an ominous sign to other developers of scripts. Juan should be encouraging innovation, and this maneuver can only put a wet blanket on it. Opinions are one thing, but erecting hard walls such as requiring signed code to access certain core methods, especially to target a specific script such as NSG, is so very wrong I don't even know where to begin. This same tactic has been tried in other places over the years and the innovation does not disappear - it often finds a welcoming home on a competing platform. Hell, the very OSes that PI development takes place on - FreeBSD and Linux - were born due to much the same situation with commercial UNIX OSes in the early 90s. They thrive today and the commercial UNIXes that were inflexible or too much all about themselves are now historical curiosities and are barely worth mentioning on CVs.

Juan, I hope that you reconsider and try to welcome this stuff instead of throwing up roadblocks. We adopted the plugin system in NINA 2.0 to facilitate innovation outside the core app and indeed that is what is happening. I strongly feel that this is benefitting NINA as a whole, even when some of the plugins implement functionality that we would never consider putting into the core application or personally aren't interested in. It has been so successful that we plan on expanding on it to other areas of the app. I hope you can recognize and accept the similar benefits for PixInsight.

 

[Zerolatitude]

Juan Conejero feels that the NSG script is competing with PixInsight's standard image preprocessing tools in the image weighting and normalization tasks. He has therefore decided not to include NSG in the official PixInsight distribution, and to block the NSG thread from this forum within the next few days.

This is unfortunate.

Whatever the reasons / discussions between Juan and John, as a Pixinsight user it is definitely a loss. The availability of so many third party scripts adds to the richness of the PI experience and is a major attraction of the platform. IMO, if 3rd party scripts are removed because of competing or overlapping functionality with PI's own modules, users, and by extension the PI platform will be the poorer for it. More choice is always preferable.

Blocking the NSG thread is definitely unusual, to say the least. This type of action is usually reserved for malware, misrepresentation etc., which NSG is clearly not. Is this policy going to apply to other popular 3rd party scripts (paid and free) like EZ, StarXterminator, GHS over a period of time if PI brings in its own competing functionality?

Hopefully the parties can sort this out so that users continue to get the best experience.

 

[Grim408]

It caught me completely by surprise, and I don't understand it.

I always had problems with LocalNormalization script on my OSC widefield data and the NSG was the savior, I have never thought that this script is a competition to the PI it was just the opposite. PI is still a dominanting software for astro photogprahy and to allow third party scripts was amazing feature to have even if it clashes with the exiting ones.

Dear Juan, I can understand that the script will not be part of PI anymore because of the similarity one has with the LN, but please don't shut down the forum thread. If the new calibration scripts will work better there is no worry, right? People will tend to use that other then some additional script, especially new users.

 

[jmurphy]

I have uploaded the latest version of NSG 2.1.1 to:

NormalizeScaleGradient

I have created a new website to host both NormalizeScaleGradient (NSG) and PhotometricMosaic (PMM): https://www.astroprocessing.com The NSG PixInsight Repository URL has also changed (see my new website for more information) Thank you!

sites.google.com

Changes:

• Improved the 'Drizzle data' tooltip.
• Removed the Minimum weight from the ImageIntegration section. This is no longer needed because the images are loaded into ImageIntegration in weight order, and it is easy for the user to toggle on/off images to include or exclude them within ImageIntegration. You can view the weights within ImageIntegration either as a prefix, or via a tooltip (Normalization data option).
• Changed the way the ImageIntegration 'Template icon' works so that if an ImageIntegration process icon is used, and its data rejection algorithm is set to 'No rejection', NSG will override this with an automatically determined rejection algorithm dependent on the number of images.

However, I have been kept out of the loop concerning the next PixInsight release, so I don't know how well the script will work in the upcoming PixInsight release. It is possible that Juan's changes may partially or fully break the script. I just don't know.

Regards, John Murphy

 

[rbotero]

I am really sorry to hear of these developments John. I hope that Juan reconsiders and allows us users to decide if we want to support the script development separately whilst still allowing you to make it compatible in new releases.
NSG has been a godsend for me imaging from the suburbs of London with varying light pollution and poor transparency conditions. Although I am looking forward to the new LocalNormalisation developments, it should be down to me as paid user of PixInsight to choose what routine fixes my background problems pre-integration. I trust I am not speaking out of tune as I don't know your discussions with Juan but I hope there's a halfway solution that benefits the end user.

Roberto