Local Normalization should default to OFF, not ON, in Weighted BatchPreprocessing

[M Covington]

In at least the latest 2 versions of Weighted BatchPreprocessing, Local Normalization defaults to "on," and it was causing a problem with my pictures -- basically, it turns minor low-level noise into dramatic color streaks in the sky background. (Illustration attached.)

[NOTE ADDED: This turned out not to be the fault of Local Normalization, but rather an incorrect setting within it, apparently left over from an earlier version whose default was different. See later postings for more about this.]

Juan Conejero's advice (https://pixinsight.com/forum/index.php?threads/local-normalization-problem.14115/#post-86421) is to use Local Normalization only when needed for a specific problem. Doesn't that mean it should default to off?

[NOTE ADDED: I see that that advice pertained to a much earlier version of Local Normalization, and now it is reasonable for it to default to on, with correct settings.]

(You will see in my picture that even without Local Normalization, there is a bit of discoloration in the sky background. This is taken with a Nikon D5500 on a C8 EdgeHD with f/7 compressor. For a long time I thought this was some kind of effect of the slightly lossy compression Nikon uses in raw files, but the sky background and also the flats are in the range well below 2100 ADU, where there should be no loss. I'm tentatively thinking that either my vignetting is itself not color-neutral, or this is some kind of sensor limitation, essentially pixel-to-pixel difference. Whatever it is, Local Normalization makes it worse. I should add that without Local Normalization, the discoloration is barely perceptible with *extreme* stretching, and may have an amplitude of only a couple of ADU.)

 

[Juan Conejero]

The link you have included in your post is from February 2020, when the LocalNormalization tool was problematic, inefficient, and had to be used with great care.

The current situation is completely different. During the last year we have invested a considerable amount of research and development work in our data reduction and preprocessing tools and pipelines. This includes very especially the LocalNormalization process, which has been completely redesigned and reimplemented, and is now a remarkably robust and accurate image normalization tool.

We now recommend applying local normalization routinely to achieve optimal results. Local normalization simplifies gradients and allows for optimal pixel rejection by making the data statistically compatible at the pixel level. For this reason local normalization is now an important component of our standard preprocessing pipeline and is enabled by default. We see no reason to change this default setting, but you can always disable local normalization if you want, although we don't recommend doing that under normal conditions.

Of course, higher accuracy always comes with added complexity. Each data set and each set of acquisition conditions poses specific problems, which must be identified and solved with appropriate process parameters. The problem you are experiencing is to be expected under the following conditions:

- You have used a single frame as normalization reference. This is not recommended, especially with low-SNR data sets. Local normalization tends to reproduce the gradients of the reference frame in all normalized images, and those gradients are then intensified in the integrated result. The recommended procedure is using a subset of the best frames to generate a local normalization reference with a relatively high SNR and averaged gradients. The latest version 2.4.5 of the WBPP script (released yesterday) includes an interactive interface to perform this operation accurately, as well as special settings applied automatically to avoid these problems.

- The normalization scale is perhaps too small for this data set. Intstead of the default scale of 512 pixels, a scale of 768 or 1024 pixels can be more appropriate with high noise levels.

If you want to upload the raw data (uncalibrated raw frames and bias/dark/flat frames as appropriate) we'll be glad to show that local normalization is now robust and leads to optimal results, while these problems can be avoided by applying the correct procedures.

 

[M Covington]

Thanks for that informative reply. I am glad to hear that Local Normalization has been improved, and I look forward to learning to use it.

At the same time, turning it on by default, but without appropriate default parameters, does not seem to be the right move, and I still think it should be off by default, leaving it to the user to turn it on and configure it.

I'm not the only one who started getting worse results when it was added and we were not aware that it was on. As I recall it was introduced in the March 15 release, although I'm not sure it was on by default in that release or if that changed later.

 

[bulrichl]

I post a copy of my reply in the CN forum here:

Did you reset WBPP choosing the option 'Reset all parameters to factory-default values?' when you tried LocalNormalization?

If not, you didn't set the default parameters, instead, the settings of your last WBPP run were used.

The default parameters are:

Generate images: disabled
Reference frame generation: Integration of best frames
Evaluation critera: PSF Signal Weight
Normalization scale: 512
Scale evaluation method: PSF flux evaluation
PSF type: Auto
Growth factor: 1.00
Maximum stars: 24576
Minimum detection SNR: 40
Allow clustered sources: enabled
Low clipping level 4.50e-05
High clipping level: 0.85

You can compare whether these values were used.

Bernd

 

[Juan Conejero]

At the same time, turning it on by default, but without appropriate default parameters, does not seem to be the right move, and I still think it should be off by default, leaving it to the user to turn it on and configure it.

The default LN parameters (those that Bernd has listed above) have been extensively tested. They represent the best set of options applicable in most cases in our opinion, based on our knowledge of the algorithms implemented as well as on numerous tests performed with many different data sets. We don't release new tools with arbitrary default parameters.

However, this is astrophotography. Nobody should expect magic solutions that work in all possible scenarios because such thing does not exist, especially when the data are problematic (short exposures, high noise, strong gradients) and very especially when we are talking about intrinsically complex and delicate problems such as local normalization. If you just want to get 'something' at the end of the process and that works for your purposes, then you can disable LN. If you don't want to explore the most accurate and versatile tools available in PixInsight, then perhaps you can use some competing products. If you want to get the best possible result out of your data, then our LocalNormalization tool is imprescindible in most cases.

I'm not the only one who started getting worse results when it was added and we were not aware that it was on. As I recall it was introduced in the March 15 release, although I'm not sure it was on by default in that release or if that changed later.

Users should realize that the kind of software development that we are doing here is not trivial. The problems we are talking about are really difficult ones, and the optimal solutions to these problems tend to be extremely complex. Our goal is always the same: design and build the most accurate and versatile algorithms and tools we are capable of. This has a serious disadvantage: complexity. I am perfectly conscious (after losing many 'wars' in this front) that understanding and managing complexity, as well as increasing knowledge, are not among the favorite ways to understand this discipline for most people. Those times are gone forever.

So, yes, we are considering the possibility to disable local normalization by default in WBPP. I really hate having to do that, but maybe it's the best option from a pure 'marketing' perspective, given the fact that we must deal with more and more competing products to keep this project alive. We are discussing this internally, no decision has been taken.

Besides all of this, my offer to work with your data is still on the table.

 

[M Covington]

Thanks, Juan. I look forward to learning how to use Local Normalization effectively. The main thing we've learned is that having it default to on, with whatever its default settings are, did not work well for me or some other people.

bulrichl, yes, see Cloudy Nights. I had never used local normalization, but I'll double-check that nothing else has left abnormal settings in place.

 

[M Covington]

IMPORTANT DEVELOPMENT. It turned out that even though I had never looked at the Local Normalization settings, one of them was non-default. When I had the problem, Local Norm. was set to use a single frame as reference. I reran it to use the integration of best frames as reference, and Local Normalization no longer causes a problem -- the results are very similar whether it's on or off. I have not spent any time figuring out how to get the best out of it yet, but I did want to rpeort this.

I don't know how the non-default setting got in there. I know that "Reset to factory defaults" changed it to "integration of best frames" as it should always have been.

Since others on Cloudy Nights reported the same image problem, I wonder if we need to be looking at how, somehow, this non-default setting could get in there, maybe carried over from an earlier version's default.

 

[robyx]

IMPORTANT DEVELOPMENT. It turned out that even though I had never looked at the Local Normalization settings, one of them was non-default. When I had the problem, Local Norm. was set to use a single frame as reference. I reran it to use the integration of best frames as reference, and Local Normalization no longer causes a problem -- the results are very similar whether it's on or off. I have not spent any time figuring out how to get the best out of it yet, but I did want to rpeort this.

I don't know how the non-default setting got in there. I know that "Reset to factory defaults" changed it to "integration of best frames" as it should always have been.

Since others on Cloudy Nights reported the same image problem, I wonder if we need to be looking at how, somehow, this non-default setting could get in there, maybe carried over from an earlier version's default.

Hi @M Covington,

when you install a new version of WBPP the global settings are migrated and maintained, as long as they are "compatible" with the new version. What happened, most probably, is that you've had the single reference frame option already set, and after the installation of WBPP 2.4.5 that option remained set.

It could be intentional to maintain the settings, but as a general rule, I would always reset WBPP after a new update has been installed to ensure that the default parameters that come with that new version are initially set.

 

[M Covington]

Another user on Cloudy Nights has reported the same experience: He had never used Local Normalization, but when he opened the settings window, it was set to use a single reference frame rather than construct the reference frame by integration. The latter is supposed to the default, and is set when you "reset all settings to factory defaults."

I am wondering if the real problem is that the process of installing some recent version of WBPP just doesn't set that default correctly.

 

[robyx]

Another user on Cloudy Nights has reported the same experience: He had never used Local Normalization, but when he opened the settings window, it was set to use a single reference frame rather than construct the reference frame by integration. The latter is supposed to the default, and is set when you "reset all settings to factory defaults."

I am wondering if the real problem is that the process of installing some recent version of WBPP just doesn't set that default correctly.

WBPP 2.4.2 had this "single best frame" value as default since the integration was still experimental and under validation, then we agreed to make the integration the new default with v2.4.3.
If for any reason, he reset WBPP a long time ago and never reset it after upgrading to v2.4.3, v2.4.4 and v2.4.5 then that value remained across these updates.

We have had quite some fast development cycles recently so WBPP 2.4.3, 2.4.4 and 2.4.5 were released close to each other, this could have well happened.

 

[M Covington]

WBPP 2.4.2 had this "single best frame" value as default since the integration was still experimental and under validation, then we agreed to make the integration the new default with v2.4.3.
If for any reason, he reset WBPP a long time ago and never reset it after upgrading to v2.4.3, v2.4.4 and v2.4.5 then that value remained across these updates.

We have had quite some fast development cycles recently so WBPP 2.4.3, 2.4.4 and 2.4.5 were released close to each other, this could have well happened.

Bingo! That has to be it. I am not at all sure what to recommend that the PixInsight team should do about this. Thanks!

In which version of WBPP did Local Normalization start defaulting to be on rather than off?

 

[robyx]

Good to hear that we probably figured it out!

One concern, if I may, about the title of the post "Local Normalization should default to OFF, not ON, in Weighted BatchPreprocessing".

We spend an immense amount of effort reapproaching the LN problem, we've redesigned it, we've integrated it into the WBPP pipeline, and we've also now introduced an interactive mode to support users in selecting the best reference frame if they would like to inspect the data and are not sure to thrust the defaults value. We took all of these risks to improve the results that you'll have in your hands for processing your images and to make your life easier during the initial data reduction phase.

Creating posts claiming that "LN should be turned off" even before receiving feedback on what is happening, is a little intellectually not respectful to the work we do.

I am sure you didn't do it on purpose and your intentions were good, but you have no idea how many posts we have that start this way, claiming that a PI tool has a problem, and in the end, they end up resolving a specific issue that has nothing to do with problems in the PI implementation.

So, in the future, If I may suggest, start a post with a brief and objective description of the topic to be discussed instead of suggesting people take an action for a problem that we've not even had the time to investigate and that damages the reputation of the work we're doing (it sounds like "LN is wrong, you should disable it")

 

[Juan Conejero]

I have little to add to what Roberto has already said above. I would just like to emphasize how important it is to be careful with the way these problems are communicated, as well as being aware of the damage that can be done to this project by spreading unjustified or ill-founded criticism or opinions.

We are professional software developers working very hard to build the best image processing platform we are capable of. As I have said many times in this forum, this is not a hobby for us, but our daily work. We take our work very seriously. Of course, we make mistakes and our code has (hopefully very few) bugs. But when we release a new version or a new tool we try to know what we are doing.

There are basically two ways of posing these questions:

(a) I have a problem, can you help me?
(b) I have a problem, your tools don't work.

Option (a) is a constructive way to solve a problem. Option (b) is a destructive way. What's worse, option (b) is a disrespectful way of judging our work beforehand without allowing us to state our opinion. When this is done in other public forums, where we cannot participate, the damage done to our project can be huge and sometimes irreversible.

A development project like PixInsight is more vulnerable than most of our users realize. The difference between the current scenario and a scenario where PixInsight no longer exists is much smaller than it may seem superficially. Keeping PixInsight alive and in constant evolution requires a lot of hard work, so please let's be more careful and respectful with the information we spread.

 

[M Covington]

I have added notes, in boldface, to the original posting. If you can change its title and would like to do so, please do so. Perhaps to "Unexpected result due to Local Normalization in WBPP." Or even "Unintended setting creeps into Local Normalization in WBPP."

Let me assure you that I did not intend to insult or denigrate your project. But it is not unreasonable, when you add a new feature that can affect existing users, to have that feature default to off. That is the angle from which I was approaching this.

"Local Normalization should default to off" does not mean "Local Normalization is bad." All I had in mind is that it requires extra attention from existing users, who might not realize it. We later learned that this was due to an incorrect default being carried over from earlier versions, and that actually, turned on with the correct defaults, Local Normalization does not cause a problem.

I'm a software developer myself and deal with similar issues (and also have some appreciation of the complexity of your task -- PixInsight is an impressive piece of work). Please keep up the good work.

 

[LeeB]

Let me assure you that I did not intend to insult or denigrate your project. But it is not unreasonable, when you add a new feature that can affect existing users, to have that feature default to off. That is the angle from which I was approaching this.

Just to throw in my 2 cents on this comment... If I took the time to "tinker" with the settings to find what worked best for my data. Then to install an update, and have those setting turned off or changed. I would be disappointed.

 

[M Covington]

Good point. There may be nothing to do but get the word out. Some people have the single-image default unintentionally, carried over from an earlier version; others have it deliberately because they made it work. As time goes on, a smaller proportion of WBPP users will be people who previously had the versions that caused the problem.

 

[Juan Conejero]

We'll extend the tool tip information for the Reference frame generation parameter on WBPP to provide more information on the difference between the Single best frame and Integration of best frames options in the next version of WBPP. Another possibility is forcing the Integration of best frames option in the incoming WBPP 2.4.6 version when transitioning from previous versions—we have to think about this carefully. We are also planning a series of tutorials that will describe the latest new features we've been implementing in the WBPP script, as well as an official WBPP documentation. However, this requires a lot of time and investing many resources that we have to share among a large amount on ongoing projects.