GradientCorrection challenge

[johnpane]

I am having trouble getting satisfactory results from the new GradientCorrection on this image.

Areas of nebulosity I want to preserve are being lost, and Structure Protection does not work because it prioritizes protecting a gradient in the lower left corner that is brighter.

For reference, here is a version corrected with DBE that seems superior in preserving nebulosity and correcting the corner gradient.

I explored many GradientCorrection parameter settings and did not find any that solve this. Suggestions welcomed!

 

[vicent_peris]

I am having trouble getting satisfactory results from the new GradientCorrection on this image.

Areas of nebulosity I want to preserve are being lost, and Structure Protection does not work because it prioritizes protecting a gradient in the lower left corner that is brighter.

For reference, here is a version corrected with DBE that seems superior in preserving nebulosity and correcting the corner gradient.

I explored many GradientCorrection parameter settings and did not find any that solve this. Suggestions welcomed!

Hi John. I'm out of home now, will take a look as soon as I come back tonight.

Best regards,
Vicent.

 

[Harry page]

Hi
Had a quick go and tried the default settings ( but outputed the masks )

there was a under correction in the lower left hand corner - which if you look at the masks is being proteded by them

so I adjusted the mask protection tolerance up a bit and all worked just fine - or just disable structure protection - but did seem to get

a better result with the mask active

both of these images have a high stretch the top left is the DBE version

Cheers

Harry

 

[Juan Conejero]

Here is my try, very similar to Harry's parameters:

 

[Juan Conejero]

For reference, here is a comparison with the original image:

I have to say this image is not an easy target. You did a really good work with DBE ;-)

 

[Marcelofig]

I have a simple question, when is it recommended to use GC, before or after SPCC? I have the impression that it is better before.

 

[ngc1535]

I have a simple question, when is it recommended to use GC, before or after SPCC? I have the impression that it is better before.

Before is the answer.


@johnpane Do you have any followup comments?

-adam

 

[tom886]

Before is the answer.

Can you elaborate more about this?
From my understanding it is a "replacement" for DBE/ABE and the general consensus was, DBE/ABE should be used after SPCC (except in cases with multiplicative gradients).
Or did I understand something complete wrong?

Cheers
Tom

 

[cloudbait]

Before is the answer.

Why? I am playing with an image, and I get better visual results using SPCC first. And when I do that, I get a really clean grouping of stars to the linear fit, but when I use GC first, the resulting SPCC has a LOT more scatter, and the correction coefficients are different.

 

[johnpane]

Thank you Harry and Juan! The flaw in my thinking was that a completely white mask would be equivalent to turning off protection.

I prefer the results with Juan's settings.

 

[johnpane]

Before is the answer.

@johnpane Do you have any followup comments?

-adam

My only comment is that this a point of considerable debate, for example see this thread.

 

[fredvanner]

Why? I am playing with an image, and I get better visual results using SPCC first. And when I do that, I get a really clean grouping of stars to the linear fit, but when I use GC first, the resulting SPCC has a LOT more scatter, and the correction coefficients are different.

That is what I would expect. I think this is yet another instance of the myth that SPCC "needs help" with backgrounds. Since SPCC corrects for background estimated over a small local region specific to each star it will almost always be better to run SPCC first. I have a sneaky suspicion that even the PI developers sometimes fall for this myth.

 

[johnpane]

Thank you Harry and Juan! The flaw in my thinking was that a completely white mask would be equivalent to turning off protection.

I prefer the results with Juan's settings.

Nonetheless, I find that GC is taking more signal away from the nebula than DBE. Here is a difference image.View attachment Image21.jpg

 

[dld]

The major advantage of DBE is that you can place your samples based on a reference image. And don't underestimate ABE; its major advantage is the model order setting. And in the era of PixInsight LE, if I am not mistaken, you could also choose a model order in DBE. Prior knowledge and the "unbiasedness" of low-order models are still indispensable (imho).

 

[ngc1535]

@fredvanner @cloudbait @tom886 @robyx @vicent_peris @Juan Conejero @johnpane

As I have mentioned over and over there is needless disagreement on this.
I asked for the information the very second the tool was teased knowing this was going to be an issue.


From the developers:

On a final note, I request (please!) that Juan/Vicent write a definitive explanation on the order of operations with respect to this tool. There are quite a few misconceptions about gradient correction in general and when it is applied (e.g. before or after color calibration) on the official forum and elsewhere. As an ambassador I would like to represent the proper information based on math, usage and technique- and not on opinion.

for this topic I think that I can give my two cents:

Speaking about SPCC, With color calibration we have primarily a multiplicative operation (the color calibration properly said) + an optional Additive operation (Background neutralization).
Let's forget Background neutralization for the moment.

With GradientCorrection we apply exclusively an additive correction that is automatically normalized (as for the "Normalize" checkbox in ABE/DBE).

Therefore SPCC and Additive gradient correction can be performed in any order you want, 'cause they are independent operations. (the only drawback in doing GC as second operation is that you will probably need to perform a new background neutralization after GC)

When you need to correct for Multiplicative gradient (I.E. Flat field missing or very strong differential extinction) then Gradient correction MUST be performed before Color calibration.

To simplify for "unskilled" users we can say that Gradient correction, in generale, should be done before color calibration: but this is a simplification of the problem.

Consider that if we have a mixed gradient (additive+multiplicative) at the moment we cannot separate the two component, so there's no correct answer, o better the answer should be:
1) multiplicative gradient
2) additive gradient and color calibration in any order (with a final background neutralization if needed).

But at the moment we don't have a tool that can separate additive and multiplicative gradients.

Edoardo Luca Radice
PixInsight Ambassador

Hi Adam,

Edoardo gave you good answers. I want to add two things:

- We are producing a video where we apply SPCC before and after applying GradientCorrection. The RGB weights of the second SPCC run are 1:0.997:0.997. This indicates that the subtractive gradient correction does not alter in any way the color calibration. Even if you have mixed multiplicative and additive gradients (the first ones will always decrease SPCC accuracy), GradientCorrection will leave the multiplicative factor in the same state.

Best regards,
Vicent.

And finally Juan tacitly agrees with the above.

So you want to claim a position on this? Fine. Take it up with the developers.
I do have an opinion on the nature of a FORUM. It is a forum of ideas that seeks to help others. Sometimes consensus is a good thing.

So to summarize.
1. GC does not affect multiplicative gradient (errors). You can do GC before or after SPCC without detriment. THIS is in direct contradiction and disagreement with Fred and Chris.
2. Furthermore, multiplicative gradients- if you want/need to correct them, require a different approach that comes *before* SPCC. (It is not clear to me if DBE with division can be helpful in this regard. However, most people don't know what errors they have.)
3. So we have GC which has any order and multiplicative errors that need to be corrected before SPCC.
4. Thus on the whole suggesting to correct gradients before SPCC (color calibration) appears to me a sensible answer.

I do not consider the above a "myth" (Fred) or "outdated or wrong" (Chris).

Bottom line, what is being explained to me makes sense. Vicent literally has examples (he says) that contradicts Fred's expectations. Increasing the scatter does not necessarily change the fit significantly I believe is what is being claimed. Showing your work is always a good thing.

Given all of this, I think I am being reasonable.

-adam

 

[cloudbait]

@fredvanner @cloudbait @tom886 @robyx @vicent_peris @Juan Conejero @johnpane

As I have mentioned over and over there is needless disagreement on this.
I asked for the information the very second the tool was teased knowing this was going to be an issue.
Fred you malign me with mythmaking.

From the developers:


for this topic I think that I can give my two cents:

Speaking about SPCC, With color calibration we have primarily a multiplicative operation (the color calibration properly said) + an optional Additive operation (Background neutralization).
Let's forget Background neutralization for the moment.

With GradientCorrection we apply exclusively an additive correction that is automatically normalized (as for the "Normalize" checkbox in ABE/DBE).

Therefore SPCC and Additive gradient correction can be performed in any order you want, 'cause they are independent operations. (the only drawback in doing GC as second operation is that you will probably need to perform a new background neutralization after GC)

When you need to correct for Multiplicative gradient (I.E. Flat field missing or very strong differential extinction) then Gradient correction MUST be performed before Color calibration.

To simplify for "unskilled" users we can say that Gradient correction, in generale, should be done before color calibration: but this is a simplification of the problem.

Consider that if we have a mixed gradient (additive+multiplicative) at the moment we cannot separate the two component, so there's no correct answer, o better the answer should be:
1) multiplicative gradient
2) additive gradient and color calibration in any order (with a final background neutralization if needed).

But at the moment we don't have a tool that can separate additive and multiplicative gradients.

Edoardo Luca Radice
PixInsight Ambassador

Hi Adam,

Edoardo gave you good answers. I want to add two things:

- We are producing a video where we apply SPCC before and after applying GradientCorrection. The RGB weights of the second SPCC run are 1:0.997:0.997. This indicates that the subtractive gradient correction does not alter in any way the color calibration. Even if you have mixed multiplicative and additive gradients (the first ones will always decrease SPCC accuracy), GradientCorrection will leave the multiplicative factor in the same state.

Best regards,
Vicent.

And finally Juan tacitly agrees with the above.

So you want to claim a position on this? Fine. Take it up with the developers.
I do have an opinion on the nature of a FORUM. It is a forum of ideas that seeks to help others. Sometimes consensus is a good thing.

So to summarize.
1. GC does not affect multiplicative gradient (errors). You can do GC before or after SPCC without detriment. THIS is in direct contradiction and disagreement with Fred and Chris.
2. Furthermore, multiplicative gradients- if you want/need to correct them, require a different approach that comes *before* SPCC. (It is not clear to me if DBE with division can be helpful in this regard. However, most people don't know what errors they have.)
3. So we have GC which has any order and multiplicative errors that need to be corrected before SPCC.
4. Thus on the whole suggesting to correct gradients before SPCC (color calibration) appears to me a sensible answer.

I do not consider the above a "myth" (Fred) or "outdated or wrong" (Chris).

Bottom line, what is being explained to me makes sense. Vicent literally has examples (he says) that contradicts Fred's expectations. Increasing the scatter does not necessarily change the fit significantly I believe is what is being claimed. Showing your work is always a good thing.

Given all of this, I think I am being reasonable.

-adam

I'm not saying anybody is right or wrong. It's a new tool, and I'm still trying to figure it out. But empirically, I'm getting better results applying SPCC first. There are two issues I'm sorting right now. The first is that without applying SPCC, I can't even see the gradient accurately (an unlinked STF just doesn't cut it). So I iteratively apply GC (using unlinked STF) until the gradient appears corrected, then when I apply SPCC residuals of it become visible again, along with some background mottling. This doesn't happen if I start with SPCC. The other issue is that in an image with lots of stars, an initial SPCC results in a tight linear grouping (I'd say the resulting linear fit has a correlation coefficient close to 1). When I apply GC and then SPCC, there is a lot more scatter of the stars (a worse R).

So far I'm only working with a couple of images (I don't often have significant gradients, so I don't have a lot of test images). I'll keep playing around and trying to improve my understanding of what's going on.

 

[Marcelofig]

In a side note to better understand this thread for less advanced users like me:


1) additive gradient: basically light pollution, either artificial like city lights or natural like the moon.

2) multiplicative gradient: basically vignetting.



Is this correct, anything else to add?

 

[ngc1535]

I'm not saying anybody is right or wrong. It's a new tool, and I'm still trying to figure it out. But empirically, I'm getting better results applying SPCC first. There are two issues I'm sorting right now. The first is that without applying SPCC, I can't even see the gradient accurately (an unlinked STF just doesn't cut it). So I iteratively apply GC (using unlinked STF) until the gradient appears corrected, then when I apply SPCC residuals of it become visible again, along with some background mottling. This doesn't happen if I start with SPCC. The other issue is that in an image with lots of stars, an initial SPCC results in a tight linear grouping (I'd say the resulting linear fit has a correlation coefficient close to 1). When I apply GC and then SPCC, there is a lot more scatter of the stars (a worse R).

So far I'm only working with a couple of images (I don't often have significant gradients, so I don't have a lot of test images). I'll keep playing around and trying to improve my understanding of what's going on.

There is a subtle distinction to be made. The principle as described appears to be a disagreement. Whether or not empirical results are better/worse - though practical- is somewhat besides the point. It is certainly important to improve the process/tool if it is not doing what it should- but it isn't what is at stake here as I understand it.
-adam

 

[cloudbait]

There is a subtle distinction to be made. The principle as described appears to be a disagreement. Whether or not empirical results are better/worse - though practical- is somewhat besides the point. It is certainly important to improve the process/tool if it is not doing what it should- but it isn't what is at stake here as I understand it.
-adam

I would expect any gradient correction applied initially to corrupt the star RGB ratios. That is, given two stars with identical spectral characteristics, once gradient correction is applied they will have different RGB ratios. Seems like that would explain the greater scatter I see in SPCC post GC.

 

[AstrGerdt]

Just to add my two cents to the discussion.

When imaging under dark skies, I see practically no difference between the order of gradient correction and color calibration. The gradient is already week from the beginning on and does not effect background calibration.

However in my home hortle 6 skies, I have gradients with irregular forms and color changes in one image. Doing any form of background neutralization based on a preview would mean defining this specific region as true background. But since the background color changes over the image, this is wrong. So I have to do gradient reduction prior to color calibration.

I guess this is part of the constant discussion about this topic. Different fields of view and locations have an impact on this matter. So while it may work for me to do it in this order, people from darker regions with smaller FOVs may prefer the opposite order. Or something completely different.

CS Gerrit