GradientCorrection challenge

[ngc1535]

I would expect any gradient correction applied initially to corrupt the star RGB ratios.

I do not think this is true given that SPCC is doing its own background subtraction *per star* before integrating the flux.
So fine, now we are talking about methods and math. You made a mathematical statement. You claim that *any* gradient correction applied will corrupt RGB ratios. I hypothesize that for additive gradients with good photometry this will not be an issue.

So...when exactly does Juan step in here? lol I am happy to be wrong... as long as it is quite final in terms of methods and math.

-adam

 

[vicent_peris]

Hi,

Some comments on this thread:

GradientCorrection performs an additive gradient subtraction. This leaves the RGB ratios on the stars untouched. The only thing that changes after the gradient correction is the additive component, not the multiplicative one. Even if the dispersion of the values (that could happen in the dimmer stars when the gradients are severe or complex), SPCC has a robust linear fitting algorithm that converges to the same RGB ratios whether you apply it before or after GC. In my tests, if I apply SPCC before GC, then again after GC, I obtain routinely almost unitary RGB ratios.

This problem is more important from a pragmatic point of view than focusing on the numbers. Sometimes it can be better to apply SPCC before GC so you know you have the right colors on the objects before the gradients are corrected. Then, after the gradient correction, you can check if the colors you're getting on the objects are the right ones. STF with unlinked RGB channels can lead to a very wrong image visualization and checking how the gradient correction is performing can be difficult.

About GC modeling the objects (and therefore subtracting them from the image). Any automated gradient subtraction technique will have some representation of the objects in the gradient model. But our solution is light years from our competition. You can already check this on the NGC7000 image in the narrowband knowledge capsule. Try to apply GC with default values to that image and you'll see almost no change; try to apply our competition products with default values and see what happens. GC is thus a robust product and this topic should be seen in perspective. Thus, currently stating "GC removes nebulae" is not right; the right statement is "GC is light years beyond the competition when preserving valuable data".

Regarding DBE, the strength of GC is that it is much less biased than a tool based on a subjective sample placement. That is the main advantage of GC over DBE. Moreover, GC can correct the gradients below extended objects and gradients that are far more complex than DBE is capable of correcting. GC models and corrects the gradients in every single pixel of the image, while DBE models the gradients only on specific and sparse areas of the image. We consider that gradient correction techniques based on background sampling are mostly a thing of the past.


Best regards,
Vicent.

 

[wadeh237]

I would expect any gradient correction applied initially to corrupt the star RGB ratios. That is, given two stars with identical spectral characteristics, once gradient correction is applied they will have different RGB ratios. Seems like that would explain the greater scatter I see in SPCC post GC.

If gradient correction is altering the RGB ratios of stars, isn't that a *good* thing?

Consider an image with light pollution (and additive gradient) that is mostly represented in the red channel. Before gradient correction, stars in the light polluted area would be too strong in the red channel, while stars in an area where the gradient is weaker would not be as much biased towards the red. That seems like a situation that would *cause* scatter in the SPCC graphs.

If, on the other hand, gradient correction were correctly applied (which may not be possible until the MARS work is done), the RGB ratios in the stars would be more consistent, and result in better color calibration.

 

[cloudbait]

I do not think this is true given that SPCC is doing its own background subtraction *per star* before integrating the flux.
So fine, now we are talking about methods and math. You made a mathematical statement. You claim that *any* gradient correction applied will corrupt RGB ratios. I hypothesize that for additive gradients with good photometry this will not be an issue.

So...when exactly does Juan step in here? lol I am happy to be wrong... as long as it is quite final in terms of methods and math.

-adam

My observation (tested with DBE, but not yet with GC) is that after applying background correction, the R:G:B ratios of stars in different parts of the image change in different ways. Obviously that is going to affect SPCC calculations. I haven't seen a visual difference, but I do see it in the scatter plot SPCC produces. This is an issue of star color calibration only, not background treatment.

My other observation is with GC only. I haven't figured out a way to accurately visualize the image without applying SPCC, so the approximation I'm seeing (using unlinked STF) when I try to apply GC is resulting in visible background errors as I continue processing (SPCC followed by stretching). That may be a real problem, or it may just be related to my not properly using GC yet.

 

[pfile]

However in my home hortle 6 skies, I have gradients with irregular forms and color changes in one image. Doing any form of background neutralization based on a preview would mean defining this specific region as true background. But since the background color changes over the image, this is wrong. So I have to do gradient reduction prior to color calibration.

i think though, in the context of SPCC, fred's argument is that the tool already does a local background subtraction around the stars that it is using to calibrate the image. thus gradient correction before SPCC is not necessary. when using the old tools, (BN+CC) i think you are right to subtract the gradients first, but the argument is that this is not necessary when using SPCC.

rob

 

[fredvanner]

the tool already does a local background subtraction around the stars that it is using to calibrate the image. thus gradient correction before SPCC is not necessary

Exactly. Not necessary, and probably more accurate (since it is a local background estimate, not "smoothed" into an image-wide background / gradient estimate).

 

[chris.bailey]

i think though, in the context of SPCC, fred's argument is that the tool already does a local background subtraction around the stars that it is using to calibrate the image. thus gradient correction before SPCC is not necessary. when using the old tools, (BN+CC) i think you are right to subtract the gradients first, but the argument is that this is not necessary when using SPCC.

rob

But the background neutralisation element of SPCC does rely on a background sample. With lots of complex colour gradients, picking a representative preview to use is nigh on impossible.

 

[pfile]

But the background neutralisation element of SPCC does rely on a background sample. With lots of complex colour gradients, picking a representative preview to use is nigh on impossible.

but is that sample related to the photometry samples or is it akin to a preview over a large area of the image? hard to know without the code.

 

[cloudbait]

But the background neutralisation element of SPCC does rely on a background sample. With lots of complex colour gradients, picking a representative preview to use is nigh on impossible.

I guess that depends on what makes a color gradient "complex". When I have gradients, they are always the same... a broadly linear gradation from red biased to green biased. There's always a zone between the extremes that appears pretty neutral. I pick that as my background reference with SPCC and I get good results. Then I clean up with a gradient reduction tool after that. At this point I have four to work with, and none of them can be said to be definitively best. Depending on the specifics of the case, any one might be the best choice. Right now, I just try different ones to get the result I like best. I'm looking forward to MARS, as that will work in an entirely new way, and might end up being the one-stop solution to gradient management.

 

[fredvanner]

With lots of complex colour gradients, picking a representative preview to use is nigh on impossible.

This is true, though I rarely have images that complex. In this case running SPCC without BN, followed separately by gradient / background subtraction is probably best (the BN step is completely independent of colour calibration in SPCC, and is performed after colour calibration anyway).

 

[chris.bailey]

This is true, though I rarely have images that complex. In this case running SPCC without BN, followed separately by gradient / background subtraction is probably best (the BN step is completely independent of colour calibration in SPCC, and is performed after colour calibration anyway).

I am slowly working my way through the LDN/LBN objects so faint and often dark. Gradient Correction, ColCal and BN are vital steps with these as it is all too easy to end up with a green cast once stretched. DBE has never been ideal as picking samples is tricky on such extended objects. I was therefore excited by the release of the GC tool. At the moment I am tending towards GC>SPCC with a carefully chosen background reference. This is giving better results than the other way around.

 

[cloudbait]

I am slowly working my way through the LDN/LBN objects so faint and often dark. Gradient Correction, ColCal and BN are vital steps with these as it is all too easy to end up with a green cast once stretched. DBE has never been ideal as picking samples is tricky on such extended objects. I was therefore excited by the release of the GC tool. At the moment I am tending towards GC>SPCC with a carefully chosen background reference. This is giving better results than the other way around.

I think this is consistent with my experience with DBE, as well... there is no clear best order. Theory is fine, but results matter. It depends on the nature of the image. If it's something tricky (typically meaning low signal), it's easy enough to try it both ways and see what's doing the best job.

 

[AstrGerdt]

Yes, you guys are correct regarding the multiplicative color calirbation of course. However, as @chris.bailey pointed out, I was referring to the BackgroundNeutralization part of SPCC explicitly. For that part, I have to agree with him, there is basically no way for me to find a background sample.

@cloudbait is right, in the middle of the image, the background is mostly close to the mean. However, with large FOVs and large nebula, that I tend to place in the center of my image of course, that location is close to unusable for a background sample.

This is only a problem for me when I'm pushing the limits with long exposure times and strong stretches, but in that case, I see a clear green cast when I don't do a gradient reduction prior to SPCC.

CS Gerrit

 

[Juan Conejero]

We have a new version of the GradientCorrection process ready. Here is the result with this 'challenge' image:

With the new simplified model feature. The gradients in this image are now modeled much more accurately without any possibility of damaging significant objects. The new version will be released next week, representing another important step forward. Stay tuned!

 

[cloudbait]

We have a new version of the GradientCorrection process ready. Here is the result with this 'challenge' image:

View attachment 22314

With the new simplified model feature. The gradients in this image are now modeled much more accurately without any possibility of damaging significant objects. The new version will be released next week, representing another important step forward. Stay tuned!

Looks good! (And thanks for providing one-click access to the model. It wasn't that hard to get it with the current version, but this is much better.)

 

[Harry page]

Hi

Outlook gradient model -Excellent

 

[ngc1535]

We have a new version of the GradientCorrection process ready. Here is the result with this 'challenge' image:

View attachment 22314

With the new simplified model feature. The gradients in this image are now modeled much more accurately without any possibility of damaging significant objects. The new version will be released next week, representing another important step forward. Stay tuned!

This looks like an improvement. Did you try on the exemplar I made available to you as well?
-adam

 

[AstrGerdt]

We have a new version of the GradientCorrection process ready. Here is the result with this 'challenge' image:

View attachment 22314

With the new simplified model feature. The gradients in this image are now modeled much more accurately without any possibility of damaging significant objects. The new version will be released next week, representing another important step forward. Stay tuned!

Hi Juan,

Looking great!

Just for our/my understanding: is this basically ABE modeling the gradient that is produced by GC? It would be interesting to get a understanding how this feature works.

Of course I fully understand if you can't disclose that (now).

CS Gerrit

 

[vicent_peris]

Hi Juan,

Looking great!

Just for our/my understanding: is this basically ABE modeling the gradient that is produced by GC? It would be interesting to get a understanding how this feature works.

Of course I fully understand if you can't disclose that (now).

CS Gerrit

Hi,

It is a simple model based on polynomial fitting. ABE does a similar approach, but GC applies this fitting with the resources and knowledge we have today. ABE is a very old and simplistic tool. We introduce a new parameter but the overall work is simplified because the gradient subtraction performed by the core algorithm is much more robust now. This means that, in cases where the simple model works well, it is easier to find the right parameters to correct the image.


V.

 

[Juan Conejero]

Hi Juan,

Looking great!

Just for our/my understanding: is this basically ABE modeling the gradient that is produced by GC? It would be interesting to get a understanding how this feature works.

Of course I fully understand if you can't disclose that (now).

CS Gerrit

Hi Gerrit,

As Vicent says, this new feature is similar to ABE in the device we use to perform a two-dimensional interpolation of the simplified gradient function. However, this interpolation is guided by a multiscale analysis instead of the simplistic statistical model implemented in the ABE tool. This allows us to perform an initial robust simplification of the gradients in the image, which greatly facilitates the work of the core gradient modeling and correction algorithm, making it more efficient and controllable. The new simplification feature applies to complex gradient cases where no bright objects cover a significant area of the image.

As you have guessed, unfortunately, we cannot disclose the algorithms we have designed and implemented in the GradientCorrection tool. Although this contradicts our customary practice and philosophy of development, we must protect the research and development work we have carried out against competing applications willing to replace us by copying our most relevant achievements.