DBE / ABE is resulting in "rainbow" colors and increased grainy noise

Aloha Everybody!

I am new to PI and I am currently stuck at a DBE / ABE issue.

My initial image looks relatively ok (see pre-DBE attachment), but when I run a DBE or ABE it turns into a very grainy image and the background becomes less smooth and very colorful. Subsequent DBE adjustments after the first run are no longer possible since the resulting layer is even more colorful. This "new" color scheme then also shows in the non-linear processing phase.

I saw this issue mentioned occacionally in CN or here, but I could not solve my issue with the offered solutions.

If you would like to take a look you can find the master light here:


I use an entry level Nikon D3400 camera and image from outside of LA at a red zone. Total integration time of the example is around 6 hours (126x3min)

Any help, suggestion, comments, would be appreciate as I am out of ideas!
 

Attachments

  • Pre DBE.jpg
    Pre DBE.jpg
    165.8 KB · Views: 258
  • Post DBE.jpg
    Post DBE.jpg
    333.8 KB · Views: 218
the artifacts that appear in the red channel are actually already visible in the pre-dbe image. i think you may have one or more bad subs?

imaging from red zone LP is really hard...

rob
 
the artifacts that appear in the red channel are actually already visible in the pre-dbe image. i think you may have one or more bad subs?

imaging from red zone LP is really hard...

rob
Thank you Rob. I did another test stack with WBPP, limiting the subs to only the darker ones that I took later in the evening. The stacked lights looks comparable to the attached one, and I did not see much of a difference between them. Both result in the same odd DBE / ABE look.
 
i think you should blink thru your subs to see if there are any bad ones... there is a very complex gradient that appears in the red channel and to a lesser extent the blue channel. the green channel is mostly fine. this is not the result of DBE but rather the result of a DBE pass that did not completely sample the background. when you have really complex gradients like this it can be difficult to completely remove them. one strategy might be to split the channels so you can work on each gradient independently.

rob
 
Thank you Rob! I will try both options and report back here.

I did also go back to another test image I took a few weeks earlier. This one was pointing as far away from any light domes as possible (north-west vs. east-south) and the exposure length was reduced to 25 seconds (vs the 3 min on my initial example).

I now notice that you can see the same red gradient in the same spot and with a similar pattern in this Master Light as well. Maybe it's a little less obvious but I think it's there. This seems to indicate that the exposure length, and intensity of light pollution might play a lesser part in the red issue. Is this possibly a debarring issue?

I added the master flat, and a single sub, to the shared drive too in case you are interested.
 
well i was mentioning that the gradient was present in the master (as you have also seen) because i wanted to point out that the gradient was not the result of DBE but was pre-existing (and not completely removed from the image.) i don't think it's a debayering issue...

sometimes these weird gradients are not always visible in the subs because their SNR is very low and its only thru stacking that they reveal themselves. but the complex shape made me think that whatever it is is changing and so could be the result of clouds or dew or something like that which might be obvious in the subs.

rob
 
i think you should blink thru your subs to see if there are any bad ones... there is a very complex gradient that appears in the red channel and to a lesser extent the blue channel. the green channel is mostly fine. this is not the result of DBE but rather the result of a DBE pass that did not completely sample the background. when you have really complex gradients like this it can be difficult to completely remove them. one strategy might be to split the channels so you can work on each gradient independently.

rob

Rob,

I wanted to give you an update: I did try to process the individual channels but was not able to get a different outcome. However, I was able to see what you meant by the chancels being impacted differently, which was interesting. Thank you for suggesting it.

Jonny from CN shared that he saw a very similar effect after he started to process the red channel of a new camera. He also shared that he was able to isolate the issue to his flats. He used an iPad to illuminate his flats and, since I do that too, he recommended that I try a different light source.

I only had a laptop to test it but I am happy to report back that changing from the iPad to the laptop resulted in a noticeable improvement.

You can still see some red tint on the right, but it's much less than before. I am not sure if this is still a flat issue or maybe now the result of local LP.

I added the new test MLight, MFlat, and single Flat images to the google drive under the "New_PC_Flat" folder. In case you are interested here is the link:


Thank you again for your help. I appreciate it!
 
that is pretty interesting as i would not have expected such a complex pattern from a flat issue.

what was your flat duration? the most common issue with panel flats is exposures which are too short - shorter than the frequency that the backlight in the panel is driven at. most LED panels have PWM frequencies in the 10-20khz range so if your flats are seconds long they should be fine as far as flicker goes.

at any rate you can't argue with the results... good that you found a solution.

rob
 
that is pretty interesting as i would not have expected such a complex pattern from a flat issue.

what was your flat duration? the most common issue with panel flats is exposures which are too short - shorter than the frequency that the backlight in the panel is driven at. most LED panels have PWM frequencies in the 10-20khz range so if your flats are seconds long they should be fine as far as flicker goes.

at any rate you can't argue with the results... good that you found a solution.

rob
I usually had the flat exposure time set on Aperture Priority on my DSLR, and I regulated the iPad screen brightness to get to at least 1/30. I did this after I checked that the iPad refresh rate was 60 khz so I thought 1/30 would be ok.

During this problem resolution process it was also pointed out to me that my flats seemed under-exposed. This surprised me since the histogram did seem to be fine in PSRaw, DigicamControl (only acquisition SW that works with my D3400) and my Nikon Capture NX-D.

I increased the exposure time (for testing) significantly for the flat I uploaded as well.

In case you are interested here is the CN link that you might find interesting:


Frank
 
well the backlight frequency and the LCD refresh rate are 2 different things. yeah the histogram thing is something that gets everyone at first - you need something like +2EV if you are going to auto-expose your flats.

will check the thread.

rob
 
Hello Pixies!

What about the fact that ABE/DBR increases the noise and that stars are bigger and less sharp after the gradient extraction? I have experienced the same difficulty on monochrome images: is it normal? Did you find a solution?

Syn
 
What about the fact that ABE/DBR increases the noise and that stars are bigger and less sharp after the gradient extraction? I have experienced the same difficulty on monochrome images: is it normal? Did you find a solution?
BackgroundModelization processes (ABE, DBE) do not add noise to an image, period. The removal of a gradient causes that a STF Auto Stretch after application of a BackgroundModelization process will be more aggressive than before. Compare the image (before/after) with the same STF settings, and you will see that there is no increase of noise. Thus there is no solution because the problem is not existent. The noise is already in the image before you apply a BackgroundModelization process.

Bernd
 
Last edited:
BackgroundModelization processes (ABE, DBE) do not add noise to an image, period. The removal of a gradient causes that a STF Auto Stretch after application of a BackgroundModelization process will be more aggressive than before. Compare the image (before/after) with the same STF settings, and you will see that there is no increase of noise. Thus there is no solution because the problem is not existent. The noise is already in the image before you apply a BackgroundModelization process.

Bernd

Would it eventually be possible that DBE sometimes shows the noise more than other times ?
result.png


Twice the same photo.
Left after DBE with the points chosen as carefully as possible : no stars inside of it, no nebulosity, etc... Resulting in a very chaotic pattern

Right after a DB with just big points at the sides of the foto.

The visible noise in the left picture is more annoying thanin the right picture, it seems to me ?
 
I wrote:
Compare the image (before/after) with the same STF settings, and you will see that there is no increase of noise.
You can achieve this by selecting the right image as current view, applying a STF Auto Stretch to it, and then dragging the blue triangle of the ScreenTransferFunction on the left image. Subsequently you can compare.

Bernd
 
I wrote:

You can achieve this by selecting the right image as current view, applying a STF Auto Stretch to it, and then dragging the blue triangle of the ScreenTransferFunction on the left image. Subsequently you can compare.

Bernd

If I remember well (its from november), I did an STF Auto Stretch on both.
But not the way you describe, that is correct. No idea where the image is to have a retry.
 
The following screen sections show a linear image (gradient removed with ABE, color calibrated with PCC and noise reduced with TGV and MMT), the STF process window and the STF parameters, for the same image but with different STF parameters.

IC1613_STF_params_01.JPGIC1613_STF_params_02.JPG

When you take a look at the stars or the galaxy, it is easily noticeable that in the second section a more aggressive STF stretch is applied. However, if you only view the background, it might seem that there is more noise in the background of the second section. This is a deception, caused by the different STF stretches.

When you want to compare two linear images (before and after applying a BackgroundModelization process), it is important that the STF parameters in the second column (Midtone balance) are the same for both images. The parameters are shown when you click on the wrench icon in the STF process window.

Bernd
 
The easy way to keep the same stretch across all images is to save the desired stf stretch as a process icon and the just drag and drop the icon onto subsequent images.
 
Hello @Dr. Frankenkeim
a small contribution from my side, I took your initial .xisf and played with it, I got the follwong results :

What has been said regarding "noise" and auto stretch is important when you want to compare some images, what I do generally is that I "memorise" the a set of STF parameter with a "drag an drop" of the STF triangle to the histogram process, and then I use the histogram triangle to drag and drop to various images to compare with the same stretch parameters. Very usefull especially when you perform some noise reduction in linear phase

Coming back to the ABE / DBE issues you are facing, I don't know what is your objective but if you think that my result is interesting, here is what I did and why I did it :
- even if I am not sure that this is mandatory, I split the R-G-B channels to work channel per channel
- gradient removal is perhaps the only process that I think I understood, it is based on a complex math problem : knowing a fonction when you know the function in a set of point
- this complex problem is simple when you make an assumption : the function is a polynom
- so when you use ABE DBE, you have to keep in mind that your samples are the set of values of the function that you are giving to the algorithm
- this is working only because the samples values must be made of something like "sky background fonction [which is a constant] and gradient fontion [which you are going to calculate then substract]"
- this is not working at all if your sample contains a part of another signal, let's say some nebulosity :-(
- in the case of your data : you have galaxies in a sky background that do not contain any nebulosity, some samples are not put because the difference in pixels values is so high that the process "thinks" it may be a "good"signal"
- but because you know that this is an actual background, you just force the "tolerance" parameter until the samples are put in the background of the image (and check that you are not putting samples on stars either)
- this is what I did, I can repeat the ABE or DBE process several time, as long as the samples are put on sky background, there is no harm in repeating the process

Usually, I am not using a lot of samples but because your red gradient was complicated, I used much more to take into account the rapid change in gradient value in some part of the image.
I dit not spend some time of the last "red transversal bar" in the left area but with the same principle, you may try to get rid of it.
This is quit fast and straightforward, writing this post took me much more time :)

Understanding this was very usefull to me, for example, at the opposite, when you face a "simple" gradient, coming from the moon for example, you may force the polynome value to 1 or 2 in order to avoid a bad modelisation due to samples that contains part of the "good signal"

I hope this will help you a little

cheers

Jean-Baptiste
 
Thank you Jean-Baptiste. That was helpful and good to see what you were able to do.

I kept thinking about why "the iPad did not work" for my flats and came across the following thread on CN:

https://www.cloudynights.com/topic/667524-preparing-color-balanced-osc-flats-in-pixinsight/

In short, DonR seemed to describe a very similar issue to the one I was experiencing above. That got me thinking...

I also use an unmodified DSLR to take my subs, and I started to analyze them with the statistics tool in PI. I saw the exact same impact that DonR described: "an artificially boosted channel in the master light"

If you take a look at the first screenshot ("standard flat example") you will see:

A1) = the underexposed flat that we discussed in this thread above

B1) = the debayered version of A1. Please note the relative value of the red channel compared to G and B as you see that it is "lower" then the B channel

C1) = the resulting master flat using ~30 of A1 equivalent frames

D1) = a standard example light used in the thread above. Please also note here the relative value of R compared to G and B

E1) = the resulting master light

If you take a look at E1 you see how PI "boosted" the red channel dramatically compared to the relative values of G and B on the example light D1. To me it finally made sense why I saw this red tint in my master light. Maybe it was less of an issue with using the iPad for flats, but more of an issue with my unmodified DSLR and the low red channel being artificially boosted in PI.

So...what to do about it? I kept reading the linked thread above, and how you could for example take separate exposures to boost the channel in question, which seemed complicated. I thought why not use a "monitor manager tool"; one that will allow you to regulate the screen brightness and the R,G, and B channels. I found the "Free Monitor Manager" that allows me to do exactly that, and you can also save your settings.

I played around with the different settings to a) get to a flat median of around 2,000 (my D3400 is 12bit) PLUS b) balance the flat R,G,B channels to match them as closely as possible.

Now take a look at the second attachment ("colorboosted flat"). A2 - E2 follow the same logic as A1 - E1 above. But it you look at B2 you see how my R,G,B channels of the flat are much more aligned compared to B1. Now take a look at the resulting master light E2: you no longer see that the red channel is disproportionately boosted compared to E1.


This fully supports DonR's post and this method really seems to work for me now

I wanted to share it with you guys and hope this will help somebody else.


C.S! Frank
 

Attachments

  • standard flat example_jpg.jpg
    standard flat example_jpg.jpg
    279.1 KB · Views: 66
  • Colorboosted flat_jpg.jpg
    Colorboosted flat_jpg.jpg
    290.9 KB · Views: 72
Back
Top