Help me understand what I'm doing wrong with DBE

[cdavid]

Hi everyone.  I've encountered a strange problem when applying DBE to a recent image I acquired with my new camera.  It is a wide field view of M106 area.  The calibrated-integrated image created in pixinsight looks very smooth with little noise when applying an unlinked STF, it does however have a bad gradient due to the horizon light pollution.  After performing a DBE and re-applying the STF now linked I have this very noisy and color blotched image.  I have tried few and many samples, varied the sample size, played with the tolerance, etc....the end result is a very noisy image.  I have not processed further as I want to get some feedback and thoughts on what might be causing this.  My experience with DBE in the past few years has never resulted in such a dramatic change in my image.  I am not sure what is going on.

Here is a screenshot of my workspace.  You can see the baseline image with STF on the left, to the right is a clone that has been set up for DBE, on the bottom are the resulting background and post-DBE image after applying an STF linked.

Here is a link to my dropbox with the XISF baseline file if anyone wants to play with it.
https://www.dropbox.com/sh/zxgq1ve0hlx8okj/AAAWB6SZSGDL2hoBK3Cu5MxVa?dl=0

thanks for any input.
Carlos

[msmythers]

Carlos.

I like to use ABE with heavy gradient images and little nebulosity. I use the 'Draw Sample -- Just Try Samples" boxes to adjust the parameters. I like to use small sample sizes with as wide of a separation to allow for proper coverage of  the sample boxes with no dark colored boxes or missing boxes in corners. Also I'm looking to see if brighter objects are not being covered by boxes just as you would with DBE.

I also crop the edges of images just to make sure I'm not trying to correct edge issues from stacking, flats or optics. Again something I do.

While not perfect here is what I got with ABE.



Mike

[MikeOates]

I have the same issue and so do many others as it’s posted here quite a lot. So I looked further into it. The main thing I noticed was the background level after DBE was a lot lower around 100 ADU where as before DBE it was (I forget, but a few hundred if not more) And just a few days ago I experimented with settings in DBE and found a solution.

If you tick the option ‘Normalize’ in DBE in the ‘Target Image Correction’ section you will find DBE still does the background subtraction, but it leaves the background at around the same level so it does not look noisy.

The same should apply to ABE as well, but I have tried that.

I am half expecting Juan to comment on this and say that’s not the way to use the ‘Normalize’ option!

Mike

[cdavid]

Thanks for your inputs Mike and Mike....

Mike S. -  That looks pretty good....I stopped using ABE a few years back and had great success with DBE until my last couple images which were shot with my new T3i camera.  It just strikes me how smooth in terms of noise the initial unprocessed image appears and how bad and full of chromatic noise the DBE image appears.

I had once been told to wean myself off ABE as DBE was more powerful.  I am just not sure why I am getting these results with it.  I would expect at least as good as ABE.

Mike O.- yes I tried that as well...it helps some but still not a great result...

I hope Juan or some other Pixi guru chimes in and points out an error in our employment of the DBE.

Thanks again!
Carlos

[Carlos Milovic]

Hi Carlos

The image is a bit small, so I can't read the parameters from it. Anyway, from guessing and inspection of the image, I'd say that you have too few samples. Try to manually place them on the parts of the background where they are missing. Don't overdo, but cover all the background. Also, I would increase the sample size, and make sure that none is placed above a star. Finally, you should increase the tolerance and maybe the shadows relaxation too. The colors in the sample (inside DBE's window) should be nearly white for the background, and darker/black for stars. I think that you may be rejecting too many valid samples.

[Harry page]

Hi

While ABE can work well in images I always encourage the use of DBE so you can understand what you are doing better ( and hence better correction )

I can see from your screen grab you do not have samples right across the image , a must in this case as you have a gradient right across the image

I needed to up the tolerance to 2.5 to get the samples inc in the model ( not red ) and fine tuned some samples to avoid bright stars

As for normalisation you should not need this option as it will tend to recreate the background before you started usally resulting in a non neutral

background , lastly Will by default create that neutral background and in the course of this remove most of the pedestal in your image

but this is not a problem as it is not removing signal ( if done correctly )

Harry

[cdavid]

Thanks Carlos, Harry

I will try your suggestions in terms of larger sample size. My screen shot was just one version of all the iterations I tried. I had also run it with many more samples and my result is similar to Harrys....

Harry you have nicely removed the gradient but I see areas of reddish noise that I was also getting.

I guess my question is why do those reddish areas of noise appear after DBE when they are not visible before?  Is this then the expected result from DBE.

Thanks again for your help.


Carlos D.

[pfile]

carlos, when you remove a smooth model of the background signal, you're left with the shot noise of the background signal… that's why the result is noisy.

for OSC images i have "cheated" with the following methods - 1) add gaussian noise to the background only, to make it look a little more pleasing 2) kill the large scales of the chrominance of the background with atrous wavelets. i think alejandro has a tutorial that covers this.

rob

edit: meant gaussian noise, not poisson, but probably the results would be similar.

[Harry page]

Hi

The mottled background is shown due to the fact the stf is to aggressive in this case ( left it to show this )

What is left is your signal -good or bad inc the noise 

Harry

[cdavid]

Thanks guys...thats helpful.
Carlos

[cdavid]

Well I continued to work it as everyone suggested.  I obtained reasonable results but I was not totally happy.  Perhaps its my noise reduction techniques but I found it difficult to completely get rid of all the background noise after DBE.

I experimented a bit and found a workflow that "I think" gave a nice result....but I await your judgement and commentary.

I am not sure if my workflow is unorthodox or just another road to Rome but here is what I did:

I skipped DBE during the linear stage and proceeded with-
1. Background Neutralization
2. Color Calibration
3. Deconvolution
4. TGV
5. Histogram Transformation

6. At this point I then used DBE with samples all the way across as suggested and it gave a much smoother result than when done during the linear stage.  Not sure if this is considered kosher but I think it worked.
7.HDRWT at 2 using Median with lightness mask and to lightness
8.Morphological Selection at .2, amount .1 x 7 iterations
9. More Noise reduction using MMT and SCNR
10. Reset Black point with HT
11. MMT for sharpening
12. Saturation Boost
13. Final MMT for noise reduction

Here is my final result, comments and critique wanted

http://astrob.in/157595/0/

Thanks again for all your help.
Carlos

[MikeOates]

Carlos,

Well to me it looks great, perhaps the background is a bit dark, but then that may be hiding any remaining noise

I will try your workflow, especially the DBE at the non-linear stage. I also have issues with noise reduction and I have not found any workflow yet that works well for me.

You used:

8.Morphological Selection at .2, amount .1 x 7 iterations

Was that applied to the whole image or through a mask? I am not sure what that process would do over all the image, what did it do on your image. I have only ever used it in combination with a star mask.

Mike

[cdavid]

Thanks Mike!

The background appears slightly darker than my usual preference but it is the result of auto-zeroing using the HT.  It you look at the Histogram it is not clipped so I'm ok with it.

Sorry for the lack of details.....yes I used masks as appropriate for all the standard processes.  Inverted Lum masks for noise reduction, Lum mask for Deconvolution, etc....

Regarding step 8-

yes, the morphological selection was done with a star-mask.  This is a method I picked up from one of Carlos M.'s posts long ago....it is an excellent way to control the stars.  I use a star-mask made of contours  for the medium to large stars and then apply these settings....using 7-10 iterations and a structure size of 3 or 5....it works  quite nicely.

I'm pretty happy with the image now looking at it the next day....usually the next day peek is "blah"

I did a plate solve for those interested.....simply amazing how many galaxies are in this area of the sky!

Cheers
Carlos

[Alejandro Tombolini]

Hi Carlos, DBE is a tool that in my experience works extremely well and most of the times you can achieve your objective.
I recommend that you continue testing it in the linear stage of the image, experimenting with the possibles settings until getting a good result.

Your processed image when stretched shows us that the background can be corrected even more. I made a try with the image you have shared.

Here you can see a video with the setting used.

BTW, this video is a pilot test for the new series of subtitled videos that we are planning 


Saludos, Alejandro.

[cdavid]

Thanks Alejandro....yes in the past with other images I have found it to work well in the linear state.  With my new camera and as you saw in the current image I found it created noise that at least with my skills I could not control.  I will look at your video, thanks again!
Clear Skies
Carlos