M106, tricky background gradients and noise

[DanielF]

Hi all.
I've been struggling a bit with the data I collected on M106 in February. The background is very uneven and noisy.
The data is a stack of 39x3min exposures and has been calibrated in PixInsight with Bias, Dark and Flats.
Camera used is an unmodified Canon 500D DSLR.

Please have a go at it and see if you can make something decent with the data.
Master can be downloaded from Endor:
Edit: New version uploaded.
http://endor.uv.es//data/public/87151ddf2da5aea8b77b553947fe0445.php?lang=en

This is what I've managed to do in Photoshop shortly after I took the images (This was before I found out about PixInsight)


/Daniel

[marekc]

Hi Daniel,

Looking at what you've done in PS so far, I think you did one thing quite right - you didn't try to pull too much out of the image. You've got a pretty even background, and although it's somewhat noisy, it's not *too* noisy looking. If you had a noisy background with large gradients, then I think you were smart to not try and stretch the image too much. You've got some of M106 visible, along with the nuclei of the nearby galaxies (and some arms, too), but you haven't made the noise too visible. For a dataset that sounds like it was less ideal than you would have liked, you seem to have done a good job of `staying conservative', IMO.

The big question now will be: What more can you do with PI, if anything? I think it will be worth trying PI's modules, to see if you can stretch a little more out of the image, without it looking too noisy. I think that Dynamic Background Extraction and the various noise-reduction strategies will allow you to get somewhat more out of the image, without looking too noisy. I predict that with a careful application of PI, you'll be able to get more than you could with PS. Hopefully you'll be pleasantly surprised!

I think your head's in the right place, though, in knowing that one can't get an *infinite* amount of detail out of an image that we wish we could have shot longer. That's a good `foundational' idea.

I'll be curious to see what you can do with these data and PI!

- Marek

[DanielF]

Hi Daniel,

Looking at what you've done in PS so far, I think you did one thing quite right - you didn't try to pull too much out of the image. You've got a pretty even background, and although it's somewhat noisy, it's not *too* noisy looking. If you had a noisy background with large gradients, then I think you were smart to not try and stretch the image too much. You've got some of M106 visible, along with the nuclei of the nearby galaxies (and some arms, too), but you haven't made the noise too visible. For a dataset that sounds like it was less ideal than you would have liked, you seem to have done a good job of `staying conservative', IMO.

The big question now will be: What more can you do with PI, if anything? I think it will be worth trying PI's modules, to see if you can stretch a little more out of the image, without it looking too noisy. I think that Dynamic Background Extraction and the various noise-reduction strategies will allow you to get somewhat more out of the image, without looking too noisy. I predict that with a careful application of PI, you'll be able to get more than you could with PS. Hopefully you'll be pleasantly surprised!

I think your head's in the right place, though, in knowing that one can't get an *infinite* amount of detail out of an image that we wish we could have shot longer. That's a good `foundational' idea.

I'll be curious to see what you can do with these data and PI!

- Marek

Hi Marek.
Thank you for your comments. I will definitely try to improve the image with the help from PixInsight but so far I've not been satisfied with my results. Probably because I'm a complete beginner at this. As soon as I can find some time and really dig into it, I will post the result here.
Hopefully someone else can have a try at the data and we can compare the results.

/Daniel

[DanielF]

Ok now I have given it a try in PixInsight.
I had to redo the preprocessing because something was not right with the first version I did. I have uploaded the new version to endor, and updated the link in the original post.

This is what I managed to do this time.
These are the actions I performed.
Dynamic Crop
DBE
ColorCalibration
HistogramTransformation
ACDNR
A bit more HT
CurvesTransformation (Saturation)
SCNR (To remove a bit red from the background)
A litte more CurvesTransformation (Saturation)

I think the background maybe got a bit more even and darker but the color definitely improved. Still it needs a lot more exposure time.

[Alejandro Tombolini]

Hi Daniel, I think you have achieved a very good result. I have tried to work on the background to smooth it as much as possible.

This is the final result:




Details of processing

Crop to correct borders


After that CBR - Amount:1 and Protect highlight: 1


Apply first ABE


Second ABE


First DBE


Second DBE


With AberationInspector (4,100,0) built a reference image to be used in BackgroundNeutralization


ColorCalibration


AberrationInspector (4,500,0) to built a Mosaic and on it generate the PSF to be used in deconvolution


Deconvolution protecting stars with starmask



HT


Color Saturation


ACDNR to Chrominance


HT


InterChannelCurves protecting the galaxy to improve the background working in Luminance and with CIE c as target.



ATWT to a duplicate of the image to built a mask to protect galaxy


HT protecting galaxy


ATWT protecting galaxy whit the previous mask


CurvesTransformation whit inverted mask protecting background


MMT with starmask


CurvesTransformation with starmask


Saturation of stars with inverted starmask to protect the background


Resample

[DanielF]

Hello Alejandro!
Thank you very much for taking the time to process my data and for presenting your steps in such detail. I really appreciate it!

I will definitely try and follow your steps and se if I can get a similar result.
I have som initial questions. The step where you apply InterChannelCurves with CIE c as target, what exactly does that do?
And the step where you use ATWT to create a mask to protect the galaxy, how does that mask differ from the previous mask created with RangeSlection?

Once again, thanks for your help.

/Daniel

[Alejandro Tombolini]

Hi Daniel,

InterChannelCurves allows you to modified a channel (CIE c in this case) in function of other that is used as reference (CIE L in this case).
I have applied STF to a preview to show what I am trying to improve. There are areas too much saturated that could not been solved with the previous aggressive ACDNR applied. See in the RealTime how it change. 
The Rangemask in this case is used to protect galaxy and affect only background with ICC.



The second mask used to protect during HT is very different from the previous Rangemask. ATWT is applied in a duplicate off the image and in this case the idea is to have a mask only with structures, (galaxy and stars) without background and protect those structures in the original image and the HT is going to affect only the background.



Hope this helps and I am happy you appreciate the processing details.

Saludos, Alejandro.

[rfriedrich]

Hi Alejandro!
I found your article regarding noisy background becaue I am suffering exactly this problem!
Having said that I am a beginner of PixInsight is very important because perhaps you could put a little more patience in your answers ;-) ... I hope there will be some ;-)

The first I did was to check which process icons you have in your screenshot - so what you used - and this I wrote down.
I got to the point of Deconvolution - where I can not get a "before-after effect" but anyway.
So I am between "Deconvolution" and the first "Historic Transformation" following your process icons in the screenshots

The point where spent the last 5 hours!!!!! is the step from Deconvolution to the first Histogram Transformation.
I have a similar photo like you, very noisy and many background pixels.

I am near to the complete disperation.....
In your explanation from one screenshot to the next the noise is gone!!!
How did you do this?
Can you please help me in this step very detailed?
Going further does not manke any sense without getting rid of this noise because it gets even worse then.

I hope you find the time to reply with a reproducable flow for me from a really noisi image to a nearly clear one.

Thank you very much
best regards
Roman

[Alejandro Tombolini]

Hi Roman,

Thanks for asking and for your interest in the processing steps.

"Before-After Effect"
At the time of Deconvolution the image is been showed with an automatic STF applied which is good to see how Deconvolution affects the image in general or the background but do not allow to see very little changes in bright areas.
Note that I have used STF in all process until Histogram transformation.
The next screen shot has deactivated the STF and is more noticeable the change made by Deconvolution in the core of the galaxy and the stars. 



You are right that Deconvolution is very time consuming, at least for me, but in some images worth the time.

Deconvolution with preview:

The result of the process in the complete image and in a preview are different, but even that you can approach your result using a preview (maybe 1/4 of the image) and then get the final approximation in the complete image.
In this case I have used a preview of the galaxy and only had to adjust a little the deringing parameters for the final image.
Tip: you can try in a very small (and fast) preview of some stars and learn how Global dark parameter in Deringing affect it, and  then you will know how to adjust it in the final image.

"the noise is gone!!!"

Unfortunately not, the noise is still there.

After deconvolution come Histograms Transformation.

First remember that at this time you have the image with and STF applied which means that if you deactivate it will see the image almost completely black.
With HT I have stretched the image to a point where the noise is not as evident as the noise that we were seen with the STF.

This is the real change after applying HT:

Before HT
   

After HT


Saludos, Alejandro.


PS: Do not despair! It's a hobby

[rfriedrich]

Hi Alejandro,
Yes, youre right, it is a hobby but it would be nice getting some little success steps with the time. This noise problem is a realy painful one in the moment because I need to figure out the right way. But PixInsight is the one tool I really want to learn because I know that this is the right one! I never worked with Photoshop, just a little Deep Sky Stacker. I will stay at this path of PixInsight, no matter of erroring in the moment.
Thank you for your great hints and descriptions. I will try them and will give my best to improve ;-)
I will keep you posted which level of improvement I got after trying to follow your description.

By the way - Would it help to reduce the problems in noise postprocessing to make BIAS and Flat files too? In the moment I just to Darkframes.
Another thing which I did not consider so far is dithering. I just leave the Mount guided by PHD Guiding and do one Photo after the other - without dithering.
What do you think? Would that help to reduce the noise problem already in the BatchPreprocessing Script?

Thank you and very best regards
Roman

[Alejandro Tombolini]

Hola Roman, (sorry if there are mistakes but I am writing whitout text corrector)

Diethering is a very good idea but in my case I do not do it for the same reason of you, I usually let the scope alone taking pictures.
Related to darks, bias and flats, it is a very important part in my workflow as my Canon 450 is extremely noisy in comparison with anothers I have use like Canon 5 D or Canon 60D.
I use masters already done.
In the case of masterbias with enought bias to eliminate all horizontal lines. If I remmember well is a superbias made with 300 bias.
I also have a gallery of masterdarks made with 40 dark each, with  different times and allways at the same iso (work with iso 800).
It does not worry too much the temperature as PixInsigh calculate the scale factors to compensate that.
I use the same master flat if the only problem is bigneting and make new ones if that day I see something extrange as dunouts.
Regarding too bathpreprocesingsprit I am not using it. I have  tried it a couple of times but I fell more confortable doing it step by step.

[rfriedrich]

Hi Alejandro,
What a day! Again 37°C here in Vienna and my first - no my second - AHA experience.
The first were your tipps previously and your explanation. The second one was a little bit hidden ;-) but you can imagine my joy about finding out that I have to reset the STF with the active photo and then apply the HT with the preview ... A small step for you experienced guys but a big step for little Roman ;-)
The noise is gone!!!
I always applied the HT to my STF Photo (not resetted) .... you can try what you get ;-)

I get closer to my target!!!! Thank you Alejandro very very much!

Now I am working on getting my picture looking a little more happier - meaning a little more with stronger colours. At the moment there is a nice structure already - of course not perfect - but it looks boring ;-)

Lets see what more functions I can discover and deepdive in this phantastic tool.
very best regards
Roman

[DanielF]

Hello again Alejandro.
I've been trying to follow your processing steps and I think I have got the hang of most of it.
The following step though I have some problem with.  I don't get the desired protection of the galaxy when using the mask previously created using ATWT. I see that you have checked the "k-Sigma Noise Thresholding". What values have you used there? When using the default values the galaxy core was smeared too much compared to the image in your step.

ATWT protecting galaxy whit the previous mask

Regards
Daniel

[Alejandro Tombolini]

I've been trying to follow your processing steps and I think I have got the hang of most of it.

Hola Daniel,

That are excellent news!

I see that you have checked the "k-Sigma Noise Thresholding". What values have you used there?

Oops!, you are right the values are:

Threshold: 3
Amount: 0.5
and it is checked "Soft thresholding"

I don't remember now that step on this work flow, but if you need to check something else, attached you will find the process used. They will be open in Workspace01. Sorry the description is in Spanish.

mmmhhh, well, for some estrange reason I have to exclude the two DBE process because the are too big and the maximum size is 512 KB.   

Saludos,
Alejandro.

[DanielF]

I don't remember now that step on this work flow, but if you need to check something else, attached you will find the process used. They will be open in Workspace01. Sorry the description is in Spanish.

mmmhhh, well, for some estrange reason I have to exclude the two DBE process because the are too big and the maximum size is 512 KB.   

Saludos,
Alejandro.

Thank you very much Alejandro! This is very helpful.
/Daniel