Unable to remove gradient using DBE

[zwik]

Hi All,

Recently I've been shooting on M45 with my new setup. All was finally working out and I've some OK data to play around with. Now I started to play around with PI to get some experience. The whole stack was done with PI. My data is shot with a modified Canon 600D (Europe)

My stack consists of 8 lights, 21 flats, 21 darkflats and 23 darks, these are just details. After stacking I did the following:

• Trying to remove the gradient using the DBE tool. I ran this tool two times
• Extracted the luminance using the ChannelExtraction tool
• Set two previews to calibrate the background and white balance using the BackgroundNeutralization and ColorCalibration tools
• Now I try to stetch using the HistrogramTransformatoin tool

Though, after using the DBE tool I run the STF AutoStretch to see if all went well. Unfortunately I see a gradient in the upper right corner as displayed below. The gradient is caused because of my OAG that's partially in the way. I've fixed this for the next time I'm shooting . However, I can fix the gradient using the the GradientExterminator plugin in Photoshop. Can you please help me fix this problem in PixInsight as well?


I've been toying around with different settings in the DBE tool but I can't seem to figure out how to get rid of this gradient. I was reading about the substraction and division options in the Target Image Correction section but I don't really see a difference.

I've been using the following tutorial to get me going. There you can also find the settings I've been using in the DBE tool.

Thanks for the help!

[oldwexi]

Hi zwik!
To analyze better what causes the dark corner in the red channel of the image i would need the
MasterFlat and the stacked MasterLight image.

This dark corner in the red channel is not a gradient. Split your RGB image in 3 channels with ChannelExtraction, Stretch the blue and red channel Image,
and you will see the upper right corner in the blue channel is o.k., the upper right Corner in the red channel ist much darker.
So, as long as your OAG is not translucent for BLUE light only -  it is NOT the OAG!
So without seeing the original data i tend to assume that your flat is not 100% o.k.. Especially your camera is modded
so check the MasterFlat if the upper right corner is brighter in the red channel.

Also the DBE tutorial you used does not fit for this kind of image, it may fit with images were less bright objects are there.
Your image is full of nebulosity and bright stars.
NEVER set the DBE Points near bright stars this creates dark Halos, dont place the points all over the Image.

If you stretch your image which got your DBE, you see dark areas around all bright stars and nebula this comes from placing
the DBE Point to near at the bright objects.

Having the requested data will help to analyze better and finally allow to help.


However this are all assumptions as long as you dont show the MasterFalt and the linear stacked MasterLight.

Gerald

P.S.: I do not recommend the usage of a painting tool like PS for astroimages -  it destroys the data and dynamic.
        Use Pixinsight for developing your image data..

[zwik]

Hi Gerald,

Thank you so much for your big reply and explanations! I will upload all the stacks later tonight so you can have a look as well but I will first try to see what's going on using the ChannelExtraction tool with the tips you gave me in your post.

Concerning the DBE tutorial, what is a better way? Should the DBE tool be used at all for this kind of objects? I removed the DBE points near bright stars and nebulosity as much as possible as this was stated in the tutorial. When toying around with the DBE tool to get a better understanding I indeed saw that when I place more points over the image there were indeed black spots and halos.

Anyway, I will look into the image again later today as I'm at work now, and inspect my master flat on all channels for errors. After that I will upload my masterflat and masterlight for your inspection and will post my findings here.

Thanks!

PS: I know PS is not the right tool for astro images but I was able to remove the "gradient" there and got frustrated I couldn't do it in PixInsight . I like to process in PI all the way!

[zwik]

I think I solved the problem. I inspected my flats but all looked fine. In each of the channels there was as much signal as in other channels. At that point I went inspecting my calibrated lights with the flats. Here I found out that the vignetting from the OAG is still there. Yes, i'm sure it's vignetting of the OAG. So I redid everything and made a new stack. I ended up with the following:

And after I did a background neutralisation on an area with less nebulosity:


Looking good! I solved it by setting the the "Calibrate" check in the "Master Flat" section of the ImageCalibration tool. I'm not exactly sure what it does but it sure seems to do the trick! Please correct me if I'm doing this wrong.

My next question is, how do I neutralize the background and set the color balance right in a nebulosity rich image? In Gerald's post I read that I shouldn't use the DBE/ABE and even BackgroundNeutarlization tools because there are no real valid area's to do this. So how should I continue from when my stack is done?

Thanks!

[pfile]

well, the thing is that the master flat needs to be calibrated or created from calibrated flat subs. this is because the statistical analysis of the flat that's necessary to compute the scaling can be thrown off if the bias signal is still present in the flat.

so you did the right thing, unless you ended up calibrating your flats twice…

rob

[zwik]

I still don't fully understand it I think because there's so much calibrating . I followed the guide on the PixInsight website to generate the master dark and flat: http://pixinsight.com/tutorials/master-frames/index.html

So I calibrated my flats with my masterdarkflats instead of a masterdark because of dSLR. And then I calibrated my lights with the calibrate checkmark on as described above. So I calibrated them twice now, which is wrong?

[oldwexi]

Hi zwik!
You misunderstood my post. I did not say dont do DBE/ABE. I wrote
dont set the DBE Points near bright stars and nebula!

For sure DBE, Background neutralization is needed.

Again, if you provide a light, a flat and  the masterflat and the masterbias i can try to analyze whats happened from the beginning.

Gerald

[zwik]

Hi Gerald!

Sorry for the misunderstanding! I will try to get around with the tools and see what happens.

I've also uploaded some of my files for you. I've uploaded a light, original flat and masterflat, a masterdarkflat and a masterdark. I don't have a masterbias because I substract bias with the masterdarkflat and masterdark.
Here are the links to my dropbox:
Light: https://www.dropbox.com/s/uv5pefnkxbtyubs/M45_LIGHT_600s_400iso_%2B27c_LPS-P2_01096stdev_20141019-05h30m12s626ms_a.fit?dl=0
flat: https://www.dropbox.com/s/5kh2saszcks6he7/M45_FLAT_Tv8%27%27s_200iso_%2B21c_LPS-P2_00296stdev_20141019-06h15m14s649ms.CR2?dl=0
masterflat: https://www.dropbox.com/s/ftv144pb6tr6auq/masterFlat.fit?dl=0
masterdarkflat: https://www.dropbox.com/s/yfq5uhi4aiifwzz/masterDarkFlat.fit?dl=0
masterdark: https://www.dropbox.com/s/xcauwtt318yncfa/masterDark.fit?dl=0

Thanks for looking into this!

[pfile]

I still don't fully understand it I think because there's so much calibrating . I followed the guide on the PixInsight website to generate the master dark and flat: http://pixinsight.com/tutorials/master-frames/index.html

So I calibrated my flats with my masterdarkflats instead of a masterdark because of dSLR. And then I calibrated my lights with the calibrate checkmark on as described above. So I calibrated them twice now, which is wrong?

it sounds like you may have calibrated the flats twice. DSLRs can be very difficult to work with since the camera firmware is always playing tricks with the "raw" data, so this may work but it's not a proper calibration flow...

however IC may have skipped the 2nd calibration if it could not find a correlation between the calibrated master flat and your master dark, which is likely. you'd have to check the console log to see.

anyway Gerald will be able to get to the bottom of it with your frames.

rob

[oldwexi]

Hi zwik!
Thanks for the files,  but i wont be able to
check them carefully before Friday noon.

A first check showed, that the flats lack very much red.
If you debayer your flat and open the histogram you will see that the green and blue channel
are nicely in the middle (14-Bit = 0.12), the red channel is only half as bright, it is at 0.05 !!!
This will produce very very red calibrated debayered lights and therefore  to much red loss in the upper right corner
where the OAG darkens all channels.

Surprisingly your light has the histogram in all 3 channel very similar!?!
What also surprises me is that the masterdark and the masterdarkflat have the same brightness...!!!
As your light is only very very little brighter than the masterdark, so the light looks like already beeing calibrated?
+
Let me know is the light really raw or already calibrated.

Anyhow your flats need to be better.

Gerald






Gerald

[oldwexi]

Hi zwik!
To answer your question how to use DBE with your image.
Find attached a link to a screenshot which shows a processing example with your single light.
http://www.werbeagentur.org/oldwexi/DBE_BG_neu.jpg
Top left
Step 1 DBE with all Points and Parameter
middle result of DBE (DBE nicely removes the black corner in the upper right)

Step 2 BackgroundNeutralization
Right  result of BackgroundNeutralization

Gerald

[zwik]

Thanks for all the replies guys! Much appreciated!

...

it sounds like you may have calibrated the flats twice. DSLRs can be very difficult to work with since the camera firmware is always playing tricks with the "raw" data, so this may work but it's not a proper calibration flow...

however IC may have skipped the 2nd calibration if it could not find a correlation between the calibrated master flat and your master dark, which is likely. you'd have to check the console log to see.

anyway Gerald will be able to get to the bottom of it with your frames.

rob

This may indeed have happened and to be honest I can't really remember what I did. I will redo everything and save the instances and save them inside the project. What is the correct way to stack and calibrate? Should I calibrate each flat with the masterdarkflat or should I calibrate my light stack (with the checkmark set to on) while integrating? Now I'm totally confused on what the correct process is . I followed this process to create my masterdark/masterdarkflat/masterflat: http://pixinsight.com/tutorials/master-frames/index.html

Hi zwik!
Thanks for the files,  but i wont be able to
check them carefully before Friday noon.

A first check showed, that the flats lack very much red.
If you debayer your flat and open the histogram you will see that the green and blue channel
are nicely in the middle (14-Bit = 0.12), the red channel is only half as bright, it is at 0.05 !!!
This will produce very very red calibrated debayered lights and therefore  to much red loss in the upper right corner
where the OAG darkens all channels.

Surprisingly your light has the histogram in all 3 channel very similar!?!
What also surprises me is that the masterdark and the masterdarkflat have the same brightness...!!!
As your light is only very very little brighter than the masterdark, so the light looks like already beeing calibrated?
+
Let me know is the light really raw or already calibrated.

Anyhow your flats need to be better.

Gerald

No worries Gernald, take your time!

I've inspected my flat as well and I don't fully agree on you. I ran some tests and the red channel always seem to lag behind. I made test flats with and without ND filter on my flatpanel and combinations with LPS and H-alpha. Here's a link to all my flats in a zip file. All are untouched. Link: https://www.dropbox.com/s/goo8rqswuknuwqc/flat%20test.zip?dl=0 . So my big question, how can I improve on my flats?

A friend talked to me a bit tonight and we talked about it. He has the same camera and also follows a similar scenario in processing. He inspected his flats and also there the red seems to lag behind. I'm not saying you're wrong but it just catches my attention and doesn't seem that much of a problem to me and my friend. If I understand correctly the correct way to calibrate lights is with non-debayered flats. So color doesn't seem a problem at all? I found this thread: http://pixinsight.com/forum/index.php?topic=5748.0 . In post #13 a process is described but it seems wrong to me to debayer first before calibrating? Especially with H-alpha.

Concerning the light I've provided, yes, it's a non calibrated image. I've only converted it to FITS format because of performance. I've checked the original CR2 file and it is exactly the same . The masterflat and masterdarkflat are indeed a tad darker compared to the light. What could have caused this? Did I stack my masterdark and masterdarkflat wrongly? I followed this guide on PixInsight.com: http://pixinsight.com/tutorials/master-frames/index.html

Hi zwik!
To answer your question how to use DBE with your image.
Find attached a link to a screenshot which shows a processing example with your single light.
http://www.werbeagentur.org/oldwexi/DBE_BG_neu.jpg
Top left
Step 1 DBE with all Points and Parameter
middle result of DBE (DBE nicely removes the black corner in the upper right)

Step 2 BackgroundNeutralization
Right  result of BackgroundNeutralization

Gerald

Thanks! I will use this .

[oldwexi]

A friend talked to me a bit tonight and we talked about it. He has the same camera and also follows a similar scenario in processing. He inspected his flats and also there the red seems to lag behind. I'm not saying you're wrong but it just catches my attention and doesn't seem that much of a problem to me and my friend. If I understand correctly the correct way to calibrate lights is with non-debayered flats. So color doesn't seem a problem at all? I found this thread: http://pixinsight.com/forum/index.php?topic=5748.0 . In post #13 a process is described but it seems wrong to me to debayer first before calibrating? Especially with H-alpha.

Hi Zwik!
Yes, calibration is done with none debayered lights! I only debayered them to show clearly WHICH color is underexposed in the flat.
If you put the histogram over your undebayered raw, there will be the same 2 histogram bumps far apart but only in B/W.

An underexposed channel in the flat gets the same curve in the Profile as the others, but if the colors are so far apart in the flat, 
it is a big difference dividing with flat values between 0.11 and 0.13,   or dividing by 0.04 and 0.06.

So if you take your raw light, the 3 histogram bumps are very near together , if you take your
flat the histogram bumps are far apart.
I would try to check why your flat is so strong greenblue and has only half red intensity.
The light you use for the flats is not white now. Try to get a better light source.

Gerald

[pfile]

actually, if you have an Ha image from an OSC for which you are only interested in the red channel, you can debayer first as long as you use SuperPixel mode, and extract the red channel as a mono file. then you are still dealing with the raw red pixel values, and if all your subs were preprocessed this way then you can make masters and calibrate the lights.

if you don't use superpixel debayering, then this is not correct as the interpolation of the missing R pixels will be done before calibration. but with superpixel, there is no interpolation. the red pixel at 0,0 goes to the output image at 0,0, the red pixel at 2,0 goes to the output image at 1,0 and so on. this is why superpixel-debayered OSC images are 1/2 the pixel dimensions of the sensor.

anyway Gerald is absolutely right. you need good signal strength in each channel of your flat (or those channels that you care about.) if the SNR of the flats is low, then this low SNR corrupts your calibrated frames.

i am not sure if PI calculates a scaling factor for each flat channel individually. certainly for CFA images this is hard to do, but not impossible. if it does, then low SNR is your only problem. if it does not, then since the signal is very low in the R channel the scaling will probably be wrong and one or more channels will be under or over corrected.

rob