Problem pre-processing Ha images taken with DSLR

[MikeOates]

I am having a problem with pre-processing Ha images taken with a Canon 500D DSLR with an Astrodon 5nm Ha filter. The flats were obviously taken with the same filter, but if those flats are used in pre-processing it seems to remove the red from the Ha image.

I am not doing anything different from a standard RGB image, should I be?

I show a screenshot of the image at different stages...

From left to right:
1. The fit of a cr2 file after batch format conversion.
2. The debayered image (This is as I would expect, if SFT is linked the image is very red, again as I would expect)
3. The debayered image after calibration (red colour removed!)
4. The master flat used in the calibration

Images 1 to 3 are shown with SFT (unlinked channels)
Images 4 is shown with SFT (unlinked)

Where can I be going wrong? This is my first Ha image, so it is quite likely I am doing something stupid. My normal flats have a blue colour and are taken with an EL panel, but they seem to turn out ok.

Mike

[georg.viehoever]

What are the statistics of your MasterFlat?

[MikeOates]

Georg,

The statistics are:

Mean: 0.0100682
Median: 0.0049982
AvgDev: 0.0061748
StdDev: 0.0100282
Variance: 0.0001006
Minimum: 0.0013107
MinPos: 3027,3177
Maximum:    0.0521923
MaxPos: 226,738

Scientific notation:

Mean: 1.006817e-002
Median: 4.998205e-003
AvgDev: 6.174782e-003
StdDev: 1.002815e-002
Variance: 1.005638e-004
Minimum: 1.310749e-003
MinPos: 3027,3177
Maximum: 5.219235e-002
MaxPos: 226,738

I have to admit, I don't know what I am looking for 

Thanks,

Mike

[astroedo]

I have had a similar problem with a CLS filter that has a strong blue cast.
In my opinion the problem lies in the G and B channels that are highly underexposed.
When you apply the flat it boosts these channels dimming the red one.
I solved taking three different flats for each color channel and than putting together with pixelmath.
Now I'm not at home, this evening I will post the pixelmath expression to merge channels in a bayered image

[georg.viehoever]

...I have to admit, I don't know what I am looking for 
...

I thought maybe something was strange with you flat. But the statistics look reasonable.

[pfile]

georg, are you sure? i'd expect the mean for a good DSLR flat to be in the 0.1 range, not 0.01. in other words a saturated 14-bit frame gives 0.25 in 16-bit space, so 0.125 is 1/2 well depth, no?

also as noted above the CLS will completely underexpose your red channel when the blue/green are well exposed. when i was using a DSLR and CLS filter i ended up using a slightly pinkish t-shirt to try to get the flat histograms centered up. there's no real need to have the histograms centered except for the fact that if one channel is underexposed you can end up injecting noise into your subs with the underexposed flat (unless you have 100s of flat subs, maybe.)

anyway you are going to throw away the green and blue channels here, so i guess you could let the blue/green channels become overexposed when taking flats. that assumes you do not use a fancy debayering method which may use the G channel to help reconstruct R. if you stick with superpixel or bilinear debayering you will be fine here.

[astroedo]

Finally at home 

As I said before in my opinion the problem is in the flat.
Your G and B Channels probably are almost black and when you apply your flat dividing your image, G and B are Boosted up and, to normalize the image, the R is dimmed.

To check this I've uploaded a pixel math Icon that, in your Bayer Matrix Image (let's suppose RGGB pattern), puts the R value in G and B channels.

Use it on your flat field and than use this new "synthetic" flat to calibrate your image: the result should be better.

Remember that this approach is, in general, WRONG: you cannot mix channel without loosing information, but in this specific case G and B channels are almost useless so is not so bad    .

If this experiment works and you want to calibrate your G and B channels properly you must consider to get three different flats with different exposures in order to have G and B flat channels properly exposed and than mix together in a single Bayer matrix flat that uses the proper pixels.

If you are interested I already have a pixel math expression which does this job.

Please let mi know if it works!

bye

[MikeOates]

Astroedo,

Thank you very much for that PixelMath icon. I tried it, but unfortunately the result was exactly the same.

Anyway, I have done more thinking on this and I have found a solution and it works very well 

The procedure is...

Create New Master Flat

• Open the master flat
• Debayer the master flat
• Extract channels just keep red
• Save this as Ha master flat

Extract Red channel from sub exposure

• Open Ha sub frame that has been calibrated but not including the flat correction
• Debayer
• Extract channels just keep the red and save the file

The above red channel extraction can be done using the BatchChannelExtraction script.

Do flat calibration using ImageCalibration...

• Point to target frames (the extracted and saved R channels)
• Select just the Master Flat section and point to the new Ha master flat.

The result is a gray Ha image ready for further processing.

Wonderful. I hope others find this of help.

I also use a CLS filter for non narrowband images, so I may find that the same process will improve my RGB images!

Mike

[pfile]

unless you use superpixel debayering, that sequence is not correct. once you have debayered with any type of pixel interpolation, the calibration results are invalid.

[MikeOates]

Thank you pfile,

I have just done a whole load of searching and reading on CFA and debayering and I am slowly beginning to understand.

The method I showed above did appear to work, but perhaps I was just lucky with that particular data and it just looks ok to the eye, but scientifically it is flawed.

I think I have a good idea now how to implement the processing that I need, I just need some time to try my ideas out. I will come back with my findings.

Mike

[pfile]

it will work in that the results may look okay, but as you say from a scientific point of view it's not correct.

anyway what you can do is while making your flats, don't pay attention to the g and b channels, just make sure R is properly exposed. then you can calibrate your images and debayer them, and then use the channel extraction script to pull out just the red channel. then you can register these mono images to your reference and stack them.

if you use the bayer RGB format then i guess you could use channel extraction to pull out the red channel from your lights, flats, bias and darks first, then put all those mono images into the calibration workflow. that would result in less disk space used. i have never tried this, but it seems like it would work.

rob

[MikeOates]

Rob,

anyway what you can do is while making your flats, don't pay attention to the g and b channels, just make sure R is properly exposed. then you can calibrate your images and debayer them, and then use the channel extraction script to pull out just the red channel. then you can register these mono images to your reference and stack them.

This I will try at some point when I get the time.

if you use the bayer RGB format then i guess you could use channel extraction to pull out the red channel from your lights, flats, bias and darks first, then put all those mono images into the calibration workflow. that would result in less disk space used. i have never tried this, but it seems like it would work.

rob

Yes this is one of the ideas I was going to try. I have now done this and it looks good, but I need more data / time to check it out further. It certainly produces much smaller files and the subsequent processing is also very fast. I am now using the superpixel debayering. The extraction of the red channel from all subs and calibration files removes all the noise from the other channels. I will report back in a few days with the process and results.

Thanks,

Mike

[pfile]

cool. one thing i realized right now is that calibration after superpixel debayering is still not correct for the G channel. since there are 2 G pixels, SP debayering averages them together. if you imagine that one of the 2 green pixels is stuck at it's max value, then you've 'corrupted' the other green pixel by averaging it with junk before attempting to calibrate it.

to be fair, after calibrating a raw bayer matrix with a stuck pixel like that, you'd probably end up averaging the good G pixel with 0, which is just the same thing. of course, using CosmeticCorrection might get your hot or cold pixel into a state that's more representative of your neighbor pixels. if you do that on the raw image, then it has less of an impact on that good G pixel. if you do CC on the debayered image, then you end up throwing away all the good data that was in the unstuck pixel.

of course, these are really picky points... but there is really only one right way to do calibration - on the raw files.

[MikeOates]

I said I would post back here with my results, so here goes...

Method to calibrate Ha subs from DSLR

Take light frames (ISO1600 10mins)
I typically take as many lights as I can in the time, you can't take too many.

Take calibration frames. I now take the following in this order:
flats (30) using an EL panel exposure @ f5 ISO200 5seconds
bias (150) fastest shutter speed (i.e. 1/4000 sec) at same ISO as the lights (ISO1600) [Cap on scope/camera]
dark flats (30) same ISO as the flats (ISO200) and same exposure length as flats (5sec) [Cap on scope/camera]
darks (30) same ISO as the lights (ISO1600) and same exposure length as lights (10min) [Cap on scope/camera]
Remember those are my exposures, they may be different for you.

The following is the calibration method I used, followed by the processing method. The end result is the Ha image of NGC 7000 & the Pelican Nebula.

I am not saying this is the correct method, but it worked for me, and you can use it as a basis for experimentation.
Also note that this is only for use with Ha images where only the red channel has data. As pfile says, this may not work for a green channel as there are two pixels involved.

Calibration

BatchDeBayer the RAW DSLR (CR2) files using the Super Pixel method.

BatchChannelExtraction the DeBayered files: Just the red channel.

The result is a mono file that has no contamination from the other two channels.
This makes files much smaller 14Mb in my case.
Do this for all subs and calibration files.
To speed this up, if all files are located in one folder they can be done in one batch for both the BatchDeBayer & BatchChannelExtraction.

Also being much smaller files this also speeds up all subsequent processing.

Intergrate the bias frames to produce the master bias:
ImageIntegration:
   No hints
   Image Integration
      Average
      No normalization
      Don't care (all weights=1)
      Scale estimator (default Iterative k-sigma)
      Select generate integrated image
      NO (Evaluate noise)

   Pixel Rejection (1)
      Winsorized Sigma
      No normalisation
      all boxes selected

   Pixel Rejection (2)
      Min/Max low: 1
      Min/Max high: 1
      Percentile low: 0.020
      Percentile high: 0.010
      Sigma low: 4
      Sigma high: 3
      Linear fit low: 5
      Linear fit high: 2.5
      Range low: 0
      Range high: 0.98

Save the integrated bias as a master bias

Now calibrate the darks using the master bias using ImageCalibration:

ImageCalibration:
   Add dark files
   Set Output dir
   Tick Master Bias and pick master bias file
   do not tick Calibrate
   do not tick Master Dark or Master Flat

Use ImageIntegration as for the master bias and save the master dark.

Next integrate the dark flats and save the master dark flat, the process being the same as the master dark & bias.

Calibrate flats using the dark flats, there is no need to use the bias as this is already in the data with the dark flat:
Only select the Master Dark, which in this case is the master dark flat just saved. Do not tick the calibrate or optimize.

integrate the calibrated flats using ImageIntegration:

Image Integration:
   Add files
   Average
   Multiplicative
   Don't care (all weights=1)
   Tick: generate integrated image

   Pixel Rejection (1)
      Percentile Clipping
      Equalize fluxes
      all boxes selected

   Pixel Rejection (2)
      Min/Max low: 1
      Min/Max high: 1
      Percentile low: 0.020
      Percentile high: 0.010
      Sigma low: 4
      Sigma high: 3
      Linear fit low: 5
      Linear fit high: 2.5
      Range low: 0
      Range high: 0.98

Calibrate the lights, using the master bias, dark and flat files, untick all calibrate and optimize boxes as these masters have already been calibrated and the darks are the same exposure as the lights.

Cosmetic Correction:
   Auto detect at default
   - hot 3.0 sigma
   - cold 3.0 sigma

Now align those calibrated lights using StarAlignment, I kept all the default settings.

The final integration is next:

   No hints

   Image Integration
      Average (Median for better rejection)
      Additive
      Noise evaluation
      Select only generate integrated image

   Pixel Rejection (1)
      Winsorized Sigma
      Scale + zero offset
      All boxes selected

   Pixel Rejection (2)
      1
      1
      0.2
      0.1
      4
      3
      5
      2.5
      0
      0.98

Note: Settings used for integration of the various calibration files are based on:
Master Calibration Frames: Acquisition and Processing http://www.pixinsight.com/tutorials/master-frames/en.html

Post Processing

ATWT with settings
1: S(3.000,1.00,2
2: S(2.000,0.50,2
3: S(1.000,0.50,1
4: S(0.500,0.25,1

K-Sigma on a default 3.00,1.00
tick soft & multiresolution

Rescale again 200% with Bilinear to allow further work on over sampled image, which will later be downsampled.
This step may not be needed, just an idea I had to get the image processed oversampled and downsample later, I will try at a later time not including this in order to compare.

Use HistogramTransformation to convert to non-linear image do not clip blacks and save.
Do a CurvesTransformation with a black end curve to effectivly remove the black spots, clumps that can appear after noise reduction with ATWT. (see curves screenshot)
Note this has to be done very carefully as it is all too easy to destroy and shadow detail. You can see in the screenshot the dark spots that I am aiming to reduce with this CurvesTransformation.

Do further HistogramTransformation to boost contrast, again without clipping the blacks. A further curves to reduce the black patches may be required with further HT or contrast adjustment via CurvesTransformation.

LocalHistogramEqualization
   Kernal Radius: 240
   Contrast: 1.5
   Amount: 0.6

UnsharpMask
   StdDev: 2
   Amount 0.8

Dark Structure Enhance Script
   Default settings

ACDNR
   StdDev: 2.0
   Amount 0.60
   Iterations 3
   Robustness: 5x5 Weighted Average
   Structure Size: 5
   Lightness Mask: Mid 0.5, Shadow 0.08, Highlights 1.0

Followed by second application of ACDNR with default settings and no lightness mask

MultiscaleMedianTransform
   8 layers
   4: +0.100
   6: +0.020
   7: +0.010
   8: +0.03

HT to reduce contrast slightly

Resampled to 50% so image is at same scale as any RGB images.

CurvesTransformation to slightly reduce contrast.

NoiseGenerator:
   Amount 0.08
   Uniform
   Preserve Median

The added noise improves the image visually where noise reduction looks too smooth in the shadows near lighter areas.

The processing is finished.

These are all the steps I took, some may not be needed but I have recorded all steps taken here, partly for my own reference and it may help someone else.

Additional info: I did not need to do any DynamicBackgroundExtraction, just as well as it would have been difficult to do for this image anyway. I did also did a slight crop just to remove the edges caused by dithering.

Photo info:

This is actually only the 4th image I have taken with a telescope and the 2nd Ha image I have ever taken. It was taken on 26th July 2013 from light polluted Manchester (UK) with a bright Moon and thin cloud. The scope is a Takahashi FSQ-106 ED, Guided with an 80mm guidescope with Lodestar camera. The image was recorded using a Canon 500D DSLR with full spectrum mod and I used an Astrodon 5nm Ha filter. There are 19 x 600 second subs, captured & guided with Maxim DL5. The mount is a Skywatcher EZ-EQ6 GT, controlled using EQMOD.

The subs were also dithered by up to 7 pixels in Maxim DL.

I will be posting the image in the Gallery section as well.

Thank you,

Mike

PS: The full size image is here: http://www.mikeoates.org/astro/ngc_7000_large.jpg

[pfile]

dslr or not, that looks darn good. thanks for posting the flow.