Poor Image Integration

[Doug]

I have had major problems in getting a final image after Image Integration, initially thinking it was a light polution issue.
I have followed Atropixel's work flow 1.6.1 - 1.6.9 using all the suggested settings.
I have also created master dark, flat and bias using the workflow suggestions
My post processing image comes out worse than one of the JPEGs taken at the time of exposure.
I imaged M27 from a dark site using a Canon 1000D taking 9 x 5 minute lights, 6 darks, 20 flats and 20 bias.

Images were captured RAW and JPEG, see attached for pre-processing JPEG.
See attached for Post Processing after Image Integration.
The HistogramTransformation only increases whiteness in the image.

I know it's not the images as other software has aligned and integrated including calibration of darks, flats and bias.
I cannot see where the processing is going wrong, but it's consumed a lot of time.
My problems seem to have started moving from the demo to full licensed version. (probably not connected)

Without going into too much detail, is there anything that I could be missing in the process.

Regards
Doug

[Philip de Louraille]

By comparing both image provided, I noticed that the post-processed one is missing a lot of the faint stars and the background is too dark. Have you performed a noise-removing step? Or moved the black point too far to the right?
Colors are also washed out in the second pic. It is obvious on the nebula but is also apparent on the numerous stars. I am thinking a bad option in the Integration?
Did you choose Average, Additive+Scaling, and what did you use for the weights?
What algorithms did you choose for Pixel Rejection 1 & 2?

[Doug]

Hi Philip

I didn't move the black point
In Image Calibration I Optimized the Master Dark.
The Post Processing image is before any Histogram Transformation or Curves Transformation

I used Average, Additive + Scaling and Noise Evaluation for the weight and Evaluate Noise checked
For Pixel Rejection 1- Linear Fit Clipping and Scale + Zero Offset and all boxes checked
For Pixel Rejection 2 - Linear Fit Low 4.000 and Linear fit High 2.000

Do you think that the problem is in Pixel Rejection?

[Philip de Louraille]

Do an integration *without* a pixel rejection.
After the calibration steps (dark, bias, flat) I presume you debayered the resulting frames?

[Doug]

I've Integrated without Pixel Rejection and it doesn't make any difference.
I have debayered using RGGB matrix for the Canon 1000D

[Philip de Louraille]

Before doing the integration step, can you open the debayered, calibrated light frames (9, if I read correctly) and do a ScreenTransferFunction on each (the Automated [A] button usually does a great job) and see what you are starting with?
I am starting to wonder if one of your masters if messing you up.

[Doug]

Under Screen Transfer Function, the calibrated files look very noisy and yellow in colour.
Prior to Calibration the converted files are faint with presence of the nebula.
As you say, one of the masters is messing up.
I have tried without masters and get a very green looking image.
I will try and find out which master by calibrating one master at a time.

[Philip de Louraille]

When you say "Prior to calibration, the converted files are faint with presence of the nebula." I am assuming you are talking about the bayered frames? (where the RGB is still encoded?)

[Doug]

The basic steps I've used are:
Convert all RAWs to FITS (under Screen Transfer Function, lights are grey with faint stars and nebula)
Integrate Bias FITS and create Master Bias
Calibrate Dark FITS
Calibrate Flat FITS
Integrate calibrated Dark FITS and Create Master Dark
Integrate calibrated Flat FITs and create Master Flat
Calibrate Light FITS with Masters (under Screen Transfer Function, these are very noise consisting of green and reds and blues)
Debayer calibrated Light FITS (under STF, these are noisy and yellow)
Align Lights
Integrate registered calibrated lights to create Master Light

Should I be working in a different order?

[Philip de Louraille]

I have always created the bayered master frames. Darks, flats & bias. Then calibrated the (still bayered) lights, then with PixInsight, unbayered the lights and then align and integrate them.
If I use Nebulosity, I will align and integrate the (still bayered) frames and unbayer at the last step of the post-processing.
BUT! and it is a big but, my frames come out of SBIG cameras under the FIT format.
Your frames come out of a Canon under a RAW format which need to be converted to FIT although I have read posts from folks stating that now, with PI, you can create the masters under the RAW format (for some cameras) and get your FITs later. But i have no experience with RAW frames as an input.

To see if one of your master frames is incorrectly corrected, bypass them.
After converting your RAWS to FITS, debayer the lights, align and integrate and STF the result.

[Doug]

I have bypassed Masters and managed to get an image after processing.
I will now start again including and calibrate my lights using Master Dark and integrate.
Next I will calibrate lights with Master Dark and Master Flat and integrate.
Finally calibrate lights with Master Dark, Flats and Bias.
I should be able to spot which Master is giving me the problem.
My Flats have been created using a ligh panel, and look uniform in colour.
This will probably take me a few days so I will keep you posted.

How do I improve on the Nebula itself?
I have tried stretching using Histogram Stretch and Colour Curves but the background and stars change colour.
I have tried Star Mask but the background is not masked from the nebula.

[Philip de Louraille]

While your image is still linear (before applying a histogram transformation), do a background neutralization.
You can then do some noise removal with the Àtrous Wavelet alg (use the denoising button)

Then apply the Histogram transformation
Then I would use the HDRWT alg to improve the nebula.
Then use the Curve Transformation to improve color, contrast
You can use the ÀTrous later on as well to remove the small noisy details.

For every image processing person, you'll find a different series of steps...

[Doug]

I have carried out tests and it looks like my Master Flat is causing the problem in Pixinsight.
See attached for a debayered flat taken using a light panel that covered the area of the dew shield.
Also attached is a Calibrated Light using the Master Flat integrated from all my flats.

What am I doing wrong as using an alternative demo software, I don't get these problems?

[Philip de Louraille]

I am not sure I understood by what you meant with here is "a Calibrated Light using the Master Flat integrated from all my flats."
The Calibrated Light means that dark noise and bias have been subtracted?
If so, the image of the nebula you attached is then corrected of those two *and* you have attempted to remove the gradient sing the Master Flat?

If you think your master flat is incorrect, do not use it and, instead, use the AutomaticBackgroundExtractor function. Just an idea.

[Doug]

I was trying to find out if it was my Dark, Flat or Bias FITS that were giving the problem.
I calibrated my lights with Master Dark subtraction only, which was ok.
I then calibrated my lights with only the Master Flat subtracted, without Dark or Bias subtraction.
As the flat images were uniform in colour , I was suprised to see the gradient in this image.

I do use DBE which didn't make much difference to this image.
I will try a few more tests to see if I can get an integrated image.