Please Help! M31 processing

[Mybluox]

Hello all,

  I'm basically a noob beginner, I started using PixInsight about 5 years ago and was doing fair. I had some equipment failures and family issues; so I haven't done anything for about 3 1/2 years.
  I'm now trying to get back into the hobby. Pixinsight has had a few upgrades since I had last used it and Debayering looks totally different than what I was used to. Now my pics are either too Blue or too Red and are really flat. I've tried different Debayering methods and have found 3 that are close.
 I use a Sbig STF-8300c camera

I have posted 3 different Exposures of M31 Andromeda Galaxy 47x300s stacked and Debayered in Dropbox.

1.) Debayered using  BGGR Up Bottom Checked and Upper left Corner for FITs in Format explorer.
Link: https://www.dropbox.com/s/1s62lanqzzlj0lg/M31%20Andromeda%20Galaxy%2047x300s%20BGGR%20Up%20Bottom%20Checked.xisf?dl=0

2.)  Debayered using  RGGB Up Bottom Checked and Upper left Corner for FITs in Format explorer.
Link: https://www.dropbox.com/s/4ynse8e1rj3zghy/M31%20Andromeda%20Galaxy%2047x300s%20RGGB%20Up%20Bottom%20Checked.xisf?dl=0

3.)  Debayered using  GRBG Up Bottom UnChecked and Lower left Corner for FITs in Format explorer.
Link: https://www.dropbox.com/s/0isgqiuukldipr9/M31%20Andromeda%20Galaxy%2047x300s%20GRBG%20Up%20Bottom%20UnChecked.xisf?dl=0

Please see what you can do with these and let me know which was the best debayering. Also please tell me the workflow and setting used; so I can learn to do by myself. I hate being stupid.

If needed I can post all captures with Flats, Darks, and Bias masters.

Thank you in advanced. I really appreciate it.

Jeff

[Mybluox]

 I have uploaded all of my calibration files, including Master Bias, Master Dark, Master Flat and all images taken to dropbox.

Link: https://www.dropbox.com/s/t61nr0zuilzswrm/M31%20Calibration%20files.zip?dl=0

It is in zip format and very large.

Thanks,

Jeff

[wimvb]

I've downloaded the first integrated image and the calibration files. When I calibrated the fits images, I got errors from image 33 onward. It seems that suddenly a pedestal is added to the fits header. Any idea why that is?
I get the same result as you when I try to integrate the images (first 32). I'm not sure which the correct Bayer setting is, so I chose BGGR for deBayering.
The resulting image has a strange colour cast and a very strong gradient that I couldn't remove with DBE without introducing posterisation.
Sorry I can't be of any more help.

[Mybluox]

I've downloaded the first integrated image and the calibration files. When I calibrated the fits images, I got errors from image 33 onward. It seems that suddenly a pedestal is added to the fits header. Any idea why that is?
I get the same result as you when I try to integrate the images (first 32). I'm not sure which the correct Bayer setting is, so I chose BGGR for deBayering.
The resulting image has a strange colour cast and a very strong gradient that I couldn't remove with DBE without introducing posterisation.
Sorry I can't be of any more help.

Image 33 onward were from a different session. I use MaximDL Ver 5 for capturing. There was 1 change I made in the different sessions. I didn't think it would make a difference. In the readout mode I had selected color in one session and Raw in the other. That might be the problem.

Thank you for trying.
Jeff

[Mybluox]

Hi all,

Here is what I did so far. I Calibrated, Debeyered, and Stacked the first 32 images in MaximDL Ver. 5. I then processed in Pixinsight. As I said I'm a noob at this and am still learning.

Here is what I have so far.

[Mybluox]

I put a copy of the unprocessed image in dropbox. If anyone is willing to process it and tell me what I am doing wrong and what I can do better.

Link: https://www.dropbox.com/s/k1dh5m386a74fp2/m31_cc.xisf?dl=0

Thank you,

Jeff

[wimvb]

It seems to me that you still have a lot of hot pixels left in the subs. These are probably the cause for the horizontal lines in your stacked image. You need a more aggressive cosmetic correction to remove them.
Try this:
load one sub into pixinsight and define a preview with several hot pixels in it.
Open cosmetic correction and choose Auto Detect.
Adjust the clipping sigma for hot pixels.
Save an instance of CC and use it in the calibration process (or batch pre processor script)
Here's a screenshot of what I got from your very first sub (m31_001.fit)

[Mybluox]

It seems to me that you still have a lot of hot pixels left in the subs. These are probably the cause for the horizontal lines in your stacked image. You need a more aggressive cosmetic correction to remove them.
Try this:
load one sub into pixinsight and define a preview with several hot pixels in it.
Open cosmetic correction and choose Auto Detect.
Adjust the clipping sigma for hot pixels.
Save an instance of CC and use it in the calibration process (or batch pre processor script)
Here's a screenshot of what I got from your very first sub (m31_001.fit)

Yes I know. I have about 15000 hot pixels.

Thank you,

Jeff

[Geoff]

Did you use data rejection while stacking?

[Mybluox]

Did you use data rejection while stacking?

Geoff,

That I don't know I was using BatchPreProcessing.

Jeff

[Mybluox]

Well I was finally able to integrate my image in PI. I didn't use the BattchPreProcessing script. I instead went through each step separately.

I haven't processed the image yet; but if any of you would like to and show me the proper steps I would appreciate it. You can download the image here:
https://www.dropbox.com/s/s8jbuj8goil09v8/M31_Intregrated.xisf?dl=0.

Thank you in advance.

Jeff

[Mybluox]

I processed the image. Here is what I have. You can download the full image here:
https://www.dropbox.com/s/5nydvwbdbatbdpf/M31_Intregrated_DBE_HDR_Sat_CS.xisf?dl=0

I have a blue line down the right side of the image. I don't know where it came from or how to get rid of it.
I'll take any suggestions. I looked at all of the unstacked images and I don't see any line.

Thanks

[wimvb]

I downloaded the unprocessed image and had a quick look at it. There are still quite a number of hot pixels that calibration and stacking didn't catch. Also the stars look odd. Have you examined the rejection images (high rejection especially)?
If you upload the original light frames and calibration files again, I can have a try at them. The zip file from your earlier post isn't available anymore.
The vertical line requires a bit of trickery to remove. Best is, of course, to avoid it alltogether. Large scale Pixel rejection is probably the best tool for this. To remove it from the integrated image, try this with pixelmath
1. Determine at what x-coordinates the line is. I measured it at x = 2639 ... x = 2641
2. Enter the following formula in PixelMath and execute

iif(x()==2639,pixel($T, x()-1, y()), $T)

3. Next, enter the following formula in PixelMath and execute

iif(x()==2641,pixel($T, x()+1, y()), $T)

4. Then for the middle line

iif(x()==2640,(pixel($T, x()+1, y())+pixel($T, x()-1, y()))/2, $T)

This replaces the outer columns of the line that is 3 pixel columns wide with the neighbouring columns, and the middle pixel column with the average of its neighbours.
However, this leaves a nasty streak. Fill it in with noise through a mask.

Create a mask using PixelMath. Enter the formula

iif(x()>=2639 && x()<=2641,1, 0)

This time set the colourspace in PixelMath to grayscale and create a new image called LineMask.
Apply this mask to the image and use NoiseGeneration to fill in the columns with some noise. Experiment with the amount. I got ok results with Poisson noise and an amplitude of 0.03.

[Mybluox]

I downloaded the unprocessed image and had a quick look at it. There are still quite a number of hot pixels that calibration and stacking didn't catch. Also the stars look odd. Have you examined the rejection images (high rejection especially)?
If you upload the original light frames and calibration files again, I can have a try at them. The zip file from your earlier post isn't available anymore.
The vertical line requires a bit of trickery to remove. Best is, of course, to avoid it alltogether. Large scale Pixel rejection is probably the best tool for this. To remove it from the integrated image, try this with pixelmath
1. Determine at what x-coordinates the line is. I measured it at x = 2639 ... x = 2641
2. Enter the following formula in PixelMath and execute

iif(x()==2639,pixel($T, x()-1, y()), $T)

3. Next, enter the following formula in PixelMath and execute

iif(x()==2641,pixel($T, x()+1, y()), $T)

4. Then for the middle line

iif(x()==2640,(pixel($T, x()+1, y())+pixel($T, x()-1, y()))/2, $T)

This replaces the outer columns of the line that is 3 pixel columns wide with the neighbouring columns, and the middle pixel column with the average of its neighbours.
However, this leaves a nasty streak. Fill it in with noise through a mask.

Create a mask using PixelMath. Enter the formula

iif(x()>=2639 && x()<=2641,1, 0)

This time set the colourspace in PixelMath to grayscale and create a new image called LineMask.
Apply this mask to the image and use NoiseGeneration to fill in the columns with some noise. Experiment with the amount. I got ok results with Poisson noise and an amplitude of 0.03.

WIMVB,

Thanks, Here is the link to the files you requested.:
https://www.dropbox.com/s/qcoq30z78j9c3tm/M31.zip?dl=0

I don't understand the Image rejection settings yet. I'm going through as many tutorials as I can; since I'm still learning.

Jeff

[wimvb]

I got the files and calibrated with the bpp script, including cosmetic correction. DeBayering was a bit problematic. I ended up removing the first and last column of each calibrated sub (image container to the rescue), and using superpixel deBayering with bggr matrix. No remaining hot pixels in the stacked image.
I found a strong gradient in the background that DBE had difficulties with. Still working on that.