Rainbow background after Dynamic Background Extraction

pzampella

pzampella

Member
Hi,

After using Dynamic Background Extraction, I am getting the background looking like an rainbow. Is this a sympthom of some error?

The problem is evident after the Dynamic Background Extracting, where the background looks like a rainbow. Attached you can se how it looks like pre and post extraction. Is this problem familiar to any of you?

I already tried to fix this with Color Calibration, or trying to reduce the chrominance noise, but after stretching looks horrible...

This are the pre-processing parameters:

  • 82 lights at 200 ISO, 60s through a Skywatcher Blackdiamond 150/750, manually dithering along the RA + Dec axis
  • 20 darks
  • 20 flats at 1/5s
  • 10 darks flats
  • 200 bias

DSS:

  • Lights: Media Kappa-Sigma Clipping (2 and 5)
  • Darks: Media Kappa-Sigma Clipping (2 and 5)
  • Flats: Media Kappa-Sigma Clipping (2 and 5)
  • Bias: Media Kappa-Sigma Clipping (2 and 5)
  • Automatic Alineation
  • Cold and hot pixels removed

Pixinsight:

I followed this tutotial for stacking: https://www.lightvor...pixinsight.html
Feel free to ask me for any other parameter that you might find useful!

Literally any recommendations will be fully appreciated!

EDIT:

These are the stacked images in:
Pixinsight: https://www.dropbox.com/s/8bmcr00mlzo8zxw/PI_Manual.fts?dl=0
DSS: https://www.dropbox.com/s/7nhz1wfwxdxt5mw/Original.fts?dl=0
 

Attachments

  • 1.JPG
    1.JPG
    182.2 KB · Views: 206
  • 2.JPG
    2.JPG
    152.7 KB · Views: 210
  • raw.JPG
    raw.JPG
    37.1 KB · Views: 173
Last edited:
it could be that the sample tolerance was too low or too high and some of the samples have either missed background signal or found too much signal (actually color noise.)

one thing to try is to split the channels, do 3 separate DBEs, then re-merge the channels and see what it looks like. you can also do further DBE to the merged image in case there are still gradients. it is sometimes hard to see a subtle gradient in a mono image. but the advantage is there's only one channel so there's no ambiguity about how well tuned your sample tolerance is.

rob
 
pfile said:
it could be that the sample tolerance was too low or too high and some of the samples have either missed background signal or found too much signal (actually color noise.)

one thing to try is to split the channels, do 3 separate DBEs, then re-merge the channels and see what it looks like. you can also do further DBE to the merged image in case there are still gradients. it is sometimes hard to see a subtle gradient in a mono image. but the advantage is there's only one channel so there's no ambiguity about how well tuned your sample tolerance is.

rob
Click to expand...

I made sure that all samples have no starts not nebulosity, and it cover the whole image. Could it be something to do with the debayer?

I see that Pixinight identifies the RAW as GBRG, but according to the camera specifications, it is RGBG (https://www.imaging-resource.com/PRODS/canon-t5/canon-t5DAT.HTM).

That image debayered with GBRG - VNG.
 
Don't worry about the bayer pattern that is detected by LibRaw, 'GBRG' is correct, corresponding to the crop that is applied by LibRaw in order to exclude the optically black region of the sensor.

'RGBG' is definitely wrong:

R G
B G

In this case there would be columns of photosites that have only green filters. In Bayer patterns the green pixels are always arranged in a diagonal, i.e. either XGGX or GXXG, (X = R or B).

Bernd
 
bulrichl said:
Don't worry about the bayer pattern that is detected by LibRaw, 'GBRG' is correct, corresponding to the crop that is applied by LibRaw in order to exclude the optically black region of the sensor.

'RGBG' is definitely wrong:

R G
B G

In this case there would be columns of photosites that have only green filters. In Bayer patterns the green pixels are always arranged in a diagonal, i.e. either XGGX or GXXG, (X = R or B).

Bernd
Click to expand...

Actually, I debayered the frames using all 3 possibilities and this is what I got. The first one is using GBRG, the second one is RGGB and the third one is RGBG.

For what I understand, the third one should be the correct one? Because it has a random noise pattern, instead of the square mosaic pattern. The weird part is that one is RGBG.
 

Attachments

  • mosaics.JPG
    mosaics.JPG
    195 KB · Views: 98
I was referring to opening a CR2 file from a Canon 1200D with PixInsight, because you showed a section of a PixInsight process console output. When a CR2 file of a 1200D is opened in PixInsight, the correct bayer pattern is 'GBRG'. This is the bayer pattern that is determined by LibRaw, and it is based on the cropping that Libraw executes in order to remove the optically black region. You can either debayer with the setting 'Auto' or with the setting 'GBRG' to get a correct result. Any other setting for bayer pattern will yield a wrong result.

Besides you don't seem to use CR2 files, but FITS files. Then the situation may be different. When I use my Canon 600D with CR2 format, the correct bayer format is 'GBRG', when I use FITS format (e.g. saved by SGP), the correct bayer format is 'RGGB'.

The pattern in your rightmost image section looks OK. However, I really doubt that it was produced by debayering in PixInsight with a bayer pattern set to 'RGBG'. Perhaps you can check your statement.

As far as I know there is not one camera which has a bayer pattern 'RGBG' or generally: 'XGXG' or 'GXGX'. These patterns would not have the green filters arranged in diagonal direction.

Bernd
 
bulrichl said:
I was referring to opening a CR2 file from a Canon 1200D with PixInsight, because you showed a section of a PixInsight process console output. When a CR2 file of a 1200D is opened in PixInsight, the correct bayer pattern is 'GBRG'. This is the bayer pattern that is determined by LibRaw, and it is based on the cropping that Libraw executes in order to remove the optically black region. You can either debayer with the setting 'Auto' or with the setting 'GBRG' to get a correct result. Any other setting for bayer pattern will yield a wrong result.

Besides you don't seem to use CR2 files, but FITS files. Then the situation may be different. When I use my Canon 600D with CR2 format, the correct bayer format is 'GBRG', when I use FITS format (e.g. saved by SGP), the correct bayer format is 'RGGB'.

The pattern in your rightmost image section looks OK. However, I really doubt that it was produced by debayering in PixInsight with a bayer pattern set to 'RGBG'. Perhaps you can check your statement.

As far as I know there is not one camera which has a bayer pattern 'RGBG' or generally: 'XGXG' or 'GXGX'. These patterns would not have the green filters arranged in diagonal direction.

Bernd
Click to expand...

I think I understand then. I was using the RAW pattern, but since at this point I have already done calibration, etc., it is a FITS file.

I will continue using the 'Auto' setting and see how it goes.
 
I guess there still remained ambiguity. The right bayer pattern depends on the files that came out of the camera. For the Canon EOS 1200D: CR2 files opened in PI -> 'GBRG', FITS files opened in PI -> 'RGGB'.

If you capture the frames in CR2 format and open the files in PixInsight, the correct bayer pattern with this camera is 'GBRG'. This doesn't change when you calibrate the light frame.

If you capture the frames in FITS format (e.g. with SGP or other acquisition software) and open the files in PixInsight, the correct bayer pattern with this camera is 'RGGB'. This also doesn't change when you calibrate the light frame. Whether the 'Auto' bayer pattern will work with FITS files or not depends on whether the acquisition software writes the FITS keyword 'BAYERPAT' to the FITS header. There is software that uses this keyword, but some software doesn't.

Data in Color Filter Array (CFA) format (= bayered data) must not be rotated, mirrored or cropped with an uneven number of rows or columns at the left or top. By such operations the bayer pattern is changed. This is one reason for confusion, since the vertical orientation in FITS files is not standardized. Different software might read a FITS file in a different way, mirroring vertically can be the consequence. This is one of the reasons that the FITS format is used as input format in PixInsight, but this format is deprecated as an output format. You should not use the FITS format when you save processed (e.g. calibrated) data, please save in the XISF format.

Bernd
 
bulrichl said:
I guess there still remained ambiguity. The right bayer pattern depends on the files that came out of the camera. For the Canon EOS 1200D: CR2 files opened in PI -> 'GBRG', FITS files opened in PI -> 'RGGB'.

If you capture the frames in CR2 format and open the files in PixInsight, the correct bayer pattern with this camera is 'GBRG'. This doesn't change when you calibrate the light frame.

If you capture the frames in FITS format (e.g. with SGP or other acquisition software) and open the files in PixInsight, the correct bayer pattern with this camera is 'RGGB'. This also doesn't change when you calibrate the light frame. Whether the 'Auto' bayer pattern will work with FITS files or not depends on whether the acquisition software writes the FITS keyword 'BAYERPAT' to the FITS header. There is software that uses this keyword, but some software doesn't.

Data in Color Filter Array (CFA) format (= bayered data) must not be rotated, mirrored or cropped with an uneven number of rows or columns at the left or top. By such operations the bayer pattern is changed. This is one reason for confusion, since the vertical orientation in FITS files is not standardized. Different software might read a FITS file in a different way, mirroring vertically can be the consequence. This is one of the reasons that the FITS format is used as input format in PixInsight, but this format is deprecated as an output format. You should not use the FITS format when you save processed (e.g. calibrated) data, please save in the XISF format.

Bernd
Click to expand...

Reading what you are saying, I must confess that I have been using the FITS format during the master generation (the tutorial I was following said that it should not be a problem). I will retry everything using the XISF format to see how it goes.

However, I just did the debayering with the Auto option and this is the result after trying to extract the background. It is not as bad as before, but the rainbow noise still remains.

I'll get back with the results of this new try :)
 

Attachments

  • 3.JPG
    3.JPG
    184.8 KB · Views: 77
It appears that you have used Local Normalization. Try integrating without it. Two other things you may try is the CanonBandingReduction script to remove banding, and the Multiscale Linear Transform with a mask, to reduce the noise from the upper layers. But before that, I suggest using the simplest possible workflow, preferably using PI solely :).
 
dld said:
It appears that you have used Local Normalization. Try integrating without it. Two other things you may try is the CanonBandingReduction script to remove banding, and the Multiscale Linear Transform with a mask, to reduce the noise from the upper layers. But before that, I suggest using the simplest possible workflow, preferably using PI solely :).
Click to expand...

Yes, I did use Local Normalization like in the image, with scale of 256 and 128. Do you see something wrong there?
 

Attachments

  • LN.JPG
    LN.JPG
    65.8 KB · Views: 99
Well, Local Normalization might be unnecessary. It is a powerful tool used when gradients vary wildly between light frames and needs care to set up properly. If not, it introduces its own gradients at the edges of the image and sometimes bloats the stars. Your "rainbow background" is another side-effect. With fewer workflow steps the task of troubleshooting gets easier. That's why I suggested using a simplest workflow.
 
Indeed, removing the local normalization step solved it! Thank you!!!
 

Attachments

  • solved.JPG
    solved.JPG
    202.1 KB · Views: 96
Ok ok, so it is A LOT better, but when I stretch it I will get this weird blueish colored background noise in grand scale. I am not sure if this has something to do with the original problem, but it certainly looks similar.

index.php


The steps perform before stretching are:
  1. Dynamic Background Extraction
  2. Background Neutralizacion
  3. Color Calibration
  4. Multiscale Linear Transform (to reduce luminance and chrominance noise)
I also tried using the SCNR set to blue (I normally use green, but since the noise is blue...), but my galaxy looses coloration.

I have tried to take away some steps in case they are the root cause of the problem, but the result is pretty similar.

Here there is a link to the original stacked image, before any editing: https://www.dropbox.com/s/9gm1qul8nskao2r/20200428.xisf?dl=0

Is there something I am still missing? Or maybe my data is the problem?
 

Attachments

  • Captura.JPG
    Captura.JPG
    116.7 KB · Views: 651
Last edited:
The horizontal banding suggests that there went something wrong in the image calibration.

pzampella said:
I followed this tutotial for stacking: https://www.lightvor...pixinsight.html
Feel free to ask me for any other parameter that you might find useful!
Click to expand...
The cited tutorial recommends to calibrate the individual dark frames with the MasterBias. (or superbias). This is bad advice, see my guide https://pixinsight.com/forum/index.php?threads/for-beginners-guide-to-pis-imagecalibration.11547/ .

Bernd
 
bulrichl said:
The horizontal banding suggests that there went something wrong in the image calibration.


The cited tutorial recommends to calibrate the individual dark frames with the MasterBias. (or superbias). This is bad advice, see my guide https://pixinsight.com/forum/index.php?threads/for-beginners-guide-to-pis-imagecalibration.11547/ .

Bernd
Click to expand...

You are right, the histogram of the master dark was being clipped to the left. I am processing everything again and I will share the results later.
 
So, you were absolutely right: the problem was the dark master calibration. This is how it looks like after stretching:

index.php


This is still far from what I want, but equipment and technique must have something to do with it ;)

In any case, thank you all for helping me, and any other recommendation is appreciated!
 

Attachments

  • Captura.JPG
    Captura.JPG
    100.6 KB · Views: 586
You must log in or register to reply here.
Back
Top