Issue with Prepocessing script

[ColinR]

Hi,

I'm having a problem with a set of frames which I am trying to process using the Batch Preprocessing script. I'm new to Pixinsight but I have already successfully calibrated a set of frames using this script. As with the previous occasion I am following the YouTube tutorial from Astro Dude,  Part 12 Batch Preprocessing.

However on this occasion, following repeated attempts to check that I am following the tutorial correctly, I end up with a red image with some yellow stars and black artifacts. The problem actually cuts in at the calibration stage. (I think that the original RAW12 frames from the Altair Astro 183C Pro TEC are OK- maybe I'm wrong? I'm also new to this type of camera, .fits files and to the latest level of SmartCap. Maybe that's the problem!)

Please see my magnified attachments to show what is going on between a captured frame and the final master light.

Bias and Flats (30 each,) were taken on the night, with Darks taken indoors the following day at set point cooling conditions. I also have the log file if needed.

I would appreciate any help on this problem.

Colin

[pfile]

not sure if i can say anything just from that screenshot. have you tried applying an STF with the channels unlocked?

rob

[ColinR]

Hi Rob,
Thanks for the reply. I'm also new to this Forum and didn't realise that I had only uploaded the last of three. Maybe the other two images will explain the immediate effect of just calibrating the images.

ColinR

[ColinR]

Problem resolved but true cause not known.
I rerun the script without Flats and it worked OK. The Flats were way left on the histogram, so maybe that was the cause, but I don't know for sure.

ColinR

[bulrichl]

Hi Colin,

omitting the flat field correction is not the proper solution.

Strong color cast may result after calibration when the intensity of one channel in the MasterFlat deviates severely from the other channels. This seems to be the case here. Compare your captured image with the calibrated image: in the calibrated grayscale image, one channel is way brighter than the others. As you can see in your master light, it is the red channel which deviates.

The reason for the strong increase in red channel intensity after flat field correction is that the red channel of the MasterFlat is considerably weaker than green and blue channels - take a look at the histogram your MasterFlat and you will notice that. This is not bad as long as 1) there is no clipping in the green and blue channel of the flat frames and 2) the intensity of the red channel of the flat frames is not that weak that a low SNR results for the red channel of the MasterFlat. If these two conditions are met, everything is OK. Try Rob's suggestion of applying the STF autostretch with the channels unlinked to your master light (i.e. deactivate 'Link channels' in STF) in order to evaluate the integration. The strong color cast is removed later in the image processing, in the Color Calibration step.

If the two conditions are not met, you will have to capture new flat frames with adjusted exposure time. The intensity of all channels shall 1) not be clipped and 2) be in the middle of the intensity range of your camera. If I understand it correctly, the camera has 12 bit but the output is scaled to the range of 0 - 65535 ADUs. So the target value for the flat frames is around 32000 ADU.

Bernd

[ColinR]

Hi Bernd,

Thanks for your reply.
Yes I completely agree that not applying Flats would be the wrong way to go. In this case I just went without flats as this image set was just a star field that I was using to practice Pixinsight.
The three previous images since moving from PS CS6 to Pixinsight, (as well as an Altair Astro 183C ProTEC and SharpCap!), a few weeks ago have all been OK. So I was thinking that the cause was in one of with these items and me.

Anyway, as I climb up the PixInsight steep learning curve, I'm sure I will come to understand the intracasies of understanding and utilising curves etc. In the meantime I have reran the data with Flats included. (for speed I included only nine of the 30 Flats, Bias and Darks with the original 12 Lights). I notice that althought remaining Red, unlike the original Master Light, this Master Light shows star detail when stretched. (See attachment). I have also attached the Histogram Transformation and the Screen Transfer Function with RGB unlinked.

Just in case this is a factor, unlike any previous Flats, (I use a multi led illumination panel with a high quality perspex diffuser), these flats were taken with a twilight sky and were at a slightly different focus point from the actual images.

My main concern of course is how to prevent this happening again, as trying to correct it afterwards is just outside my current understanding envelope. But I will get there!

So thanks again for your input, I don't know if my attachments will shed any light on what caused the red to be so far from the blue/green.

Colin

[bulrichl]

Hi Colin,

the information that you provided so far is not sufficient. Could you please upload the MasterFlat, MasterDark (in xisf format) and one light frame (in FITS format) to a filehoster and disclose the link?

Bernd

[ColinR]

Hi Bernd,

These are the files you asked for. I've shared a link in Drop Box which should allow downloading.

I greatly appreciate the time you are providing to help me understand the cause of this problem. Hope the files are of use to you.

https://www.dropbox.com/s/zcbkx9h4zvtpso8/dark-BINNING_1-EXPTIME_240.xisf?dl=0

https://www.dropbox.com/s/wdijejcm8y9zsfa/flat-BINNING_1.xisf?dl=0

https://www.dropbox.com/s/t4b3hahoqmzolps/REF_Capture_00002_a.fits?dl=0

[bulrichl]

Hi Colin,

there are two problems with your MasterFlat (see the appended image showing the debayered MasterFlat and its histogram):
- the red channel of the MasterFlat is severely clipped, and
- the amplifier glow shows up negatively.

Your settings for the integration of the calibrated flat frames are correct. So my conclusion is: the flat frames are not calibrated correctly.

How did you calibrate the flat frames (which calibration master files, and which settings did you use)?

Please upload one uncalibrated and one calibrated flat frame.

Bernd

P.S.: I confirmed that the driver scales the output from 12 bit to the range of 0 - 65535 (multiplied by factor 16), accordingly what I wrote about the target value of the uncalibrated flat frames is correct: around 32000 ADU. I suspect that your flat frames are considerably underexposed. However, note that this is not the main point here, it is just not optimal.

[ColinR]

Hi Bernd,

Please see the files you asked for below.

https://www.dropbox.com/s/m35nbnz54yh9st9/Capture_00002_c.xisf?dl=0

https://www.dropbox.com/s/kvoxuhuoj5w2u2p/Capture_00002.fits?dl=0

I processed the set of files using the Preprocessing script. I am sure I left the settings at defaults. I could run this again and take screen grabs of the setting views?
I've now processed a total of four image sets, if you include this one, using both the manual DIY method and the Preprocessing method, using the YouTube videos by Astro Dude. None of the others had the Red problem.

I used a 2" Astronomik CLS CCD filter throughout.

Also these flats, (with the Red problem), were taken at a guessed exposure in twilight and the spots will probably be out of focus stars. After this session I decided to start reading the SharpCap instructions!

I hope this helps. And once again I greatly appreciate the time and trouble you are going to to work on this problem.

Colin
PS ADU? Is this analogous to ISO settings on a DSLR. No doubt I will learn as I delve further into the complexities of the latest level of SharpCap!

[bulrichl]

Hi Colin,

as expected, the red channel is severely clipped in the calibrated flat frame (as it is in the MasterFlat), see the appended image "Cal_Flat_Statistics". Nearly 25 % of the pixels have zero intensity, this is virtually the complete red channel. So the fault occurred during calibration of the flat frames.

Take a look at the FITS Header of the calibrated flat frame (see appended image "Cal_Flat_FITS_Header"). You calibrated 4-s flat frames with MasterBias and the 240-s MasterDark, using the following settings:

'Master Bias' section enabled; "bias-BINNING_1.xisf", 'Calibrate' checked (*)
'Master Dark' section enabled; "dark-BINNING_1_EXPTIME_240.xisf", 'Calibrate' checked, 'Optimize' unchecked
'Master Flat' section disabled

[(*) It is not reasonable to enable 'Calibrate' for the MasterBias, because you don't use Overscan calibration, so there is no calibration of the MasterBias. This setting is not causing any trouble though.]


These settings result in subtracting the 240-s MasterDark from each of the 4-s flat frames. Of course this is not a correct calibration of the flat frames.


There are 3 posibilities to make it better:

1) Use dark frame optimization.
In the first step of the calibration, the MasterBias will be subtracted from the flat frames. In the second step, the SCALED difference (MasterDark - MasterBias) will be subtracted from the flat frames. Settings:
'Master Bias' section enabled; "bias-BINNING_1.xisf", 'Calibrate' unchecked
'Master Dark' section enabled; "dark-BINNING_1_EXPTIME_240.xisf", 'Calibrate' checked, 'Optimize' checked
'Master Flat' section disabled

2) Use only the MasterBias for flat frame calibration.
The corresponding settings are:
'Master Bias' section enabled; "MasterBias", 'Calibrate' unchecked
'Master Dark' section disabled
'Master Flat' section disabled

3) Use a MasterFlat-Dark for flat frame calibration.
In this case you capture dark frames with the exposure time of the flat frames (here: 4 s) and integrate them to the MasterFlat-Dark. The flat frames are calibrated with the following settings:
'Master Bias' section disabled
'Master Dark' section enabled; "MasterFlat-Dark", 'Calibrate' unchecked, 'Optimize' unchecked
'Master Flat' section disabled

Case 3) is preferable. If you follow that route, you'll need no bias frames at all. For light frame calibration, use the following settings:
'Master Bias' section disabled
'Master Dark' section enabled; "MasterDark", 'Calibrate' disabled, 'Optimize' disabled
'Master Flat' section enabled; 'Calibrate' disabled


By the way, this subject is described in detail in my Guide to PI's ImageCalibration, https://pixinsight.com/forum/index.php?topic=11968 .

-----

As I expected, your flat frames are way underexposed, see appended image "Uncal_Flat_Histogram". The peaks should be about in the middle of the horizontal axis of the histogram.

I have still one question: what is the mean of your MasterBias (see ImageStatistics)?

Bernd

[ColinR]

Bernd,

Please find attached the statistics for the Master Bias as requested.
I will study your reply and your document and get back to you with my understanding of the strategies for going forward.

One thing at this early stage of understanding is that I do not understand the 1.2 One Shot Color (OCS) cameras in relation to "true sided". For example I could not find any reference to "Coordinate origin" or "Upper left corner (Up-bottom)" in the Format Explorer.
I did find reference to "Upper left corner..." in the mouse over description in Format Explorer/FITS/Implementation but I don't know how to find out what is the actual setting for any particular file - FITS, XISF or otherwise. 

And thanks again for this help, it really is appreciated.

Colin

[bulrichl]

Hi Colin,

> For example I could not find any reference to "Coordinate origin" or "Upper left corner (Up-bottom)" in the Format Explorer.

In Format Explorer, double click on 'FITS'. A dialog box ('FITS Format Preferences') is opened which allows you to adjust the settings that PixInsight will use for files in FITS format. In the section 'Miscellaneous Options', you'll find the settings for 'Coordinate origin'. Usually the default setting [Upper left (up-bottom)] is OK. However there is capturing software that mirrors the image. That is one reason that I suggest to use always the same capturing software. E.g. when you use different capturing software for taking light frames and dark frames, they would not match at all if one capturing software mirrored the images and the other not.

A mirrored image is not true-sided any more. You could argue: I don't mind, there is no lettering in the universe. I beg to differ. The recognition of an object is disturbed when it is shown in a mirrored view. Therefore I suggested to test (with the capturing software that you are going to use) whether the images are displayed true-sided in PixInsight, and if this is not the case, toggle the 'Coordinate origin'. This can be done at daylight and has to be done only once with a new camera or if you are going to change your capturing software. When you toggle the 'Coordinate origin' setting, calibration Master files that have been produced before are not usable any more.

Bernd

[ColinR]

Bernd,

Thanks for all your help, I have learned a lot since you started to help me. Yes, I have found the FITS Format Preferences as instructed. If I have understood this correctly, my images in Pixinsight are "flipped". Maybe I'm not understanding this correctly because I attribute this to imaging with an astrocam and a refractor telescope? I correct this in PS CS6 in the final TIFF master file.
As for all images being the same I will be using SharpCap for all Astrocam imaging, going forward. It has been quite a jolt from my comfort zone of a DSLR!

One last thing I'm still unsure about: As I said, I have processed the images in question using BatchPreprocessing and cannot see how to check the settings as instructed in your excellent article, which I have been studying. For example there is no Dark-Flat option that I can see in BatchPreProcessing. Am I correct in saying that these options and setting will be available when I chose to manually calibrate the files myself? (I've already done a manual calibration as I said. But I wasn't paying attention to all the settings options, I was just following the YouTube video from Astro Dude.)

Thanks again for all your help.

[bulrichl]

Hi Colin,

no, the image of a refractor is 'true-sided'. Please don't confuse this term with 'upside down'. Example: appended image "Mirrored". Image 1 is the original, image 2 is vertically mirrored and image 3 is upside down. Image 3 is true-sided as well, because it can be transferred to the original by a 180° rotation. Image 2 is not true-sided, because it cannot be transferred to the original image by a rotation.

--------

Regarding the preprocessing script, I cannot really help since I never used it.

For a full calibration with new Master files, I do the following steps:
1) Integrate the dark frames to the MasterDark, integrate the flat-dark frames to the MasterFlatDark (ImageIntegration; Average, No normalization, Don't care (all weights = 1); Winsorized Sigma Clipping, No normalization),
2) Calibrate the flat frames with the MasterFlatDark (ImageCalibration; Section MasterDark enabled; MasterFlat-Dark selected, both 'Calibrate' and 'Optimize' options unchecked)
3) Integrate the calibrated flat frames to the MasterFlat (ImageIntegration; Average, Multiplicative, Don_t care (All weights = 1), Winsorized Sigma Clipping, Equalize fluxes)
4) Calibrate the light frames with the MasterDark and the MasterFlat (ImageCalibration; parameters as described in Reply #10, case 3).

A good MasterDark can be used for a while. I don't capture dark frames very often. A MasterFlat can also be reused for a while provided that the camera was not rotated relative to the scope, the focus is not changed much, and there are no new dust particles. So for the calibration, steps 1) to 3) have to be performed rather rarely.

I guess it is a good compromise to perform steps 1) to 3) "manually" (not with the script). The calibration masters should be checked. Then it might be comfortable to use the batch preprocessing script for light frame calibration (using the light frames, MasterDark and MasterFlat, no MasterBias), CosmeticCorrection, Debayering, and Registering. The ImageIntegration can also be run in the script, but take its result only as a first orientation. Subsequently it will usually take some additional trials with ImageIntegration, because you will have to tweak the Pixel Rejection parameters.

Bernd