Vertical Pattern Noise

[roodyj]

I have recently had problems with the images from my ZWO 1600MM cooled camera. All of my images demonstrate a vertical pattern of noise bars. It appears that the bias frames also contain the noise.

This link https://drive.google.com/open?id=1NEGNaI0_bIDgVLv1Gdg0MMAXuO5sHkA0 points to the latest bias images I have taken, along with an integration of all 75 images using the recommended settings in the PI Image Integration process.

I initially contacted ZWO technical support about this problem. However, I have been told by them that the integration process I am using is most likely the culprit. Since I only use PI, my posting is in the hope that someone reading this may  somehow help me correct my settings. FWIW, I have already tried every combination of Rejection Algorithm settings I could think of.

Please give me some insight on what may be the real problem here (yes I am a 'newbie').

Thank you,
Jim

[ngc1535]

Hi,

Hmmm...
I note you have made available a FIT file instead of an XISF file... Was there a reason for this format?
Next, the HISTORY of that file looks odd to me. There are hundreds of lines of image integration stuff. I would expect there to be just a few lines of History. In addition, your sigma factors of 7 for high and low will reject virtually nothing and leave you with basically a mean image.  That is true if I am reading the History correctly. Finally, in looking in detail at the image- it appears quite noisy for a average of that many frames.

So my guess is that something is going on. Can you make available a small sample of your original biases?
The header doesn't report the temperature (other than 0, which I assume you didn't take these biases at).
-adam

[bulrichl]

When I compare this MasterBias of a cooled ASI1600MM to a FlatDark of my ASI294MC Pro, it doesn't look unusual to me for a CMOS camera: with my camera, the subframes show a (random) horizontal banding which is averaged out by stacking, leaving a vertical banding in the integration which is the "fixed pattern noise" of the sensor. This fixed pattern noise is contained in the light subframes as well and will be removed by the calibration process.

@Adam, it is quite normal that the data scaleEstimate, locationEstimates, imageWeights, zeroOffsets, rejectedLow and rejectedHigh are stored for each subframe in the FITSHeader of the integration, this is the same with integrations saved in XISF format. I agree that the sigma values used for integration are probably slightly too high - I leave them at the default values (with Winsorized Sigma Clipping low sigma: 4, high sigma: 3). However this is not supposed to be the problem here.

@Jim, I guess your issue is not related to this MasterBias but to your preprocessing workflow. Can you please describe it in detail?

Bernd

[roodyj]

The FITs versus XISF files are simply the format saved by SGPro. Also, I am not sure that the folks at ZWO have access to PI XISF. The original bias file are also  saved as FITs files on the google link. I will save several as XISF also so they can be compared:

https://drive.google.com/drive/folders/1G_Gt3_-1SK0NDvncQ8YK_J_PO9e0bTue?usp=sharing

My workflow is very simple:

Use SGPro to take the bias frames (or any frames, actually)
Process them directly in PI using the Integration Process
     (I am using the process Icon from Warren Keller's set)
For astro image frames, use PI Batch PreProcess
     (I have also tried manual preprocess instead of using the BPP

Jim

[bulrichl]

Yes, SGP saves the subframes in FIT format, but Adam was referring to your MasterBias which is in FIT format as well. However, the format is not connected with your issue.

I don't detect any abnormality comparing a single bias frame with the MasterBias.

You showed neither an uncalibrated / calibrated light frame nor an integration after calibration with the MasterBias, debayering and alignment. Is the "vertical noise pattern" that you see in the light frames really banding (i.e. "fixed pattern noise")? I cannot tell you without seeing these images, it could be walking noise (caused by hot / warm pixels in connection with drift) as well. It is not possible to give useful advice without further information.

Generally a full calibration needs dark frames, flat frames and flat-darks (or bias frames). I guess with a CMOS sensor you will not succeed without using at least a MasterDark. A calibration without MasterFlat can only give a decent result if vignetting is negligible and you use perfectly clean optics. So I suggest that you take a lot of dark frames, make a MasterDark and conduct the light frame calibration with this MasterDark instead of the MasterBias. If this minimalist aproach is free of vertical pattern, you are one step ahead. However, presumably you will need a MasterFlat in addition.

Perhaps you should take a look at my guide https://pixinsight.com/forum/index.php?topic=11968.0.

Bernd

[roodyj]

So I suggest that you take a lot of dark frames, make a MasterDark and conduct the light frame calibration with this MasterDark instead of the MasterBias. If this minimalist aproach is free of vertical pattern, you are one step ahead. However, presumably you will need a MasterFlat in addition.

Thank you for your information, @Bernd. I do have MasterFlats for each filter. Normally I do not use darks since the ASI1600MM is very low noise and others have said that darks are unnecessary for good calibration. Live and learn.

I will have to start from scratch and follow your detailed guide. With the weather the way it is locally here, I have lots of time to polish up my PI skills based on expert's inputs.

Thank you again, and I will let you know about improvement as I progress.

Jim

[roodyj]

The latest bias integration that I have attempted is:

https://drive.google.com/file/d/1Y8gRD82l_uvTB7yPEI2inYVg_PXYb-BZ/view?usp=sharing

This series of frames was captured with APT instead of SGPro, to eliminate the SGPro as the culprit. The integration was done simply using no Superbias or other tricks. I also have changed the camera location to my office, indoors, rather than on my mount to eliminate any electrical noise from the other items (focuser, guidecam, mount, etc.). I carefully checked and set the Integration Process options to what I believe are correct.

Same vertical pattern noise as before.

It seems that the only constant has been the ASI1600MM. My feeling is that the camera is causing the vertical noise, not the processing software.

I would appreciate any further input on this topic to help my claim of hardware failure.

Thanks to you all,
Jim

[bulrichl]

1) Maybe you misunderstood what I was trying to say: the vertical pattern in a master bias of a CMOS camera is not unusual and needs not be wrong. There are a few examples of MasterBiases of an ASI1600 available in the internet, all showing a vertical pattern. What you see is the "fixed pattern noise" of the sensor. So you should not strive to produce a MasterBias without it - it will be impossible.

2) The two MasterBiases that you uploaded differ considerably from each other. Actually the bias offset is differing by 177 ADUs! Here the statistics:

            A_Integration_of_Bias_1x1_75_frames     Bias_50frames_0_002sec_exposures_integration
count (%)   100.00000                               100.00000
count (px)  16389120                                16389120
mean        321.908                                 145.347
median      321.920                                 145.280
stdDev      2.024                                   2.177
avgDev      1.973                                   2.148
MAD         1.898                                   1.898
minimum     303.573                                 123.520
maximum     365.653                                 182.080

This difference could be due to the fact that one series is taken with APT and the other with SGP or because different settings of gain and offset were used. This is for sure: Very odd results might appear if different image acquisition software is used for taking light, bias, dark and flat frames in one project. You should not do that. Stick to one application for image acquisition, at least within one project! Of course gain and offset settings have to be the same. Please check which reason is applying in your case.

Conclusion:
It is very important that all calibration frames match your light frames. This means: same images acquisition software, same settings in the image acquisition software (gain, offset, camera temperature, if applicable exposure time). Furthermore, using appropriate settings in the precalibration workflow (calibration and integration of calibration frames during the preparation of master calibration frames) is necessary.

Bernd

[roodyj]

Will apply your advice. Thanks again.

Jim