Lights & darks calibration issue

[Startraveller]

Hi all,

I'm having issues calibrating my light frames with dark frames. Have tried both BPP and Image Integration, using raw darks and master dark frame. Have also tried playing around with the 'calibrate' and 'optimize' tickboxes in the ImageCalibration module, and have also tried calibrating without bias frames. When I calibrate with bias frames alone, there is no issue. This is my second library of 300s darks, which I created as the first one was giving me the same issue. I've uploaded photos of an autostretched dark-calibrated image, alongside a single dark frame and master dark frame in both PNG and XISF formats here (couldn't do it on the forum for some reason): https://drive.google.com/drive/folders/0B12lFcbG64vNYk9tcm9IUVJSTzA?usp=sharing. Any help would be appreciated!

[bulrichl]

Hi Startraveller,

your camera is a ZWO ASI 1600 MMC, isn't it? For some reason your calibrated image is clipped severely. This is the Statistics:

H_SMosaic_300sec_1x1_Ha_frame1_c
            K
count (%)   0.18870
count (px)  30927
mean        4753.275
median      1828.444
stdDev      9810.908
avgDev      4104.815
MAD         1567.111
minimum     0.800
maximum     64370.822

Only 19 % of the pixels have non-zero value, the rest (81 %) being 0. Maybe (I'm just speculating):
- you have exposure to light (= light leakage) when taking your dark frames or
- the temperatures of light frame and dark frame acquisition did not match at all (temperature of darkframe acquisition >> temperature of light frame acquisition).

I propose to repeat the calibration with the following settings (your MasterDark shall be uncalibrated, i. e. NO previous subtraction of the MasterBias):
--------------------
MasterBias checked
Calibrate unchecked

MasterDark checked
Calibrate checked
Optimize checked

MasterFlat unchecked
--------------------

If that doesn't result in an unclipped calibrated light frame, there is something seriously wrong with your dark frames, and you should make sure:
- that there is no light leakage when taking the dark frames and
- that the correct temperature has been set when taking the dark frames.

By the way, your MasterBias looks like it probably should (a narrow peak with maximum at 306 ADU), but your MasterDark looks suspicious to me (large variations of the ADU values). I don't have a 1600 MMC, but I think the MasterDark should look different. So check the conditions of your dark frame acquisition. And if you actually have light leakage, probably also the light frames could be affected.

In this context maybe it's appropriate to mention that black plastic caps are not sufficient as a light shield when dark frames are taken in the daytime. They are partially permeable for IR radiation. So it's a good idea to take the dark frames in a really dark room.



Bernd

[Startraveller]

Hi Startraveller,

your camera is a ZWO ASI 1600 MMC, isn't it? For some reason your calibrated image is clipped severely. This is the Statistics:

H_SMosaic_300sec_1x1_Ha_frame1_c
            K
count (%)   0.18870
count (px)  30927
mean        4753.275
median      1828.444
stdDev      9810.908
avgDev      4104.815
MAD         1567.111
minimum     0.800
maximum     64370.822

Only 19 % of the pixels have non-zero value, the rest (81 %) being 0. Maybe (I'm just speculating):
- you have exposure to light (= light leakage) when taking your dark frames or
- the temperatures of light frame and dark frame acquisition did not match at all (temperature of darkframe acquisition >> temperature of light frame acquisition).

I propose to repeat the calibration with the following settings (your MasterDark shall be uncalibrated, i. e. NO previous subtraction of the MasterBias):
--------------------
MasterBias checked
Calibrate unchecked

MasterDark checked
Calibrate checked
Optimize checked

MasterFlat unchecked
--------------------

If that doesn't result in an unclipped calibrated light frame, there is something seriously wrong with your dark frames, and you should make sure:
- that there is no light leakage when taking the dark frames and
- that the correct temperature has been set when taking the dark frames.

By the way, your MasterBias looks like it probably should (a narrow peak with maximum at 306 ADU), but your MasterDark looks suspicious to me (large variations of the ADU values). I don't have a 1600 MMC, but I think the MasterDark should look different. So check the conditions of your dark frame acquisition. And if you actually have light leakage, probably also the light frames could be affected.

In this context maybe it's appropriate to mention that black plastic caps are not sufficient as a light shield when dark frames are taken in the daytime. They are partially permeable for IR radiation. So it's a good idea to take the dark frames in a really dark room.



Bernd

Thanks for your help Bernd, appreciate it. I've checked the FITS data for my raw darks, and there was some variation in temperature between the darks (maximum of 1.3 degrees), but I don't think that would cause such drastic clipping.

I've tried your method, and still getting clipping. I agree that it must be the darks causing this issue; I'll rebuild my darks library, making sure there's absolutely no light leakage + that temperature is completely uniform and see how things go.

[bulrichl]

I've uploaded photos of an autostretched dark-calibrated image, alongside a single dark frame and master dark frame in both PNG and XISF formats here

Can you please upload a single uncalibrated dark frame in XISF format? I could not find that on Google Drive.

Virtually a matter of course: are you sure that all light, bias and dark frames have been taken with the same settings of gain and offset values? The calibrated light frame shows unity gain (= 139), so you can have a look at the FITS headers of the bias and dark frames and compare.

Bernd

[Startraveller]

I've uploaded photos of an autostretched dark-calibrated image, alongside a single dark frame and master dark frame in both PNG and XISF formats here

Can you please upload a single uncalibrated dark frame in XISF format? I could not find that on Google Drive.

Virtually a matter of course: are you sure that all light, bias and dark frames have been taken with the same settings of gain and offset values? The calibrated light frame shows unity gain (= 139), so you can have a look at the FITS headers of the bias and dark frames and compare.

Bernd

Single raw dark uploaded: https://drive.google.com/drive/folders/0B12lFcbG64vNYk9tcm9IUVJSTzA?usp=sharing. Yes, I've checked the FITS headers for all images. They're all at gain 139, and my offset is automatically set to 21 whenever unity gain is applied

[bulrichl]

The comparison of the statistics of the single dark with your MasterDark shows that the MasterDark for some reason seems to be scaled, and that is not what you want:

Dark_300sec_1x1_UG_frame1         MasterDark_300s_20C_1x1_UG
K                                 K
count (%)   100.00000             98.91889
count (px)  16389120              16211936
mean        2920.9                16388.585      <==
median      2976.0                16721.606      <==
stdDev      503.8                 5038.851       <==
avgDev      395.9                 3784.080       <==
MAD         320.0                 2965.084       <==
minimum     720.0                 3.888          <==
maximum     65504.0               65533.914

From this finding I guess there are two possibilities:
1) your dark frames are very different from each other or
2) your dark frames are fine, but the integration of the dark frames went wrong.

Have a look at the histogram of all of the 30 dark frames: is the maximum of the peak at about the same location (the uploaded single dark frame's maximum is at 3100 ADU)?

If the answer is no, the dark frames are erroneous. Delete the dark frames and the MasterDark and take new dark frames.

If the answer is yes, delete the MasterDark and make a new one from your existing dark frames.

Bernd

[bulrichl]

I'm curious: did you succeed in calibrating your light frames with a new MasterDark without clipping? What was the solution of your problem?

Bernd

[Startraveller]

I'm curious: did you succeed in calibrating your light frames with a new MasterDark without clipping? What was the solution of your problem?

Bernd

Hi Bernd,

I took a new set of darks and that seemed to fix the problem. I now have a new issue, however! All of my images, whether LRGB/SII/OIII seem to have black spots everywhere; Ha doesn't seem to have this issue, not to that extent anyway. I've tried it without darks and with darks, and the problem persists. Attaching a couple of XISFs of M31 here (64x30s frames at Gain 300, Offset 50 in R channel): https://drive.google.com/open?id=1H6JPMMc8C3C6iUxHBGIhcFquPnEtmzhl. Any ideas? I tried altering the sigma clipping parameters, tried Linear Fit clipping, neither of these worked. I also tried different rejection parameters + no rejection for my darks integration (have taken a new set of these just a day ago) with no luck.

Cheers

[bulrichl]

I don't quite understand what you did. What does it mean "I've tried it without darks and with darks, and the problem persists."?

From the histograms of the XISF files, "M31_Darks" and "M31_NoDarks" it is clear that no clipping occurred in both files. I don't see black spots in the M31 images either. In "M31_Darks" the standard deviation is lower than in "M31_NoDarks", but I cannot comment on that because I don't know how the images were processed. I think the difference is in the calibration of the subframes but will not speculate about that. If you don't describe your settings in the calibration step for both images in detail I canot tell you any more.

The first image (preview of O3) looks quite different, but neither histogram nor statistics can be applied because it is only a screendump. Again I don't know how the image was processed. You wrote that you tried different settings in the image integration. However, I presume the problem will not be in the integration but still in the calibration.

So what can I say any more? Unfortunately nothing.

Bernd

[Startraveller]

I don't quite understand what you did. What does it mean "I've tried it without darks and with darks, and the problem persists."?

From the histograms of the XISF files, "M31_Darks" and "M31_NoDarks" it is clear that no clipping occurred in both files. I don't see black spots in the M31 images either. In "M31_Darks" the standard deviation is lower than in "M31_NoDarks", but I cannot comment on that because I don't know how the images were processed. I think the difference is in the calibration of the subframes but will not speculate about that. If you don't describe your settings in the calibration step for both images in detail I canot tell you any more.

The first image (preview of O3) looks quite different, but neither histogram nor statistics can be applied because it is only a screendump. Again I don't know how the image was processed. You wrote that you tried different settings in the image integration. However, I presume the problem will not be in the integration but still in the calibration.

So what can I say any more? Unfortunately nothing.

Bernd

Sorry about that Bernd, wrote the post in a bit of a rush. What I meant was that I attempted to integrate M31, firstly by calibrating with darks, and then not using any darks; this was to see whether it was my darks causing a problem.

For the processing of the images, all I did was calibrate them in BPP using master dark and master bias frames, followed by registration, and then integration using Winsorized Sigma with the default values of Low 4.000 and High 3.000. I didn't use any cosmetic correction. Someone later on mentioned that there is no need to use bias frames as I'm not using flats; I tried using only a master dark, which did not make a difference.

The black spots and lines tend to be more around the peripheries, and become a lot more obvious in other areas of the image once I've run DBE.

I've added a new XISF to the folder, IC1805_OIII: https://drive.google.com/open?id=1H6JPMMc8C3C6iUxHBGIhcFquPnEtmzhl. If for example you look at the top right corner, there are quite a few black lines present; once I run DBE, these lines also become prominent in other areas of the image. The image was processed using BPP to calibrate using a master dark and master bias, followed by registration and then integration using  Winsorized Sigma with the default values of Low 4.000 and High 3.000. I didn't use any cosmetic correction.

I hope that clarifies things somewhat! Thanks for your help so far Bernd, much appreciated.

[bulrichl]

Regarding the image "IC1805_O3":

Please take a look at the histogram and the statistics of the image in PixInsight. There is no trace of clipping in that image, it just looks fine in both histogram and statistics.

However, when viewing the image at 1:1 I discern dark streaks, particularly in the weaker exposed parts at the edges. These streaks resemble the bright streaks that you get with DSLRs, caused by hot pixels and drift between exposures, when you don't dither. If the streaks had the same orientation in different parts of the image I could imagine that they are caused by dead pixels or unequal pixel sensitivity, and drift between exposures would cause streaks in the integration. However, the orientation of the streaks differ in different parts of your image. Therefore this possibility is excluded.

I can only imagine that the issue still is a fixed pattern noise of the sensor.

Then it would be either correlated to your calibration process (dark signal non-uniformity) or to the fact that you did not apply a flat frame correction (photo response non-uniformity). Please specify all the settings used in the calibration process (best as a screen dump).

Bernd

[Startraveller]

Regarding the image "IC1805_O3":

Please take a look at the histogram and the statistics of the image in PixInsight. There is no trace of clipping in that image, it just looks fine in both histogram and statistics.

However, when viewing the image at 1:1 I discern dark streaks, particularly in the weaker exposed parts at the edges. These streaks resemble the bright streaks that you get with DSLRs, caused by hot pixels and drift between exposures, when you don't dither. If the streaks had the same orientation in different parts of the image I could imagine that they are caused by dead pixels or unequal pixel sensitivity, and drift between exposures would cause streaks in the integration. However, the orientation of the streaks differ in different parts of your image. Therefore this possibility is excluded.

I can only imagine that the issue still is a fixed pattern noise of the sensor.

Then it would be either correlated to your calibration process (dark signal non-uniformity) or to the fact that you did not apply a flat frame correction (photo response non-uniformity). Please specify all the settings used in the calibration process (best as a screen dump).

Bernd

Thanks Bernd. I was doing a high dither every 5 frames. I'll post a screendump later on as I'm about to head out, but my workflow has never involved flats, which could well be the reason. I've done some imaging overnight and will be doing flats to perform a comparison. I've read a lot about optimal ADU values for flats; do you have any insight into this at all?