trouble with calibration.
- Thread starter dpdallolmo@aol.com
- Start date
dpdallolmo@aol.com
Well-known member
Thanks,
So my question is... if a master flat is created after calibrating flat frames, shouldn't it (master flat) show up in the calibration diagram of the light frames?
That is, the diagram should show something like this:
light----master dark----master flat----calibrated light.
today my diagram shows no master flat in the diagram...only master dark...peculiar no?
So my question is... if a master flat is created after calibrating flat frames, shouldn't it (master flat) show up in the calibration diagram of the light frames?
That is, the diagram should show something like this:
light----master dark----master flat----calibrated light.
today my diagram shows no master flat in the diagram...only master dark...peculiar no?
I took a look at the files and tried to calibrate the light frames with PixelMath, using the following expression (I used shorter identifiers for this expression, L for Light, MD for MasterDark, MF for MasterFlat):
(L - MD - a/65535) / MF*mean(MF) + p/65535
p = 100 (this is a pedestal in order to ensure positive values.
and varied the variable a. An approximately correct flat field correction was performed with a = 605.
This result means, that the light frames contain a pedestal of 605 DN that is not contained in the MasterDark. I don't know the reason for this finding for sure. However, it is probable that the offset value which is set in the camera driver was different for light frames and dark frames.
The FITS header of the FITS files does not contain an offset keyword, so I can only conclude indirectly. At gain 0, an offset value of 10 produced a bias of 154 DN. The light frames seem to have a bias that is greater by 605 DN. Assuming that differing offset settings are causing the issue, on can estimate the offset value that was set for the light frames:
offset(lights) = offset(darks) + 605 DN /154 DN * offset(darks) = 49 (that is: approximately 50)
with offset(darks) = 10
So if you want to process your light frames at hand, you can try to capture new dark frames with an offset of 50 set in the camera driver, make a new MasterDark and try whether this works.
Bernd
(L - MD - a/65535) / MF*mean(MF) + p/65535
p = 100 (this is a pedestal in order to ensure positive values.
and varied the variable a. An approximately correct flat field correction was performed with a = 605.
This result means, that the light frames contain a pedestal of 605 DN that is not contained in the MasterDark. I don't know the reason for this finding for sure. However, it is probable that the offset value which is set in the camera driver was different for light frames and dark frames.
The FITS header of the FITS files does not contain an offset keyword, so I can only conclude indirectly. At gain 0, an offset value of 10 produced a bias of 154 DN. The light frames seem to have a bias that is greater by 605 DN. Assuming that differing offset settings are causing the issue, on can estimate the offset value that was set for the light frames:
offset(lights) = offset(darks) + 605 DN /154 DN * offset(darks) = 49 (that is: approximately 50)
with offset(darks) = 10
So if you want to process your light frames at hand, you can try to capture new dark frames with an offset of 50 set in the camera driver, make a new MasterDark and try whether this works.
Bernd
dpdallolmo@aol.com
Well-known member
WOW,
That is complicated. Thank you so much for or all you time and effort!!!
I will try what you say and let you know.
Thank you again,
Dan
That is complicated. Thank you so much for or all you time and effort!!!
I will try what you say and let you know.
Thank you again,
Dan
I am not recommending this procedure for image calibration. The root cause for the trouble is a different setting for the capturing of light and dark frames, probably different offset setting. This should be investigated and it should be ensured that the SAME settings are used in future for the capturing of light and dark frames. There could be a camera driver issue, or an issue with the capturing software, or user error. As long as the FITS header does not contain the FITS keyword OFFSET it is hard to find out what is going wrong.
Probably it is advisable to update SGP to the current v4.0.0.680 as well (in order to include the FITS keyword OFFSET).
Bernd
Probably it is advisable to update SGP to the current v4.0.0.680 as well (in order to include the FITS keyword OFFSET).
Bernd
dpdallolmo@aol.com
Well-known member
Hello Bernd and Robyx,
I want to again thank you for all the help you have given me. I am a bit embarrassed to say that I finally got around to doing some earlier recommendations and those recommendations solved the problem. I updated SGP as suggested and I also updated ASCOM and the driver for the QHY camera. I re-took Darks and Bias frames and now when using WBPP....Everything worked...all the way to image integration!
I don't know if the camera driver, SGP or ASCOM was the problem but with all updated, everything works fine. I suspected it wasn't an issue of the way I took darks, flats, flat darks or lights since everything worked fine in the past and then abruptly changed. So, I will say goodbye and thank you and will get back to processing unfinished data.
All the best,
Dan
I want to again thank you for all the help you have given me. I am a bit embarrassed to say that I finally got around to doing some earlier recommendations and those recommendations solved the problem. I updated SGP as suggested and I also updated ASCOM and the driver for the QHY camera. I re-took Darks and Bias frames and now when using WBPP....Everything worked...all the way to image integration!
I don't know if the camera driver, SGP or ASCOM was the problem but with all updated, everything works fine. I suspected it wasn't an issue of the way I took darks, flats, flat darks or lights since everything worked fine in the past and then abruptly changed. So, I will say goodbye and thank you and will get back to processing unfinished data.
All the best,
Dan
Hi Dan,
I am glad that after all it now works for you.
The feature that offset values of CMOS cameras can be changed within a sequence is a relatively new one. IIRC the ASCOM platform did not support the setting of offset values at all until version 6.5. Then the ASCOM camera drivers had to be updated, and the capturing software as well, because camera driver and capturing software have to communicate (e.g. about the available range of offset values). This all seems to be solved now, but still some compatibility issues of certain driver versions and capturing software exist, see e.g. https://forum.sequencegeneratorpro....event-settings-via-ascom-asi-2600mm-pro/15593 . I guess that these are teething troubles and expect that reliability will stabilize.
I think it is important that gain and offset values are stored in the FITS header in order that one is able to check whether light and calibration frames have been captured with identical settings. Trouble-shooting is thereby facilitated considerably.
Bernd
I am glad that after all it now works for you.
The feature that offset values of CMOS cameras can be changed within a sequence is a relatively new one. IIRC the ASCOM platform did not support the setting of offset values at all until version 6.5. Then the ASCOM camera drivers had to be updated, and the capturing software as well, because camera driver and capturing software have to communicate (e.g. about the available range of offset values). This all seems to be solved now, but still some compatibility issues of certain driver versions and capturing software exist, see e.g. https://forum.sequencegeneratorpro....event-settings-via-ascom-asi-2600mm-pro/15593 . I guess that these are teething troubles and expect that reliability will stabilize.
I think it is important that gain and offset values are stored in the FITS header in order that one is able to check whether light and calibration frames have been captured with identical settings. Trouble-shooting is thereby facilitated considerably.
Bernd
dpdallolmo@aol.com
Well-known member
Hello Bernd,
After redoing my library of darks and flat darks (new library created August 3-5, 2021)and after updating SGP, ASCOM and QHY driver, I was able to calibrate images that were taken and were unable to be calibrated with my older darks and flat darks (old library created in April 2021). What puzzles me is, if I try to calibrate images taken before I had this issue, with the new darks and flat darks, I have the same problem you saw before. That is, the calibrated images show no object and the image looks like a colored hot pixel map. I'm sending a link to google drive that contains 3 light frames (Iris Nebula), 3 flat frames, a master dark taken in April, a master dark taken in August and 2 master flat darks (same time exposure as the flats)...one taken in April and one taken in August.
drive.google.com
could you tell me if there is a difference in the darks or flat darks. Visually, they look the same although I know this is a poor method of evaluation.
I still don't understand why I am struggling with calibration.
Thank you ,
Dan
After redoing my library of darks and flat darks (new library created August 3-5, 2021)and after updating SGP, ASCOM and QHY driver, I was able to calibrate images that were taken and were unable to be calibrated with my older darks and flat darks (old library created in April 2021). What puzzles me is, if I try to calibrate images taken before I had this issue, with the new darks and flat darks, I have the same problem you saw before. That is, the calibrated images show no object and the image looks like a colored hot pixel map. I'm sending a link to google drive that contains 3 light frames (Iris Nebula), 3 flat frames, a master dark taken in April, a master dark taken in August and 2 master flat darks (same time exposure as the flats)...one taken in April and one taken in August.
Pixinsight images - Google Drive
drive.google.com
I still don't understand why I am struggling with calibration.
Thank you ,
Dan
dpdallolmo@aol.com
Well-known member
Also what I notice, while running WBPP, that during debayering, the Gaussian noise estimates are zero
dpdallolmo@aol.com
Well-known member
And, the noise estimates are K sigma. when the process works with older files the noise estimates are MRS
dpdallolmo@aol.com
Well-known member
Pixinsight images - Google Drive
drive.google.com
dpdallolmo@aol.com
Well-known member
dpdallolmo@aol.com
Well-known member
Pixinsight images - Google Drive
drive.google.com
dpdallolmo@aol.com
Well-known member
sorry, I needed to change the "restricted" category
Dan
Dan
Hi Dan,
MasterFlatDark (0.21 s)
April: mean: 153.1 DN, median: 152.6 DN
August: mean: 761.5 DN, median: 761.1 DN0
MasterDark (300 s)
April: mean: 154.2 DN, median: 152.9 DN
August: mean: 761.7 DN, median: 760.2 DN
The difference between the calibration frames taken in August and in April is about 608 DN. This compares favorably with the value (605 DN) that I evaluated with PixelMath (post #43).
My reasoning was and is: during capturing images in April, a low offset value was set in the capturing software or in the camera driver. In August, a much higher offset value was used. A difference of 608 DN is like day and night. It is all-important that the light frames and all calibration frames are captured with the same gain and offset value. According to the FITS header, the gain values (EGAIN = 1) are matching. Unfortunately, despite of the update of the SGP version there is no OFFSET keyword in the FITS header. Admittedly it is not only the SGP version that matters, the ASCOM platform version and the camera driver version have to support setting the offset value as well.
Which ASCOM platform version are you using?
Are you using the QHY ASCOM camera driver or native camera driver, and which version?
Your light frames of the Iris Nebula (captured on July 16, 2021) have a mean value of about 170 DN. Therefore they are matching the old calibration files of April. If you try to use the new MasterDark for calibration of these light frames, the difference (light - MD) becomes negative except for stars and brightest regions of the nebula. This result also suggests that the change of the offset value occurred after July 16, 2021).
If you want to check the compatibility of light frames and dark frames or flat-darks, please use the Statistics process for mean and median values and the histogram (HistogramTransformation) with an appropriate horizontal zoom (a value of about 40 might be right). If the mean value of a dark frame or flat-dark is greater than the mean value of a light frame, they are not compatible.
Bernd
MasterFlatDark (0.21 s)
April: mean: 153.1 DN, median: 152.6 DN
August: mean: 761.5 DN, median: 761.1 DN0
MasterDark (300 s)
April: mean: 154.2 DN, median: 152.9 DN
August: mean: 761.7 DN, median: 760.2 DN
The difference between the calibration frames taken in August and in April is about 608 DN. This compares favorably with the value (605 DN) that I evaluated with PixelMath (post #43).
My reasoning was and is: during capturing images in April, a low offset value was set in the capturing software or in the camera driver. In August, a much higher offset value was used. A difference of 608 DN is like day and night. It is all-important that the light frames and all calibration frames are captured with the same gain and offset value. According to the FITS header, the gain values (EGAIN = 1) are matching. Unfortunately, despite of the update of the SGP version there is no OFFSET keyword in the FITS header. Admittedly it is not only the SGP version that matters, the ASCOM platform version and the camera driver version have to support setting the offset value as well.
Which ASCOM platform version are you using?
Are you using the QHY ASCOM camera driver or native camera driver, and which version?
Your light frames of the Iris Nebula (captured on July 16, 2021) have a mean value of about 170 DN. Therefore they are matching the old calibration files of April. If you try to use the new MasterDark for calibration of these light frames, the difference (light - MD) becomes negative except for stars and brightest regions of the nebula. This result also suggests that the change of the offset value occurred after July 16, 2021).
If you want to check the compatibility of light frames and dark frames or flat-darks, please use the Statistics process for mean and median values and the histogram (HistogramTransformation) with an appropriate horizontal zoom (a value of about 40 might be right). If the mean value of a dark frame or flat-dark is greater than the mean value of a light frame, they are not compatible.
Bernd
dpdallolmo@aol.com
Well-known member
Thank You, Bernd.
I am using the latest ASCOM platform (6.5 SP1). I also Updated SGP ( 4.0.0.680 (64-bit)) and the QHY all in one driver (.21.07.15.14). All were updated in early August 2021 per our discussion, When the camera is connected to SGP and I open the camera settings, the window that opens is ASCOM QHY Camera set up....see attached. This is how I am set. I have no idea why the settings changed in mid July since I didn't intentionally change anything. Any suggestions are appreciated.
Dan
I am using the latest ASCOM platform (6.5 SP1). I also Updated SGP ( 4.0.0.680 (64-bit)) and the QHY all in one driver (.21.07.15.14). All were updated in early August 2021 per our discussion, When the camera is connected to SGP and I open the camera settings, the window that opens is ASCOM QHY Camera set up....see attached. This is how I am set. I have no idea why the settings changed in mid July since I didn't intentionally change anything. Any suggestions are appreciated.
Dan
Attachments
dpdallolmo@aol.com
Well-known member
Bernd,
please ignore the previous screenshot. I have 2 laptops that I use for imaging and I realized the offset settings were different on each laptop. The laptop used to make the new library of darks and flat darks has settings below. I went through ang made sure all my settings (both laptops) match what you see in this attached image.
please ignore the previous screenshot. I have 2 laptops that I use for imaging and I realized the offset settings were different on each laptop. The laptop used to make the new library of darks and flat darks has settings below. I went through ang made sure all my settings (both laptops) match what you see in this attached image.
Attachments
OK. I'm glad that you found the reason why you got light and calibration frames with incompatible offset values. Now you will know how to avoid this complication.
Nevertheless your software setup doesn't seem to allow requesting and setting the offset value. The offset value seems to be defined in the ASCOM camera driver only, and the capturing software cannot even request this value. I know that the current versions of SGP and the ASCOM platform that you mentioned in post #58 can achieve this job when an appropriate ASCOM camera driver is used. It works flawlessly e.g. with cameras of ZWO and an older ASCOM camera driver (v6.5.1.5), and it doesn't work with newer versions of the ASCOM camera driver (v 6.5.1.7 or v6.5.1.8). So it might be worth asking QHY which ASCOM camera driver is suitable.
Bernd
Nevertheless your software setup doesn't seem to allow requesting and setting the offset value. The offset value seems to be defined in the ASCOM camera driver only, and the capturing software cannot even request this value. I know that the current versions of SGP and the ASCOM platform that you mentioned in post #58 can achieve this job when an appropriate ASCOM camera driver is used. It works flawlessly e.g. with cameras of ZWO and an older ASCOM camera driver (v6.5.1.5), and it doesn't work with newer versions of the ASCOM camera driver (v 6.5.1.7 or v6.5.1.8). So it might be worth asking QHY which ASCOM camera driver is suitable.
Bernd