Calibration Conundrum

[jeffweiss9]

I've been shifting over to doing my full calibration in PI following Vicent Peris' "Master Calibration Frames: Acquisition and Processing" tutorial.  However, I've noticed that I'm getting very different Blue Master Flats than I got with my former calibrations done with AIP4WIN.  My blue flats themselves (taken using a pretty good light-box) average 0.4 intensity while the red and green flats average in the 0.7 to 0.8 intensity range.  When I integrate my calibrated flats to make a Master Flat, however, this average 0.4 intensity is preserved with the procedure in the tutorial using multiplicative normalization, while AIP4WIN generates master flats that all have the same average intensity, which is the higher one.   The PI procedure doesn't seem correct to me as 1) I have Astrodon Gen2 Series I RGB filters that are supposed to be used with 1:1:1 weights with my SBIG ST2000XM camera and 2) PI doesn't know if I changed the light box intensity or the exposure time of the flats, but isn't it making an implicit assumption that the source for the flats is constant?  In this case, I had the same intensity and exposure time for LRGB filters but I had to increase both exposure time and light-box intensity setting to make my Halpha flats.  Neither the blue nor the Halpha (average intensity 0.13) are calibrating correctly (totally washed out subs) while the other filters with intensities 0.7 to 0.8 appear to have calibrated successfuly. Suggests something is wrong in my making the master flats for those two filters but what might be wrong here?
 
  Second question: One step in the tutorial that is very unclear is on using ImageCalibrate to calibrate the lights procedure so I may be making the wrong guess there.  Specifically, I have Calibrate unchecked for both the Master Bias and the Master Dark sections (and optimize also unchecked for Master Dark), while I then have the Master Flat section checked with Calibrate unchecked there.  I presume calibrating the lights must use the Master Dark and Master Bias frames, in addition to the Master Flat, but that all three Calibrate boxes and the Optimize box should be UNCHECKED.   Is that correct?
  Thanks very much.
-Jeff

[jeffweiss9]

   PI calibration is simply failing for me on Ha and Blue frames while apparently working reasonably for L,R, and G frames from the same data collection.  The problem appears to have to do with the PI calibration technique
                                  http://pixinsight.com/tutorials/master-frames/en.html
or at least my instantiation of it in the attached psm file, assuming that is correct.
   When I calibrate the same data in AIP4WIN (advanced calibration option) using the same darks, flats, and bias frames (also AIP4WIN uses flat-darks), the Ha and Blue data calibrates successfully.  I'm at a loss as to what is going on because the calibrated frames coming out of PI for these two filters don't seem to display in any reasonable way in either PI or AIP4WIN so they apparently are defective in some way.  I'd love to send one of the large fits files coming out of PI calibration if Vicent or someone at PI or elsewhere wanted to try to look at them.  But I'm unable to get any program to do anything with them so obviously I've managed to find a way to completely break PI.
   Thanks very much in advance.
-Jeff

[vicent_peris]

Hi Jeff,

I'm sorry, I don't understand completely your problems. Please could you upload some light and calibration frames to make some tests?


Best regards,
Vicent.

[jeffweiss9]

Hi, Vicent-
 Thanks very much. I have just used YouSendIt.com to send you two Ha light frames and the corresponding two calibrated frames plus the master flat -- all of which were created using the process icon settings in the psm file attached to my previous post.  Let me know if you need anything else and I'll send it too you.  The data itself I believe is pretty good.  Here's a link to the image I finished with it last night using PI calibration for L,R,G and AIP4WIN calibration for B and Ha:

http://www.astrophotogallery.org/member-galleries/p7052-ngc-2264-cone-nebula.html

I'd love to learn what I'm doing wrong and switch over to just doing them in PI.  A hint here is it is the lower-intensity images from those two filters that were giving the problem where I couldn't find anything wrong with the other calibrated images in data with the other filters.  But that may have nothing to do with it and it's just cockpit error on my part.

Clear skies,
-Jeff

[jeffweiss9]

    Not much luck so far, so I try once again over here in "my own thread" on calibration problems as the problem I am having is different and separate from Marek's since my 16 bit unsigned bias frames don't look abnormal:
    Clicking the "calibrate" checkbox of the enabled dark section of the ImageCalibration tool did enable calibrated lights for my Ha, B subs that are no longer obviously "bad"; i.e, they are no longer gray under STF with only bright stars visible (I believe due to a large part of the range having negative values as in the calibrated lights previously sent).  However, my only basis for saying that they are now "good" is that the registered stacks I made with PI from them for Ha, B and, as a control, remade with the "calibrate" box checked for Lum also, all compare closely to the registered stacks of those channels that I made through AIP4WIN "advanced" calibration (which also uses flat darks) and AIP4WIN registration/stacking. (All relevant data was collected at a camera-controlled -20 deg C).  Due to the sketchiness of the single "calibrate lights" paragraph of the "Master Calibration Frames..." tutorial, there is still much ambiguity about the correct settings in the IC tool for that particular step of the PI calibration process.  Therefore, I am asking for some further elaboration on what those IC tool settings should be.
    Here are my present guesses for the correct settings in the IC tool to do proper calibration of light frames:

Master Bias section checked; calibrate UNchecked; not mentioned in the tutorial
Master Dark section checked; calibrate CHECKED, optimize UNchecked; neither mentioned in the tutorial
Master Flat section checked; calibrate UNchecked, as specified in the tutorial

If I'm understanding the logic inferred from the earlier sections of the tutorial, these settings translate into saying that you are to calibrate the MasterDark by subtracting the MasterBias and then use both the bias, the bias-subtracted MasterDark and the MasterFlat to calibrate the light frames (but not change the already-calibrated MasterFlat).  I'm most uncertain about whether the optimize box should be unchecked or checked during this step.

I would really be delighted if Vicent or anyone could confirm or correct these settings as I believe it is a straightforward question that is the boiled-down essence of my confused earlier attempts to describe the problem above. I'm sure its my fault for not posing the question clearly enough previously and someone can clarify this.

Thanks very much.
-Jeff

P.S. For some of us, a math equation for the calibrated lights in terms of the various inputs might help our understanding but that's just a suggestion, if possible, for some future time.  Right now, I'd just like to get the missing requested information so I can switch over to using PI for calibration of my data.

[vicent_peris]

Master Bias section checked; calibrate UNchecked; not mentioned in the tutorial
Master Dark section checked; calibrate CHECKED, optimize UNchecked; neither mentioned in the tutorial
Master Flat section checked; calibrate UNchecked, as specified in the tutorial

If I'm understanding the logic inferred from the earlier sections of the tutorial, these settings translate into saying that you are to calibrate the MasterDark by subtracting the MasterBias and then use both the bias, the bias-subtracted MasterDark and the MasterFlat to calibrate the light frames (but not change the already-calibrated MasterFlat).  I'm most uncertain about whether the optimize box should be unchecked or checked during this step.

Hi Jeff,

your settings are good. The only point is the dark optimization. It must be activated if your master dark are not at the same temp or doesn't have the same exposure time. If your master dark has the same exp and temp than your light frames, optimization is not mandatory, but I highly recommend it.

So you understood very well my article.  Anyway, I will revise it and will add some screen shots asap. Thank you for your suggestions!


Regards,
Vicent.

[jeffweiss9]

Vicent-
 Thanks so much for the clarification and I will include the optimization as well for nominally same-temperature data. (I suppose I can now just restrict my dark library to just the longest exposure time that I use and take full advantage of the scaling.)  There was a hint that also enabling optimization was correct from overlaying the cursor on the "optimization" line in the IC MasterDark section and, unlike some of the other steps, the little dialog box did not suggest it was to be done only "at the beginning of the calibration process".  But since overlaying the cursor on the "calibration" line of the same section does have the "at the beginning.." statement without mention of the additional invocation at the end, that confused me.
 Hope this helps making things clearer for folks.
Clear skies,
-Jeff

[jeffweiss9]

Vicent-
  I was hoping that closed this subject out but I found unfortunately that checking the optimization box for the calibrating the lights for my data, but keeping everything else unchanged, actually ended up letting through a substantial number of additional hot pixels in the calibrated frames where there were very few previously.  So maybe I have another problem.
-Jeff
 

[vicent_peris]

Vicent-
  I was hoping that closed this subject out but I found unfortunately that checking the optimization box for the calibrating the lights for my data, but keeping everything else unchanged, actually ended up letting through a substantial number of additional hot pixels in the calibrated frames where there were very few previously.  So maybe I have another problem.
-Jeff
 

Hi,

please give me this information:

- Temperature and duration of your light frames.
- Temperature and duration of your dark frames.
- How much dark and bias frames did you acquire to create the masters.
- In the console output of the calibration process, which are the dark scaling factors?
- I would like to see a JPEG of one of your calibrated light frames (stretched, of course). Just to check how serious is the problem of hot pixels.


Regards,
Vicent.

[jeffweiss9]

Hi, Vicent-
 Surely:
- Temperature and duration of your light frames.
          CCD Temp from fits header: -20.0 deg C, 480 sec
- Temperature and duration of your dark frames.
          CCD Temp from fits header: -20.0 deg C, 480 sec
- How much dark and bias frames did you acquire to create the masters.
          10 dark frames and 10 bias frames
- In the console output of the calibration process, which are the dark scaling factors?
          dark scaling factors for the 8 Ha frames:  .084, .091 .111, .074 .063 .094 .064 .090
- I would like to see a JPEG of one of your calibrated light frames (stretched, of course). Just to check how serious is the problem of hot pixels.
          I emailed two messages to your hotmail account each with a yousendit.com link to fits files:
               1. first calibrated Ha light frame with optimization checked
               2. first calibrated Ha light frame with optimization unchecked
Thanks,
-Jeff