PixInsight versus AstroArt for Preprocessing

[rodmichael]

I'm just learning and just getting started (at milepost zero) with AP and PI.  Please have patience, especially if I have used some of the jargon incorrectly.

In the IP4AP Tutorials Warren Keller presents two different ways to accomplish Dark frame reduction:
1.  With AA5 he seems to suggest a preference for using a Master Bias in combination with a Defect Map as a substitute for Dark frame reduction.
2.  With PI he sticks with Dark frame reduction and uses cosmetic correction (I think PI's version of a Defect Map) optionally.

The AA6 methodology (including making a Defect Map) seems much more straight-forward and easily accomplished than the PI methodology for Dark frame reduction.  In PI creating a Defect Map seems much more complex and I'm still not sure I understand it.  I like the idea of using a Master Bias frame (obtained at a camera temperature appropriate for the lights) as opposed to having to have a library of Master Darks at different temperatures and exposures.  Although, I guess I can have a library of Master Darks all at a single exposure (longer than my longest light exposure) obtained at multiple camera temperatures.  But is there a point to that if I can use very short bias exposures to obtain 40 or 50 bias frames at my desired temperature to make a Master Bias and then combine that with a single Defect Map frame obtained from a single light frame?

If I have this all wrong in concept, don't hesitate to tell me so.

[Duncan]

There's many different ways that you can do calibration in PixInsight, and many parameters that you can tune to improve the result. As inputs to the process, I use both bias frames and dark frames, and I don't consider either to be remotely optional. I also noticed that you didn't mention flats. Are you capturing flats also?

I guess you could say that cosmetic correction is PixInsight's version of a defect map; it does include a defect map function. It's actually much more sophisticated than that, though, and incorporates multiple different correction methods that can be used independently or all at the same time.

I have plenty of questions of my own about calibration, btw, and am preparing a post about that. I do have a calibration process flow that seems to work very well, though, and I can at least explain some of the options that you have.

To start with, though, it would probably help to understand what equipment you have. What camera and optics are you using?

[akulapanam]

It completely depends on what type of camera you have. Generally CCD Sony chips don't need dark frames and can actually add noise whereas KAF chips and CMOS chips do need dark frames.

I suggest you give this a read. http://blog.astrofotky.cz/pavelpech/?p=1252 that was presented at CEDiC last year.

[rodmichael]

There's many different ways that you can do calibration in PixInsight, and many parameters that you can tune to improve the result. As inputs to the process, I use both bias frames and dark frames, and I don't consider either to be remotely optional. I also noticed that you didn't mention flats. Are you capturing flats also?

I guess you could say that cosmetic correction is PixInsight's version of a defect map; it does include a defect map function. It's actually much more sophisticated than that, though, and incorporates multiple different correction methods that can be used independently or all at the same time.

I have plenty of questions of my own about calibration, btw, and am preparing a post about that. I do have a calibration process flow that seems to work very well, though, and I can at least explain some of the options that you have.

To start with, though, it would probably help to understand what equipment you have. What camera and optics are you using?

BACKGROUND.  At this point this is all an intellectual exercise for me.  I have owned a C8 with an equatorial wedge since 1979 with which I did off and on observing and made a few meager attempts at piggy-back AP over the years as I worked through my career.  I retired 5-1/2 years ago and I am 70+ now.  4-months ago I decided that AP will be a principal retirement hobby for me.  I have been on an acquisition binge since then.

It's winter and my P.O.D. and permanent pier are on the way from Wayne Parker and not actually in place yet.  I plan to be operational by April after pouring some concrete and assembling everything.  In the mean time I am trying to learn AP, primarily with an Ip4AP subscription and "dry-labbing."  My SW selections are still in a bit of flux, but currently I think I'll primarily use SharpCap for Polar Alignment (drift alignment seems too problematic on which to spend a lot of time), Stellarium as my planetarium, SGP and MetaGuide for image acquisition, AA6 for preprocessing, and PixInsight for postprocessing.  On the side (in "reserve") I have BackYardEOS, Nebulosity, PHD2, and DSS.  But my current learning focus is the other SW packages.

TO ANSWER YOUR EQUIPMENT QUESTIONS:  I have a Celestron 11" RASA with which I will be using primarily a QSI 683WS (KAF 8300 sensor) with a 5-position FW (Astrodon 1.25" LRGB and SHO filters).  I have a Celestron CGX mount; an Optec QuickSync motorized focuser; an Orion 80mm Short-Tube OTA and a LodeStar X2 for autoguiding; and a TelRad finder.  I will likely do both LRGB and NB photography, eventually.  I will also see what I can do with a modified Canon EOS 80D as a OSC camera, perhaps that as a first part of actually doing some real AP and photo processing.  All the above components are currently sitting in one of our guest BRs gathering dust.  DSOs will likely be my initial principal targets (my equipment selections now tend to dictate that).

I have devised a t-shirt flat device for the RASA and I plan to do flats concurrent with image acquisition, probably in the early evening before or in the morning after image acquisition against the sky opposite the setting or rising sun.  Per my initial post, my question is whether or not I should develop a Master Dark frame library (Darks taken at an exposure greater than my longest light exposure at a variety of camera temperatures) that can be scaled using a Master Bias (a second reduction library).  Or can I develop only a Master Bias library of Bias frames taken with very short exposures at the same variety of camera temperatures that (I think Warren Keller says in his IP4AP "Soup to Nuts" tutorials on AstroArt [SN6 and SN11]) I can then use with a Defect Map (as the term is used in AA6) in place of dark frames for both Light frame and Flat frame adjustment/calibration.  The Master Bias frame library would be substantially less time consuming to develop than would the Master Dark library.

Does that clarify my question and situation sufficiently?  Thanks much for taking the time!

[chris.bailey]

My experience of a KAF8300 chip (~500 hours) is that darks are beneficial. Scaling darks does not always adequately deal with hot pixels (and the 8300 will have lots). Now you can either build a library of matching darks (duration and temperature) and not scale or scale them and then deal with hot pixels with a defect map or Cosmetic Correction (I always use the latter). The KAF8300 likes a cold set point, I always run mine at -20C so you only need a dark library by duration (I use 300, 600 and 1200). You may also find you have a partial or full column defect (mine now has one full and one partial) and Cosmetic Correction repairs that too. I also find big benefit in having a dither of several pixels (I use 5). In respect of Bias I have not found it to vary significantly with temperature if at all so a single master bias will cover all images.

Chris

[rodmichael]

My experience of a KAF8300 chip (~500 hours) is that darks are beneficial. Scaling darks does not always adequately deal with hot pixels (and the 8300 will have lots). Now you can either build a library of matching darks (duration and temperature) and not scale or scale them and then deal with hot pixels with a defect map or Cosmetic Correction (I always use the latter). The KAF8300 likes a cold set point, I always run mine at -20C so you only need a dark library by duration (I use 300, 600 and 1200). You may also find you have a partial or full column defect (mine now has one full and one partial) and Cosmetic Correction repairs that too. I also find big benefit in having a dither of several pixels (I use 5). In respect of Bias I have not found it to vary significantly with temperature if at all so a single master bias will cover all images.

Chris

Thanks for all the information and insights.

However, I'm still wondering about Warren Keller's suggestion that a Master Bias + a Defect Map (as AstroArt defines the term) can substitute for Dark frame reduction.  In his tutorials on IP4AP, alongside presenting a lot of PI methodology, Warren Keller discusses AstroArt 5 preprocessing (Soup to Nuts #6 and #11)and says the following:

"Before putting time into making many Dark frames, bear in mind that most artifacts, except for hot pixels, are contained in the bias [frames].  Consider Master Bias subtraction and the averaged subtraction of a defect map in lieu of Darks."

This makes a lot of sense to me and it is what prompts my question in a user forum where I see little mention of this methodology.  BTW, AA also offers capability to deal with column and row defects.

I'm still wondering if any of the practitioners on this forum have tried this methodology and what recommendations you may have.

Of course, it may be that one of my first projects as a budding astrophotographer will be to give this methodology a try for myself, even perhaps comparing it to dark frame reduction and "Cosmetic Correction" if I can develop a suficient understanding of that preprocessing step in PI.

[Duncan]

Thanks for all the information and insights.

However, I'm still wondering about Warren Keller's suggestion that a Master Bias + a Defect Map (as AstroArt defines the term) can substitute for Dark frame reduction.  In his tutorials on IP4AP, alongside presenting a lot of PI methodology, Warren Keller discusses AstroArt 5 preprocessing (Soup to Nuts #6 and #11)and says the following:

"Before putting time into making many Dark frames, bear in mind that most artifacts, except for hot pixels, are contained in the bias [frames].  Consider Master Bias subtraction and the averaged subtraction of a defect map in lieu of Darks."

This makes a lot of sense to me and it is what prompts my question in a user forum where I see little mention of this methodology.  BTW, AA also offers capability to deal with column and row defects.

I'm still wondering if any of the practitioners on this forum have tried this methodology and what recommendations you may have.

I think many (most) people here will have tried all combinations of the above, plus several others, at one time or another. Personally, given the time taken to acquire good image data, I wouldn't even think about dropping dark frames from my processing just to speed things up a little...

Right now I'm going back over some old data (right now some stuff from 2014) with the archived calibration files, and fine tuning the calibration process just to get a slightly cleaner starting point. I have bias, dark and flat frames, and then I run a cosmetic correction process, taking into account the column defects that I can see in contemporaneous dark frames.

Generating a decent bias and dark frame set really doesn't take much time. I refresh my bias and dark frames about every 3-6 months (the camera chip ages noticeably on that sort of timescale). It takes a single overnight run to capture a new set.


For your setup (very nice!) the QSI camera has a thermoelectric cooler with good regulation (I have the older QSI583 myself, which has the same KAF-8300 image sensor). I would strongly recommend finding a temperature that you can comfortably hit under all reasonable conditions and standardize on that. That way you won't have to worry about creating a calibration library with multiple different temperatures. I use -10C. Chris above recommends -20C. Depending on your local temperature conditions, you may struggle to hit -20C with the cooler in that camera (just based on my own experience with the 583). You should experiment a little with this; run the camera indoors at room temperature, and see if you can get to -15C or -20C without the cooler running at 100% power.

You can always start with a standard set point of -10, and then go colder later on. Realistically, there are a lot of things you are going to have to get ironed out before dark current becomes your biggest problem...

Most importantly: start taking images as soon as you can. You're going to find there's a lot of bugs in your setup that you have to work out, and a lot of procedures that you have to master, one at a time. The sooner you get outside and start switching things on and finding what works the better.

Good luck!

[rodmichael]

...
I think many (most) people here will have tried all combinations of the above, plus several others, at one time or another. Personally, given the time taken to acquire good image data, I wouldn't even think about dropping dark frames from my processing just to speed things up a little...

Right now I'm going back over some old data (right now some stuff from 2014) with the archived calibration files, and fine tuning the calibration process just to get a slightly cleaner starting point. I have bias, dark and flat frames, and then I run a cosmetic correction process, taking into account the column defects that I can see in contemporaneous dark frames.

Generating a decent bias and dark frame set really doesn't take much time. I refresh my bias and dark frames about every 3-6 months (the camera chip ages noticeably on that sort of timescale). It takes a single overnight run to capture a new set.


For your setup (very nice!) the QSI camera has a thermoelectric cooler with good regulation (I have the older QSI583 myself, which has the same KAF-8300 image sensor). I would strongly recommend finding a temperature that you can comfortably hit under all reasonable conditions and standardize on that. That way you won't have to worry about creating a calibration library with multiple different temperatures. I use -10C. Chris above recommends -20C. Depending on your local temperature conditions, you may struggle to hit -20C with the cooler in that camera (just based on my own experience with the 583). You should experiment a little with this; run the camera indoors at room temperature, and see if you can get to -15C or -20C without the cooler running at 100% power.

You can always start with a standard set point of -10, and then go colder later on. Realistically, there are a lot of things you are going to have to get ironed out before dark current becomes your biggest problem...

Most importantly: start taking images as soon as you can. You're going to find there's a lot of bugs in your setup that you have to work out, and a lot of procedures that you have to master, one at a time. The sooner you get outside and start switching things on and finding what works the better.

Good luck!

Thanks, Duncan.  You're the guy new to PI but who's been imaging for the past 12 or 13 years, correct?  I appreciate your thoughts and advice.  Will keep you all apprised of what's going when I can get up and going.

I do have one more question that I think you can answer:  When we speak about camera temperature do we speak about its absolute temperature, i.e., ambient temperature minus the relative cooling, or are we talking about only the relative cooling in degrees?  It would seem to me that we have to use absolute temperature and not just relative cooling.  If that's true, then one would still have to make Master Darks that are both temperature and exposure specific.

[Duncan]

Yes, I'm (fairly) new to PixInsight, by which I mean I've been using it for about a year, but I've been imaging for about 13 years.

For the temperature, yes, what matters is the absolute temperature.

You won't need to make calibration sets at multiple temperatures as long as you choose a temperature within the capability of the cooling system under all reasonable conditions.

The QSI cameras (like most astro-imaging cameras) has regulated cooling, meaning you set the temperature you want and the software in the camera runs a control loop that maintains exactly that temperature. So when you set the temp to -10C, it will cool it to exactly -10.0 C and hold it there indefinitely. The only exception would be if the ambient is so high that the cooler is no longer able to maintain the necessary differential. Unless you're in Arizona that's not likely to be a problem. Hitting -15 is reasonable for that camera most of the time. Hitting -20 may be a stretch of the cooler's ability, depending on your typical ambient.

[rodmichael]

Yes, I'm (fairly) new to PixInsight, by which I mean I've been using it for about a year, but I've been imaging for about 13 years.

For the temperature, yes, what matters is the absolute temperature.

You won't need to make calibration sets at multiple temperatures as long as you choose a temperature within the capability of the cooling system under all reasonable conditions.

The QSI cameras (like most astro-imaging cameras) has regulated cooling, meaning you set the temperature you want and the software in the camera runs a control loop that maintains exactly that temperature. So when you set the temp to -10C, it will cool it to exactly -10.0 C and hold it there indefinitely. The only exception would be if the ambient is so high that the cooler is no longer able to maintain the necessary differential. Unless you're in Arizona that's not likely to be a problem. Hitting -15 is reasonable for that camera most of the time. Hitting -20 may be a stretch of the cooler's ability, depending on your typical ambient.

Excuse me for being dense.  So if I understand you correctly, the temperature at which one sets the cooling mechanism is the absolute temperature of the CCD sensor in operation.

On the QSI website the specs for the QSI 683 say the following:
1.  Thermoelectric CCD Cooling:  Temperature regulation +/- 0.1°C, @ 0°C  to -40°C CCD temperature
2.  In free air, Fans @ Full Speed:  Typically 45°C below ambient air temperature with 85% cooling power

So I'm still confused (and/or dense).  Is it a cooling device relative to ambient temperature or to an absolute temperature at the chip?

[Duncan]

The set point is absolute.

When it says cooling to -40 degrees C CCD temperature, that's the lowest set point it will allow you to program (that's actually already outside the absolute max ratings for the KAF-8300 per the TrueSense datasheet).

When it says 45 degrees difference to ambient typical, that's the maximum differential is says it can maintain. The QSI600 series may have better cooling capability than my 583 - I've never been able to get that sort of differential.

Most importantly, that first spec line says:

Temperature regulation +/- 0.1°C

that's the accuracy of the control  loop. The other parameters are telling you the range of temperatures that it can be set to, subject to the ambient conditions. The regulation is telling you how accurately it can maintain the temperature you set. So if you set -14.3C, provided that value is within the range of it's capabilities, the control system will maintain exactly that within +/-0.1C.

[rodmichael]

The set point is absolute.

When it says cooling to -40 degrees C CCD temperature, that's the lowest set point it will allow you to program (that's actually already outside the absolute max ratings for the KAF-8300 per the TrueSense datasheet).

When it says 45 degrees difference to ambient typical, that's the maximum differential is says it can maintain. The QSI600 series may have better cooling capability than my 583 - I've never been able to get that sort of differential.

Most importantly, that first spec line says:

Temperature regulation +/- 0.1°C

that's the accuracy of the control  loop. The other parameters are telling you the range of temperatures that it can be set to, subject to the ambient conditions. The regulation is telling you how accurately it can maintain the temperature you set. So if you set -14.3C, provided that value is within the range of it's capabilities, the control system will maintain exactly that within +/-0.1C.

Thanks for sticking with me, Duncan.  I have made up a spreadsheet to help select a camera cooling setpoint based on average or usual nighttime temperatures.  Does it look OK to you?  For instance, where I am in South Dakota, I would probably select a cooling setpoint of -15C or -20C.  Historically, our nighttime temperatures are between -4C and 16C (23F and 58F) between April and November.  So a -10C or -15C setpoint would bracket those averages with a pretty good buffer.

What do you think?

[rodmichael]

Oops.  There's an error in the first spreadsheet.  The second line of Ambient Temperatures should be Degrees F not C.

[Duncan]

Looks about right...

If the camera can really maintain a 45 degree C differential from ambient, then you should have no difficulty hitting -15 or -20 unless you're in Arizona in the summer. There's no substitute for actually firing it up and seeing how it does, though. I'd suggest plugging it in indoors at room temperature and see how hard it has to work getting down to -20. If you can to that under room temperature conditions, you're good to go.

[chris.bailey]

"Before putting time into making many Dark frames, bear in mind that most artifacts, except for hot pixels, are contained in the bias [frames].  Consider Master Bias subtraction and the averaged subtraction of a defect map in lieu of Darks."

This makes a lot of sense to me and it is what prompts my question in a user forum where I see little mention of this methodology.  BTW, AA also offers capability to deal with column and row defects.

I'm not too sure I agree with Warrens statement on a global basis. Having done quite a lot of testing, My KAF8300 chip builds significant dark signal over my 'normal' imaging times. I have another Kodak chipped camera that shows similar results. Both are significantly 'cleaner' at -20c compared with -10c. I do mostly Narrowband imaging so sub-exposures of 20 or 30 minutes are not uncommon. Whats more that dark signal is not even across the chip from top to bottom or around the edges and that 'gradient' becomes more pronounced with time. Bias subtraction will do nothing to deal with this. Sure you could try and deal with this later on with DBE or the like but I am of the mind it is better to deal with unwanted 'stuff' at source.

Now with my two Sony chipped cameras, the dark current is insignificant (subtracting a Master Bias from a Master Dark shows pretty much zero value results), even over 30 minutes, and shows little variation. For these CCD's, I rarely use darks at all and largely rely on the dither to remove errant pixels in the stack i.e. I don't even bother with a defect map or cosmetic correction. A lot of my target objects are on the faint to very faint end of the spectrum. A robust calibration workflow is essential to having any hope at all of teasing what little signal there is above background levels. Bright objects are a lot more forgiving of workflow.

I suspect many of us do things slightly differently and one of the beauties of PI is that is flexible enough to allow for these subtle variations.