Darks and Lights at differenct temperatures

[niteman1946]

New at this, so bear with me.

I'm following the procedure per this thread
 http://pixinsight.com/forum/index.php?topic=2570.msg19019#msg19019


1.  How do I adjust for various temperature darks taken at one session?  Are there special switches in the operators  ----
      a)  For dark calibration?
      b)  For master dark integration?
      c)  For light calibration?

2.  And how are different temperature lights managed?

There's more, but I'll wait for some guidance on this before moving on.

Thanks,

Mark

             

[Philip de Louraille]

Wow, that is a little tougher. Do CCD camera dark noise respond linearly to temperature changes??

[niteman1946]

Sorry Philip.  It's a Canon DSLR.

Am I in the wrong forum?


Mark

[pfile]

no you are in the right forum.

so what you need to do is create your masters by integrating all your bias subs together and all your dark subs together. the dark subexposures should all be the same length (to match your lights) and ideally, at or near the same temperature. as far as i understand it, dark current goes linearly with time, but is nonlinear in temperature.

so when it comes time to calibrate your lights, you want to tick "calibrate" and "optimize" in the master dark area. "calibrate" will subtract the master bias from the dark, and "optimize" will scale the dark in order to minimize total noise in the calibrated light sub.

here's where the problem lies - the scaling is linear. this means that the scaling will do a much better job of matching your (say) 10 minute dark at 10C to a 5 minute light at 10C than it will matching your 10 minute dark at 10C to your 10 minute light at 25C. in my case i've seen my dark get scaled up so much that it "punches" holes where there were hot pixels in the dark.

vincent peris mentioned that they are working on nonlinear scaling for darks in a future version of the ImageCalibration module:

http://pixinsight.com/forum/index.php?topic=2788

[pfile]

i guess i didnt really answer your question, but i'm not sure i know the answer. i don't know if a master dark whose subs vary in temperature is a good idea. not sure it makes sense to try to normalize your darks to compensate for temperature differences because of the nonlinear dependence on temperature.

but i understand where you are coming from - without putting your DSLR in the refrigerator while taking darks, it's hard to get the dark temperatures to match your lights. i have a makeshift cooling setup for my DSLR but it's open-loop and pretty much tracks the air temperature, so i have the same problem.

[niteman1946]

Thanks pfile and Philip.

Actually pfile, you did answer my question on the darks.  Up to some point, I can tick the Master Dark calibrate box in the Light frame calibration operation.  And this should compensate for some temp difference between darks and lights (and lights/lights, and darks/darks ?).  Alternatively, if I followed Jerry Lodribuss' advice, I would take lots of flats at different temps, and match them up to the lights' temp.

The other question was....If I were to take lights (and darks) from multiple sessions, how would I go about integrating all the lights together?  That is, integrating lights that might have significantly different temperatures?

A few initial observations on my trial run of PI.
1.  I am glad there has been chatter about PI on Cloudy Nights.  As you know a major complaint is lack of documentation.  But with some effort, I have been able to latch onto a few (sometimes contradictory) forum threads and Harry's videos to trudge through.  FWIW, I use IRIS and never understood it either (just followed Jim Solomon's cookbook).
2.  PI is a resource hog.  It runs out of memory regularly and requires a computer reboot.  It's also slow compared to IRIS.
3.  My first trial yielded decent results -- NGC2359 and better than the same files processed in IRIS.  But my second capture of the same object yielded good results in IRIS, but purple and blue speckles in PI.  I went back twice to re-process to make sure I didn't mis-switch something in PI.  Third time was the charm -- I ticked the calibrate box in the Master Dark section of Light calibration.
Speckles disappeared!

Mark

[pfile]

Thanks pfile and Philip.

Actually pfile, you did answer my question on the darks.  Up to some point, I can tick the Master Dark calibrate box in the Light frame calibration operation.  And this should compensate for some temp difference between darks and lights (and lights/lights, and darks/darks ?).  Alternatively, if I followed Jerry Lodribuss' advice, I would take lots of flats at different temps, and match them up to the lights' temp.

well the 'calibrate' box just makes the master dark scalable by subtracting the master bias. it's the 'optimize' box that scales the dark to match the noise in the light. but yes, that will compensate for time and temp differences between your master dark and your lights. currently it does a better job compensating for time differences than it does temperature differences.

The other question was....If I were to take lights (and darks) from multiple sessions, how would I go about integrating all the lights together?  That is, integrating lights that might have significantly different temperatures?

i think if your lights have wildy varying temperatures (like more than 5C difference vs. the temperature of your master dark) then you may end up with hot pixels being overcompensated for. probably the most correct thing to do (as you point out above) is to get a dark library going, and bin your lights into 5C groups and calibrate them separately with different master darks that match the light temperatures. having said that, i don't have much of a dark library and only once did i notice the problem mentioned in that thread. if you are dithering your subs those 'punched out' pixels will not be as much of a problem.

5C is just kind of a guess by the way... it probably really depends on your camera. it might require some experimentation.

once all the lights are properly calibrated with their corresponding darks, it should be perfectly fine to just integrate them all together in one go.

A few initial observations on my trial run of PI.
1.  I am glad there has been chatter about PI on Cloudy Nights.  As you know a major complaint is lack of documentation.  But with some effort, I have been able to latch onto a few (sometimes contradictory) forum threads and Harry's videos to trudge through.  FWIW, I use IRIS and never understood it either (just followed Jim Solomon's cookbook).
2.  PI is a resource hog.  It runs out of memory regularly and requires a computer reboot.  It's also slow compared to IRIS.
3.  My first trial yielded decent results -- NGC2359 and better than the same files processed in IRIS.  But my second capture of the same object yielded good results in IRIS, but purple and blue speckles in PI.  I went back twice to re-process to make sure I didn't mis-switch something in PI.  Third time was the charm -- I ticked the calibrate box in the Master Dark section of Light calibration.
Speckles disappeared!

Mark

on #1, all i can say is PI is moving in the right direction.
#2 ... maybe on windows? i'm running macosx and while the 32-bit version occasionally runs out of memory, it's never crashed my computer.
#3 that's good to hear. i've never used anything but deepskystacker or pixinsight to calibrate and stack and PI is definitely much better than DSS at this task, though more difficult to set up. of course, there is this script now:

http://pixinsight.com/forum/index.php?topic=2861.0

[niteman1946]

Thanks pfile for your reponse.  Here are some results from today's effort --

1.  Figured out the logic of multiple sessions.
   a)  Calibrate Lights for each session with their respective darks -- one session at a time.
   b)  Debayer Lights from all sessions at same time.
   c)  Register all Lights at same time.
   d)  Integrate all lights at same time (in theory).

My computer runs out of memory somewhere between 25 and 55 lights.  I wound up breaking the Lights into three groups for initial integration, and saving the three integrated masters.  Then integrated these three with no normalization, nor pixel rejection.  Seemed to work.

Followed Harry's tutorials through Morphilogical Transformation.

When I compare this work to previous IRIS/GIMP processing, I'm disappointed.

It lacks the snap and brightness of the Iris work. I's softer and smokier.

Is there a magic button?

Mark
 

[Enzo De Bernardini]

Hi Mark,

My computer runs out of memory somewhere between 25 and 55 lights.

Try to decrease the "Buffer size (MB)" parameter value from 16 to, say, 8 MB on ImageIntegration process. This can avoid the memory problem.

Greetings,

Enzo.

[Cleon_Wells]

Mark, on the PI crash and reboot issue, try keeping the Task Manager for Windows and the Terminal window for the Mac open in the back ground and end/kill the hung process after it crashes, it always works for me, knock on wood.8>)

Is the Magic Button similar to the Easy Button I’ve never found?   LOL

Have you tried the ATrousWaveletTransform process and maybe a little Curves Transformation  on your Histogram Transformation stretched Master, Harry has a video on ATrousWaveletTransformation process, the multi, 1_2_4_8_16_ect_R  (resolution layer) sharpening or blurring tool,  Noise Reduction, DeRinging , Dynamic Range and the Layer Preview pull down  and … and… the Easy Button ----NOT---….
When I use the ATrous Process with its Real-Time PreView window, I’ll also drag the open Histogram Transformation window to the side with the ATrous Real-Time preview screen selected. The Histogram window allows me to watch the image Histogram change with different ATrous adjustments, who knows maybe I’ll learn something. LOL
Cleon
 

[niteman1946]

Try to decrease the "Buffer size (MB)" parameter value from 16 to, say, 8 MB on ImageIntegration process. This can avoid the memory problem.

Thanks Enzo.

I dropped the buffer size from 16mb to 10mb, and all 55 images integrated.

It did slow the process down by reducing the size of the interations.

Good stuff to know.

Mark
 

[pfile]

Followed Harry's tutorials through Morphilogical Transformation.

When I compare this work to previous IRIS/GIMP processing, I'm disappointed.

It lacks the snap and brightness of the Iris work. I's softer and smokier.

Is there a magic button?

Mark
 

not sure exactly what steps are outlined in harry's tutorial... but the advice upthread to try atrous wavelets to sharpen your image is good advice. i would say that it's a good idea to make a luminance mask for your image by extracting the luminance, then adjusting the histogram of the luminance mask to only leave white pixels over the brightest (and thus highest SNR) portions of your image. i also use atrous to blow away the first 2 or 3 pixel scales from the mask to smooth it out a little. sometimes i'll subtract the star mask from the luminance before smoothing it, or put the star mask over the luminance and use atrous to remove most of the scales for just the stars to get rid of them, before subsequently smoothing the whole luminance mask.

then you can apply some bias to maybe the first 4 layers of your image with the mask protecting the low SNR/dark parts of your image. you can experiment with per-layer noise reduction in the atrous tool also.

[niteman1946]

Thanks Cleon and pfile.

Yes what I need is the Easy Button, but of course.....

What would work great for me is a Brightness and Contrast slider similar to what's in Adobe Elements and Gimp.  Maybe the creators of PI can add that.

The ATrous/Wavelet (etc.,etc.) is a new chapter in the book.  I'll take a hard look at it, and again I appreciate the help.

BTW, my full integration of the 55 lights does seem to respond better to all the massaging it has been getting today.

Mark