Batch Preprocessing Question

[f11]

I posted this question previously in the Tutorials section, but got no response - so I figured maybe I posted in the wrong place.  So I'm posting it again here, in somewhat abbreviated form to suit a more general audience:

If you refocus during a session (lets say, every four or six images), and subsequently take a set of flats for each specific focus setting, it's going to get REAL messy doing the calibration inside the batch preprocessing script.  How would you associate flats and lights with the same focus settings out of large set of loaded files?

What do others do to take into account flats taken at different focus settings when it comes to calibration - do you calibrate the lights in separate runs of the preprocessor based on the focus setting? Or load all files and just "X" out the flats and lights for each focus setting?  Or can the preprocessor check the light and flat file names and parse out which flats should be applied to which lights?

If I gotta run each set of lights and flats separately by focus, then fine - but PI seems to be so thoroughly thought out, I figured maybe there's already a built-in solution I just haven't found.  I apologize in advance if this reposting is annoying - next time I'll choose the section more carefully before asking a question.

Thanks for any help!! 
Rod

[sreilly]

IO fail to understand why refocusing would require the need to take new flats. Unless the filters have changed in some way or the optics have, your flats images should be identical throughout the night. I use one set of flats, all focused, for an entire imaging session, in fact for the entire project. I re-use the flats if the optical train is the same as well as the rotator angle.

Have you actually seen a difference from one flat to another from temperature change and the need to refocus? If so, could you post an example. I'm not saying this isn't true but rather that I've never seen this using my equipment.

-Steve

[pfile]

the thing is though, infinity focus is infinity focus. if the tube length changes due to temperature changes, then your images are out of focus until you refocus. in other words, there are a range of focuser positions that yield infinity focus depending on the tube temperature.

in practice unless you have like an f/2 scope, i doubt that the focus changes are going to manifest at all in your flats, as long as you are close to focus. i've done flats the next morning when the tube has definitely warmed up but the focuser position was set for a colder tube - in other words the flats were slightly out of focus - and have seen no problems. i've also left the temperature compensation going so that in theory the flats should have been taken at infinity focus... again no noticeable effect. but that's at f/8 and the critical focus zone is almost 70 steps, where i see a -10 step/degreeC slope. it's very likely that in both cases the scope has been in "perfect" focus.

[mschuster]

Rob,
I use the ImageCalibration process to do something similar. Although I don't have different flats for different focus positions, I do have different flats for different nights, and 4 to 7 nights worth of calibrated lights go into a single integration. I don't use the preprocessing script, I just do multiple runs with ImageCalibration as appropriate, making a master flat for each night and using it to calibrate the lights from the same night. I think you could do the same thing.
Mike

[pfile]

yah, i still do it all by hand mainly because i am used to it and with process icons it's not terribly hard. although lately i'm finding that my flats are "good" for longer amounts of time, and i don't really know why. maybe the new camera is less susceptible to picking up dust, or maybe the ccd - filter distance is larger.

the original question was "how do i account for flats taken at different focus points" and i guess i'm trying to say the question does not really make sense. all frames should be taken at infinity focus, wherever that happens to be based on the tube temperature. and depending on the speed of your telescope, you probably can't tell the difference between a flat taken between two temperature extremes without refocusing.

rob

[f11]

pfile, Steve and Mike... thanks for the input.  Really appreciated!

Just for completeness, my system is a 10" f/8 RCX400 OTA mounted on a Paramount MX in a permanent installation.  Camera is ST2000XM with CFW8 and AO8 and focused using a FeatherTouch system.

It was my understanding that ANY change to the optical system required a new set of flats - which is a MAJOR PITA to accommodate, but I thought it was a necessity.  I wondered why I couldn't find anything on this forum about it, and now I know - it appears I understood incorrectly.

Steve, I took this "requirement" (read elsewhere some time ago) at face value - it makes some sense if focusing causes some rotation of the camera train, or significant distance shift between dust particles and focal plane... my system isn't perfect and despite my best efforts to prevent ANY optical train movement during imaging, if it happens, theoretically it should affect the optics.  If I recall, my critical focus zone is something like 140 microns, and with the temperature swings possible in a few hours imaging where I live (central Alberta Canada), FocusMax can come up with significant moves in focus from one run to another taken an hour apart.  But I've never done an objective test to see if flats taken at significantly different focus settings are actually different.

I have a few nasty dust donuts fairly close to the focal plane (judging by their small size and dark shadows), so flatting makes a difference... which is why I didn't question the advice: I just followed it.  Now that somebody has questioned it, I'll have to go back to determine what the situation REALLY is by deliberately flatting a test image with flats taken at different focus settings.

If I can jettison this need for a deluge of flats every session, it would sure take some of the tediousness (and hard drive consumption) out of my workflow.

What I'm hearing is that folks use the standalone ImageCalibration function (not the batch processing function) to make their masters... I'll try that... the test images I created using the batch processor to "do it all" came out very white clipped and I had the feeling the masters created weren't doing their jobs... probably my fault, but I just loaded files and let 'er rip. 

Rod

[mschuster]

Rod,

You might also check out doing a PixelMath "flatMaster1 - flatMaster2 + 0.5", uncheck "rescale results", apply an STF to the difference and look at the results. There will be changes in intensity (ImageCalibration deals with these overall changes correctly), but any dust spot changes will show up also.

I sometimes set up for a week, don't change filter wheel position (Ha only) and I usually don't mess with the setup. You would think flats would stay the same from night to night. They don't. Dust seems to settle out over the week slowly or something.

Mike

[pfile]

mike - i know. i always stretch the flats and i can see the various dust motes, etc. all i can say is that the lack of new flats was glaringly obvious in my old setup, and in my new setup not so much. i'm just really lazy and don't always take new flats. i was surprised that from the standpoint of a "visual" inspection, i do not see any changes over the course of about a week. in my old setup just a couple of nights would yield wildly different flats with overcorrection, etc. i'm pretty sure this is only because in my DSLR the LP filters are about 2cm from the sensor, and so any dust casts a smaller, darker shadow on the sensor. with the new camera and filter wheel, the filters are probably 4-5cm from the sensor. the donuts get spread out more.

i agree that the right thing to do is take new flats every day, but i'm just lazy sometimes. believe it or not i've even removed and replaced the camera multiple times and the flats still seem to work okay. i'm sure i'd get better results with proper technique though.

anyway WRT flats, the most important things are: flats per-filter and flats per-position angle, and as mike points out, flats per-time depending on how much dust you think is settling on your filters and ccd. the position of the focal plane at infinity is changing with temperature, so as far as i understand it, if you are tracking this temperature change then your flats should match subs taken at the beginning of the night as well as at the end of the night, because the flats also have been taken at the (perhaps new) infinity focus position. i'm assuming here that your focuser is doing temperature compensation. but even if it's not, just on first principles if this were a huge problem i think as you say there would be methods for dealing with it.

i've been using PI for like 3 years and the batch preprocessing script only appeared lately. it does work really well and i have used it but i just find myself reverting to my old ways. i really have to go in and read the code for the script to understand, for instance, how it calibrates the flats. i know the script includes some logic for seeking out darks that approximately match the frame being calibrated. but since flats are relatively short, i think a lot of people calibrate their flats with a master bias only. it's this uncertainty that keeps me from using the script more often. if i do it myself, i know exactly what happened.

[f11]

Mike,

I followed your suggestion of using PixelMath to do subtract operations on four PI-generated flat-masters that varied in their focus settings and the day they were taken, corresponding to four different sets of lights with those focus settings and dates,  per your equation - covering all  6 combinations (assuming a-b = b-a).  Then I applied the STF Autostretch to each, and then used Statistics to give me some data on each.

First, as you guys suggested, NO dust donuts were visible on any of the six.  Thats good news! 

Here's the stats from each of the six S-F's after the STF Autostretch application to each...

Code: [Select]

S-F  Mean    Med   AvgDev  StdDev  Var    Min    MinPos    Max    MaxPos
1-2  32684  32684  42.358  53.103  2820  32351  0279,483  32950  1335,0535
1-3  32385  32385  42.496  53.271  2838  31958  0279,483  32631  0556,0670
1-4  31767  31767  42.940  53.861  2901  31422  0279,483  32028  0394,0755
2-3  32469  32469  42.496  53.286  2839  32077  0276,311  32767  1165,1023
2-4  31851  31851  42.943  53.829  2898  31590  1130,677  32109  1509,0791
3-4  32149  32149  42.995  53.894  2905  31880  1509,451  32421  1415,0854

Am I correct in concluding that  the above S-F's are pretty much identical for the purposes of flatting any of the associated lights, without regard to the focus settings they were taken at?  IOW: you guys were exactly right - creating flats for each focus position is a waste of time and disk space.

Thanks for setting me straight!
Rod