Calibration and align batch operations

[kwiechen]

Latest version 0.6 works well.
Problems:
- when selecting the output directory, I get a very long dialog, see screenshot.

The problem is in line 1445 in the GUI section

this.dialog.outputDirEdit.text = engine.outputDirectory = this.text = dir;
should be
this.dialog.outputDirEdit.text = engine.outputDirectory = dir;

Kai

[Juan Conejero]

this.dialog.outputDirEdit.text = engine.outputDirectory = this.text = dir;
should be
this.dialog.outputDirEdit.text = engine.outputDirectory = dir;

Ouch! I'll release a fixed version ASAP. Sorry!

As for including cosmetic correction in the script, I am not sure. How would you implement this feature? What parameters and/or working mode(s) of the CosmeticCorrection tool (or other tools) should be included?

[kwiechen]

The script was originally written to batch call the powerful IC, II and StarAlignment tools as a simple wrapper to save some time. Image integration of calibrated and aligned light frames is one step where it is necessary to fine tune parameters of the II module and therefore the final integration step is not part of this script. If the CosmeticCorrection tool (unfortunately I have not used it yet) requires fine tuning of a set of parameters, inclusion into this script should also be avoided. If this is an simple automatic task it may be OK.   

Kai

[mmirot]

I am just looking for a simple way to fix a bad columns and hot pixels by averaging with neigboring pixels. I have not played with the defect map module and cosmetic correction module to know what to suggest.

MaxIM DL uses a dark frames with thresholds to detect hot/cold pixels, clusters and make a map. Also bad columns or allows the user to manually entery coordinates or use a mouse. 

Once a defect map is generated a median of surround pixels works.

A non value as a place holder to mask the data for the integration stack would be even better.

Max

[mmirot]

Once a map is made it require little or no intervention. Settings can be pretty simple to work nicely.

I find Maps can generally be used for a year or more. Most cameras age,  developing new bad pixels over time. It is slow process mediated by cosmic ray hits.

Max

[Harry page]

Hi

A defect map is a must simple to do    if there is no defect map the script becomes a little usless 

Harry

[georg.viehoever]

I think a defect map would be sufficient. It needs to be applied to the lights after calibration and before debayer and register, needs to be able to handle CFA images, and should be able to handle multiple neighboring defect pixels. The current DefectMap tool lacks CFA capability, the CosmeticCorrection process has CFA and works nicely using the Auto Detect filter...probably similar to what DSS does (but I could not get MasterDark based defect detection working for some reason).

Defect map handling can be done with the current tools, but since it needs to be done right in the middle of the calibration process, it is not easily integrated into the flow of the current script.

To illustrate the need: I usually need defect maps for my wide fields, because my mount does not do enough automatic dithering (otherwise called periodic error/tracking error  ) for short focal lengths. Integration of the calibrated stack clearly shows hot pixels. When I activate rejection to a sufficient degree, I loose SNR. See screenshot of integration of 52 lights, each 120 seconds@85mm:
- left: stacking without rejection and defect map: High SNR, but traces from hot pixels in red and green
Gaussian noise estimates:
?sR = 1.076e-004
?sG = 7.923e-005
?sB = 7.929e-005
Reference SNR increments:
??s0R = 4.9153
??s0G = 6.2299
??s0B = 6.2170
Average SNR increments:
??sR = 4.9063
??sG = 6.3918
??sB = 6.3783

- center: stacking with linear fit rejection (sigma high=3): Lower SNR, no traces. Also relatively slow (2353 sec)
Gaussian noise estimates:
?sR = 1.317e-004
?sG = 8.875e-005
?sB = 8.991e-005
Reference SNR increments:
??s0R = 3.9821
??s0G = 5.5467
??s0B = 5.4732
Average SNR increments:
??sR = 3.9747
??sG = 5.6908
??sB = 5.6152

- right: stacking with defect map, using Cosmetic correction with Autodetect, Hot Sigma 3): Higher SNR, no traces, fast (281 secs)
Gaussian noise estimates:
?sR = 1.076e-004
?sG = 7.923e-005
?sB = 7.929e-005
Reference SNR increments:
??s0R = 4.9153
??s0G = 6.2299
??s0B = 6.2170
Average SNR increments:
??sR = 4.9063
??sG = 6.3918
??sB = 6.3783
Georg

[troypiggo]

Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.

[kwiechen]

Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.

The script selects the most exposed dark frame when processing a set of light frames with different exposure times.

I think a defect map would be sufficient. It needs to be applied to the lights after calibration and before debayer and register, needs to be able to handle CFA images, and should be able to handle multiple neighboring defect pixels. The current DefectMap tool lacks CFA capability, the CosmeticCorrection process has CFA and works nicely using the Auto Detect filter...probably similar to what DSS does (but I could not get MasterDark based defect detection working for some reason).

Defect map handling can be done with the current tools, but since it needs to be done right in the middle of the calibration process, it is not easily integrated into the flow of the current script.

Including a call to the CosmeticCorrection module with 'Use Auto detect' may be possible. Adding all parameters of this complex module will heavily bloat the script. So it may be better to use the batch script to generate master frames and calibrate the light frames only and thereafter do further optimization manually.

Kai

Kai

[troypiggo]

Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.

The script selects the most exposed dark frame when processing a set of light frames with different exposure times.

I see.  Shouldn't the script separate or group the light frames according to exposures similar to binning or filters?  Then apply the calibration in those groupings rather than assuming all the same?

[Harry page]

Hi

Other apps just let you create a master defect map and use this in its calibration , Nothing complicated is required here 

Harry

[troypiggo]

Not sure if you're replying to me, Harry? I thought the BatchPreprocessing script just used scaled darks. I haven't used defect maps to date as far as I'm aware.

[georg.viehoever]

Including a call to the CosmeticCorrection module with 'Use Auto detect' may be possible. Adding all parameters of this complex module will heavily bloat the script. So it may be better to use the batch script to generate master frames and calibrate the light frames only and thereafter do further optimization manually.

A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

[kwiechen]

A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

So, we need a DefectMap module with CFA capabilities and the defect map is selected by the user?

Kai

[Harry page]

A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

So, we need a DefectMap module with CFA capabilities and the defect map is selected by the user?

Kai

Yes Please

Harry