Author Topic: Calibration and align batch operations  (Read 135813 times)

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #240 on: 2012 April 06 11:36:29 »
Latest version 0.6 works well.
Problems:
- when selecting the output directory, I get a very long dialog, see screenshot.

The problem is in line 1445 in the GUI section

this.dialog.outputDirEdit.text = engine.outputDirectory = this.text = dir;
should be
this.dialog.outputDirEdit.text = engine.outputDirectory = dir;

Kai




Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7096
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #241 on: 2012 April 06 12:31:36 »
Quote
this.dialog.outputDirEdit.text = engine.outputDirectory = this.text = dir;
should be
this.dialog.outputDirEdit.text = engine.outputDirectory = dir;

Ouch! I'll release a fixed version ASAP. Sorry!

As for including cosmetic correction in the script, I am not sure. How would you implement this feature? What parameters and/or working mode(s) of the CosmeticCorrection tool (or other tools) should be included?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #242 on: 2012 April 06 13:15:02 »
The script was originally written to batch call the powerful IC, II and StarAlignment tools as a simple wrapper to save some time. Image integration of calibrated and aligned light frames is one step where it is necessary to fine tune parameters of the II module and therefore the final integration step is not part of this script. If the CosmeticCorrection tool (unfortunately I have not used it yet) requires fine tuning of a set of parameters, inclusion into this script should also be avoided. If this is an simple automatic task it may be OK.   

Kai


Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #243 on: 2012 April 06 13:23:55 »
I am just looking for a simple way to fix a bad columns and hot pixels by averaging with neigboring pixels. I have not played with the defect map module and cosmetic correction module to know what to suggest.

MaxIM DL uses a dark frames with thresholds to detect hot/cold pixels, clusters and make a map. Also bad columns or allows the user to manually entery coordinates or use a mouse. 

Once a defect map is generated a median of surround pixels works.

A non value as a place holder to mask the data for the integration stack would be even better.

Max

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #244 on: 2012 April 06 13:30:38 »
Once a map is made it require little or no intervention. Settings can be pretty simple to work nicely.

I find Maps can generally be used for a year or more. Most cameras age,  developing new bad pixels over time. It is slow process mediated by cosmic ray hits.

Max


Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1457
    • View Profile
    • http://www.harrysastroshed.com
Re: Calibration and align batch operations
« Reply #245 on: 2012 April 06 14:04:00 »
Hi

A defect map is a must simple to do  :-*  if there is no defect map the script becomes a little usless  :surprised:

Harry
Harry Page

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
    • View Profile
Re: Calibration and align batch operations
« Reply #246 on: 2012 April 06 15:37:04 »
I think a defect map would be sufficient. It needs to be applied to the lights after calibration and before debayer and register, needs to be able to handle CFA images, and should be able to handle multiple neighboring defect pixels. The current DefectMap tool lacks CFA capability, the CosmeticCorrection process has CFA and works nicely using the Auto Detect filter...probably similar to what DSS does (but I could not get MasterDark based defect detection working for some reason).

Defect map handling can be done with the current tools, but since it needs to be done right in the middle of the calibration process, it is not easily integrated into the flow of the current script.

To illustrate the need: I usually need defect maps for my wide fields, because my mount does not do enough automatic dithering (otherwise called periodic error/tracking error  :-[) for short focal lengths. Integration of the calibrated stack clearly shows hot pixels. When I activate rejection to a sufficient degree, I loose SNR. See screenshot of integration of 52 lights, each 120 seconds@85mm:
- left: stacking without rejection and defect map: High SNR, but traces from hot pixels in red and green
Gaussian noise estimates:
?sR = 1.076e-004
?sG = 7.923e-005
?sB = 7.929e-005
Reference SNR increments:
??s0R = 4.9153
??s0G = 6.2299
??s0B = 6.2170
Average SNR increments:
??sR = 4.9063
??sG = 6.3918
??sB = 6.3783

- center: stacking with linear fit rejection (sigma high=3): Lower SNR, no traces. Also relatively slow (2353 sec)
Gaussian noise estimates:
?sR = 1.317e-004
?sG = 8.875e-005
?sB = 8.991e-005
Reference SNR increments:
??s0R = 3.9821
??s0G = 5.5467
??s0B = 5.4732
Average SNR increments:
??sR = 3.9747
??sG = 5.6908
??sB = 5.6152

- right: stacking with defect map, using Cosmetic correction with Autodetect, Hot Sigma 3): Higher SNR, no traces, fast (281 secs)
Gaussian noise estimates:
?sR = 1.076e-004
?sG = 7.923e-005
?sB = 7.929e-005
Reference SNR increments:
??s0R = 4.9153
??s0G = 6.2299
??s0B = 6.2170
Average SNR increments:
??sR = 4.9063
??sG = 6.3918
??sB = 6.3783
Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline troypiggo

  • PixInsight Addict
  • ***
  • Posts: 258
    • View Profile
Re: Calibration and align batch operations
« Reply #247 on: 2012 April 06 20:59:51 »
Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #248 on: 2012 April 07 03:45:53 »
Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.
The script selects the most exposed dark frame when processing a set of light frames with different exposure times.

I think a defect map would be sufficient. It needs to be applied to the lights after calibration and before debayer and register, needs to be able to handle CFA images, and should be able to handle multiple neighboring defect pixels. The current DefectMap tool lacks CFA capability, the CosmeticCorrection process has CFA and works nicely using the Auto Detect filter...probably similar to what DSS does (but I could not get MasterDark based defect detection working for some reason).

Defect map handling can be done with the current tools, but since it needs to be done right in the middle of the calibration process, it is not easily integrated into the flow of the current script.


Including a call to the CosmeticCorrection module with 'Use Auto detect' may be possible. Adding all parameters of this complex module will heavily bloat the script. So it may be better to use the batch script to generate master frames and calibrate the light frames only and thereafter do further optimization manually.

Kai

Kai

Offline troypiggo

  • PixInsight Addict
  • ***
  • Posts: 258
    • View Profile
Re: Calibration and align batch operations
« Reply #249 on: 2012 April 07 04:07:54 »
Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.
The script selects the most exposed dark frame when processing a set of light frames with different exposure times.

I see.  Shouldn't the script separate or group the light frames according to exposures similar to binning or filters?  Then apply the calibration in those groupings rather than assuming all the same?

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1457
    • View Profile
    • http://www.harrysastroshed.com
Re: Calibration and align batch operations
« Reply #250 on: 2012 April 07 04:59:12 »
Hi

Other apps just let you create a master defect map and use this in its calibration , Nothing complicated is required here  :-*

Harry
Harry Page

Offline troypiggo

  • PixInsight Addict
  • ***
  • Posts: 258
    • View Profile
Re: Calibration and align batch operations
« Reply #251 on: 2012 April 07 05:50:17 »
Not sure if you're replying to me, Harry? I thought the BatchPreprocessing script just used scaled darks. I haven't used defect maps to date as far as I'm aware.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
    • View Profile
Re: Calibration and align batch operations
« Reply #252 on: 2012 April 07 10:58:41 »
Including a call to the CosmeticCorrection module with 'Use Auto detect' may be possible. Adding all parameters of this complex module will heavily bloat the script. So it may be better to use the batch script to generate master frames and calibrate the light frames only and thereafter do further optimization manually.
A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #253 on: 2012 April 07 12:52:44 »
A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

So, we need a DefectMap module with CFA capabilities and the defect map is selected by the user?

Kai


Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1457
    • View Profile
    • http://www.harrysastroshed.com
Re: Calibration and align batch operations
« Reply #254 on: 2012 April 07 12:55:49 »
A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

So, we need a DefectMap module with CFA capabilities and the defect map is selected by the user?

Kai

Yes Please

Harry
Harry Page