Calibration and align batch operations

[kwiechen]

am i correct in assuming that this script will not work for a DSLR user since it relies on some set of FITS keywords? thanks.

The current version can group and calibrate FITS files from a monochrome CCD using sets of FITS keywords to group files. If I can can get test data and information about differences
in the workflow it should be easily possible to use FITS from color OSC. It my be in addition possible to use DSLR data, if the ImageCalibration/ImageIntegration tools can import the
raw files. But in order to proceed I need some information about workflow differences.

Best

Kai

[kwiechen]

0.0.7g should have removed the bug when clearing the light frame list.

@Max: to use IRAF designation of light frames you may try to comment out line 3 in CalibrateAlign-def and uncomment line 6. But it is not possible to mix files with different IMAGETYP designations. If you have for example light frames with wrong IMAGETYP values you can try the BatchConvertImagetype script posted already some time ago (use with backup data!!!). Rotator support: do you include the correct pierside values to the flat files?

@Rob: in order to modify bufferSize and/or stackSize the vars are written to lines 305/306 in CalibrateAlign-engine. Try to reduce the values and if this can solve the problem with ImageIntegration crashes on PC's with low RAM, we can expose these vars to the user interface.

Kai

[mmirot]

0.0.7g should have removed the bug when clearing the light frame list.

@Max: to use IRAF designation of light frames you may try to comment out line 3 in CalibrateAlign-def and uncomment line 6. But it is not possible to mix files with different IMAGETYP designations. If you have for example light frames with wrong IMAGETYP values you can try the BatchConvertImagetype script posted already some time ago (use with backup data!!!). Rotator support: do you include the correct pierside values to the flat files?


Kai

The image type designations is now under control not that I know not to use IRAF or Mira key words.
I used the batch script to fix these. It might be nice to widen its usage by allowing editing of additonal keywords ie pierside, filter, bin.


Yes, we are only trying to match flats to lights.

The  pierside key word will work nicely for rotators.
For MaxIm Dl you just have to the scope setting on german equitorial and the Maxim DL observatory connected to the mount.
It generates a Pierside : ' East ' or ' West ' entry.

I was a little confused at first because it based on the which side the OTA is located on rather than where the scope is pointing in the sky.  There is a option in MaxIM dl to switch it to enter the scope pointing. However,  as long as the pierside entry consistant the convention does not matter.
If your not using Maxim the adding a pierside key word through a batch to only one side of the merdian should work for matching porposes.

I will post a fits header with the pierside format or send you a few images.

Max

[pfile]

am i correct in assuming that this script will not work for a DSLR user since it relies on some set of FITS keywords? thanks.

The current version can group and calibrate FITS files from a monochrome CCD using sets of FITS keywords to group files. If I can can get test data and information about differences
in the workflow it should be easily possible to use FITS from color OSC. It my be in addition possible to use DSLR data, if the ImageCalibration/ImageIntegration tools can import the
raw files. But in order to proceed I need some information about workflow differences.

Best

Kai

IC can definitely handle CR2 files directly. II can not, you have to convert the CR2s for your bias/darks to .fits manually first. maybe you can run IC with no calibration masters and that would convert the CR2 to fits, but i've never tried that and suspect it would not work.

BackyardEOS can write EXIF data to your file indicating what it is (light, dark, bias) and it's duration, etc. BYE can also save your files with descriptive filenames (frame type, ISO, duration, etc) but none of these things are going to show up in the FITS header as far as i know. IC would have to try to carry this info into the FITS, or the fits would have to be post-processed to create keywords from the filenames. i think BYE is going to support writing FITS soon and it's likely that the same data written to the EXIF would show up as FITS keywords.

but even if it can write fits i'd probably stick with CR2 since i prefer to have what the camera gave me rather than something that's been modified...

[georg.viehoever]

...raw files. But in order to proceed I need some information about workflow differences....

The DSLR workflow is described in http://pixinsight.com/forum/index.php?topic=2570.msg19019#msg19019 . The .CR2 files dont contain much useful information, except maybe the exposure time. The way I usually work is that I rename the image files such that they are named light*.CR2, bias*.CR2, dark*.CR2, flat*.CR2. I then follow the workflow linked above.

Georg

[mmirot]

Kai some thoughts...

We should have a check box to calbrate only. This is especially helpful when testing the calibration functions.

An option to ignor Fits Key words all together is a good idea too.  I use MaxIM and want the automatching and checking.
There will be others that don't have Maxim Dl that will be happy to do one image set at time putting in the correct/ matching bias,  dark exposure and flat.   It will be easier than running the batch file to add key words.  ( Not that I am suggest we don't have this too)

I would be very nice to deselect or remove selected images from the lists.

Have you though about adding in integration? It could be nicely if you have the option to pause after registration than select a ROI to tweek it.

Max

[kwiechen]

Max,

1) do you think about 'calibrate only and skip registration'?

2) We need some kind of metadata to group files. It may be possible to rewrite this to be able to manually add this data (for example frame/image type and/or exposure time) and to bypass evaluation of the FITS header. I am thinking about this to support for example DSLR data.

3) The internal data structure to select/deselect images is in place. I have not put this to the user interface because of problems with interaction (modal dialog)

4) Light frames need some tweaking of integration parameters to get the best SNR and pixel rejection possible (http://pixinsight.com/forum/index.php?topic=3736.0)

Kai

[mmirot]

1) Calibrate only is great.

2) Ideally we want meta data from the fits to match.  It allows for all your cool features,
However, I was wondering if there was an additional way to allow for data to be processed when the proper FITs key words are not present.


3) good

4) You mean integration needs tweaking. I agree.   What I suggest is the script can run through with the initial parameters.
The script can then rerun at the integration step using different settings as need.

 When I use the integration module most of the time I am pretty good close to my initial settings.

However, if want the best combination of SNR and rejection I choose a smaller region of interest and do some iterations. This saves a lot of time because of smaller area and all the data is in memory too.
 If this script has persistence and the data is still in memory then I think it could iterate just like the module.  This would Rock !


Btw is this the right link? The link does not discuss this.

Thanks

Max
 

[johnrt]

This script looks great, but will not work for me. I cannot add any files to the script window. I navigate to the files, bias, dark light etc etc, select them and click open and nothing gets added to the files box in the script window.

I'm using a Mac OSX 10.0.72 32bit version.

[mmirot]

We can help with that.
It is a the key word recognition problem I was discussing above. 
There is a typebatch script to add the fits words for flat , dark, bias, light. 
Just select the files and add the appropriate file type.
Note:  recommend you make copies to all file first.
 It makes changes to the header entries and there is no undo.

Btw, What software did use to acquire them?

[johnrt]

They were all captured using Artemis Capture - with an Atik 314+.

We can help with that.
It is a the key word recognition problem I was discussing above. 
There is a typebatch script to add the fits words for flat , dark, bias, light. 
Just select the files and add the appropriate file type.
Note:  recommend you make copies to all file first.
 It makes changes to the header entries and there is no undo.

Btw, What software did use to acquire them?

[mmirot]

The batchconverttype script is on first page on this thread
Try adding the key words see if this works

max

[kwiechen]

0.0.7h adds a control on the 'Light' tab to select the interpolation algorithm and the clamping threshold value. So try to use this, if the script gives ringing artefacts with the default settings (Auto=Lanczos-3, clamping 0.3). Attached is an example with the default values (left side) and bicubic spline/clamping threshold 0.1 (on the right) with a 2 x binned chrominance frame of M106.

Again, the script needs FITS files from monochrome CCDs with keywords/values in the header to work (IMAGETYP, XBINNING, FILTER, EXPTIME). To get this to work with other files (OSC, DSLR, files without header info) I will need complete example data. If there are any problems please provide me with the camera type, info about software for data aquisition and example data.

Kai

[Alejandro Tombolini]

Hi Kai

Congratulations for the script, it looks exellent.

I am interested in the glass of wine that you mention in the script, that why I have uploaded a comple set of image of a Canon XSi (450)

10 Images
10 Darks
10 Bias
10 Flats (BTW are de images that do not have name.)

All the images are adquired with Canon' Software (EOS Utility)

Let me know what other way could I colaborate for this wonderfull script to work with DSRL images. 

I shared the folder with you:

http://pixinsight.org/files/?repository_id=my-files&folder=%2FDataExamplesForCalibrationAndAlignmetScript

If it does not work this is de zip with all files: 

http://pixinsight.org/files/data/public/56df5e4990afea3e1230620797dabbbb.php?lang=es 

(Let me know if it works)

Saludos. Alejandro.

[astrospotter]

Hello,
A week or so back I reported seeing issues after using image integration on frames that were calibrated and registered with the tool, version f.  I now have version g that fixes the issues with file lists so thanks.

Anyway, that was in the end my own failure to setup pixel rejection on the lights when doing image integration.  Tracked it down over the weekend.

So all is well and the tool functions better than ever as far as the other improvements from f to g version.

Thanks so much for all your time and efforts.  PixInsight was sorely lacking in a cal script to ease the many mechanical steps and checks that can messup or take forever to calibrate/register before this script.