PixInsight Forum

Software Development => New Scripts and Modules => Topic started by: mmirot on 2011 December 06 18:40:09

Title: Calibration and align batch operations
Post by: mmirot on 2011 December 06 18:40:09

Kai has started a nice little script.

http://pixinsight.com/forum/index.php?topic=3638.0

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2011 December 15 14:42:00
Here the latest version of the script
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2011 December 17 05:43:13
this small update prevents from loading duplicate files.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2011 December 19 15:40:48
Kai, Max,

Please don't think I'm ignoring Kai's excellent work. I am well aware of it (as well as Nikolay's modules, Enzo's script, etc.) and as soon as I can invest the necessary time, I'll review everything you're doing and we'll support these developments as they deserve.

Right now I am extremely busy finishing a new version of the PI Core application (around Qt 4.8) and a new batch of documentation. Thank you for your nice work, please keep it up!
Title: Re: Calibration and align batch operations
Post by: mmirot on 2011 December 21 18:28:34
Thanks Juan,
I have using kai's script over a week now it works nicely

Max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 02 11:14:01
Juan, Max,

thank you for your encouragement!

Some parts of the script were rewritten to use a less complex data structure and there were minor modifications at the GUI part.

Kai

 
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 02 19:30:46
Some file selections don't show up in the list of images .
This is true of masters created with the last version.

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 03 03:24:06
Can you please give me some files for testing? My files were aquired using MaximDL and some of my flat files have an IMAGETYP 'Light Frame' set.
The master frames from older versions of the script do not contain the IMAGETYP keyword. The last version of the script uses strict checking of the IMAGETYP key.

If this is the problem you can try the small script attached to set IMAGETYP. The script will not ask before writing to your fits header data - so please don't forget to make a  backup.


Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 03 16:19:33
My fits were made by Maxim Dl too.  The Masters were created by the 6d version

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 03 16:50:00
 My wish list for script in future versions

 1  Add integration. Unselect align or integration processes when desired too
 2. Deselect individual fits files from list in addition to clear all
 3. Select the reference image from the image list , double click or default to first file ( easier than set path)
 4. Single image or an image named master in fits key word automatically identified as a master
     No need to check use master
 5.  Persistent setting and paths. They should remain the same next time script is run
 6.  I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)

this just my 2 cents

Max


Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 03 16:53:01
Kai, Max,

Please don't think I'm ignoring Kai's excellent work. I am well aware of it (as well as Nikolay's modules, Enzo's script, etc.) and as soon as I can invest the necessary time, I'll review everything you're doing and we'll support these developments as they deserve.

Right now I am extremely busy finishing a new version of the PI Core application (around Qt 4.8) and a new batch of documentation. Thank you for your nice work, please keep it up!

1.74 is done come out and play Juan.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 04 04:33:50
Max,

the script was initially written as a prototype to better understand the PI process of calibration/alignment and integration. And it is my first try with javascript. There are limitations
of the script language preventing me from adding complex functions to this (mainly lack of a debugger, the limited degree of interactivity with the modal behaviour of the script dialog).

If possible, I would try to port this to a C++ module and then enhance functionality - but if I correctly understand it is up to now not possible to invoke PI modules or their properties/methods from another PCL module ????.


Thank you for your feedback!


 1  Add integration. Unselect align or integration processes when desired too

I can try to add this - the integration control is separated in the script code and it may be possible to add this dynamically for all filter settings to the Light tab.
This was removed from the initial version because I thought about using different sets of integration parameters for well sampled L data and more or less poorly sampled RGB data.

 2. Deselect individual fits files from list in addition to clear all

I have planned to do this - I will need some more information about the TreeBox object in order to give a visual feedback to the user.
 
 3. Select the reference image from the image list , double click or default to first file ( easier than set path)

Hm, I need again more info or code examples for TreeBox.

 4. Single image or an image named master in fits key word automatically identified as a master
     No need to check use master

Yes, I thought about adding a new FITS keyword to the master frames.

 5.  Persistent setting and paths. They should remain the same next time script is run

I need some information about object persistance in PJSR. This should be added with priority.

 6.  I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)

Unfortunately I have no rotator yet - but I have thought about adding this to my equipment. Of course it is possible to add this to the data when reading the FITS files but I will need
example data.

Best,

Kai
Title: Re: Calibration and align batch operations
Post by: Jack Harvey on 2012 January 04 07:31:12
This script is very cool!  THe above suggestions are all good ones.  I did have a bit of trouble learning to use the script (no reference docs<G>.)  So I made a short cheat sheet for myself - others may find it helpful?

CalibrateAlign Script Cheat Sheet

1.  Download CalAlign script 6d from PI forum.  Open script editor File>Open and navigate to the   CalAlign script on your computer.  Execute>Compile and Run gives you a window to work with.

2.  Select Bias and load either raw bias frames or bias master (check use master bias if so)

3.  Select Dark and repeat process in step 2

4.  Select Flat and again repeat process in step 2.  Load flats for all the filters you will load in Lights

5.  Select lights and load all the light frames you wish (can load all filters)

6.  Select a registration master for the Reference Image drop down

7.  Select a destination folder for the Output drop down

8.  Press OK and now go get a cup of coffee or a glass of wine<G>.  The script will output
     the calibrated and registered light frames in the output folder in subfolders for each filter.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 04 10:48:59
No hurry,

Progressing nicely.   Hopefully, we can get some of the other actual programers to give it a try.
Perhaps they can suggest some code.  This is very useful script.

A few more on the wish list.
Handle color camera images.

Images that don't contain the key words. Most of us use MaxIm and other programs that contain the standard Fits key words for time, bin, and filter.  However, there will be a few with the odd acquistion program that does not put these in.

Low priorty options, ie, bells and whistles.
Dark flat option or scaling darks for longer duration flats.

Add cosmetic correction module to script. It can be helpful to remove fixed hot columns and pixels after calibration. 
I need less sigma rejection when I do this for my hot columns.

Max

Max,

the script was initially written as a prototype to better understand the PI process of calibration/alignment and integration. And it is my first try with javascript. There are limitations
of the script language preventing me from adding complex functions to this (mainly lack of a debugger, the limited degree of interactivity with the modal behaviour of the script dialog).

If possible, I would try to port this to a C++ module and then enhance functionality - but if I correctly understand it is up to now not possible to invoke PI modules or their properties/methods from another PCL module ????.


Thank you for your feedback!


 1  Add integration. Unselect align or integration processes when desired too

I can try to add this - the integration control is separated in the script code and it may be possible to add this dynamically for all filter settings to the Light tab.
This was removed from the initial version because I thought about using different sets of integration parameters for well sampled L data and more or less poorly sampled RGB data.

 2. Deselect individual fits files from list in addition to clear all

I have planned to do this - I will need some more information about the TreeBox object in order to give a visual feedback to the user.
 
 3. Select the reference image from the image list , double click or default to first file ( easier than set path)

Hm, I need again more info or code examples for TreeBox.

 4. Single image or an image named master in fits key word automatically identified as a master
     No need to check use master

Yes, I thought about adding a new FITS keyword to the master frames.

 5.  Persistent setting and paths. They should remain the same next time script is run

I need some information about object persistance in PJSR. This should be added with priority.

 6.  I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)

Unfortunately I have no rotator yet - but I have thought about adding this to my equipment. Of course it is possible to add this to the data when reading the FITS files but I will need
example data.

Best,

Kai
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 04 10:49:50
Thanks for post the instructions Jack.

Max

This script is very cool!  THe above suggestions are all good ones.  I did have a bit of trouble learning to use the script (no reference docs<G>.)  So I made a short cheat sheet for myself - others may find it helpful?

CalibrateAlign Script Cheat Sheet

1.  Download CalAlign script 6d from PI forum.  Open script editor File>Open and navigate to the   CalAlign script on your computer.  Execute>Compile and Run gives you a window to work with.

2.  Select Bias and load either raw bias frames or bias master (check use master bias if so)

3.  Select Dark and repeat process in step 2

4.  Select Flat and again repeat process in step 2.  Load flats for all the filters you will load in Lights

5.  Select lights and load all the light frames you wish (can load all filters)

6.  Select a registration master for the Reference Image drop down

7.  Select a destination folder for the Output drop down

8.  Press OK and now go get a cup of coffee or a glass of wine<G>.  The script will output
     the calibrated and registered light frames in the output folder in subfolders for each filter.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 04 21:13:41
 Version 7a seems to work fine
 Max
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 January 05 04:42:54
Guys, would anyone care to comment on pros/cons off this latest cal script versus the older PreProcessingPipeline script?

Thanks,
Rob
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 05 06:37:22
The script is not yet as sophisticated as the pipeline.  It does not integrate the aligned lights currently.  Hopefully, this will be added shortly.

However, it is much easier to set up and use. I find the UI much more intuitive.

I is geared a more to those that use a monochrome camera right now and have the FITs key words Bin, exposure, and filter.
MaxIm does this for all monchrome cameras automatically as do other acquistion programs.

The nice part of the script is it will automatically sort out darks, flats, bias by these criteria make masters and calibrate matching lights. That is if you give a bias set with bin2, darks with bin2 &10min and flats with bin 2 with a red filter;
you can expect the script to create masters for bias bin2, Master dark bin2 & 10min and a bin 2 red flat.

If Red ,bin2, 10min image are in the light list they are then automatically matched to the appropiate masters, calibrated and then aligned. It outputs a folder with each type of image produced masters, calibrated and registered.

You can list as many combination of images as you like with various bins, exposure times and filters in it and it will figure out the matching calibration.

The masters can be reused for future calibrations with script to save time.

Max



Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 05 10:51:33
Jack, Max,

thank you for this contribution and feedback!

@Jack: can I integrate this nice doc into the help label of the next script version?

@Max: I have sorted the wish list a little bit:


1.  Persistent setting and paths. They should remain the same next time script is run
< I will try to add this to the next version.


2. Single image or an image named master in fits key word automatically identified as a master. No need to check use master
<There should be no problem to add a new custom keyword MASTERFRAME or something like this with a corresponding value and reuse these files automatically.

3 Images that don't contain the key words. Most of us use MaxIm and other programs that contain the standard Fits key words for time, bin, and filter.  However, there will be a few with the odd acquistion program that does not put these in.
<For older data not containing the fits header keywords, I would prefer to extend the small BatchConvertImagetype script to set all keywords necessary to be handled by the CalibrateAlign
script.

4 Handle color camera images.
<I have never used PI to process OSC/DSLR data. Can you please give me some details regarding the difference when calibrating/integrating OSC images compared to monochrome data? Is it necessary to debayer/split these images into different channels before? If we have
DSLR images how can I get for example the exposure time? To handle images from color cameras I can provide a mode ignoring image metadata (binning is meaningless from my point of view,
filter information can be taken from the debayer pattern)

5 Dark flat option or scaling darks for longer duration flats.
<Dark frame scaling is already set by default in this script. The dark flats can be added to the dark tab and will be used.

6  Add integration. Unselect align or integration processes when desired too
<I can try to add this - the integration control is separated in the script code and it may be possible to add this dynamically for all filter settings to the Light tab.
This was removed from the initial version because I thought about using different sets of integration parameters for well sampled L data and more or less poorly sampled RGB data.

<But I am not convinced that this is a good solution. This excellent document (Jordi Gallego, Image integration techniques: Increasing SNR and outlier rejection with PixInsight, slides for a
presentation given at the VI Seminario de Astrofotografía de Cielo Profundo, Madrid, November 20th, 2010.) shows the importance of fine-tuning integration parameters

7: Add cosmetic correction module to script. It can be helpful to remove fixed hot columns and pixels after calibration.
I need less sigma rejection when I do this for my hot columns.
<The CosmeticCorrection module is not installed in the last version of PI (Windows 7, x64), so I can not test this.


8. Deselect individual fits files from list in addition to clear all
9. Select the reference image from the image list , double click or default to first file ( easier than set path)
<Points 8 and 9 will require some rewrite of the internal structure. The TreeBox is only updated to provide visual feedback to the user in the current state of the script.

10. I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)
<Is there a 'standard' to handle this? And here I will need some test data.


Best,


Kai


Title: Re: Calibration and align batch operations
Post by: Jack Harvey on 2012 January 05 11:12:24
Yes feel free to add it to help
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 05 15:12:06
Jack, Max,

thank you for this contribution and feedback!

@Jack: can I integrate this nice doc into the help label of the next script version?

@Max: I have sorted the wish list a little bit:


1.  Persistent setting and paths. They should remain the same next time script is run
< I will try to add this to the next version.

Excellent
2. Single image or an image named master in fits key word automatically identified as a master. No need to check use master
<There should be no problem to add a new custom keyword MASTERFRAME or something like this with a corresponding value and reuse these files automatically.
Great


3 Images that don't contain the key words. Most of us use MaxIm and other programs that contain the standard Fits key words for time, bin, and filter.  However, there will be a few with the odd acquistion program that does not put these in.
<For older data not containing the fits header keywords, I would prefer to extend the small BatchConvertImagetype script to set all keywords necessary to be handled by the CalibrateAlign
script.

Good Idea a batch script crossed my mind too
4 Handle color camera images.
<I have never used PI to process OSC/DSLR data. Can you please give me some details regarding the difference when calibrating/integrating OSC images compared to monochrome data? Is it necessary to debayer/split these images into different channels before? If we have
DSLR images how can I get for example the exposure time? To handle images from color cameras I can provide a mode ignoring image metadata (binning is meaningless from my point of view,
filter information can be taken from the debayer pattern)

I am clueless here. I used a color CCD only once in my life. We need help from the peanut gallery on this one
5 Dark flat option or scaling darks for longer duration flats.
<Dark frame scaling is already set by default in this script. The dark flats can be added to the dark tab and will be used.

Good.  I assumed it was just using a bias subtract on flats as a default.
You might want see about a use "bias" option. There can be calibration problems if the dark noise in the flat is too low.  See Juans coments on this from today. http://pixinsight.com/forum/index.php?topic=3677.15
6  Add integration. Unselect align or integration processes when desired too
<I can try to add this - the integration control is separated in the script code and it may be possible to add this dynamically for all filter settings to the Light tab.
This was removed from the initial version because I thought about using different sets of integration parameters for well sampled L data and more or less poorly sampled RGB data.

<But I am not convinced that this is a good solution. This excellent document (Jordi Gallego, Image integration techniques: Increasing SNR and outlier rejection with PixInsight, slides for a
presentation given at the VI Seminario de Astrofotografía de Cielo Profundo, Madrid, November 20th, 2010.) shows the importance of fine-tuning integration parameters

You are right to be concerned. I agree the best way to do this is to link the parameters to each filter.  Once the all setting are persistant then you might be able to use a ROI just like the integration module. I think the files may still be in the cache after the script is run.  You could just uncheck calibrate, and align funtions then run the integration part of the script again with tweeking the ROI and settings the same as the module.   ::)

7: Add cosmetic correction module to script. It can be helpful to remove fixed hot columns and pixels after calibration.
I need less sigma rejection when I do this for my hot columns.
<The CosmeticCorrection module is not installed in the last version of PI (Windows 7, x64), so I can not test this.

This is way down on my wish list    ;)

8. Deselect individual fits files from list in addition to clear all
9. Select the reference image from the image list , double click or default to first file ( easier than set path)
<Points 8 and 9 will require some rewrite of the internal structure. The TreeBox is only updated to provide visual feedback to the user in the current state of the script.



10. I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)
<Is there a 'standard' to handle this? And here I will need some test data.
I will send a couple FITs overs soon

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 07 10:08:47
Any programmers out there?    May be someone can provide Kai support on these issues ???

Path and setting persistence.

Calibration of color cameras

The TreeBox

Max
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 January 07 10:20:15
Quote
Any programmers out there?

If I can be considered a programmer, I'm starting to take a look at Kai's script. First let me get acquainted with the script and I'll try to improve it and provide support on those topics.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 07 10:45:24
Thanks for the attention Juan!
--------------------------------------------------------------------------------------------------------------------------------------------------
BTW, Kai I for correction of hot columns we should be using the defect map which is a module
Attached is  a Fits header with MaxIM's Rotator entry. In the this it is at 313.00 degrees.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 09 08:05:40
Thanks for the attention Juan!
--------------------------------------------------------------------------------------------------------------------------------------------------
BTW, Kai I for correction of hot columns we should be using the defect map which is a module
Attached is  a Fits header with MaxIM's Rotator entry. In the this it is at 313.00 degrees.

Max

Thanks Juan
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 11 12:05:29
007b adds some kind of persistence. The code was splitted for better maintenance.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 12 21:50:19
Seems to work the same. I can't tell what is persistant.

Note: I you check master ; the file is removed if it was selected first.


Max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 12 23:19:59
The persistence settings save the state and values of controls - but not the file lists. If you check 'use master' the file list
in the current tab will be removed (tooltip), because the current state of the script is not able to recognize master frames from their fits headers.

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 18 11:53:42
I am looking forward to the next version of the script. I have new image at last.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 19 06:35:04
Small note.
I was trying to calbrate and align a few images from the west side of the mount. Most were obtained on the east side. So I set up to align and calibrate the westerly light frames, six green images and two reds to the reference frame.  The script stopped before calibrating the two red frames. The process window said it could not continue because there were less than three images.
I realise three will be required to integrate a master etc, but calibrate and alignment sure go ok.

Sure would help to recogize the rotator angle in the fits for calibration. I would not have to run the scipt twice.

Max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 19 12:44:43
Max,

I have never thought about east and west side when aligning the light frames because StarAlignment works perfectly if I mix everything. Next, I will clean up the code of v0.0.7b and try to remove this limit from the calibration and alignment part.

@Rotator angle: if I understand this correctly you would like to make flat/light subgroups for a fixed angle value (in addition to the filter subgroups)?

Best,

Kai



Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 19 14:30:47
You understand correctly the rotation angle is moved 180 on either side of the meridian after the flip.  I need to calibrate with the flat where the rotation angle and filter is the same as the light.

Max
Title: Re: Calibration and align batch operations
Post by: Mike Reid on 2012 January 26 09:44:26
Excellent work Kai!

I have a couple of minor issues with 007b to report.

1) In getMasterDarkFrame the delta is seeded with 1000 seconds.  I tested the first time with Ha data that had 1800 second exposure time and I had 1800 second darks.  I had no short darks to match the flats so nexptime never got set tnio anything other than it's initialed value of 0 and I got a crash.  I hacked mine by increasing the initial delta to 2000 but you might want to add a check here.

2) On line 468 I had to change,
    masterFlatPath = engine.getMasterFlatFrame(binnning, info.filter);
to
    masterFlatPath = engine.getMasterFlatFrame(binnning, trim(info.filter));
because info.filter was something like '    Ha'.


Thanks,
Mike
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 26 12:10:08
Mike,

thank you for testing this! Currently I try to clean up the code and I will try to resolve these issues.

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 26 16:47:20
If understand correctly the script dark subtracts flats using scaling. I recommend an option to bias subtract the flats. Often the darks noise is so low with short exposures on the flats the bias works better. Too much error in the scaled dark.

Max
Title: Re: Calibration and align batch operations
Post by: Mike Reid on 2012 January 27 15:05:07
Max,

Yes, you do understand correctly.  I think, rather than adding another option, it would be good to simply treat the master bias as a 0 second master dark.  This way Kai's algorithm that looks for the closest dark in exposure time to the flat would pick the master bias in this case.  He currently finds the dark master with the smallest absolute value difference in exposure time from the exposure time of the flat.  So if the case of my 9.8 second flat the master bias would be chosen in preference to the 1800 second master dark.

Mike
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 January 29 02:56:13
Dark scaling is enabled by default in the script. If there is no significant dark current in very short exposed flats, the ImageCalibration module
warns about no convergence in noise evalulation and (if I have understood this correctly) no dark is subtracted. But if there is no sigificant dark current in the flat I tend to ignore this.
I think it is not correct to use a bias for dark subtration as the bias will be subtracted twice.

Kai



Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 January 29 05:10:17
If the script defaults to a bias subtraction when there is no convergence that sound fine.

Juan,

What do you think?

Max
Title: Re: Calibration and align batch operations
Post by: Mike Reid on 2012 January 29 14:47:56
I think it is not correct to use a bias for dark subtration as the bias will be subtracted twice.

Well, not exactly.  The difference is that the bias that is used as a short dark would be scaled to match the flat before subtraction where the "real"  bias is simply subtracted.

I don't actually care which of these solutions is implemented.  In my version of the code I changed the max delta to 2000 instead of 1000 and now the script doesn't crash but it's scaling a 30 minute dark frame to do dark subtraction of a 9.8 second flat.  Maybe it would be better to just not do dark subtraction of the flats if the difference between the exposure time of the flat and the nearest dark is too great.

Mike

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 02 17:56:50
Please post your code Mike

Max
Title: Re: Calibration and align batch operations
Post by: Mike Reid on 2012 February 02 20:08:33
Max,

I'd rather not post a copy of Kai's script.  But the two bug fixes I made were simple.  Referring to version 007b.  These are the changes to CalibrateAlign-007b.js,

1. Change line 199 from,
              var delta1 = 1000;
      to
              var delta1 = 2000;

2. Change line  468 from,
              masterFlatPath = engine.getMasterFlatFrame(binnning, info.filter);
    to
              masterFlatPath = engine.getMasterFlatFrame(binnning, trim(info.filter));

Any other changes that I have made were just to experiment with my own ideas for enhancements.  If any of these pan out I'll forward them to Kai for his consideration.

Thanks,
Mike
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 February 02 23:37:56
I'm having some trouble with this script on OSX 64 bit.  Script version 007b.

My FITS files are captured in MaxIm.  When I use this script to "add files" I select the files, but they don't appear in the script's window.  Doesn't matter if they're the bias, light or whatever.  Just can't seem to add them.

Looking in the process console window there are warnings  about line 120: reference to undefined property info.imagetype and TypeError: info.imagetype is undefined.

EDIT: I notice that the master bias file I had previously created in PI does not have an ImageTyp header, but the individual bias frames captured in MaxIm do.  Not sure how/why I'm getting the error when trying to add masters, though.

EDIT 2: I tried it again, and it seems that it will add the subs ok.  Only seems to break if trying the masters option.
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 February 03 05:28:20
Yes, I've had some issues in both versions on Mac OSX (Lion). I'm unable to open files! The error reads:

*** Error [222]:(path) /CalibrateAlign-007b.js, line 120: TypeError: info.imagetype is undefined
Title: Re: Calibration and align batch operations
Post by: Jack Harvey on 2012 February 03 07:55:31
FWIW, I am running ver 006d on my Mac 64 without any problems.
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 February 03 20:46:52
It's worth something, Jack!  :D

I tried 006d on Mac 32 and got this error when choosing a Bias:

Processing script file: (path)/CalibrateAlign-006d.js
*** Error [222]: (path)/CalibrateAlign-006d.js, line 350: TypeError: engine.FrameSets[0][parseInt(info.binning) - 1] is undefined


I'll try 64 next.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 February 05 19:19:30
Just did a test on my OSX 64bit again.  Selected individual frames for bias, darks, and lights.  Didn't have any flats.

The script went through the master bias and dark creation ok it seems, created a /masterframes folder in the root dir of my hard drive and in there it saved masterbias_binning_1.fit and masterdark_binning_1_exptime_1800.fit.  All seemed good, although not too keen on this folder in root of hard disk, would prefer somewhere else.  Assume that is configurable somewhere, still getting used to things.

But it seems when the script started trying to calibrate the lights, I got this error message:

Quote
Writing file:
/masterframes/masterdark_binning_1_exptime_1800.fit
Writing FITS image: 32-bit floating point, 1 channel(s), 3326x2504 pixels: 100%
searching for master dark frame with 240s - available master dark frame with 0s
*** Error [000]: /Users/troy/Applications/pixinsight/CalibrateAlign-007b.js, line 479: Error: <p><b>ImageCalibration</b></p><p style="white-space:pre;">Couldn't execute this instance in the global context.</p><p>Reason: Missing or invalid master dark frame.</p>

Note also that the html tags were there in the console window too.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 06 01:24:23
Unfortunately I cannot test on MacOS - currently I am cleaning up the code, so I hope that the problems will be rexolved.

There is an error in 0.0.7b when using darks with an EXPTIME longer than 1000s - this was adressed by Mike a few days ago. The next version will fix this.

Kai

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 09 11:06:00
Attached is 0.0.7d - after code clean up and minor enhancements to the user interface.

The script needs FITS input with proper keywords (IMAGETYP, XBINNING, EXPTIME, FILTER) to group calibration and light frames. I have tested
this with FITS frames aquired with MaximDL.

Kai

Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 February 09 15:31:24
Still having the issue of master frames not being accepted on OSX64 here.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 09 23:13:36
can you please give me an example data set?

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 10 07:47:17
Did you generate the master frames with the current script version?
Some generated with some of the older versions may not work.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 10 20:52:25
Kai,

I tried the rotator version you modified for me.  It appears to recognize the rotator degrees and sort the columns fine.

However, when run I get
"  doCalibrate - masterBiasPath: C:/Users/max/Rotator script/masterframes/masterbias_XBINNING_1.fit
*** Error [000]: C:/Kai Script PIX/CalibrateAlign-09022012/CalibrateAlign-engine.js, line 367: Error: Math.max(): invalid array length "


Max


Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 11 04:33:13
Max,

there is a problem in the getDarkFrame function when the array length of EXPTIMES avalilable is 1. I will try to fix this. Do you apply dark frames with different exposure times?

Kai
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 February 13 23:21:58
can you please give me an example data set?

Kai

G'day Kai,

Sorry haven't gotten back to you sooner, haven't had access to the data.  I think that the problem is that my master files were not generated by your script.  I had created them manually before I discovered your script.  So if the script has to have certain FITS headers in the master frames to work, that would be the problem.  Is that the case?

Personally, I don't think it should rely on the FITS keywords.  If someone has chosen to use a master, it could come from anywhere and not necessarily have any keywords.

Did you generate the master frames with the current script version?
Some generated with some of the older versions may not work.

Max

G'day Max,

No, as I mentioned above, I manually created the masters before I discovered Kai's wonderful script.

I might do some testing by creating masters now using the script and trying those.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 14 00:54:05
Hi,

the master frames should be prepared using the script because the FITS keywords are written to these files. But I think your error message shown is caused by a problem in the function getMasterDarkFrame(), when you apply dark frames with one EXPTIME only. I have fixed this in the current version of the script (I will post the fix this evening).

To use (master) files from another source I have posted a small batch script some weeks ago to set FITS keywords/values on a number of files.

Best,

Kai
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 February 14 01:39:17
Thanks mate. I did a test of creating masters using your script, then using it as master in the script separately. It did work that way. So your script is certainly requiring something it is adding to the files it creates.

Issues:
- creating the master without applying it to any lights caused problems. Next time tried to run the script, got error message in console about incomplete /tmp/calalign.cfg file. Manually deleted it and script ran fine. Be nice to be able to create masters without having to apply it to lights without error?
- creating master as per above, it gave error message about not being able to find/create /masterframes due to permissions. Maybe don't try to create folders in root, but home dir where you know you have permission? Or better yet, make it a user definable variable?
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 14 02:45:29

- delete the cfg-file and rerun the script generating masters only. If there are any errrors please send me the console log.

- you can select the output directory - the out files are then gropued below.  Of course you will need write access there.

Kai

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 14 12:04:36
0.0.7e should fix 2 problems when selecting master dark frames (single EXPTIME and exposure of light frames > available master dark)

Kai
Title: Re: Calibration and align batch operations
Post by: oo_void on 2012 February 14 14:56:50
Just tried this script for the first time with no luck ... I get following error when attempting to add flats or lights. 

*** Error [222]: /Users/p139jeb/Library/PixInsight/CalibrateAlign-engine.js, line 89: TypeError: FrameType[group.Imagetype] is undefined
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 15 00:53:17
Hi,

camera, software? The script ist splitted in several files that are included at runtime. The global var FrameType is defined in CalibrateAlign-def

Kai

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 25 11:08:41
0.0.7f adds some improvements to the user interface.
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 February 25 16:20:22
I've been unable to give this script a run on XP32 unfortunately.  As Troy has commented earlier, you can't create suitable Master frames without using the script itself it appears, however when I try to combine a reasonable number of frames for bias or darks I run out of memory. 

The usual way around this is to reduce the default Buffer Size during image integration from 16mb downwards.  Would it be possible to expose that parameter to the script.

I'm quite disappointed - PI REALLY need these sorts of facilities currently, so thanks for all the hard work!
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 25 19:35:24
Kai,

The light frames file don't show up after selecting them.  Bias, darks, flats select fine.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 25 19:49:28
Never mind. I think the problem is on my side not your code.  O:)

Max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 26 06:05:30
Rob, Max,

thank you for testing this.

@Rob: the script was tested with Win7 64bit. I will test with winxp 32 and of course we can expose more parameters to the script.

@Max: can you give me some hints or data about the rotator keyword support. If possible I can merge this into the current version.

Kai
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 26 09:27:22
Hi Rob,

I have tested with 30 bias raw frames binning 1 from my QSI583ws without any problems using Windows XP 32bit/2GB RAM.

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 26 11:04:29
Rob, Max,


@Max: can you give me some hints or data about the rotator keyword support. If possible I can merge this into the current version.

Kai

Sure,

I don't want go forward with rotator code just yet. I found that CCDAP5 with MaxIM DL appears have round off error occasionally after re- initializing the rotator.
That is the one image set may be a degree off under the Rotator key word,  i.e now you have 354 and 355 degrees on frames that are theoretically identical rotation.
This won't work well !


MaxIM DL inserts a FITs key  Pierside ' East '   or ' West '
Also the there is Flipstat key word.
These might be better since East verses West or Flipped verses not  are all we need.

Another approach is to have the function work through the image title.  The non case sensitive word East or West in the title  ... M51_East ... M51east etc.

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 26 11:12:57
1
 I found a problem with the imagetyp Key word.  My recent set of images taken with MaxIM DL did nt get recognized because the contained imagetyp ' light'  rather than ' light frame '

I am sure that this is due to a update/ with setting changes in my software. Surely, this will cause problems for someone latter on too.  Can we make it less restrictive so both work?

2

7f the clear function with light frames did not work.

I hit reset the files go away but then you can't open a directory and add them in again.

Max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 26 12:41:55
Max,

flexible handling of IMAGETYP values is on my TODO list (for example to use IRAF conventions).

The clear function is unfortunately broken. I will try to fix this.

It is possible to group files following different sets of FITS keywords, therefore there should be no problem to use 'Pierside'

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 26 18:14:40
Yep had the IRAF option checked. No longer a problem.

Max
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 February 27 03:37:06
Hi Rob,

I have tested with 30 bias raw frames binning 1 from my QSI583ws without any problems using Windows XP 32bit/2GB RAM.

Kai

Hi Kai

Yes, I was being greedy - using about 41 frames bias and darks.
One other problem - the script seems to crash if you don't specify any flats or master flat.  Surely should be an option that don't have to specify flats?

(BTW, I was able to get the script going happily using my dual-boot Ubuntu 64bit setup)
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 27 07:44:40
Rob,

You will be a lot happer with a 64 bit system with at least 6-8 GIGs of memory. I makes a big difference. 32 bit is limits ram access to 2 GIGs

Max
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 February 28 01:54:10
Yes, definitely on the wish-list to upgrade Max, but feeding the family comes first  :moneyinmouth:
Title: Re: Calibration and align batch operations
Post by: astrospotter on 2012 February 28 21:22:17
After looking at this a month ago it was 'nice' but still so very easy to make an operator error or in several cases to figure out in an intuitive way.

Now with version 7f my hat is off and I thank you SO much for the efforts of putting together this extremely valuable and key piece of missing workflow to PixInsight.   Looking much better now.   I see you have the keyword recognition to avoid operator error and the layout as well as logic in the tool checkboxes is far more intuitive.

I had a problem with what looks like hot pixel issues so will have to revisit in case I operated it wrong but the workflow and so on is much improved.  I'll try to sort out what went wrong.  I went back to MaximDL and figured out where I had gone wrong there (had skipped an align that I thought was automatic but required that step separately).

Over the last few days I have been struggling with both MaximDL and DeepSkyStacker both of which have had assorted serious issues that have led to inability to form masters of quality.

IN ANY CASE, THANK YOU!!!     This will make a truly important contribution to everyone's workflow.

Mark Johnston
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 February 28 22:52:05
am i correct in assuming that this script will not work for a DSLR user since it relies on some set of FITS keywords? thanks.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 29 04:21:56
am i correct in assuming that this script will not work for a DSLR user since it relies on some set of FITS keywords? thanks.

The current version can group and calibrate FITS files from a monochrome CCD using sets of FITS keywords to group files. If I can can get test data and information about differences
in the workflow it should be easily possible to use FITS from color OSC. It my be in addition possible to use DSLR data, if the ImageCalibration/ImageIntegration tools can import the
raw files. But in order to proceed I need some information about workflow differences.

Best

Kai

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 February 29 04:55:27
0.0.7g should have removed the bug when clearing the light frame list.

@Max: to use IRAF designation of light frames you may try to comment out line 3 in CalibrateAlign-def and uncomment line 6. But it is not possible to mix files with different IMAGETYP designations. If you have for example light frames with wrong IMAGETYP values you can try the BatchConvertImagetype script posted already some time ago (use with backup data!!!). Rotator support: do you include the correct pierside values to the flat files?

@Rob: in order to modify bufferSize and/or stackSize the vars are written to lines 305/306 in CalibrateAlign-engine. Try to reduce the values and if this can solve the problem with ImageIntegration crashes on PC's with low RAM, we can expose these vars to the user interface.

Kai
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 February 29 07:41:05
0.0.7g should have removed the bug when clearing the light frame list.

@Max: to use IRAF designation of light frames you may try to comment out line 3 in CalibrateAlign-def and uncomment line 6. But it is not possible to mix files with different IMAGETYP designations. If you have for example light frames with wrong IMAGETYP values you can try the BatchConvertImagetype script posted already some time ago (use with backup data!!!). Rotator support: do you include the correct pierside values to the flat files?


Kai

The image type designations is now under control not that I know not to use IRAF or Mira key words.
I used the batch script to fix these. It might be nice to widen its usage by allowing editing of additonal keywords ie pierside, filter, bin.


Yes, we are only trying to match flats to lights.

The  pierside key word will work nicely for rotators.
For MaxIm Dl you just have to the scope setting on german equitorial and the Maxim DL observatory connected to the mount.
It generates a Pierside : ' East ' or ' West ' entry.

I was a little confused at first because it based on the which side the OTA is located on rather than where the scope is pointing in the sky.  There is a option in MaxIM dl to switch it to enter the scope pointing. However,  as long as the pierside entry consistant the convention does not matter.
If your not using Maxim the adding a pierside key word through a batch to only one side of the merdian should work for matching porposes.

I will post a fits header with the pierside format or send you a few images.

Max
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 February 29 08:47:14
am i correct in assuming that this script will not work for a DSLR user since it relies on some set of FITS keywords? thanks.

The current version can group and calibrate FITS files from a monochrome CCD using sets of FITS keywords to group files. If I can can get test data and information about differences
in the workflow it should be easily possible to use FITS from color OSC. It my be in addition possible to use DSLR data, if the ImageCalibration/ImageIntegration tools can import the
raw files. But in order to proceed I need some information about workflow differences.

Best

Kai

IC can definitely handle CR2 files directly. II can not, you have to convert the CR2s for your bias/darks to .fits manually first. maybe you can run IC with no calibration masters and that would convert the CR2 to fits, but i've never tried that and suspect it would not work.

BackyardEOS can write EXIF data to your file indicating what it is (light, dark, bias) and it's duration, etc. BYE can also save your files with descriptive filenames (frame type, ISO, duration, etc) but none of these things are going to show up in the FITS header as far as i know. IC would have to try to carry this info into the FITS, or the fits would have to be post-processed to create keywords from the filenames. i think BYE is going to support writing FITS soon and it's likely that the same data written to the EXIF would show up as FITS keywords.

but even if it can write fits i'd probably stick with CR2 since i prefer to have what the camera gave me rather than something that's been modified...

Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 February 29 08:55:49
...raw files. But in order to proceed I need some information about workflow differences....
The DSLR workflow is described in http://pixinsight.com/forum/index.php?topic=2570.msg19019#msg19019 . The .CR2 files dont contain much useful information, except maybe the exposure time. The way I usually work is that I rename the image files such that they are named light*.CR2, bias*.CR2, dark*.CR2, flat*.CR2. I then follow the workflow linked above.

Georg
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 01 07:26:51
Kai some thoughts...

We should have a check box to calbrate only. This is especially helpful when testing the calibration functions.

An option to ignor Fits Key words all together is a good idea too.  I use MaxIM and want the automatching and checking.
There will be others that don't have Maxim Dl that will be happy to do one image set at time putting in the correct/ matching bias,  dark exposure and flat.   It will be easier than running the batch file to add key words.  ( Not that I am suggest we don't have this too)

I would be very nice to deselect or remove selected images from the lists.

Have you though about adding in integration? It could be nicely if you have the option to pause after registration than select a ROI to tweek it.

Max




Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 01 12:33:48
Max,

1) do you think about 'calibrate only and skip registration'?

2) We need some kind of metadata to group files. It may be possible to rewrite this to be able to manually add this data (for example frame/image type and/or exposure time) and to bypass evaluation of the FITS header. I am thinking about this to support for example DSLR data.

3) The internal data structure to select/deselect images is in place. I have not put this to the user interface because of problems with interaction (modal dialog)

4) Light frames need some tweaking of integration parameters to get the best SNR and pixel rejection possible (http://pixinsight.com/forum/index.php?topic=3736.0)

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 01 17:09:42
1) Calibrate only is great.

2) Ideally we want meta data from the fits to match.  It allows for all your cool features,
However, I was wondering if there was an additional way to allow for data to be processed when the proper FITs key words are not present.


3) good

4) You mean integration needs tweaking. I agree.   What I suggest is the script can run through with the initial parameters.
The script can then rerun at the integration step using different settings as need.

 When I use the integration module most of the time I am pretty good close to my initial settings.

However, if want the best combination of SNR and rejection I choose a smaller region of interest and do some iterations. This saves a lot of time because of smaller area and all the data is in memory too.
 If this script has persistence and the data is still in memory then I think it could iterate just like the module.  This would Rock !


Btw is this the right link? The link does not discuss this.

Thanks

Max
 

Title: Re: Calibration and align batch operations
Post by: johnrt on 2012 March 02 01:51:08
This script looks great, but will not work for me. I cannot add any files to the script window. I navigate to the files, bias, dark light etc etc, select them and click open and nothing gets added to the files box in the script window.

I'm using a Mac OSX 10.0.72 32bit version.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 02 08:02:55
We can help with that.
It is a the key word recognition problem I was discussing above. 
There is a typebatch script to add the fits words for flat , dark, bias, light. 
Just select the files and add the appropriate file type.
Note:  recommend you make copies to all file first.
 It makes changes to the header entries and there is no undo.

Btw, What software did use to acquire them?
Title: Re: Calibration and align batch operations
Post by: johnrt on 2012 March 02 14:28:08
They were all captured using Artemis Capture - with an Atik 314+.


We can help with that.
It is a the key word recognition problem I was discussing above. 
There is a typebatch script to add the fits words for flat , dark, bias, light. 
Just select the files and add the appropriate file type.
Note:  recommend you make copies to all file first.
 It makes changes to the header entries and there is no undo.

Btw, What software did use to acquire them?
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 02 21:17:09
The batchconverttype script is on first page on this thread
Try adding the key words see if this works

max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 04 13:14:21
0.0.7h adds a control on the 'Light' tab to select the interpolation algorithm and the clamping threshold value. So try to use this, if the script gives ringing artefacts with the default settings (Auto=Lanczos-3, clamping 0.3). Attached is an example with the default values (left side) and bicubic spline/clamping threshold 0.1 (on the right) with a 2 x binned chrominance frame of M106.

Again, the script needs FITS files from monochrome CCDs with keywords/values in the header to work (IMAGETYP, XBINNING, FILTER, EXPTIME). To get this to work with other files (OSC, DSLR, files without header info) I will need complete example data. If there are any problems please provide me with the camera type, info about software for data aquisition and example data.

Kai

(http://img12.imageshack.us/img12/8632/interpolation.jpg)
Title: Re: Calibration and align batch operations
Post by: Alejandro Tombolini on 2012 March 04 21:31:05
Hi Kai

Congratulations for the script, it looks exellent.

I am interested in the glass of wine that you mention in the script, that why I have uploaded a comple set of image of a Canon XSi (450)

10 Images
10 Darks
10 Bias
10 Flats (BTW are de images that do not have name.)

All the images are adquired with Canon' Software (EOS Utility)

Let me know what other way could I colaborate for this wonderfull script to work with DSRL images. 

I shared the folder with you:

http://pixinsight.org/files/?repository_id=my-files&folder=%2FDataExamplesForCalibrationAndAlignmetScript

If it does not work this is de zip with all files: 

http://pixinsight.org/files/data/public/56df5e4990afea3e1230620797dabbbb.php?lang=es 

(Let me know if it works) ???

Saludos. Alejandro.
Title: Re: Calibration and align batch operations - Light integration rejection
Post by: astrospotter on 2012 March 04 22:37:15
Hello,
A week or so back I reported seeing issues after using image integration on frames that were calibrated and registered with the tool, version f.  I now have version g that fixes the issues with file lists so thanks.

Anyway, that was in the end my own failure to setup pixel rejection on the lights when doing image integration.  Tracked it down over the weekend.

So all is well and the tool functions better than ever as far as the other improvements from f to g version.

Thanks so much for all your time and efforts.  PixInsight was sorely lacking in a cal script to ease the many mechanical steps and checks that can messup or take forever to calibrate/register before this script.
Title: Re: Calibration and align batch operations
Post by: robhawleyastro on 2012 March 05 07:46:18
How do you handle dark flats currently?

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 05 11:58:09
Hi,

@Alejandro: thank you very much for your example data - download was OK. I will need some time trying to update the script. In its current state image metadata is read from the FITS header when the files are added and subsequently grouped. I have removed all other dependencies to read FITS header data of individual files in other parts of the script. So it should be possible to add the data needed (exposure time, image type) using a new dialog window.

@robhawleyastro: dark frame scaling is on by default - I use short exposed dark frames as flat darks. Simply add these and they will be processed.

 Kai
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 March 05 18:04:07
I'm so excited to give this script a try, but I'm still stuck at square zero with getting it to work on my Mac. I've tried both the 32 bit and 64 bit installs and still get errors like this when trying to load any image:

*** Error [222]: /Users/itech/Documents/ Astronomy/PIScripts/CalibrateAlign-007g/CalibrateAlign-engine.js, line 89: TypeError: FrameType[group.Imagetype] is undefined

Do I need some kind of Developer's kit loaded? I know it *can* work on Macs because Jack Harvey is getting it to work... I'm willing to try anything.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 05 23:15:34
Please see my previous comment to 0.0.7h

Again, the script needs FITS files from monochrome CCDs with keywords/values in the header to work (IMAGETYP, XBINNING, FILTER, EXPTIME). To get this to work with other files (OSC, DSLR, files without header info) I will need complete example data. If there are any problems please provide me with the camera type, info about software for data aquisition and example data.
(http://img12.imageshack.us/img12/8632/interpolation.jpg)

Kai

Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 06 13:26:52
@jlake: I have just tested FITS files from other sources. In some cases the IMAGETYP keyword is not present in the header leading to the error above. If other keywords are present it may help to add IMAGETYP/value using the BatchConvertImagetype script posted previously in this thread. However the latest script 0.0.7h will only work with FITS files from mono CCDs.

Best,

Kai

Title: Re: Calibration and align batch operations
Post by: robhawleyastro on 2012 March 07 10:58:02
I do most of my work in narrowband and thus have very little signal.  Would it be possible to have the scripts work in 64-bit? 

Before I start a long discussion on the merits of 64 bit -> I am not sure it will improve the result, but I am sure it will only cost time (not that much) and disk space.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 07 12:52:35
I have been running it in 64 bit windows. Works fine.

Max
Title: Re: Calibration and align batch operations
Post by: robhawleyastro on 2012 March 07 18:59:48
No I mean could there be a way to use 64 bit math.

I also use the 64 bit version of Pixi exclusively
Title: Re: Calibration and align batch operations
Post by: robhawleyastro on 2012 March 07 20:07:20
The terminology used in Image Calibration is "sample format= 64 bit" and in Integration "generate 64 bit result"
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 08 03:21:26
Rob

64 bit rarely offers any advantage.  Your are actually just filling up your drive and taking much longer to process images.  If you are going to use a 64bit images wait until the integration step. The extra precison is totally unnecessary until this point unless you have a camera able to produce data at 64 bits. Not likely:)

I suggest you post a new thread on this in another forum.

Juan's comments should be informative

Max
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 16 02:57:38
Quote
64 bit rarely offers any advantage.

Absolutely. A 64-bit format is only necessary in the context of very large HDR compositions. Note that with 'large' here I don't mean a large set of images necessarily, but a large dynamic range requiring more than 107 discrete values, which is what the 32-bit floating point format provides. In these cases the 32-bit unsigned integer format (232 or about 109.63 discrete values) can be used in PixInsight, except in really extreme cases. We use 64-bit floating point instead because it avoids integer-to-floating-point-to-integer conversions that happen internally in most processes when they work on 32-bit integer images.

Other than these monster HDR compositions and some image analysis tasks, the 64-bit floating point format is normally not necessary.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 16 03:02:52
Just a note to inform everybody that I'm working on a revision of Kai's script, with the purpose to get it ready for release as an official update. As Kai knows I wanted to start doing this a few days ago, but other commitments have prevented me to do so. Hopefully I'll have a time window today.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 16 07:57:37
Thanks Juan,

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 16 08:03:38
What are you planning to do ?

Max
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 16 15:55:19
Hi

I dont want to be a damp squib here , but from what I am reading all the files have to have correct fits headers for this script to work  ???

Never put the correct info in a Fits header in my life , nor will a lot of people if they do not have maxim  :-[  I need to be free to select any files I like  :surprised:

Am I right  O:)

Harry
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 16 16:43:00
Hi,

some additional code to use files without FITS keywords will be added.   

Kai

Title: Re: Calibration and align batch operations
Post by: JamieInCLT on 2012 March 16 16:51:01
Very cool, please make it work directly with RAW files, too :D, and not just fits
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 16 17:47:25
Very cool, please make it work directly with RAW files, too :D, and not just fits
+1
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 16 18:43:10
It is great to see such interest in Kai's project.  8)

Max
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 March 17 14:18:10
I'm finding that if I use this script to generate master frames only, no lights, the next time I run it I get an error:

Code: [Select]
Processing script file: /Users/troy/Applications/pixinsight/CalibrateAlign.js
using config file /tmp/CalibrateAlign.cfg
*** Error [000]: /Users/troy/Applications/pixinsight/CalibrateAlign-settings.js, line 11: Error: File I/O Error: Unexpected end of file: /tmp/CalibrateAlign.cfg

I can manually delete the /tmp/CalibrateAlign.cfg but it's not really the desirable behaviour.
Title: Re: Calibration and align batch operations
Post by: astropixel on 2012 March 17 16:10:21
I second requests for handling raw files.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 19 00:55:41
I have added some lines to make the script ready to use FITS files without keywords and OSC FITS. I will need some time to test and to debug this. Is there any difference when calibrating flat/light frames from OSC cameras compared to a monochrome CCD (detect CFA ??)?

Kai

 

Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 19 03:02:29
Hi all,

I have made a first revision of Kai's script. You'll find it attached to this post.

First of all I want to say a big thanks to Kai for his nice initiative with this script.  His work is excellent and the script is clear and well designed and structured. It has been a pleasure working on it.

The first change you'll notice is a new name: Batch Preprocessing. I think this script has the potential to become a complete preprocessing pipeline for PixInsight, so a more general name was required in my opinion.

The revised version of the script does not rely so heavily on FITS headers. Now you can manually add bias, dark, flat and light frames that the script will load as such without further questions (you are supposed to know what you are doing in such case). Of course, the "Add Files" button is still there and works by selecting frame types automatically, based on standard FITS keywords. Both "Bias Frame" and "BIAS" keyword flavors are now supported.

The independence on FITS headers should make the script much more flexible. It should work now with raw frames generated by applications that don't store frame type keywords (which is a very bad practice IMO) and opens the door to support DSLR and OSC raw frames with little effort.

A change that all of you will love for sure is that the script can now generate instances. I have replaced Kai's settings system based on configuration files (which was efficient and very well designed by the way) with a standard import/export system based on the Parameters PJSR object. This means that the script integrates tightly with the platform, so you can generate process icons with your preprocessing pipelines and store them in XPSM icon files and XOSM project files.

The script now disables ImageIntegration's file cache for integration of bias, dark and flat frames. This greatly improves efficiency, since it prevents calculation of unnecessary noise estimates.

I have simplified the image integration parameters section. Integration and rejection normalization cannot be selected now. This is because these parameters are always fixed as either "no normalization" and "equalize fluxes", depending on frame types (the latter for flat frames), and changing them makes no sense. I have removed CCD Noise Model rejection because it is only of academic interest. I have added support for min/max rejection because it can be necessary sometimes. Finally, I have removed the image weighting parameter because calibration raw frames are never weighted. If you think that one of these parameters should be restored, please argument why, and we'll put it back.

Another change is the way master frames are managed from the GUI. Previously when you selected one of the "Use as master" options the corresponding list of files was cleared. This no longer happens, so you are free to enable or disable these options. This is more friendly in my opinion, and although it increases the risk of doing things wrong, will lead to increased flexibility when we implement a more complete management of file lists (e.g., when we can remove selected frames).

Finally I have made a GUI revamp with new buttons, some reorganization, more graphical elements and extensive use of Qt CSS styling. During this process I have also discovered a couple of minor bugs in the PixInsight Core application (some faulting PJSR routines), for  which I have found good workarounds, fortunately.

The script is perfectly usable right now. It is rather stable and robust, and yields excellent results as far as I have tested it. A lot of work remains to be done, however. These are just a few pending tasks:

* URGENT: Decide on a software license to release this script as official, via the update system. Our policy is to never release source code without a clearly stated license. I suggest using our PCL License v1.0 (http://pixinsight.com/license/PCL_PJSR_1.0.html), but this is of course Kai's decision.

* Add support for DSLR and OSC Bayer CFA raw frames. Not difficult now that the script doesn't depend on FITS headers.

* Add more file management features: remove selected frames, use enabled/disabled frame states, manage selections, etc.

* Improve diagnostics and error information.

* Make the script more robust against failed processes.

Hope you like it!
Title: Re: Calibration and align batch operations
Post by: erikgu on 2012 March 19 04:26:47
Hi !

Thank's Juan. Realy good news.  :D

Erik G.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 19 05:16:50
Juan,

thank you very much for your work!

Quote
* URGENT: Decide on a software license to release this script as official, via the update system. Our policy is to never release source code without a clearly stated license. I suggest using our PCL License v1.0, but this is of course Kai's decision.

Please take the PCL License as suggested.

Best,

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 19 08:18:33
Great can wait to try it out. Sounds good.

Max
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 19 09:16:44
A dream becomes true  8)
Title: Re: Calibration and align batch operations
Post by: ruediger on 2012 March 19 09:39:26
Very good! Thanks for the great work! Once debayering is included, this script will really speedup a typical dslr workflow.

One minor thing: the new csv file type as reference image type is not included. I have not really tried out this new alignment feature, but it looks very helpful if you have mosaic generation in mind.

Rüdiger
Title: Re: Calibration and align batch operations
Post by: bitli on 2012 March 19 11:29:26
The greatest think since slice bread.  Combined with the Blink process, this makes PI very useable.  I will no longer have to choose between spending a night doing manual process or having to switch to another program.
Thanks you very much.
-- bitli
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 19 11:32:27
Hi

If It works I will make a vid , will stop all the moaning hopefully

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 19 12:59:56
Hi

I get this error  :-\

PI calibrates the files and saves them ( good so far ) but falls over at the registration  :o

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 19 13:02:26
Hi

Bit closer

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 19 14:46:17
Hi

This is very very close  :-*

Can we make the script remember the last setting when closed i.e what files were selected , better than starting from scratch when redoing the same files

Harry



Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 19 14:47:57
Harry, create a process instance (blue triangle) to save the settings.
Georg
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 19 14:52:37
Hi

Yes I did see this  :laugh: , perhapes a personal thing I like simple to reuse not a game killer though

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 19 17:03:06
Hi,

Here is version 0.2 of the Batch Preprocessing script. News:

* Added preliminary support for CFA mosaiced images. OSC camera users should have no problems to calibrate and align their images with the script as it is now. DSLR camera users still need to convert their camera raw frames (CR2, NEF, etc.) into FITS files with the standard BatchFormatConversion script before using Batch Preprocessing. Only bias and dark frames need to be converted; flat and light frames will work in their original camera format without problems. These limitations should be removed in the next versions of the script.

* Added new GUI controls to implement CFA support.

* A new check box to enable/disable dark frame optimization (applied to calibration of flat and light frames).

* Published under PCL v1.0 license.

Please test it and report any errors here. Enjoy!
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 March 19 18:16:27
Hi,

Could it be possible to add a crop option to this script? Sometimes it's better to crop the borders of the image just after the calibration process. For instance, if you use an overscan enabled camera, the overscan areas can be extremely noisy after calibration. This can affect the registration and integration process.


Regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 19 19:10:07
I found no defects using add images as long as I started with a dark frame set of the same exposure.
The script created a usable dark master that worked subsequent operations. Other Master are fine too.

Wow...Lots of nice improvements.!

Calibration goes 3x faster without the cache for integration too. Can that work with registration?

I tried to use a single long dark frame selected with optimize checked but the calibration results were incorrect.
 I usually use a set of equal exposure darks so perhaps I am not doing this correctly?

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 19 19:17:12
I would add cosmetic correction to the wish list.
I have annoying hot column.  It would be nice to fix it prior to registration and integration.

Max

Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 19 23:12:37
...Yes I did see this  :laugh: , perhapes a personal thing I like simple to reuse not a game killer though
I agree: the way, for instance,the AnnotationScript keeps the settings from a previous run is also very convenient. Maybe having both would combine the best from both worlds...
Georg
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 20 00:29:46
A prior version preserved both settings and paths . Juan will add this soon ... Trust me ;)


Max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 20 00:45:14
Hi Juan,

looks great! It would have lasted a very very long time for me to bring this into a comparable state with tight integration into the PI system.

I have played a little bit with v0.1.

1) The script needs all file slots to be filled and a reference image selected. It should be possible to run the script with calibration frames only

2) The reference image was selectable simply by clicking in the file list in 007h

3) Checking of the masterframe checkboxes select only the first frame in the list as a master. The script fails therefore to calibrate and align with predefined masters (for example with 4 channels LRGB).

I will test v02 with 'fresh' data as soon as possible.

Best,

Kai
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 20 01:41:13
Hi

A save masters check box would be good , to save any newley created master bias / darks / flats for future use  :-*

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 20 03:16:32
Hi Kai,

Glad you like it.

Quote
1) The script needs all file slots to be filled and a reference image selected. It should be possible to run the script with calibration frames only

The script requires you to specify a registration reference image only if the 'Calibrate only' checkbox is unchecked. If you check it, then no registration reference is (should be) required.

By 'all file slots filled' what do you mean besides the reference image?

Quote
2) The reference image was selectable simply by clicking in the file list in 007h

It is selectable in 0.1, too, by double clicking one of the light frames. A single click is too prone to mistakes IMO, and we probably will implement more functionality associated with clicking frame items on the treebox structures.

Quote
3) Checking of the masterframe checkboxes select only the first frame in the list as a master. The script fails therefore to calibrate and align with predefined masters (for example with 4 channels LRGB).

If the L, R, G and B images have the correct filter identified in their FITS headers, there should be no problem. Can you verify this?.

Of course we have to improve master frame management. How about an option to manually specify a filter for each frame? In this way we'd remove another FITS header dependence, and the script would be more flexible.

Quote
I will test v02 with 'fresh' data as soon as possible.

Great. We need lots of tests with different data samples to make this script 100% robust.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 20 03:20:16

Quote
A save masters check box would be good , to save any newley created master bias / darks / flats for future use

Masters are generated during script execution and stored on the 'master' subdirectory of the root output directory.
Title: Re: Calibration and align batch operations
Post by: astropixel on 2012 March 20 04:22:40
Great work - this is going to simplify things enormously.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 20 05:52:43
Hi

Was using masters all ready so missed this  :surprised: , but excellent  ;D

How about a defect map Please  :-*

Harry


Quote
A save masters check box would be good , to save any newley created master bias / darks / flats for future use

Masters are generated during script execution and stored on the 'master' subdirectory of the root output directory.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 March 20 06:20:47
Hi,
One suggestion: Please verify geometry of ALL the images before start doing anything. For error I selected an incorrect image in the flat frame list and I got this error after integrating all the bias frames. The waiting time of this error could be avoided if the geometry of all the images is checked at the beginning.


Regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 March 20 06:23:11
Hi again,

Could it be possible to process images from different filters at the same time? If so, could it be possible to specify the FITS keyword containing this information?


Regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 March 20 06:36:35
And another suggestion:

Remove the "Calibrate only" checkbox and put three: "Calibrate", "Register" and "Integrate". Along with these checkbox, put options to select the keywords to identify each image.

For instance, you can have images from a lot of filters which are calibrated and registered. With these options we can integrate each image with its corresponding filter. In this case, we would check only "Integrate".

Or you could have a lot of images from different objects which are only calibrated. You check only "Register" and "Integrate" and after the process you would have an image of each object in each filter.


Regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 20 09:35:59
I intend to calibrate 5 120" darks, 5 180", 5 300" and 16 600", with the option "calibrate only" enabled. When the sript  has integrate the tree fisrt sets of data, and intend to integrate the last set of 16 darks of 600" it appear in the console "integration pixel number row: 0 -> 1251 : 0%". The programm freezes here and crash,  saying "runtime error ... pixinsight.exe This aplication has requested the runtime to terminate in a unusual way".
Then I should to reinstall the sript. 
This crash happens allways;  nevertheless I can callibrate the last set alone without problem.
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 20 09:50:07
Hello:

when the sript (and PI) freezes the console says: "using 1233 concurrentpixel stack(s) = 1023,58 Mg"

I can calibrate the 4 sets of darks with a set of 9 600" images, instead of 16; (when the sript crashes the images are 16).
Is there matter of memory?

Regards
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 20 09:51:10
Hi again,

Could it be possible to process images from different filters at the same time? If so, could it be possible to specify the FITS keyword containing this information?


Regards,
Vicent.

Vincet,

If you use "add files" it should match up calibration images automatically by FITs key words already bin, filter etc.
We even had tested version that was matching a rotation key word ( to match flats with a rotator )

Multiple filters sets should process in parallel ok now.  Not all programs put in the same fits key words thats why there are other boxs to do it maunually. Right now they are using the MaxIm Dl standard.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 20 09:58:44
Btw, we should also have a manual method to match multilple image sets.
That is,  manually assign filters , bin, exposure etc so the they match up correctly for calibration and can be calibrated/ processed in one sitting.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 20 10:16:23
Hi again,

Could it be possible to process images from different filters at the same time? If so, could it be possible to specify the FITS keyword containing this information?


Regards,
Vicent.

Vincet,

If you use "add files" it should match up calibration images automatically by FITs key words already bin, filter etc.
We even had tested version that was matching a rotation key word ( to match flats with a rotator )

Multiple filters sets should process in parallel ok now.  Not all programs put in the same fits key words thats why there are other boxs to do it maunually. Right now they are using the MaxIm Dl standard.

Max

Sorry for the misspelling Vicent.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 20 13:05:48


BTW the bug / problem I had yestersay has gone , good work ;D

Harry
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 20 13:14:50


Quote
1) The script needs all file slots to be filled and a reference image selected. It should be possible to run the script with calibration frames only

The script requires you to specify a registration reference image only if the 'Calibrate only' checkbox is unchecked. If you check it, then no registration reference is (should be) required.

By 'all file slots filled' what do you mean besides the reference image?

I have found this in v02, it works as expected.

Quote
2) The reference image was selectable simply by clicking in the file list in 007h

It is selectable in 0.1, too, by double clicking one of the light frames. A single click is too prone to mistakes IMO, and we probably will implement more functionality associated with clicking frame items on the treebox structures.


This works too, I have overlooked the change.

Quote
3) Checking of the masterframe checkboxes select only the first frame in the list as a master. The script fails therefore to calibrate and align with predefined masters (for example with 4 channels LRGB).

If the L, R, G and B images have the correct filter identified in their FITS headers, there should be no problem. Can you verify this?.


The FITS keywords are not rewritten to the masters in v02 (function() doIntegrate

The last dev version has

*******************

// write FITS header in master frames
        if(result == true)
        {
         var img = ImageWindow.windowById("integration");
         var keywords = LoadFITSKeywords(frameset[0]);

         var postfix = "";
         for(var i = 0; i < keywords.length; i++)
         {
            for(var j = 0; j < FrameType[imagetype].length; ++j)
            {
               if(FrameType[imagetype][j].indexOf(trim(keywords.name)) != -1)
                  postfix += "_" + trim(keywords.name) + "_" + trim(keywords.value);
            }
         }

         img.keywords = keywords;

         path = checkDir(engine.savePath + "/masterframes");
         path = path + '/' + masterframe[imagetype] + postfix + ".fit";

           img.saveAs(path, false, false, false, false);
           img.forceClose();

         engine.addFile(path, true);
        }

*************************

to be able to reuse predefined masters simply by 'File add'

Quote
Of course we have to improve master frame management. How about an option to manually specify a filter for each frame? In this way we'd remove another FITS header dependence, and the script would be more flexible.

In my opinion an excellent option. For the last dev version I have had written a small dialog to add FITS key/values and thus to bypass the FITS header in a flexible way. This was based on sets of FITS keys in the global section ordered by mono/OSC and frame type. I have to clean up this a little bit, but it will be next weekend.


Best,

Kai

Title: Re: Calibration and align batch operations
Post by: sleshin on 2012 March 20 13:47:00
Just got around to downloading and trying the script, Juan's latest update. Works great and is a tremendous addition to PI. Many thanks to Kai and Juan for getting this done.

Steve
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 20 14:05:11
Hi

Often do not use darks , I use a bias the script will not let you select the same bias file for the Bias and dark >:(

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 20 18:21:43
New version 0.3 attached:

- New custom frames functionality.

- Improved error management with a cumulative 'error stack'.

- Regression fix: FITS headers are now correctly written to generated master calibration frames.

Stay tuned! :)
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 20 21:33:10
custom .... ?. How does this work?


max
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 21 01:14:31
Hi Juan,

New version 0.3 attached:
- Regression fix: FITS headers are now correctly written to generated master calibration frames.
Stay tuned! :)

Thank you very much for this fix. I will test the new version with another set of fresh data this evening. v02 is much more performant when integrating calibration frames. The integration into the PI framework is very interesting for me - do you have some documentation for the PJSR framework?

Best

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 21 06:31:02
V3 works well.

It would be helpful to keep the image list and settings the next time the script is run.
Please add button to deselect individual files too.

Still trying to figure out custom?

Max
Title: Re: Calibration and align batch operations
Post by: ruediger on 2012 March 21 09:36:36
New version 0.3 attached:
The following applies to DSLR workflow:

If you only want to calibrate bias files and don't check "calibrate only" on the lights tab, script terminates with error:
BatchPreprocessing/BatchPreprocessing-engine.js, line 1094: Error: File I/O Error: Invalid or empty file name
The "calibrate only" should be visible always and not only on the lights tab.

Forgetting to set output directory:
BatchPreprocessing-engine.js, line 1066: Error: File I/O Error: Invalid or empty file name

Both errors close the dialog, so all other settings are lost, which is annoying.

When debayering light frames, calibrated lights are also stored on disk in RGB format, overriding my prefererred CFA monochrome setting from Format Explorer, three times larger with no advantage compared to monochrome.

Speed up suggestion: instead of "for all lights { read; calibrate; write }; for all calibrated_lights { read; debayer; write }" shorter:
for all lights { read; calibrate; debayer; write }", thus eliminating unnecessary temp files and disk i/o. Does it matter? Well, yes... calibrated 5D II files are >170MB each in RGB :)

Already mentioned, but important and please don't forget:
Allow RAW directly on each tab, so no need to do BatchFormatConversion first


Keep up the good work!

Rüdiger
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 21 11:56:15
Hi

Still can not select the same file for Bias and Dark  :sad:

Harry
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 21 12:10:29
Feedback for version 2 (did not try version 3 yet):
- I saw a problem when I wanted to add light frames. Got output like "0 of 52 frames added". Second try worked fine. Could not reproduce this.

Wishlist (probably luxary, but nevertheless nice)
- is it possible to add make like functionality (i.e. if the calibrated and registered light already exists, and nothing else changed, dont repeat the process for this image)
- tabular output, in particular for registration. Might help to identify outliers.
- tabular output of FWHM for all lights. Important to filter out bad images.
- Would be nice if it could open an Image Integration process as a last step, with basic entries alfready done.

Georg
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 21 12:32:21
Would be possible a general option for "create masters" or so? (integration of bias, dark and flat, without the light frames to calibrate) Now the metod  seems a bit confuse   :surprised:


Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 21 13:57:07
Feedback for version 2 (did not try version 3 yet):
- I saw a problem when I wanted to add light frames. Got output like "0 of 52 frames added". Second try worked fine. Could not reproduce this.

Wishlist (probably luxary, but nevertheless nice)
- is it possible to add make like functionality (i.e. if the calibrated and registered light already exists, and nothing else changed, dont repeat the process for this image)
- tabular output, in particular for registration. Might help to identify outliers.
- tabular output of FWHM for all lights. Important to filter out bad images.
- Would be nice if it could open an Image Integration process as a last step, with basic entries alfready done.

Georg

I think the Blink Module is the place to add a FWHM function.
Images need to be fully reviewed and screened first.
We risk adding to much complexity to the script.

Don't get me wrong Georg. It would great to have this as part of PI. I think the blink module is the logical step to look, analyse and sort images.

I think Juan plans to add integration.

Max



Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 21 14:13:00
Would be possible a general option for "create masters" or so? (integration of bias, dark and flat, without the light frames to calibrate) Now the metod  seems a bit confuse   :surprised:

create masters should work, if the light tab is left blank. I am using this to generate masters for reuse.

Feedback for version 2 (did not try version 3 yet):
- I saw a problem when I wanted to add light frames. Got output like "0 of 52 frames added". Second try worked fine. Could not reproduce this.

Wishlist (probably luxary, but nevertheless nice)
- is it possible to add make like functionality (i.e. if the calibrated and registered light already exists, and nothing else changed, dont repeat the process for this image)
- tabular output, in particular for registration. Might help to identify outliers.
- tabular output of FWHM for all lights. Important to filter out bad images.
- Would be nice if it could open an Image Integration process as a last step, with basic entries alfready done.

Georg


Personally I think the script should do calibration and alignment only because this could be done with a set of parameters preset. Checking the calibrated lights should be done with another tool. The script dialogs are somewhat limited in interactivity due to their modal behavior.

Best,

Kai

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 21 15:15:53
Hi

I have just done a cal run on some Red subs all ok I then deleted the red flats and red lights and selected the green flats and green lights and the script fell over
say could not intergrate less than three subs ( flats ) !!!!!!!!

Happens every time

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 21 15:38:58
Hi

Ok  after the first run the use master ( flat in this case) box gets automaticaly checked to use the newley created master flat , if you do not uncheck this box first
before loading up new Flatsfor the second run  the script does not like it  :yell:

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 21 15:46:49
Quote
Personally I think the script should do calibration and alignment only

I completely agree. Integration of light frames requires trial-and-error and analysis work to achieve optimal results. It is a delicate task that cannot be scripted on a general basis, especially given the complexity of our ImageIntegration tool (which will grow in a near future)..

Quote
- is it possible to add make like functionality

This would be nice.

Quote
- tabular output, in particular for registration. Might help to identify outliers.

Also agreed.

Quote
- tabular output of FWHM for all lights.

As Max has noted, this should be part of a different tool (probably implemented as a C++ module, not as a script for performance reasons). I guess Nikolay is already working on such tool. This task should be performed to filter out bad images before calibration and alignment.

Thanks for the ideas!

To everybody: Please bear in mind that this script is in its early development stages---it is still version 0.3 after all!

For monochrome CCD and color OSC images the script is fully usable right now (besides particular problems and special requirements), and provides nice results. However, it is definitely still too crude for DSLR, although it can be also used without problems if bias and dark frames are converted to FITS before the script. We have decided to push the script hard to achieve a first release version as soon as possible.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 21 15:56:36
Replying to myself:

Quote
This task should be performed to filter out bad images before calibration and alignment.

Actually, after alignment to facilitate finding of stars on the images.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 21 16:06:23
No script for integration...I disagree :'(

Iteration should not be reason to avoid and an automated approach.

Besides I hoping you can find  a way to iterate with this script based on the aligned files.

Are the images written to a cache?
Can a ROI be implemented on a initial integration?

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 21 16:10:51
What does custom do?

Max
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 22 00:27:22
Continuing with wish list:

- When are asked for integration above, the idea was to get an IntegrationProcess generated with all the necessary files already added - not doing the actual integration. The standard dialog is perfectly usable to doing the repeated tries that are necessary.
- Region of interest: Especially for tests it is useful to be able to use only a sub-rectangle of the images (due to speed).
- DefectMap/Cosmetic correction generated from darks. Also note the the current PI processes cannot really be used with images that need to be debayered (because you cannot use the direct neigbor pixels to estimate a value for a defective pixel, and after debayer it is too late...)
- for DSLR: derive the category of a frame (bias, dark, flat, light...) from a naming pattern
- I am not sure I like the tabbed layout: no other piece of PI uses it. A single list that is structured like a tree is more like it, an may better support actions such as changing the category of an image.

It almost feels like Christmas  ;)
Georg
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 22 02:03:23
Kai and Juan
I will add another wish... The "add files" control structures great the files according the Fits headers; but I miss the control of "temperature" in the cooled regulated CCD cameras for the Dark and light frames.
It seems that the dark current varies enormously with the temperature of the sensor; so the control of the dark frames according the temperature seems important for an accurate dark sustraction.

Thanks to both, and now specially to Kai! (without jelousy of our master frame Juan)
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 22 02:12:41
Please also look into a cross-incompatibility to the ImageSolver script, see http://pixinsight.com/forum/index.php?topic=3966.msg27894#msg27894
Georg
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 22 08:33:06
When are asked for integration above, the idea was to get an IntegrationProcess generated with all the necessary files already added - not doing the actual integration. The standard dialog is perfectly usable to doing the repeated tries that are necessary"

Work for me but how do you handle the different filters generated from aligned calibrated subs?

Max

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 23 12:16:12


A whole day has gone by and no action  >:D

Harry
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 March 23 12:53:42
I hear the sound of Harry's foot tapping ..... ;D

Max
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 24 15:29:16

" days in a row and nothing  >:D , the stress is killing me  :surprised:

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 26 12:02:29
New version 0.3a attached:

- Fixed a problem with for ... in constructs due to new methods of the Array object defined as enumerable properties.

Information for developers

The problem: Adding new methods to the Array object with usual definitions such as:

Array.prototype.foo = function()
{
   // ...
};

is problematic because the newly defined methods are created as enumerable properties of Array. Subsequent for ... in constructs, such as:

var a = new Array;
// ...
for ( var i in a )
{
   // ...
}

will most likely fail because *all* enumerable properties of Array.prototype will be iterated along with the elements of the 'a' instance. For more information on the for ... in statement and its problems please refer to the following document on Mozilla's Development Network:

https://developer.mozilla.org/en/JavaScript/Reference/Statements/for...in

Quoted from the above document:

for..in should not be used to iterate over an Array where index order is important. Array indexes are just enumerable properties with integer names and are otherwise identical to general Object properties.  There is no guarantee that for...in will return the indexes in any particular order and it will return all enumerable properties, including those with non–integer names and those that are inherited.

Because the order of iteration is implementation dependent, iterating over an array may not visit elements in a consistent order. Therefore it is better to use a for loop with a numeric index when iterating over arrays where the order of access is important.


So the bottom line is: do not use for ... in in your scripts. I recognize that I use these constructs sometimes, so I have to apply this rule to myself. By the way, even worse than for...in is the 'with' statement, which is considered poison in ECMAScript-5:

https://developer.mozilla.org/en/JavaScript/Reference/Statements/with

The solution to this problem is defining the new Array methods with the Object.defineProperty function (https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Object/defineProperty), which is part of ECMAScript 5th Edition. This is what I have done in version 0.3a of the script. Fortunately, PixInsight's JavaScript runtime (= Mozilla's SpiderMonkey JS engine version 1.8.5) is fully compliant with the ECMA 262-5 standard.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 26 12:07:36
Hi

In this script , what are we applying to the flat , a bias or a scaled dark ?

Harry
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 26 12:10:46
Hi Juan,

thank you for this fix and the explanation. I have avoided for .. in because of unpredictable results.

Best,

Kai



 
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 26 12:13:32
Hi

Yes thanks Juan , ain't got a scooby what you said but sounds very important  ;D :-*

Harry
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 26 14:13:17
Hi Juan,

I have tested v03a with new data  -  works without any problems (QSI583ws, MaximDL)

Kai

Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 26 15:50:59
several days ago I wrote
Quote
I intend to calibrate 5 darks of 120" , 5 of 180", 5 of 300" and 16 of 600", with the option "calibrate only" enabled. When the sript  has integrate the tree fisrt sets of data, and intend to integrate the last set of 16 darks of 600" it appear in the console "integration pixel number row: 0 -> 1251 : 0%". The programm freezes here and crash,  saying "runtime error ... pixinsight.exe This aplication has requested the runtime to terminate in a unusual way".
Then I should to reinstall the sript. 
This crash happens allways;  nevertheless I can callibrate the last set alone without problem
Quote

Now it happens the same, except that isn't necessary to reinstall the script  :(

Thanks
Fco Bosch
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 27 00:22:19
Can you give some information about the operating system and system RAM?

Kai

Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 27 04:53:26
Hello Kai: my system is Windows 7, 32 bits, and my Ram is 4 Gb.
I think is a "normal" configuration in a laptop. The files are all of 16.327 kb of size, and are in total 31 darks.

Thans you!

Fco Bosch
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 27 05:12:47
Most likely memory shortage.
- Monitor memory usage using Windows Task Manager
- Try reducing the Buffer- and StackSize parameters
- Do yourself a favor, and upgrade to 64bits Windows. It helped a lot with my laptop, which is similar to your system.
Georg
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 27 07:36:11
But isn't posiible to manage these parameters with the sript ...  :(

Now I realize that a similar problem was reported month ago with image integration http://pixinsight.com/forum/index.php?topic=3877.msg26615#msg26615 (http://pixinsight.com/forum/index.php?topic=3877.msg26615#msg26615); and this http://pixinsight.com/forum/index.php?topic=3545.msg25044#msg25044 (http://pixinsight.com/forum/index.php?topic=3545.msg25044#msg25044), to that the first links; and others. The mine seems simply another instance of the same matter...
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 27 17:03:49
Hi there!

New version 0.4 attached to this post. This version is able to calibrate and align DSLR and OSC raw frames directly, without any previous conversion to the FITS format:

* Direct integration of CFA images (CR2, NEF, DNG, OSC, etc.) for master bias and dark generation. Requires the latest versions of the ImageIntegration and DSLR_RAW modules, which we have just released.

* Exposure times acquired from EXIF metadata for file formats different from FITS (e.g. DSLR_RAW and TIFF).

* New file list management functions: remove selected and invert selection (useful to handle master frames).

* Improved diagnostic information generation.

* Many minor bug fixes.

Please test it and report any bugs and problems that you find. Enjoy!
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 28 03:11:13
But isn't posiible to manage these parameters with the sript ...  :(

Now I realize that a similar problem was reported month ago with image integration http://pixinsight.com/forum/index.php?topic=3877.msg26615#msg26615 (http://pixinsight.com/forum/index.php?topic=3877.msg26615#msg26615); and this http://pixinsight.com/forum/index.php?topic=3545.msg25044#msg25044 (http://pixinsight.com/forum/index.php?topic=3545.msg25044#msg25044), to that the first links; and others. The mine seems simply another instance of the same matter...

I think a 64bit OS will solve the problems. You can also try to edit the script II parameters manually (see some pages below)

Hi there!

New version 0.4 attached to this post. This version is able to calibrate and align DSLR and OSC raw frames directly, without any previous conversion to the FITS format:

* Direct integration of CFA images (CR2, NEF, DNG, OSC, etc.) for master bias and dark generation. Requires the latest versions of the ImageIntegration and DSLR_RAW modules, which we have just released.

* Exposure times acquired from EXIF metadata for file formats different from FITS (e.g. DSLR_RAW and TIFF).

* New file list management functions: remove selected and invert selection (useful to handle master frames).

* Improved diagnostic information generation.

* Many minor bug fixes.

Please test it and report any bugs and problems that you find. Enjoy!


Excellent news. Use of image metadata directly from DSLR files will help in several other projects.

Best

Kai

 
Title: Re: Calibration and align batch operations
Post by: NKV on 2012 March 28 07:01:14
I guess Nikolay is already working on such tool. This task should be performed to filter out bad images before calibration and alignment.
Yes, it's true.  (http://pixinsight.com/forum/index.php?topic=3646.0)Also it will show field curvature in 3D.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 28 09:43:12
Hmm, Voronoi diagrams? What are you doing with Voronoi diagrams? :)
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 28 09:57:10
Hi

I can not get my images to calibrate correctley  :yell:
I am using master bias / darks prepared in astroart will this have a effect if so why ?

Regards Harry
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 28 10:59:41
Hi

I can not get my images to calibrate correctley  :yell:
I am using master bias / darks prepared in astroart will this have a effect if so why ?

Regards Harry

console log, error messages?

Kai

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 28 11:04:42
Hi

Sorry , should have made it clearer  :-[

The script works , just does not correctly calibrate my images ?

Harry
Title: Re: Calibration and align batch operations
Post by: NKV on 2012 March 28 11:05:57
Hmm, Voronoi diagrams? What are you doing with Voronoi diagrams? :)
I'm use Delaunay triangulation to building a grid of FWHM/roundness/etc in the image.
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 28 11:23:58
...Please test it and report any bugs and problems that you find. Enjoy!

Script works very nicely for me. I did not observe the change color balanced observed by someone else. My remaining wishlist:
- Integration of DefectMap/CosmeticCorrection
- Due to frequent memory shortage problems: You may want to make the integration parameters BufferSize and StackSize configurable. Even better: Let the user only specify maximum memory consumption, and let the script estimate good BufferSize, StackSize parameters from this and the number of images.
- some progress indicator (at least on the Processing Console, something like "5 of 99 images processed")
- you may want to reconsider the layout: most tools have a vertical layout, this script has a horizontal one. Since I arrange my image to fit the vertical tools, the script does not really fit in.

Fantastic work!

Georg
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 28 11:36:40
Hi

This is what I get , these files work in Astro art  >:D

Harry
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 March 28 13:54:14
Quote
- Due to frequent memory shortage problems: You may want to make the integration parameters BufferSize and StackSize configurable. Even better: Let the user only specify maximum memory consumption, and let the script estimate good BufferSize, StackSize parameters from this and the number of images.

I have these memory shortage and have followed the advices of Georg and Kai:

1) I have loweerd the values of BufferSize and StackSize
2) directly in the code of the script.

Now my "normal" configuration woks well (Win 32 and 4 GB) , without problems of memory, and for me isn't more matter.

But, do you think that a new user or one that aproaches for first time PI soll crak the sript? I think sincerely that isn't good for PI nor for the user (that most had a "normal" configuration).

Fco Bosch
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 28 14:15:28
But, do you think that a new user or one that aproaches for first time PI soll crak the sript? I think sincerely that isn't good for PI nor for the user (that most had a "normal" configuration).

I agree. Maybe it is possible to determine a default config for those parameters based on machine characteristics (such as available RAM, OS).
Georg
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 March 29 02:56:13
The script is still not released and windows xp 32 is not officially supported by PI (so far as I know). So I have only written these vars to the script code. To avoid overloading of the GUI components I have not yet exposed these vars by adding controls. Again I think a 64bit OS will solve the problems.

Hi

This is what I get , these files work in Astro art  >:D

Harry

Looks like no calibration at all. Is it the integration result of the /registered/ folder?

Kai

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 29 03:19:08
The script is still not released and windows xp 32 is not officially supported by PI (so far as I know). So I have only written these vars to the script code. To avoid overloading of the GUI components I have not yet exposed these vars by adding controls. Again I think a 64bit OS will solve the problems.

Hi

This is what I get , these files work in Astro art  >:D

Harry

Looks like no calibration at all. Is it the integration result of the /registered/ folder?

Kai

Hi

Yes they are the rgistered files I was using ?

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 29 04:24:17
Hi Harry,

This is what I get , these files work in Astro art  >:D

Are you using master calibration frames generated in other applications? In such case they probably won't work with our calibration tools due to incompatible floating point ranges. If you are interested in calibrating your images with PixInsight, then you should generate your master frames with PixInsight.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 29 05:14:03
Hi

I have now tried to create all master in pixinsight and still get the same result ???

I will upload sample if you like


Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 29 06:38:11
Could you upload a calibration set to PI Files?
Title: Re: Calibration and align batch operations
Post by: cornyyy on 2012 March 29 08:07:18
Hi Kai,
Hi Juan,

thanks for that skript. It worked well for me.
I did the update with the new DSLR-Raw algorhythm to use your new released skript. Unfortunately my calibrated and stacked pics became orange-blue instead of normal colors (see examples attached). I left all the settings untouched and tried the calibration with my M51-frames acquired last week.

So i really need help! Can you tell me something about the problem? How can i fix it or is it possible to undo the update to do the calibration and get results looking similar to my results last week?

THX a lot!
Cornel

PS: I´m using an EOS 20d mod.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 29 11:09:00
Could you upload a calibration set to PI Files?

Hi uploaded for your consideration

Harry
Title: Re: Calibration and align batch operations
Post by: erikgu on 2012 March 29 22:34:36

Hi and thanks for the script!

 I tested it on wedensday. Excellent and simple to use, but the result was not as expected.  I got the same orange-blue image as Cornel.  I am using a Canon EOS60d, unmodified. The script was executed with default settings.

My PC didn't start this morning,  >:D so a could not upload the image. I can, if of interrest when the pc is up and running.

Erik G.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 March 30 00:17:55
Hi Kai,
Hi Juan,

thanks for that skript. It worked well for me.
I did the update with the new DSLR-Raw algorhythm to use your new released skript. Unfortunately my calibrated and stacked pics became orange-blue instead of normal colors (see examples attached). I left all the settings untouched and tried the calibration with my M51-frames acquired last week.

So i really need help! Can you tell me something about the problem? How can i fix it or is it possible to undo the update to do the calibration and get results looking similar to my results last week?

THX a lot!
Cornel

PS: I´m using an EOS 20d mod.


Hi,

Which is the exposure of your flats?

Regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 30 02:50:43
Version 0.4a attached. I have added a 'signed-is-physical' input hint to image calibration, which should make the script compatible with 16-bit signed calibration frames generated by some applications.

Harry, could you please check if the script works now well with your raw data?
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 30 09:44:19
Version 0.4a attached. I have added a 'signed-is-physical' input hint to image calibration, which should make the script compatible with 16-bit signed calibration frames generated by some applications.

Harry, could you please check if the script works now well with your raw data?

Hi Juan

sorry still no luck  :'(

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 30 12:57:05
Quote
sorry still no luck

The only problem I can figure out with your raw data is signedness of raw frames. The 'signed-is-physical' hint fixes that, so if you still have the same problem after regenerating your master frames with version 0.4a, then I have no idea what's going on. I'd need you to upload several bias, dark and flat frames so I can generate the masters.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 30 15:04:48
Hi

Have uploaded a set of files  :D

There has always been a problem with Astro art files , even with the fits setting altered they sometimes still need rescaling to be displayed properley ?

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 March 31 00:25:48
Hi Harry,

Thank you for the calibration files. An additional 'signed-is-physical' hint was missing for the image integration process, which will be added in the next version. However, even with this improvement your calibration files don't work.

The main problem is in the bias frames. Your bias frames have median values of 0.03 on average, while your dark frames have 0.016. So bias frames are about twice stronger than darks in your calibration set. When your master bias is subtracted, it leads to severe clipping that destroys all the calibrated frames. There are no PEDESTAL keywords in your FITS files. A pedestal of about 1400 ADU would be necessary to prevent a massive amount of negative values after bias subtraction, but this is obviously wrong because it would lead to saturation of many bright image structures. Note that normal pedestals are always in the 50 - 200 ADU range, and are only intended to fix relatively small amounts of negative values after bias subtraction. Pedestals are seldom necessary in practice, in fact.

I don't know what happens with your calibration files. If this set is representative of the data you normally acquire, and you get good results consistently with the software that creates them, then there is something it does that we don't know how to do. Our calibration routines are working perfectly with data acquired with the rest of applications, including Maxim DL and Nebulosity. We use our ImageCalibration, ImageIntegration and StarAlignment tools all the time with data acquired with professional telescopes and CCD cameras without any issue. I'll keep trying, but right now I don't have any clue as to what's going on with your data.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 31 00:41:18
Hi

I do see there is little differance between the darks and BIAS ( hence why I do not use darks and why I want to use them for darks and Bias)

Also do not try and optimise the darks ( uncheck box ) this will not destroy the cal frames  :-*

In 16 bit numbers both maxim and Astro art see my bias at about ave of 1060 ?

So what do you think

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 31 00:45:43
Hi

Just checked the bias and dark in Pixinsight and as you mention it appears the dark and bias differance  is 2 but maxim and astroart do not see this so there must be someway they read the fits differently  :o

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 31 00:54:54
Hi

The dark has used bzero note , I think this is causing the problem over the bias which is not  :-*

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 March 31 01:05:16
Hi

One for you to look at PI will open the same  file diffentley if the fits option "signed integer" option is checked or unchecked even if there is no Bzero in the fits header


Harry
Title: Re: Calibration and align batch operations
Post by: ian straton on 2012 March 31 12:58:46
Hi, I'm new to PI so might need a bit of hand holding here, I tried using this script to calibrate and register some images I took recently however having loaded the file lists and hit run I got the following error message: "<file path> The DSLR_RAW format cannot perform incremental file read operations."  and the script failed.

I am using a canon 550d which outputs the cr2 RAW format.

Its not clear to me whether this is a bug in the script or whether I have to convert my RAW files to another format before using this script?
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 March 31 13:53:46
Ian, make sure you have the latest updates.
Title: Re: Calibration and align batch operations
Post by: ian straton on 2012 March 31 14:49:09
huh, PI updated itself earlier today but I just checked manually and there were two more updates available, installed those, restarted PI and now the script works! yay!
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 March 31 14:55:34
I'm having trouble with this latest version.  I took a bunch of lights, flats, and biases last night.  Already had a dark library.  All images taken with Maxim.  Ran Batch Preprocessing before I went to bed.  When I woke up there was an error, and was getting the little B&W pizza wheel in OSX PI so couldn't save the history.  Had to quit PI.

Started PI up again.  The bias, dark, and flat masters had been created successfully by the looks.  But the first calibration of light frames looks like where it failed.  Tried running Batch PP again using the masters this time.  Failed again with the attached error messages.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 March 31 18:51:26
Sorry to followup my own post.  Maxim has added parenthesis in the filename.  I think that may have something to do it.  I have a lot of files to rename, but will see if that fixes it.

Edit: Yep, renaming the files worked.  Sorry for the detraction.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 01 04:40:28
Hi,

I have been using this script and its predesessors for a while to calibrate and align the frames from my mono CCD. Compared to the routines in MaximDL, I used before, PIs tools gave much better results using the default settings in terms of noise and accuracy of alignment (no more colored and misaligned stars). The script was primarily written to reduce the time for preprocessing with PI.

Now I have tested Juan's wunderful integration and additions to use OSC and DSLR data. The script works without any problems and the final integration gives again much better results compared to DSS (chrominance noise, artefacts and outlier rejection) when using simply default settings. Thank you very much to Alejandro Tombolini for the DSLR test data - it was really a pleasure to work with it.

Best,

Kai



Title: Re: Calibration and align batch operations
Post by: cornyyy on 2012 April 01 04:52:13
Hi Kai, Vicent and Juan,

thanks for your help. I think i figured out the problem with the orange-blue color.

If i use the skript in total and doing the integration in final i´ll get this freaky colors.
If I check the calibrate only button, doing the debayering as next and finally the registration i´ll get normal collors. So I think the problem should be solved when changing the steps of debayer first and alignment as the second step?

Cheers Cornel
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 01 05:05:46
Hi

CCD stack, maxim and Astroart can calibrate my light with these frames , so Pi is not getting the correct information out of my Images , Why I leave to you  >:D

Regards Harry

Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 01 05:22:22
Version 0.4a attached....
This update is still working very nicely for me. Dont know what happens to Harry and some others. My wishlist:
- same as in http://pixinsight.com/forum/index.php?topic=3654.msg28079#msg28079
- the calibrated files contain FITS keywords generated from RAW, such as DATE-OBS, EXP-TIME. They are lost in the debayered files. I would like to see them transported to the debayered images (should be a property of the debayer process).
- I see the BITPIX value is "-32". Is this intentional?
Georg
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 01 16:51:34
IMPORTANT NOTE: This version of the Batch Preprocessing script does not work with the current core version (1.7.5.779) due to a bug. Please use version 0.4a of the script (http://pixinsight.com/forum/index.php?topic=3654.msg28135#msg28135) until we release a new version of the PixInsight Core application (tomorrow). The latest versions of the FITS and Debayer modules will also work with version 0.4a of the script.

New version 0.5 attached:

* Improved diagnostic messages.

* Added a FITS orientation option. The script preserves now the original orientation of most images by default.

* Several minor bug fixes.

Related updates:

* With the latest version of the FITS format module, the Batch Preprocessing script should be compatible with signed integer calibration frames generated by some camera control applications (i.e., Harry's data set works with this version of the script).

* The latest version of the Debayer tool preserves all existing FITS header keywords and adds a few COMMENT and HISTORY keywords with information on the applied mosaic pattern and debayering algorithm.

Hope you like it.
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 01 23:18:30
When trying to use the latest version (0.5, all updates applied), I got a message about something wrong with FITS when the console displayed
"Writing master bias frame:
C:/Users/Public/Pictures/Astro/2011_03_08_orion_85mm/fits/master/bias-BINNING_1.fit"
On a second run it worked...

I like the blue triangle!

Georg
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 01 23:42:27
Hi Georg,

Is this error something I have to worry about, or just a sporadic error? Have you seen a similar problem again?
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 01 23:52:17
Yes, I am afraid  >:D it is reproducible. Restarted fresh PI, did the run again. Error dialog is "FITS: Invalid Image Identifier".
Georg
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 02 03:32:06
Oops! I forgot that the current public version has a bug that breaks direct file generation in PJSR (the bug is naturally fixed in the development versions that I use). I'll release an update to the core tomorrow. In the meanwhile, please use version 0.4a of the script (http://pixinsight.com/forum/index.php?topic=3654.msg28135#msg28135).

Version 0.4a will also calibrate Harry's files with the latest version of the FITS module, and FITS keywords will also be preserved by the latest Debayer module.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 02 06:13:47
Great work Juan,

You and Kai are going to be very popular once this is released  :cheesy:

I hope you can add bad pixel map / cosmetic correction to the script. 
When I get ride of my bad columns first the rejection routines in integration are much easier to apply.

I was working with Kai to add a function to match flats for those of us with rotators. My rotator does a 180 on the other side of the merdian to capture the guide star automatically.  The flip requires flats to match the lights from each side of the merdian. 

We had a version that match the lights and flats with the Fits header automatically. It worked but a more general method would be perferrable. Could we add a function where we could manually select a sub group of lights and flats frames in the set?

Also is there anyway to add integration? I realise we need to optimise integration with iteration. Could we add a ROI and give the script the tools to integrate too?

Max

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 02 10:26:31
Hi

Ok fully tested with mono images and all is ok  ;D ;D ;D ;D ;D ;D :-* :-* :-* :-* :-* :-* :-*

The fits mod has also seemed to fix the other  outstanding problems I had loading images  ;D ;D ;D ;D :-* :-* :-*

So good this Improved version , go on tell us how it was improved  ;D

Harry

BTW great work , this will help PI greatly
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 02 10:50:31
Quote
go on tell us how it was improved

Shhh. I would have to kill you if I told you 8)

I am happy to know you can calibrate your images in PixInsight, at last ...  >:D

Quote
BTW great work , this will help PI greatly

Thank you so much. Seriously now, I agree and I am particularly happy in this case because this has been born as a community initiative. Kai has made a terrific and brave work starting out this project.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 02 10:58:45
Hi

Tested with OSC sxvf m25 and all is great  ;D :laugh: :cheesy:

I can't cope with things working  :o

Harry
Title: Re: Calibration and align batch operations
Post by: slang on 2012 April 03 13:28:47
Hi.

I have tested this several times with .cr2 output from an EOS450D, and it has worked SWEET. Very very good. I am very very happy.

Absollutely AWESOME.

Cheers -
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 03 17:11:33
Great work Juan,

You and Kai are going to be very popular once this is released  :cheesy:

I hope you can add bad pixel map / cosmetic correction to the script. 
When I get ride of my bad columns first the rejection routines in integration are much easier to apply.

I was working with Kai to add a function to match flats for those of us with rotators. My rotator does a 180 on the other side of the merdian to capture the guide star automatically.  The flip requires flats to match the lights from each side of the merdian. 

We had a version that match the lights and flats with the Fits header automatically. It worked but a more general method would be perferrable. Could we add a function where we could manually select a sub group of lights and flats frames in the set?

Also is there anyway to add integration? I realise we need to optimise integration with iteration. Could we add a ROI and give the script the tools to integrate too?

Max

Juan,

Again great work. Any thoughts on the above questions?

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 04 08:12:48
Matching a subset of flats to a subset lights could be helpful for situations other than a rotator.


You may have several sets of flats from different session dates to avoid dust doughnuts.

 The ability to link the flats with a set of light and sures beats running the script over for each set.

Max
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 04 23:41:04
Oops! I forgot that the current public version has a bug that breaks direct file generation in PJSR (the bug is naturally fixed in the development versions that I use). I'll release an update to the core tomorrow. In the meanwhile, please use version 0.4a of the script (http://pixinsight.com/forum/index.php?topic=3654.msg28135#msg28135).
The latest version works nicely with the new core released yesterday. I noted however that the orientation of the integrated image changed: Compared to previous versions, it is mirrored along the x-axis (no, I did not play with the FITS settings, source images are CR2). Dont know if this is a problem.

Most important missing feature for me is defect map handling.

Georg
Title: Re: Calibration and align batch operations
Post by: chris_todd on 2012 April 05 08:02:59
Hi.

I have tested this several times with .cr2 output from an EOS450D, and it has worked SWEET. Very very good. I am very very happy.

Absollutely AWESOME.

Cheers -

I can also confirm that this seems to work well on my Mac with the latest core PI updates I installed last night.  As a noob who has only been through the DSLR_RAW tutorial procedures a few times, I can say this script is very helpful.  Thanks!  52 minutes from Canon RAW to calibrated lights, nice!  Time for two glasses of wine (but then ImageIntegration gets all screwed up), LOL.  (2.5GHz Intel Core 2 duo, 4GB RAM MacBook Pro)
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 05 08:33:43
New version 0.6 attached:

 * Core version >= 1.7.6 required. Enforced with a conditional #error directive.

 * New automatic load/save settings functionality. The script now remembers most working parameters across sessions. Note that this does not include file lists, since in my opinion file lists should be managed with process icons.

 * New Reset Preprocessing Engine dialog. This dialog opens by clicking the Reset button. It allows you to select whether you want to clear everything (including file lists), load stored settings since the last session, or reset parameters to factory-default settings.

Thanks to everybody for the nice words. I'm glad you find this script useful.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 05 10:13:01
Hi

where did 0.5 go  >:D

And I will join in on the defect map chorus  ;D

Harry
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 05 13:56:36
New version 0.6 attached:

 * New automatic load/save settings functionality. The script now remembers most working parameters across sessions. Note that this does not include file lists, since in my opinion file lists should be managed with process icons.

 

How does a file list become as process icon?
I am not sure everyone will automatically how this done.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 05 14:10:54
Hi

where did 0.5 go  >:D

And I will join in on the defect map chorus  ;D

Harry

I was hoping Juan would engage in discussion what features he might consider adding before release. 

I don't see integration happening since Juan does think it will work effectively. I respect this point of view.

However, it seems like such as waste of time to have all those registered images flowing through memory to disk. Openning many images takes my system quite a bit of processing time.

I think there could be clever way load them into the integration module. Perhap it is not a real time saver

.... Juan is ignoring me  :-\

Max


Max
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 06 08:23:11
Latest version 0.6 works well.
Problems:
- when selecting the output directory, I get a very long dialog, see screenshot.
- By default Up-bottom FITS is activated. This also flips the orientation when working purely with RAW (.CR2 files). Unexpected.

Wishlist:
- Defect/Cosmetic correction handling
- some way to control the memory consumption of the embedded image integration processes (see some posts above for proposals)
- improved dialog layout

Georg
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 06 11:36:29
Latest version 0.6 works well.
Problems:
- when selecting the output directory, I get a very long dialog, see screenshot.

The problem is in line 1445 in the GUI section

this.dialog.outputDirEdit.text = engine.outputDirectory = this.text = dir;
should be
this.dialog.outputDirEdit.text = engine.outputDirectory = dir;

Kai



Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 06 12:31:36
Quote
this.dialog.outputDirEdit.text = engine.outputDirectory = this.text = dir;
should be
this.dialog.outputDirEdit.text = engine.outputDirectory = dir;

Ouch! I'll release a fixed version ASAP. Sorry!

As for including cosmetic correction in the script, I am not sure. How would you implement this feature? What parameters and/or working mode(s) of the CosmeticCorrection tool (or other tools) should be included?
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 06 13:15:02
The script was originally written to batch call the powerful IC, II and StarAlignment tools as a simple wrapper to save some time. Image integration of calibrated and aligned light frames is one step where it is necessary to fine tune parameters of the II module and therefore the final integration step is not part of this script. If the CosmeticCorrection tool (unfortunately I have not used it yet) requires fine tuning of a set of parameters, inclusion into this script should also be avoided. If this is an simple automatic task it may be OK.   

Kai

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 06 13:23:55
I am just looking for a simple way to fix a bad columns and hot pixels by averaging with neigboring pixels. I have not played with the defect map module and cosmetic correction module to know what to suggest.

MaxIM DL uses a dark frames with thresholds to detect hot/cold pixels, clusters and make a map. Also bad columns or allows the user to manually entery coordinates or use a mouse. 

Once a defect map is generated a median of surround pixels works.

A non value as a place holder to mask the data for the integration stack would be even better.

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 06 13:30:38
Once a map is made it require little or no intervention. Settings can be pretty simple to work nicely.

I find Maps can generally be used for a year or more. Most cameras age,  developing new bad pixels over time. It is slow process mediated by cosmic ray hits.

Max

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 06 14:04:00
Hi

A defect map is a must simple to do  :-*  if there is no defect map the script becomes a little usless  :surprised:

Harry
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 06 15:37:04
I think a defect map would be sufficient. It needs to be applied to the lights after calibration and before debayer and register, needs to be able to handle CFA images, and should be able to handle multiple neighboring defect pixels. The current DefectMap tool lacks CFA capability, the CosmeticCorrection process has CFA and works nicely using the Auto Detect filter...probably similar to what DSS does (but I could not get MasterDark based defect detection working for some reason).

Defect map handling can be done with the current tools, but since it needs to be done right in the middle of the calibration process, it is not easily integrated into the flow of the current script.

To illustrate the need: I usually need defect maps for my wide fields, because my mount does not do enough automatic dithering (otherwise called periodic error/tracking error  :-[) for short focal lengths. Integration of the calibrated stack clearly shows hot pixels. When I activate rejection to a sufficient degree, I loose SNR. See screenshot of integration of 52 lights, each 120 seconds@85mm:
- left: stacking without rejection and defect map: High SNR, but traces from hot pixels in red and green
Gaussian noise estimates:
?sR = 1.076e-004
?sG = 7.923e-005
?sB = 7.929e-005
Reference SNR increments:
??s0R = 4.9153
??s0G = 6.2299
??s0B = 6.2170
Average SNR increments:
??sR = 4.9063
??sG = 6.3918
??sB = 6.3783

- center: stacking with linear fit rejection (sigma high=3): Lower SNR, no traces. Also relatively slow (2353 sec)
Gaussian noise estimates:
?sR = 1.317e-004
?sG = 8.875e-005
?sB = 8.991e-005
Reference SNR increments:
??s0R = 3.9821
??s0G = 5.5467
??s0B = 5.4732
Average SNR increments:
??sR = 3.9747
??sG = 5.6908
??sB = 5.6152

- right: stacking with defect map, using Cosmetic correction with Autodetect, Hot Sigma 3): Higher SNR, no traces, fast (281 secs)
Gaussian noise estimates:
?sR = 1.076e-004
?sG = 7.923e-005
?sB = 7.929e-005
Reference SNR increments:
??s0R = 4.9153
??s0G = 6.2299
??s0B = 6.2170
Average SNR increments:
??sR = 4.9063
??sG = 6.3918
??sB = 6.3783
Georg
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 April 06 20:59:51
Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 07 03:45:53
Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.
The script selects the most exposed dark frame when processing a set of light frames with different exposure times.

I think a defect map would be sufficient. It needs to be applied to the lights after calibration and before debayer and register, needs to be able to handle CFA images, and should be able to handle multiple neighboring defect pixels. The current DefectMap tool lacks CFA capability, the CosmeticCorrection process has CFA and works nicely using the Auto Detect filter...probably similar to what DSS does (but I could not get MasterDark based defect detection working for some reason).

Defect map handling can be done with the current tools, but since it needs to be done right in the middle of the calibration process, it is not easily integrated into the flow of the current script.


Including a call to the CosmeticCorrection module with 'Use Auto detect' may be possible. Adding all parameters of this complex module will heavily bloat the script. So it may be better to use the batch script to generate master frames and calibrate the light frames only and thereafter do further optimization manually.

Kai

Kai
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 April 07 04:07:54
Just noticed a slight bug in the FITS header interpretation.  I have 16 Ha FITS files.  The first one is 480s exposure time, the other 15 were all 240s.  The BatchProcessing scripts interpretted them all as 480s if I select the first one in the same group as the other 15.  If I select all except that first one, the script correctly interprets them all as 240s.
The script selects the most exposed dark frame when processing a set of light frames with different exposure times.

I see.  Shouldn't the script separate or group the light frames according to exposures similar to binning or filters?  Then apply the calibration in those groupings rather than assuming all the same?
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 07 04:59:12
Hi

Other apps just let you create a master defect map and use this in its calibration , Nothing complicated is required here  :-*

Harry
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 April 07 05:50:17
Not sure if you're replying to me, Harry? I thought the BatchPreprocessing script just used scaled darks. I haven't used defect maps to date as far as I'm aware.
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 07 10:58:41
Including a call to the CosmeticCorrection module with 'Use Auto detect' may be possible. Adding all parameters of this complex module will heavily bloat the script. So it may be better to use the batch script to generate master frames and calibrate the light frames only and thereafter do further optimization manually.
A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 07 12:52:44
A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

So, we need a DefectMap module with CFA capabilities and the defect map is selected by the user?

Kai

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 07 12:55:49
A defect map is the minimum that is necessary. Everything else is luxary on top of the basics.
Georg

So, we need a DefectMap module with CFA capabilities and the defect map is selected by the user?

Kai

Yes Please

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 07 12:59:12
New version 0.7 attached:

 * Added batch cosmetic correction functionality.

 * CSV text files are now selectable as registration references -- useful to register against a synthetic star field.
 
Cosmetic correction works by selecting an existing CosmeticCorrection process icon. The script automatically finds CC icons and presents them in a combo box. Additional diagnostic checks have been also implemented to detect most related pitfalls. I am sure some of you won't agree with this system (Harry, I'm hearing you), but as I see it, this has three important pros:

- It allows fine tuning CosmeticCorrection using the nice real-time functionality that Nikolay has implemented.
- It allows for much more refined and accurate corrections than a simple defect map.
- It does not clutter the script's interface.

and just one con:

- It is not a simple one-parameter thing.

But look, for simple one-thing things there are other applications out there. We are not that way. Please check it and let me know if it works.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 07 13:02:49
Hi

Cool I will have a go , I am not adverse to better it was that it " seemed to be a problem "

BTW whats all this listening to users  >:D  " Tiger and spots "   O:)

Harry 
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 07 13:05:55
I forgot to mention something important. The CosmeticCorrection icon used in the script is just a template to provide functional parameters; it does not need to be defined completely. In particular, it does not need to have a set of target frames defined---in fact, existing target frames in the icon will be ignored and the script's calibrated frames will be used instead.

The same happens with the output directory, the output postfix, and the CFA option of CC; the script will force these parameters to the corresponding subdirectory in the calibration tree, to the string "_cc" (no prefix), and to the state of its "CFA images", respectively. Also, the executed CC instance will overwrite possibly existing output files.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 07 13:11:30
Quote
Cool I will have a go

Great, let me know if you like it.

Quote
BTW whats all this listening to users

You know we love you  O0
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 07 13:30:46
Hi

Getting this , will not run

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 07 15:31:42
This happens because your light frames have the keyword FILTER=Ha1.00 in their headers, but your flat frames don't. The script tries to match sets of images acquired with the same filters.

The solution is easy though: add your flat frames as custom files, and specify: image type = flat, filter = Ha1.00. It works fine this way (I've checked).
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 07 17:13:00
New version 0.7 attached:

 I am sure some of you won't agree with this system (Harry, I'm hearing you),

Your the boss Juan ;) 


When is better a bad thing?  If we wanted mediocre we all would be someplace else.

Seriously, I have not tried it yet but sounds great.

Max
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 07 20:05:35
Hi

The script did not do this before  :surprised:  Not sure this is an improvement

I thought add files was going to look at the fits header !!!

Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 08 02:37:20
Hi

The script falls over a lot when selecting files , I will try and repeat it >:(

Also I hate having to make sure all files need the correct name header etc  :yell: :yell: :yell: :yell:

But the cosmetic thing works quite well  :-* :-* :-*

Regards Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 08 03:00:44
Hi

If you use custom to add more than one type of file it corrupts the script !!

when you use custom to say add a flat , make sure you have the flat tab selected as it puts the file in the selected tab ?? so if you have the darks selected it puts in in there  :-[

I understand why you want to make a script which trys and calibrate all subs in one go , but as I said ages ago let me decide which files to use , why do we need custom !!! just have a tick box to force the script to use selected files ( after a user warning if you like )

Harry
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 08 05:29:56
New version 0.7 attached:
 * Added batch cosmetic correction functionality.
 * CSV text files are now selectable as registration references -- useful to register against a synthetic star field.
...
 
This is working beautifully for me. Look at this (http://otaku-madnes.deviantart.com/art/bunny-nebula-185888563) beautiful result  ;)

Seriously, I have never been able to produce a decent and reproducible calibration with PI, even when following all those recipes posted on this forum. I really don't know what I did wrong again and again, there must be some magic ingredient  (http://pixinsight.com/forum/index.php?topic=3654.msg28285#msg28285) that is now encoded in those scripts. To give you an example, have a look at the screenshot: The best I had been able to do with a wide field from the Grand Canyon until 2 weeks ago was the left hand image - this was after days with PI, and I still had not figured out how to get rid of those hot pixels. I figured out how to eliminate the hot pixels last week only last week, after another 5 hours of experiments. Now the whole job including the hot pixels runs in just 1.5 hours after a couple of mouse clicks, see right hand side.

Thanks a lot for the script  :-* .

I am probably in danger of stretching my luck, but may I propose 3 additional functionalities for the script?

- for non-fits files, add a functionality that determines the image type based on the name (bias*, flat*, dark*,light*), similar to what you do based on keywords for fit files in StackEngine.prototype.addFile . Saves a lot of mouse clicks for those poor DSLR users.
- add a functionality that does an inital integration. I know that ultimately I have to tweak the integration parameters by repeating those runs, but at least the script would provide a basic result when I wake up in the morning (and would have filled the file cache). Also, this would bring the script to the same functionality level as DSS - which I would finally retire. Maybe you can choose the same route as for CosmeticCorrection: let the user provide a process icon for integration.
- I know that the script cannot provide all the functionalities that the basic processes can. For those of us that want to fine tune the calibration after a first run of the script, it would be helpful if the script were able to produce the basic process icons it used to do its job. These could be used as a starting point for the manual tweaking process.

I am a happy man today  :),
Georg
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 09 07:13:57
I processed a 150 images last without any problems.  :)

A couple notes:

Cosmetic correction works great but I don't see why this is greyed out when calibrate only is checked. It may be desirable to produce calibrated and Cal_CC images only at times.

Dark Master flat file name : Can temp be added to the name for clarity?

I think retention of paths would have helped several times last night.
I had a trucated file that caused an error. I had go back a reselect everything from scratch just to remove the single bad image file and start over.

Also, I split my work into two script runs since I had multple nights. The weather got warmer so the lasts few nights were performed at -30C  rathar than -35. I was using the same masters except for the dark. Also, I was aligning to the same image and save file to the same directory. If it retained all the paths it would have been easier.

My last request:
It would be very helpful to select a flat and light subsets. This is for nights with a different flats set or rotator meridan flips as I explained previously. It could be done through the custom button I think.

Max







Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 10 08:19:44
Version 0.8 attached:

* Optional integration of registered light frames (master light frame generation). This is enabled by default, and there are specific integration parameters for lights. Take into account that this is only intended as a preview of the final image, not as a final result. A careful manual integration process is always required in order to get the best possible image out of the calibrated and registered data.

* Rejection maps are now stored with the master images in multiple image FITS files (using IMAGE FITS extensions). This option is now enabled by default. Since the ImageCalibration tool does not support multiple image files, rejection maps are only stored for master lights, not for master bias, dark and flat frames, by this version of the script. This will be fixed soon with a new version of the ImageCalibration module.

* For FITS files without IMAGETYP keywords and other file formats, the script can now infer image types from file names with 'bias', 'dark', 'flat' and 'light' substrings (case insensitive). For FITS files, keep in mind that the IMAGETYP keyword has precedence over file names, so if the keyword value is wrong the selected image type will also be wrong, irrespective of a correct file name.

* Improved FITS header generation. More complete information on image integration results is now stored as FITS keywords.

* Some GUI refactoring.

Hope you like it!
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 10 08:54:54
...Optional integration of registered light frames (master light frame generation).
...can now infer image types from file names with 'bias', 'dark', 'flat' and 'light'
:-* Did I mention that I love you?  :-* Will you marry me?  ;)

* Rejection maps are now stored with the master images in multiple image FITS files (using IMAGE FITS extensions).
I am not sure that rejection maps for master images are really necessary. I believe it is sufficient to fix bad pixels in the calibrated lights only. Or what are they good for?

Fantastic. Never expected this in my wildest dreams.
Georg
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 10 08:59:20
Hi

Before I start winging again  >:D I would like to say thanks for this , a much needed item In Pixinsight

Now winge begin

I still can not select the same file twice ( i.e master bias instead of darks )   still hate I have to mess about with file names  :yell: :yell: :yell: :yell:

Small bug -the scroll bar on the consol disapears after running the script  resizing the consol makes it reapear  and could the final stack be left on the screen

for inspection

Much Love from your PI users

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 10 11:25:33
Quote
Will you marry me?

You already married me when you bought your PixInsight license  >:D

Quote
I am not sure that rejection maps for master images are really necessary.

For master bias and dark frames, probably not. For the master flat frame it may be necessary, especially for sky flats (rejection of stars). Perhaps we should have two independent options for generation of rejection maps, one for lights and the other for calibration frames?
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 April 10 11:39:13
I am right there with you all, trying this amazing piece of work as it gets hammered out. Fantastic, just fantastic, so much easier to set up a run and then analyze the results.

I had an OSX bug on the last version (will check this one shortly) where the Output Path changed into a clone of the Image Registration Reference path after a dialog box was worked with.

This will be fixed soon with a new version of the ImageCalibration module.

Looking forward to this new version. A while back, Vicent mentioned something about using Multiscale within ImageIntegration to take care of moving dust donuts. Any chance that this will be in there? I have some very challenging frames that flats are having a hard time with.

Edit: Ah, this new version will not work on Macs because Cosmetic Correction is not compiled for OSX yet, right?
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 10 11:41:14
Quote
I still can not select the same file twice ( i.e master bias instead of darks )

I don't understand why you want to do this, so please clarify. If you don't use dark frames, simply don't select any dark frame and the script will generate and apply just a master bias. Or is there something else that I am missing?

One of the purposes of this script is to simplify some tasks by making it very difficult to make mistakes. Duplicate frames and incorrectly selected frames are two common pitfalls. I don't see any real advantage in allowing them.

Quote
still hate I have to mess about with file names

One of the key elements of any good data reduction procedure is keeping all the data well organized. The script tries to enforce the best possible organization and file structure. Correct file names and frame types stored in FITS headers are essential items; this is a very basic requirement for any decent image acquisition software IMO.

With the latest version of the script you just have to be a bit careful to name your images: just include the 'bias', 'dark', 'flat' and 'light' bits as appropriate in the file names and everything should work automatically.

Quote
Small bug -the scroll bar on the consol disapears after running the script

Yes, I have seen this. It is a small bug in the PI core application, not in the script. I'll fix it in the next core update.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 10 12:15:51
Hi

I did not know about not needing to select a dark if not needed  :-* the warning message put me off  :laugh: so I will stop winging about this  :-[ ;D  but info in the warning might help like " No Dark frames selected If you continue a master bias will be used instead "  Or the like

But the bit about having the word " Flat" in the file name does not work though  >:D   

Regards Harry
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 April 10 12:29:45
With the latest version of the script you just have to be a bit careful to name your images: just include the 'bias', 'dark', 'flat' and 'light' bits as appropriate in the file names and everything should work automatically.

Out of curiosity, how does the script handle weird filenames with multiple filter labels in them? It's bad practice, but I'm working with a guy who gets filenames like this out of his automated observatory software:

GreenFilter_Red001.fits    or    DarkFrame_Bias7.fits

It drives me crazy while organizing and I just end up batch renaming all the files anyway, but if this script will pull filename and/or FITS header info, I can just load these messy files into the hopper and let the script organize them, yes? It's bad practice, but it'll save me a lot of time when working with his files.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 10 12:34:50
This happens because your light frames have the keyword FILTER=Ha1.00 in their headers, but your flat frames don't. The script tries to match sets of images acquired with the same filters.

The solution is easy though: add your flat frames as custom files, and specify: image type = flat, filter = Ha1.00. It works fine this way (I've checked).

Ahhhh. Great.

If I understand this correctly,  I could using this match to flats for the meridian flip with a rotator etc????

Max


Title: Re: Calibration and align batch operations
Post by: bitli on 2012 April 11 00:32:53
Just one think:

Thank you very much for this script.

And congratulation to the new weds  :)

-- bitli
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 11 04:29:41
Quote
And congratulation to the new weds

Well, this script is more to prevent divorce before marriage. Some bridegrooms were getting a bit 'aggressive'...  ::)

Thanks! :)
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 11 07:13:30
Boy I must be divorced.

I can't even get a response out of Juan.  :sad:  :sad:

No respect. No respect at all.  ::)

Max

Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 11 07:40:11
Quote
No respect. No respect at all.  ::)

 :angel:

Quote
I could using this match to flats for the meridian flip with a rotator etc?

I think so. Just use a 'virtual filter' of your choice to match flipped flats with flipped lights added as custom frames. Should work ;)

Quote
Cosmetic correction works great but I don't see why this is greyed out when calibrate only is checked. It may be desirable to produce calibrated and Cal_CC images only at times.

Agreed. I'll see how to change this in the next version.

Quote
Also, I split my work into two script runs since I had multple nights. The weather got warmer so the lasts few nights were performed at -30C  rathar than -35.

Why? Unless there are significant seeing differences (in which case the first thing to do is to remove bad frames from the whole set), out dark frame optimization routine will scale your darks automatically. So I think you can calibrate/align/integrate the whole set without problems (and you'll get a higher SNR improvement by integrating more frames in the same stack).
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 11 09:28:45
Thanks Juan,

It all sounds great.  Looks like Kai and you made my preprocess pipeline dreams come true.
 :)

Quote
Also, I split my work into two script runs since I had multple nights. The weather got warmer so the lasts few nights were performed at -30C  rathar than -35.

Why? Unless there are significant seeing differences (in which case the first thing to do is to remove bad frames from the whole set), out dark frame optimization routine will scale your darks automatically. So I think you can calibrate/align/integrate the whole set without problems (and you'll get a higher SNR improvement by integrating more frames in the same stack).
[/quote]

I guess I don't understand the optimization routine fully. I aways thought that darks and lights of the same temp were more accurate to scale rather matching different temps,  -30 and -35 etc.

On my current camera it accumlates more 5-10x radioactive/ray hits due a kodak manufacting problem with the coverslip. Otherwise I might just take a long exposure a each temp.

I usually take a set of 25 darks at the same exposure and temp. as the lights. I am doing to much work?

(the original comment was about retaining paths I think?)

Max
Title: Re: Calibration and align batch operations
Post by: sleshin on 2012 April 14 15:41:59
Just tried latest version and when "calibrate only" is checked, light frames are not calibrated. Loaded Bias, Darks, Flats, and uncalibrated Lights. The flats were calibrated and the master Bias, Darks, and Flats were properly done. But the script stopped there and didn't calibrate the lights. With "Calibrate Only" unchecked all works fine. Lights are calibrated and registered. Didn't enable Cosmetic Correction or Image Integration so can't comment on that.

Steve
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 15 06:00:40
minor problem

lines 650, 651 (engine.doLight)

if ( this.calibrateOnly )
      return;

should  be moved in front of line 744

... move here ...

      var SA = new StarAlignment;

and the CalibrateOnly checkbox works


Kai


Title: Re: Calibration and align batch operations
Post by: sleshin on 2012 April 15 07:30:25
Thanks Kai. Is a corrected version of the script available somewhere?

Stevem
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 April 15 09:33:00
One *very* minor bug -- a misspelling of 'light' as 'lihgt' (or similar) in one of the error messages (if you don't have flat frames selected to calibrate a certain filter of light frames).
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 15 11:50:50


Is 0.9 ready yet  >:D

Harry
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 17 17:33:28
Let's finish this puppy  ;)
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 April 18 13:22:17
I think I found a bug in version .8. 

Here's how I can reproduce it.

If you exit the script and then use the process icon, you will note the files are not present.  If the option "Export calibration files" is now checked, you add the files again and save a new process icon everything is saved properly.

Craig
Title: Re: Calibration and align batch operations
Post by: erikgu on 2012 April 19 02:08:29

Hi, i did a script test yesterday where i included Cosmetic correction. Everything worked perfectly. Previouse teste gave som strange colors, now is everthing ok. Thanks to the excellen work that has been done.

One question, is the processing log saved or is it only displayed in the processing window? If not saved, would it be possible to the save it, could be controled as an option, the location of the log file could be the output reference directory.

Erik G
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 19 10:21:16
As far as I know it is the same as any other PI operation which only shows what happened in the processing window.

I don't know if there is a way to record the processing window output into a file.

Max
Title: BatchPreprocessing goes official
Post by: Juan Conejero on 2012 April 19 10:37:09
Hi there,

We have released version 1.0 as an official update a while ago. From now on this script is part of the official PixInsight distribution and will always be released through the update system. For the convenience of developers, I have attached the complete source code to this post.

Version 1.0 comes with the following features:

- Rejection maps stored as multiple image files for all master frame types (not just for master light frames), thanks to a bugfix in the ImageCalibration process (released as an independent update, also today).

- When appropriate, the cosmetic correction and deBayering tasks are applied irrespective of the 'calibrate only' state. When 'calibrate only' is enabled, only the image registration and integration tasks are not executed.
 
- Prevent errors caused by invalid file name characters in filter names when used as FILTER postfixes. For example, an 'exotic' filter name such as "DLR/2{45}[1]" is now represented as "FILTER_DLR_2_45_1_" in the file name of the corresponding master frame.
 
- Several bug fixes, including the confirmed bugs that have been reported on this thread.

- The script's change log is now released as a separate text file.

Congratulations to Kay Wiechen and to all the users supporting this great community effort.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 19 11:06:50

Hi

Many thanks to both of you  :D I am really sure this will be a boon to pixinsight and will stop people bashing my ears  >:D

I know its simple to use but I will do a video to add to the list  :-* :-* :-* and  :-* :-* :-*


Harry
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 April 19 13:10:52
I agree with Harry, a big thank you to both Kai and Juan.  But I'd like to also thank everyone involved with pushing this along to completion.  The PI community is great and I'm glad to be a new member. 

This is a terrific addition to the PI tool set and will be very helpful to new users and experts alike.  Automation while retaining customization is always tricky, but is always welcomed on repetitive tasks like this.

Great work on making this happen!
Craig
Title: Re: Calibration and align batch operations
Post by: Josh Lake on 2012 April 19 17:06:41
Congratulations and thank you, this one is truly excellent. It has already saved me hours of busy work and organization, and I love it when it 'just works'.

While I still think it's profitable for new users to go through the intense calibration process manually a few times to see what's going on, this is a real boon to PI's loyal fans and users.
Title: Re: BatchPreprocessing goes official
Post by: balt on 2012 April 19 19:33:26
Hi Juan and Kai,

excellent idea, however, when I try to run it on my OS X 1.7 (latest) version, the script balks stating I need version 1.7.6. So I logged into the downloads website thinking that there might be a new beta version, but there isn't?

Cheers

- Balt
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 20 06:40:03
This script deserves a formal release annoucment. This should include an email with some notes on its features and set up.
It is really a major advancement a needs proper attention.

This will be a big step foward for a great many users.

Thanks to everyone who help make this happen

Max
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 20 07:00:53
+1
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 April 20 08:51:35
bug report: 20 DSLR images, Ha filter. calibrated with master dark and master bias only, no flats.

1) is the default value of the fits direction flag in the script consistent with the pixinsight fits reader default direction? my master calibration frames gave me the dreaded "no correlation between dark and light frame" until i unchecked the fits direction box. all my masters had been previously prepared by hand and i don't think i've changed the fits reader settings.

2) the calibrated and debayered frames could not be registered by the script, however, if i put all the calibrated/debayered frames into StarAlignment manually, they were registered with no problem using the default StarAlignment parameters. i can only guess here that the script is calling StarAlignment with some different set of parameters, possibly the noise thresholds.

should i upload the CR2 files to the Wookie planet?
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 20 08:56:02
bug report: 20 DSLR images, Ha filter. calibrated with master dark and master bias only, no flats.

1) is the default value of the fits direction flag in the script consistent with the pixinsight fits reader default direction? my master calibration frames gave me the dreaded "no correlation between dark and light frame" until i unchecked the fits direction box. all my masters had been previously prepared by hand and i don't think i've changed the fits reader settings.

2) the calibrated and debayered frames could not be registered by the script, however, if i put all the calibrated/debayered frames into StarAlignment manually, they were registered with no problem using the default StarAlignment parameters. i can only guess here that the script is calling StarAlignment with some different set of parameters, possibly the noise thresholds.

should i upload the CR2 files to the Wookie planet?

yes, please upload these files.

Kai

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 20 11:13:08
This script deserves a formal release annoucment. This should include an email with some notes on its features and set up.
It is really a major advancement a needs proper attention.

This will be a big step foward for a great many users.

Thanks to everyone who help make this happen

Max

Hi Agree

I am writting the video and will have it done tommorow, perhapes a small delay till then  :surprised:

Harry
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 20 11:53:36
I'll wait for the movie then.

Max 8)

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 20 16:45:43
You might make another video on the blink module
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 April 21 08:47:00
bug report: 20 DSLR images, Ha filter. calibrated with master dark and master bias only, no flats.

1) is the default value of the fits direction flag in the script consistent with the pixinsight fits reader default direction? my master calibration frames gave me the dreaded "no correlation between dark and light frame" until i unchecked the fits direction box. all my masters had been previously prepared by hand and i don't think i've changed the fits reader settings.

2) the calibrated and debayered frames could not be registered by the script, however, if i put all the calibrated/debayered frames into StarAlignment manually, they were registered with no problem using the default StarAlignment parameters. i can only guess here that the script is calling StarAlignment with some different set of parameters, possibly the noise thresholds.

should i upload the CR2 files to the Wookie planet?

yes, please upload these files.

Kai

i uploaded my master bias, master dark, and 4 CR2 files to the "forum shared files" area, under the folder called "pfile_unsolvable_image". i wanted to change the name of the folder but it seems that you can't. anyway have a look at those files. i have tracking problems so please don't laugh at the star shapes :)
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 21 08:53:27
.... i have tracking problems so please don't laugh at the star shapes :)
I can assure you: You are not alone   :)
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 21 12:14:47
Hi

Video as promised   http://www.harrysastroshed.com/Calibration.html (http://www.harrysastroshed.com/Calibration.html)

Harry
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 April 21 12:47:44
bug report: 20 DSLR images, Ha filter. calibrated with master dark and master bias only, no flats.

2) the calibrated and debayered frames could not be registered by the script, however, if i put all the calibrated/debayered frames into StarAlignment manually, they were registered with no problem using the default StarAlignment parameters. i can only guess here that the script is calling StarAlignment with some different set of parameters, possibly the noise thresholds.

I have put the master bias/dark and  the 4 light frames into the v1.0 script - no problem to calibrate and register. Can you please send an icon instance of the script with your settings?

Kai
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 April 21 17:40:36
very, very strange. i just tried to use the same calibration masters (together with a new master flat that i calibrated and integrated myself) on some lights that i captured last night.

now when i run, the script says that the master dark has no CFA pattern, and warns that this is contrary to the settings. it then tries to apply the dark and i only see one channel (K0) with a crazy scaling factor.

however, if i use the same calibration masters in ImageCalibration, the fits reader says it detected a CFA pattern in the dark.

i will try to reproduce the 1st problem and make an icon instance, as well as an icon instance for this 2nd problem.
Title: Re: Calibration and align batch operations
Post by: uli t on 2012 April 22 15:12:34
Hi,

tried that new script w/o having darks. Does not work. I get the following error: no master calibration frames or overscan regions have been specified.

Does ist only work with darks?

Uli
Title: Re: Calibration and align batch operations
Post by: Fco. Bosch on 2012 April 23 11:43:46
Quote
tried that new script w/o having darks. Does not work. I get the following error: no master calibration frames or overscan regions have been specified.

Does ist only work with darks?

Yes, nowadays it only works with darks. Mind that this is a script for "calibration", aligment and integration, and there isn't calibration without darks, that you can build at home with the time exposure and temperature that you wish or need, and to preserve in a library for future use. So the images will be better; this is the aim of the script.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 23 11:47:28
Hi

The script works without darks as long as you have a bias selected -- it will use this

Harry
Title: Re: Calibration and align batch operations
Post by: uli t on 2012 April 23 14:00:35
Hi Harry,

thank you, works fine.
Quote
Yes, nowadays it only works with darks. Mind that this is a script for "calibration", aligment and integration, and there isn't calibration without darks, that you can build at home with the time exposure and temperature that you wish or need, and to preserve in a library for future use. So the images will be better; this is the aim of the script.

I am dithering, so I think, I don't need Darks. (DSLR EOS 450Da). At least, I try at the moment. There had been great discussions about that topic.

Maybe later I'll try with Darks to see the difference.
Thanks
Uli
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 24 10:11:12
Hi

Juan are you going to give this script a push , would be well worth your time  ;D

Harry
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 April 26 20:03:09
I ran a BatchProprocessing run tonight and noticed this in the console output:

Loading master calibration frames:
Loading calibration frame image:
O:/fits/20120322_pi_master_gen/master/bias-BINNING_1.fit
Reading FITS: 32-bit floating point, 1 channel(s), 3906x2602 pixels: 100%
Loading calibration frame image:
O:/fits/20120424_rho_ophiuchi2/master/dark-BINNING_1-EXPTIME_180_55F_100subs.fit
** Ignoring 2 additional image(s) in master calibration frame.
Reading FITS: 32-bit floating point, 1 channel(s), 3906x2602 pixels: 100%
** Warning: The file format reports no CFA pattern for the master dark frame, but it is being forced as per process instance parameters.
Loading calibration frame image:
O:/fits/20120424_rho_ophiuchi2/master/flat-BINNING_1_35mm_f56_100subs.fit
** Ignoring 2 additional image(s) in master calibration frame.
Reading FITS: 32-bit floating point, 1 channel(s), 3906x2602 pixels: 100%

Notice the statement I bolded.  These subs are from a Canon 400D DSLR, so I checked the CFA Images box, but it appears the master dark is being generated as a monochrome image instead of RGB.  Is that by design or should the masters be generated as RAW RGB images instead for DSLR images?

Craig
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 April 26 20:11:49
yeah that's what was happening to me as well. i flaked out on the full bug report since i've been too busy with work.
Title: Re: Calibration and align batch operations
Post by: jeffweiss9 on 2012 April 27 06:40:38
    I tried the neat  CosmeticCorrection Module only, since I was in mid-course on some processing when I found this thread and watched Harry's new video, but I have the following question:
    How does one determine the "right" number of hot pixels to remove in operating the sliders in the Module?
    I thought that would be simply a matter of using the module to count the number of hot pixels in the reference MasterDark and adjust the slider by whatever means to get roughly that count.  However, all the choices I tried had me ending up using 2.5 to 3 sigma on "autodetect" since it seemed to make visible hot pixels in a sample imaging sub go away (like in the movie).  That must have been way too many false hot pixels, however, since when the RGB was later combined I ended up with a lot of random uni-color stars suggesting a lot of those 'hot pixels' I removed must have been, in fact, faint, smaller stars (unless I removed to few and suddenly got an additional batch of hot pixels showing up).   Any suggestions for a reliable way to gauge those settings, as I obviously didn't find one ? 
  I previously have used CosmeticCorrection SCRIPT with its default settings and then hand-edited; i.e. cloned, the several (5-20, say) residual hot pixels out of the L, R, G, B images before continuing by comparing them to a test RGB to spot the uni-colored stars ). The video gave me the idea that the MODULE would do a better job.  I know I really should be using dithering to reduce this problem in the first place (on deck to happen).
 Thanks very much.
-Jeff
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 27 06:45:58
I generally determine the "real" hot pixels by doing a cosmetic correction on my masterDark. I later apply the same parameters to lights.
Georg
Title: Re: Calibration and align batch operations
Post by: jeffweiss9 on 2012 April 27 08:17:00
Thanks, Georg-
   That's what I thought ought to be the way to do it.  I didn't mention that I use library darks (I'm a field imager; no obs) that typically are up to a few months old so I don't expect the hot pixels to match perfectly.  However, in this case of imaging on a (freakishly) warm night, I went to a new temperature for at least some of the data and took darks for that temperature the next night at home.  I didn't notice any difference in this problem for those subs with 'fresh' darks, however, compared to the ones with 2 month old library darks.
   But I'll go back to getting the settings from my MasterDarks.  I think I was diverted from that because AutoDetect gave what looked like more complete elimination of hot pixels in the test sub but, unlike the manual entry, autodetect doesn't show the quantity of hot pixels its removing, which presumably was much larger than it should have been.
   Other suggestions still appreciated.
Oh yes and thanks to KWA for authoring this very useful PI enhancement.
-Jeff
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 April 27 09:19:29
Yesterday I also noticed the new multi-image master dark files being generated by this script are not working with CosmeticCorrection.  The module simply errors out when trying to load it - I'm sure confused by the extra images stored within it. 

My work around for now is to open the master dark in PI and save it under a new name, then use CC with this new file.

But, I'm wondering if the BatchProcessing script should save all the rejection maps and masters as individual files instead of the single multi-image one that is currently being generated.  Or, should CosmeticCorrection be updated to interpret both file types?

Craig
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 27 09:38:56
Hi
In my vid I used the master dark as the Map and previewed it on a image so I could make sure I was not removing to much i.e less is more  :D

We also use the words "Hot pixels" to much , most chips have very few true hot pixels and most are warm Pixels and therefore How low do you go with

the sigma setting before it causes other defects ;D

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 27 10:34:23
Juan are you going to give this script a push , would be well worth your time  ;D

My maximum priorities now are PI Core version 1.8 and PCL version 2.0. I am working like crazy on these. PI 1.8 has to be released and fully stabilized before Summer.

I think this script is right now pretty stable and reasonably feature complete, isn't it? Nice video by the way ;)
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 27 10:38:10
Hi

Sorry " Give it a push "  in English  slang Means  " Advertise it " ;)

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 27 10:45:14
Quote
Sorry " Give it a push "  in English  slang Means  " Advertise it "

Oops!  :P  Yeah, I plan on sending a mass email announcing it. Maybe tonight! :)
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 27 10:49:33
Quote
** Warning: The file format reports no CFA pattern for the master dark frame, but it is being forced as per process instance parameters.

No problem at all with this. It is just a warning, not an error. It happens because the FITS format cannot tell us whether a file stores a CFA image or not. The BatchPreprocessing script 'forces' the ImageCalibration tool to load a CFA image unconditionally, hence it issues a warning because it has no way to verify if the FITS file stores a CFA image.
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 April 27 11:27:09
Quote
** Warning: The file format reports no CFA pattern for the master dark frame, but it is being forced as per process instance parameters.

No problem at all with this. It is just a warning, not an error. It happens because the FITS format cannot tell us whether a file stores a CFA image or not. The BatchPreprocessing script 'forces' the ImageCalibration tool to load a CFA image unconditionally, hence it issues a warning because it has no way to verify if the FITS file stores a CFA image.

Great, thanks for the explanation Juan.
Title: Re: Calibration and align batch operations
Post by: jeffweiss9 on 2012 April 27 12:19:04
Hi, Harry-
 If 'warm pixels' are more the problem than just 'hot' ones, how does one know how low to go with sigma in CosmeticCorrection?  In your video you suggested you had a specific quantity you dialed sigma to get. Was that just from Trial & Error for that image or is there a way to estimate it in advance?
Much thanks.
-Jeff
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 27 12:23:31
Hi

Sorry it was good old trial and error , There is no magic sigma number its different for every chip/camera  ;D

I just found that if I made it more aggresiive I lost good data   and as you said before what you do not remove with Cosmetic correction I remove with
a sigma stack ( yes I dither ) :P

Harry
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 27 12:57:48
Here Juan.......................

Calibrate and Preprocessing Script V1.0
Release notes.

Designed by Kia and Juan.

The script allows for automatic sequential calbration, alignment and intitial integration of image sets. It is able to handle mutlple simultainious image sets of various combinations of binning and filters.  It does so by accessing the already robust modules contained in the PixInsight Plateform.
It elimates much of the confusion with calibrating step by step.  Master frames frames are created and can be reused.   Most of the important alignment and integration setting are avialible through the script. 

Other features include:
 * Cosmetic correction application module to remove defects such as hot/cold pixels and columns
*  direct calbration of color one shot color ccd images
*  direct calibrtion of raw files.
*  neatly sorts the output into the directory and subfolders* 
    automatic recognition of image types when used with images obtained with MaxIm Dl
 
Images parameters such as  light, bias, dark, flat, filter, and bin can be recognised automatically through their fits header when acquired through MaxIM DL.  This done the by selecting the first add images button. Images will be automatically sorted into type ( light, bias, flat) , bin and filter.

These images sets can also be easily be added with individual buttons for bias, dark, light and flat.
The add custom can be used to assign key words used in calbration for matching flats to light.
For example no key words for filters are entered in the fits by your acquistion software. No problem enter add  "Red" to your list of lights and flats and the red files will be matched up.  A similar action could be used to designate west and east flats and lights from the same pierside when using a rotator. 

An instance of script can be saved by dragging the arrow and creating an icon. This is very useful since it save all the paths and parameter for reuse.

The cosmetic correction script will be performed after calibration when checked
A saved icon has to gernerated first from the cosmetic correction module. The Icon can be reused.

Check Boxes

 bias master, dark master, flat master - check when the image loaded is a master
 CFA - check color camera
 Fits order - change for certain acquistion software.
 Rejection maps - check for when needed


What else?

Max
 




Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 April 27 13:10:24
Automatic typing also works by looking at name patterns (instead of FITS keyword)
Title: Re: Calibration and align batch operations
Post by: FunTomas on 2012 April 27 13:42:10
...My maximum priorities now are PI Core version 1.8 and PCL version 2.0. I am working like crazy on these. PI 1.8 has to be released and fully stabilized before Summer...
Thank you Juan for your hard work. What can we look forward to? Could you tell us a little bit?
Title: Re: Calibration and align batch operations
Post by: Alejandro Tombolini on 2012 April 27 14:15:14
I am dithering, so I think, I don't need Darks. (DSLR EOS 450Da). At least, I try at the moment. There had been great discussions about that topic.

If you have an EOS 450 you definitively need darks.

Saludos.
Alejandro. 
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 27 14:22:33
Automatic typing also works by looking at name patterns (instead of FITS keyword)

You mean if the file name is      IC443_light_Red_600sec_-35_Westpierside_serial001.fit

That will get sorted somehow?

Explain how that works. I missed that.


Max
Title: Re: Calibration and align batch operations
Post by: jeffweiss9 on 2012 April 27 19:40:38
Thanks, Harry-
Trial and Error.  Where have I heard that before?
Clear skies,
-Jeff
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 April 27 23:57:37
Thanks, Harry-
Trial and Error.  Where have I heard that before?
Clear skies,
-Jeff

EEErrrr    don't know as I will " Play with the settings " in future  >:D

But If there was a magic setting for every tool there would be no need for the user , be boring then  8)
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 28 02:50:29
Hi Max,

Quote
Here Juan.......................

Wow, thank you so much for this nice description. You have saved me the work to write something like this to send a massive email to all PI users. I'll send a formal announcement of BatchPreprocessing today using your description.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 April 28 03:36:45
Quote
Thank you Juan for your hard work. What can we look forward to? Could you tell us a little bit?

Version 1.8 will be more focused on developers than on users. These are some of my goals for PI 1.8 as a development platform:

- Interprocess communication will work for processes (it now works only for file formats). This means that any module will be able to manage instances of any installed process, irrespective of which module implements it. A module will be able to create a process instance, populate it with parameters, execute it, and communicate with it. This will extend the development capabilities of PixInsight considerably, in ways that we probably cannot anticipate.

- Interprocess communication will also work for process interfaces. A module will be able to communicate with any existing tool window on the platform, even with windows defined by other modules.

- Core networking functionality exposed to modules. Thanks to this, a module can perform download/upload network operations without requiring third-party libraries, just as the JavaScript runtime can do now.

- (Still under evaluation) The entire PJSR exposed to modules. This means that a module can define and execute a JavaScript script. This functionality has some important security implications that I am still evaluating, so it perhaps will not be available in version 1.8.0. It will be available during the 1.8 cycle.

- Extended PJSR functionality with more core objects such as Workspace, Icon, ProcessIcon, ImageIcon, ProcessInterface, etc. This will allow scripts to fully manage the entire PixInsight workspace.

- PCL 2.0. This is actually unrelated to PI version 1.8, but I also want to have it ready before Summer. This new version will introduce lots of changes and improvements. (e.g. pixel iterators, a much more powerful ImageVariant class, improved container classes, optional C++11 support, etc.) This deserves a separate thorough description.

From the user's perspective, version 1.8 will be more focused on robustifying and stabilizing the platform than on new features. There will be some niceties though:

- Improved Process Explorer with more flexible and customizable favorite process collections.

- Improved web browser with file download capabilities.

- New image navigation floating window. It will allow you to see a reduced version of the current image with navigation capabilities, similar to the Blink tool.

- Better and more powerful options to organize and manage icons.

- New workspace "present windows" and "hide windows" features, similar to some Mac OS X, KDE and Windows 7 features. Present windows will temporarily show all image windows organized as a grid. Hide windows will temporarily hide all image windows in the current workspace to allow you access and organize your hidden icons.

- New and improved console commands.

- An installer application for Linux and FreeBSD. PixInsight 1.8 will integrate with most X11 desktops (automatic generation of menu items, workspace launcher icons, etc.). The new PI installer application with run in superuser mode to install PI on the standard /opt directory by default (user configurable). The PI updater application will use standard setuid bits on UNIX to run in privileged mode (similar to Windows UAC technology, but much simpler and more secure).

- Lots of performance improvements and bug fixes.

- More :)

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 28 04:25:21
Hi Max,

Quote
Here Juan.......................

Wow, thank you so much for this nice description. You have saved me the work to write something like this to send a massive email to all PI users. I'll send a formal announcement of BatchPreprocessing today using your description.

Your welcome Juan,


It was best contribution that I could offer besides beta testing. Glad you like the note . I think PI users will love the script. I do.

Max

Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 28 20:36:26

My second try- lots of spellllllling mistakes fixed and some editing :surprised:
I did the last one from memory.
I was away from my usual CPU and did not have PI to look at the script.

Max
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Batch Preprocessing Script V1.0
Designed by Kia Wiechen and Juan Conejero.

Release notes.

The script allows for automatic sequential calibration, alignment and initial integration of image sets.
It is able to handle multiple simultaneous image sets with various combinations of binning and filters. 
It does so by accessing the already robust modules contained in the PixInsight Platform.
It eliminates much of the confusion with calibrating step by step.  Master frames are created and can be reused. 
Most of the important alignment and integration settings are available through the script. 

Other features include:
 * Cosmetic correction application module to remove defects such as hot/cold pixels and columns
*  Calibration of color one shot color CCD images
*  Direct calibration of DSLR raw files.
*  Neatly sorts the output into a directory and subfolders * 
*  automatic recognition of image types when used with images obtained with MaxIm DL
 
Images types and parameters such as light, bias, dark, flat, filter, and bin can be recognized automatically
through their FITS header when acquired through MaxIM DL. 

This done the by selecting the first add files button. Images will be automatically identified and sorted as light, dark, bias, flat, bin and filter.

Images sets can also be easily be entered into the script with individual "add" buttons for bias, dark, light and flat.

The "add custom" can be used to assign key words used in calibration for matching flats to lights.

Perhaps no key words for each filters has been created in the FITS header by your acquisition software.
No problem enter add  "Red" to your list of lights and than the flats and the Red images will be matched for calibration.
 Repeat the action for each filter as needed. 

A similar action could be used when obtaining images using a camera rotator.   
Typically an object will have 180 degree rotation to reacquire the guide star after a crossing the meridian on a GEM mount.
In this case just add a key word such as "West" to the flats and light acquired on the west pier side.

An instance of script can be saved by dragging the arrow and creating an icon. This is very useful since it saves all the paths and parameter for reuse.

The cosmetic correction script will be performed after image calibration when checked.

An icon has to be generated first from the cosmetic correction module.
This is done by setting up the module and dragging the arrow to save an instance of the CC module.
The icon can then be selected by the script. 

Don't forget to save the icon for future use. 
It can be used again next time you run the script using the same camera.

There is check box for limiting the script to calibration only. The alignment and integration are skipped.

Integration Settings
Settings parallel those on the integration module
Different settings can be applied for each category of master and for the light frames

Options Settings



 CFA. Stands for Color Filter Array.  Check color cameras
 Optimize dark- this the dark optimization option available in the calibration module
 Rejection maps - check this when you desire the map obtain from integrating the masters frames or the integration of the lights
 Export calibration files- files names exported as generated instances
 Up Bottom FITS order - Should be checked for most popular acquisition software packages.
 Use master bias, Use master dark, Use master flat, these should be checked when the image loaded is a master that was produced previously.

Registration Reference Image - select the path of a reference image to register all images in the data set.
 
Output Directory- Select or create an directory to write processed images from this script.

Diagnostics - Check for problems before running the script. It will spot things you forgot.

Run- Runs script

Exit- Closes script


Max
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 April 29 03:09:27
Help - I've been trying to get to know the script with a simple scenario of 5 binned bias, 5 binned darks, a few flats for R,G and B and a few lights for R,G and B.
When the script gets to the "Begin calibration of flat frames" section, it loads the master bias, then the master dark

Then it says "Loading master calibration frames:"
Then "Loading calibration frame image:"  (followed by name of the master bias frame)

Then gives error:
(name of master bias): Multiple images cannont be used as calibration frames



I get this whether I feed in multiple bias frames and let if make a master, OR just select the master the script made previously and tick the "Use Master Bias" and "Use Master Dark" boxes.

Can anyone help shed some light on what I'm doing wring?
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 29 03:56:21
  You are generating masters so don't check use master
Check diagnostic before going to run

Max
Title: Re: Calibration and align batch operations
Post by: astropixel on 2012 April 29 04:46:19
Brilliant Max! This makes the DSLR_RAW work flow redundant... great stuff!

I have just processed a set of RAW files in very quick time, far quicker than following the DSLR_RAW workflow. The only thing is, I have a very nice monochrome image. Where do I go from here or have I missed something - I was expecting an RGB.

Interestingly, dark frames that produced very high scaling factors and poor results with optimization checked, processed without issue using the script and optimization selected. The encouraging thing, is that there were only 10 darks (cooled dslr).

Regards,

Rowland
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 April 29 04:53:30
  You are generating masters so don't check use master
Check diagnostic before going to run

Max

I always end up with this error though Max if I start with raw frames and no master options ticked, OR if I use the masters previously generated and have master bias and dark ticked. 

What is this error message supposed to be telling me guys?  Any assistance appreciated.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 29 08:41:39
Brilliant Max! This makes the DSLR_RAW work flow redundant... great stuff!

I have just processed a set of RAW files in very quick time, far quicker than following the DSLR_RAW workflow. The only thing is, I have a very nice monochrome image. Where do I go from here or have I missed something - I was expecting an RGB.

Interestingly, dark frames that produced very high scaling factors and poor results with optimization checked, processed without issue using the script and optimization selected. The encouraging thing, is that there were only 10 darks (cooled dslr).

Regards,

Rowland

Thanks but I am just the messenger. Kia and Juan wrote the script.

It debayers  the image so it should generate R G B calibrated and registered images.  Check the folders for the R G B output . I don't have color camera. May be someone else can help

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 29 08:46:34
  You are generating masters so don't check use master
Check diagnostic before going to run

Max

I always end up with this error though Max if I start with raw frames and no master options ticked, OR if I use the masters previously generated and have master bias and dark ticked. 

What is this error message supposed to be telling me guys?  Any assistance appreciated.

Sounds ok to me . I have not seen this myself perhaps someone else has an answer


Max
Title: Re: Calibration and align batch operations
Post by: Luigi on 2012 April 29 12:55:47
I just wanted to say thanks very much for this script! Excellent work! It took the script less than 20 minutes to do what would have taken me much longer, and, I didn't have do much except enjoy a nice Rioja and watch the show  ;D

Usage questions:

1) My 'mix' of lights includes exposures of differing lengths as we have no debloomer  tool >:D If I 'pile' all of the lights into the tool, it creates a single master light frame. Since my intention is to use HDR Composition,  do I just need to run the tool separately for each exposure length for which I want to create a master light? The tool does create a separate dark master for each integration time, so maybe I have missed a setting?

2) Since I have exposures of different lengths, the script generated dark masters for each of my exposure times. This is not a related to the tool per se, but am I wasting my time capturing these darks of different lengths?

Between this tool and the solver/annotater script, it's like Christmas in April!
Title: Re: Calibration and align batch operations
Post by: kwalker on 2012 April 29 13:21:44
Fantastic new script.
you don't know how long I've been waiting to ditch deepsky stacker :)
Handled my DSLR lights,darks ,flats ,and bias files and didn't kill the hard work collecting the data
Thanks guys
Title: Re: Calibration and align batch operations
Post by: kerrywaz1 on 2012 April 29 14:14:29
I am looking forward to using this script, but I think we need a really good tutorial to get us off the ground (at least I do  :laugh:).

How 'bout it Harry?

 
Title: Re: Calibration and align batch operations
Post by: Luigi on 2012 April 29 14:36:27
He's already done one!

http://www.harrysastroshed.com/preprocessing.html
Title: Re: Calibration and align batch operations
Post by: astropixel on 2012 April 29 15:22:11
Sigh... I have my life back- thank you all... but, I can't find an RGB file after Preprocessing ???
Title: Re: Calibration and align batch operations
Post by: Luigi on 2012 April 29 16:20:40
All of the masters for each color are in the 'master' folder; the script does not combine R,G, and B masters into an RGB, if that's what you mean.

In <Output_Directory>/master I have:
light-FILTER_<color>-BINNING_2.fit, for example.
Title: Re: Calibration and align batch operations
Post by: astropixel on 2012 April 30 04:47:42
Finger trouble - CFA must be selected for OSC cameras - of course ???
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 April 30 05:06:23
Help - I've been trying to get to know the script with a simple scenario of 5 binned bias, 5 binned darks, a few flats for R,G and B and a few lights for R,G and B.
When the script gets to the "Begin calibration of flat frames" section, it loads the master bias, then the master dark

Then it says "Loading master calibration frames:"
Then "Loading calibration frame image:"  (followed by name of the master bias frame)

Then gives error:
(name of master bias): Multiple images cannont be used as calibration frames



I get this whether I feed in multiple bias frames and let if make a master, OR just select the master the script made previously and tick the "Use Master Bias" and "Use Master Dark" boxes.

Can anyone help shed some light on what I'm doing wring?

It would appear that if "Generate rejection maps" is selected, then bias (and dark) masters are saved as "multiple images" which include not only the integrated master, but ALSO the rejection low and high images.  These multiple image files appear to be incompatable for subsequent calibration however (even trying to use them manually using the Image Calibration process fails with the error I gave above).  By unticking "generate rejection maps" I was back in business.  (After searching the script for the error text I realised it was a PI error, which led me to try manual image calibration....)

This seems to make "Generate rejection maps" a fairly dangerous option however - perhaps the rejection maps could be saved separately somehow, or simply left open on the screen after the script has run? 
Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 April 30 08:34:26
Wow this is a great script, thanks heaps.  ;D

I have been using it for a few hours and unfortunately I have a few issues that I hope you can resolve for me.

When loading images it does not seem to be reading the FITS header information of the image files.
For example it always loads images under the heading “Binning 1” even though in the FITS header of the image it reads binning 2 under the XBINNING or YBINNING fields. 

Also it never loads my light frames automatically although the FITS image type “IMAGETYP” field is set correctly, value = 'Light Frame'.

Another issue is that after the script has completed I only ever get 1 Master Light named “light-BINNING_1.fit” in the master folder. No others are listed even though I have added light frame images of Red, Green, Blue and Luminance images to compile.

Hope you can point me in the right directing to resolve these issues.
Regards
Andrew
Title: Re: Calibration and align batch operations
Post by: Luigi on 2012 April 30 08:54:51
Can you post the FITS header of one of your 'color' frames? What software are you using to capture your images?

The script uses the 'FILTER' keyword for creating the files by color.

I ran this with an assortment of LRGB frames plus bias, dark, and flats and all seemed OK.
Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 April 30 15:29:28
Louis, yes here is the FITS header from one of my Blue images. My capture software is Maxim DL

Regards
Andrew
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 April 30 19:41:40
Louis, yes here is the FITS header from one of my Blue images. My capture software is Maxim DL

Regards
Andrew

You can alway and the types of images manually. As long as your not mixing bin 2 with bin 1 images, it should be fine. 

MaxIm dl images should be fine. Juan may have a suggestion

Max











Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 April 30 22:06:14
hi Max, yes I can load these manually and have tried this, but I still only get one master light which is always labelled “light-BINNING_1.fit”, even though I have manually loaded red, green blue images to process (all same binning).

I have tried renaming the light frames with the word “light” in the filename, this works and the script will automatically load them. But the script will not load them automatically based solely on the FITS header info.

Also still cannot change the heading “Binning 1” no matter if I have the correct binning in either the FITS header or the filename.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 02 07:36:26
Hi Andrew,

This problem is strange. The FITS header looks correct, and all tests we have made with Maxim DL images have worked fine so far.

We need a sample of your data to understand and fix the problem. It would be great if you could upload a calibration set. Just 3 or 4 light and flat frames for each filter plus 3 bias and 3 darks would suffice. Do you have an Endor file server account?
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 02 07:41:11
Rob,

Quote
Then gives error:
(name of master bias): Multiple images cannont be used as calibration frames

Quote
These multiple image files appear to be incompatable for subsequent calibration however (even trying to use them manually using the Image Calibration process fails with the error I gave above).

This happens because you are using an old version of the ImageCalibration module. You need ImageCalibration version 1.1.2.83, which we released April 29th as an update. Have you applied all the available updates to your PI installation?

With the latest version of ImageCalibration, multiple FITS work without problems.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 02 07:55:24
Quote
1) My 'mix' of lights includes exposures of differing lengths as we have no debloomer  tool >:D

But we have the *only valid* deblooming tool: HDRComposition :)

Quote
If I 'pile' all of the lights into the tool, it creates a single master light frame. Since my intention is to use HDR Composition,  do I just need to run the tool separately for each exposure length for which I want to create a master light?

Yes, since you need a separate master light for each exposure time.

Quote
2) Since I have exposures of different lengths, the script generated dark masters for each of my exposure times. This is not a related to the tool per se, but am I wasting my time capturing these darks of different lengths?

Within reasonable limits, our dark optimization routine allows you to use a single master dark frame for a wide range of exposure times and temperatures. If you can invest that time in acquiring more signal, then my advice is don't waste it in acquiring more darks.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 02 08:53:47
Quote
1) My 'mix' of lights includes exposures of differing lengths as we have no debloomer  tool >:D

Within reasonable limits, our dark optimization routine allows you to use a single master dark frame for a wide range of exposure times and temperatures. If you can invest that time in acquiring more signal, then my advice is don't waste it in acquiring more darks.

I have a question.

If I was to process two sets of images simultainously, one set at -30 and another set at -25, would the script be able to choose the best dark master for optimization?

I am adding a dark master at both temps at the same exposure as the lights.

I was thinking there would be some benefit to have a true match in temp since this harder to optimize than just scaling at the same temp.

Perhaps I am wrong.

 How close does the temp and time between the darks and lights be to get the best results?

Max



Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 May 02 11:38:25
Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources  ;)

Craig
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 May 02 11:41:44
I have a processing questions for multiple nights and different aperture settings.

Let's say night 1 has subs using a lens at F5.6 and night 2 the lens was at F4.0.  Does the BatchProcessing script allow matching up the flats and lights from each night?  If so, how is this accomplished?

If not, would it be possible to add this functionality?

Thanks, Craig
Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 May 03 00:48:43
Hi Andrew,

This problem is strange. The FITS header looks correct, and all tests we have made with Maxim DL images have worked fine so far.

We need a sample of your data to understand and fix the problem. It would be great if you could upload a calibration set. Just 3 or 4 light and flat frames for each filter plus 3 bias and 3 darks would suffice. Do you have an Endor file server account?

Juan, yes I have just received access to Endor file server. Where do you want me to upload the files to?
 
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 03 20:40:26
Hey Juan is there a way access the overscan feature with the script

Max
Title: Re: Calibration and align batch operations
Post by: jjongmans on 2012 May 05 03:03:53
Every time when I run the script, I get the following error: ImageWindow.windowById(): invalid view identifier

All I do is; select bias and light frames, select reference file, select output directory, run the script

OUTPUT:
run --execute-mode=auto "/Applications/PixInsight64.app/Contents/src/scripts/BatchPreprocessing/BatchPreprocessing.js"

Processing script file: /Applications/PixInsight64.app/Contents/src/scripts/BatchPreprocessing/BatchPreprocessing.js

*********************************************************************
* Begin integration of bias frames
*********************************************************************

ImageIntegration: Global context

* Loaded cache: 697 item(s)

Opening files:
/Volumes/NAS/... .../QSI/calibration frames/bias frames/bias_-20c_1x1-1_001.fit
Weight          :     1.00000
/Volumes/NAS/... .../QSI/calibration frames/bias frames/bias_-20c_1x1-2_001.fit
Weight          :     1.00000
/Volumes/NAS/... .../QSI/calibration frames/bias frames/bias_-20c_1x1-3_001.fit
Weight          :     1.00000
/Volumes/NAS/... .../QSI/calibration frames/bias frames/bias_-20c_1x1-4_001.fit
Weight          :     1.00000
/Volumes/NAS/... .../QSI/calibration frames/bias frames/bias_-20c_1x1_001.fit
Weight          :     1.00000

Integration of 5 images:
Pixel combination ......... average
Output normalization ...... none
Pixel rejection ........... none

* Using 1261 concurrent pixel stack(s) = 255.99 MB
Integrating pixel rows:     0 ->  1260: 100%
Integrating pixel rows:  1261 ->  2503: 100%
9.349 s

*********************************************************************
* End integration of bias frames
*********************************************************************

Writing master bias frame:
/Users/... .../master/bias-BINNING_1.fit

This is the point where I get the error.

PixInsight Standard Edition
Version 01.07.06.0793 Starbuck

Mac OS X 10.7.3


What to do?
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 May 05 06:07:31
I have had similar problems when using files from NAS storage with win 7. Can you please try to load the files from the local HDD?

Kai

Title: Re: Calibration and align batch operations
Post by: jjongmans on 2012 May 05 06:47:50
Copied all the files to my local HDD, tried to run the script again, same result..  ImageWindow.windowById(): invalid view identifier
Title: Re: Calibration and align batch operations
Post by: bitli on 2012 May 06 02:14:27
I had the same problem as quoted below and I think that I found the problem/solution (see below):

...
I have been using it for a few hours and unfortunately I have a few issues that I hope you can resolve for me.

When loading images it does not seem to be reading the FITS header information of the image files.
For example it always loads images under the heading “Binning 1” even though in the FITS header of the image it reads binning 2 under the XBINNING or YBINNING fields. 

Also it never loads my light frames automatically although the FITS image type “IMAGETYP” field is set correctly, value = 'Light Frame'.

Another issue is that after the script has completed I only ever get 1 Master Light named “light-BINNING_1.fit” in the master folder. No others are listed even though I have added light frame images of Red, Green, Blue and Luminance images to compile.
...
Andrew

There is a bug at line 250 of BatchPreprocessing-helper.js. The line read:
Code: [Select]
      if ( ext != ".fit" && ext != ".$" )but the calling code in BatchPreprocessing-engine.js around line 385 reads
Code: [Select]
   if ( ext == ".fit" || ext == ".fits" || ext == ".fts" )
The test  at line 250 of BatchPreprocessing-helper.js must accept files with extensions 'fits' and 'fts' also.

The workaround is naturally to rename your files in '.fit'.
Hope this helps

-- bitli

Title: Re: Calibration and align batch operations
Post by: steinplanet on 2012 May 06 03:45:34
Hello,

My name is Willi and I live in Germany near Luxembourg.
Please excuse my bad English but I am still learning. ::)
For a few days I use now PixInsight. A really powerful program!

I have a question about the Batch Preprocessing.
Can I use Flatdarks instead of BIAS?

Greetings from Saarlouis
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 06 17:44:07
Flat darks are not necessary unless you use exposures longer than 45 seconds


Max
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 May 06 21:30:57
Rob,

Quote
Then gives error:
(name of master bias): Multiple images cannont be used as calibration frames

Quote
These multiple image files appear to be incompatable for subsequent calibration however (even trying to use them manually using the Image Calibration process fails with the error I gave above).

This happens because you are using an old version of the ImageCalibration module. You need ImageCalibration version 1.1.2.83, which we released April 29th as an update. Have you applied all the available updates to your PI installation?

With the latest version of ImageCalibration, multiple FITS work without problems.

I have that update now Juan, but perhaps wasn't loaded at the time - will investigate further.  Many thanks,
Rob
Title: Re: Calibration and align batch operations
Post by: RobF2 on 2012 May 06 22:20:06
By the way, this script is a game-changer for me with PI.  So many older projects I had put off finishing processing (knowing the data collection had been marginal) are now straight-forward to finish using an instance of the script with bias/darks/flats for every binning/filter loaded.  Only have to load up lights and turn it loose now while sorting out other things in the background.  Great to see Juan and the rest of the PI user base pulling in behind too.   O0

Thanks so much again Kai
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 07 02:06:10
Hi

The latest version seems to have lost the ability to save any image  files when you save a "new instance" :yell:





Harry
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 07 05:56:59
Hi

To correctley debayer a image using the Module I need to select GRBG , but with the script I need BGGR a bit confusing  :sad:


Harry
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 May 07 05:58:51
Harry,
this may be due to the FITS orientation option. Most likely your images are also rotated. Try to toggle the entry in the script GUI.
Georg
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 07 06:16:22
Hi

Indeed this does debayer the image correctly, But the image is loaded up the same orientation what ever way the box is checked  >:D

Makes my comment in my video " check setting with the debayer mod " slightly redundant

It would make more sense for the images to be treated the same inside the script and outside ( by default anyway )

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 07 12:12:46
Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 07 12:17:51
Hi

Spent all day trying to get it to work inc rebooting machine and now it works  :yell:

Will keep my eye on  it and see if it happens again

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 07 12:24:30
It would make more sense for the images to be treated the same inside the script and outside ( by default anyway )

Since this script is intended to make things simpler, and since most users don't (want to) know what this orientation stuff is all about, I avoided that behavior on purpose. By default, the script loads all FITS files with the 'normal' orientation used by all amateur image acquisition applications (which is the opposite to the professional orientation used by default in PixInsight). This 'forced setting' prevents a common source of errors (wrongly oriented images, wrong CFA patterns) and makes the whole preprocessing task more consistent.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 07 12:30:56
Quote
There is a bug at line 250 of BatchPreprocessing-helper.js. The line read:
Code: [Select]

      if ( ext != ".fit" && ext != ".$" )

but the calling code in BatchPreprocessing-engine.js around line 385 reads
Code: [Select]

   if ( ext == ".fit" || ext == ".fits" || ext == ".fts" )

The test  at line 250 of BatchPreprocessing-helper.js must accept files with extensions 'fits' and 'fts' also.

Thank you for catching this. I'll include a fix in the next release. However I don't think the original problem is due to ".fts" extension, is it?
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 07 12:34:17
It would make more sense for the images to be treated the same inside the script and outside ( by default anyway )

Since this script is intended to make things simpler, and since most users don't (want to) know what this orientation stuff is all about, I avoided that behavior on purpose. By default, the script loads all FITS files with the 'normal' orientation used by all amateur image acquisition applications (which is the opposite to the professional orientation used by default in PixInsight). This 'forced setting' prevents a common source of errors (wrongly oriented images, wrong CFA patterns) and makes the whole preprocessing task more consistent.

Hi Therefore my Files should be debayered the same in the script and by the debayer module and they are not  ^-^ I have to uncheck the box to achieve this , opposite to what it should be  >:D

Harry
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 07 12:40:14
Quote
Every time when I run the script, I get the following error: ImageWindow.windowById(): invalid view identifier

This shouldn't happen. The bug must be in either the ImageIntegration or the Debayer process. To understand and fix this problem I need to know:

- The full file names of the files you're working with.
- The filter name(s).

Thanks, and sorry for the inconvenience.
Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 May 07 15:05:42
There is a bug at line 250 of BatchPreprocessing-helper.js. The line read:
Code: [Select]

      if ( ext != ".fit" && ext != ".$" )

but the calling code in BatchPreprocessing-engine.js around line 385 reads
Code: [Select]

   if ( ext == ".fit" || ext == ".fits" || ext == ".fts" )


The test  at line 250 of BatchPreprocessing-helper.js must accept files with extensions 'fits' and 'fts' also.

The workaround is naturally to rename your files in '.fit'.
Hope this helps

-- bitli

bitli, thanks for this. I tried what you suggested  but unfortunately this did not fix the problem  :sad:


Hi Andrew,

This problem is strange. The FITS header looks correct, and all tests we have made with Maxim DL images have worked fine so far.

We need a sample of your data to understand and fix the problem. It would be great if you could upload a calibration set. Just 3 or 4 light and flat frames for each filter plus 3 bias and 3 darks would suffice. Do you have an Endor file server account?


Juan, yes I have received access to Endor file server. Where on the server do you want me to upload the files to?
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 07 15:08:24
Quote
Where on the server do you want me to upload the files to?

Create a folder on "Forum Shared Files" and upload your files there.
Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 May 07 16:38:21
Quote
Where on the server do you want me to upload the files to?

Create a folder on "Forum Shared Files" and upload your files there.

hi Juan, I have uploaded a zip file called “atheelen_NGC5128.zip” containing the sample pictures requested into the folder titled “Calibration_Align_Batch_Operations”.

Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 08 11:39:56
Hi

So anybody else see what I do  :sad:

Harry


It would make more sense for the images to be treated the same inside the script and outside ( by default anyway )

Since this script is intended to make things simpler, and since most users don't (want to) know what this orientation stuff is all about, I avoided that behavior on purpose. By default, the script loads all FITS files with the 'normal' orientation used by all amateur image acquisition applications (which is the opposite to the professional orientation used by default in PixInsight). This 'forced setting' prevents a common source of errors (wrongly oriented images, wrong CFA patterns) and makes the whole preprocessing task more consistent.

Hi Therefore my Files should be debayered the same in the script and by the debayer module and they are not  ^-^ I have to uncheck the box to achieve this , opposite to what it should be  >:D

Harry
Title: Re: Calibration and align batch operations
Post by: astropixel on 2012 May 08 11:41:27
I'm used to be being ignored :o
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 08 11:43:54

Sorry did you say something  >:D :-*

Harry


I'm used to be being ignored :o
Title: Re: Calibration and align batch operations
Post by: erikgu on 2012 May 09 05:36:17

Hi !

I dont know if the Preprosessing script is a correct place to put Canon Banding Reduction as an inital and optional prosessing step but could it be ?

I have done this by use of an ImageContainer with the lights and a Process container with CBR, Rotate 90, CBR and the rotate back to initial orientation so its not a big deal, but it could have been nice if it was included in the script as a step before calibration. Default settings could be used in CBR.

Erik G
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 May 09 06:24:44
Erik,

currently CBR is not suitable for processing RAW frames, so you need to apply it to the calibrated+debayered (and possibly already stacked) frames.  See the following links for some discussions:

http://pixinsight.com/forum/index.php?topic=3522.msg24377#msg24377
http://pixinsight.com/forum/index.php?topic=4214.msg29600#msg29600

Georg
Title: Re: Calibration and align batch operations
Post by: Alejandro Tombolini on 2012 May 09 07:11:20
(and possibly already stacked) frames. 

CBR works perfect in stacked images (Canon 450 and 5D at least), it is only a matter of find the correct values, specially 1/sigma factor when are bright areas. No need to be applied in individual frames.
Saludos, Alejandro.
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 May 09 08:04:56
Sorry, my statement probably was misleading :'( . What I wanted to say is that the current CBR implementation needs debayered images, not RAW images. The earliest point where the BatchProcessing process  produces debayered images is after calibration, after registration, and after integration. So you currently cannot use CBR earlier in the process.

Also, the benefit of doing it at an earlier stage currently is unclear, see the links.

Georg
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 09 09:29:48

Sorry did you say something  >:D :-*

Harry


I'm used to be being ignored :o

I am still waiting to hear if we add overscan to the script. I
 know NKV and Vicent would like this too.

Max
Title: Re: Calibration and align batch operations
Post by: erikgu on 2012 May 09 21:31:46
Hi, sorry for wanting CBR on a wrong stage in the processing order. I think from the test i did that running the script before registration gave a better result than on the stacked image. Might be because i stacked images taken over several days with not exactly the same camera orientation. Result of the the registration would then give not exactly horizontal and vertical bandings, correct ?

Erik G
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 May 10 08:46:10
Hi, sorry for wanting CBR on a wrong stage in the processing order. I think from the test i did that running the script before registration gave a better result than on the stacked image. Might be because i stacked images taken over several days with not exactly the same camera orientation. Result of the the registration would then give not exactly horizontal and vertical bandings, correct ?

Erik G

yes, that's right. if you have enough nights each with their own subtly different PA the banding can get all crazy. but if you have enough different PAs you can make stacks of each night, do CBR, and then stack the stacks...
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 11 04:41:05
Hi everybody,

I have just released version 1.05 as an update (for developers, the full source code is also attached to this post). This is a quick bugfix release while we prepare a more substantial update during the next days:

- FITS files with the .fts extension were not properly recognized in previous versions due to a bug in the File.loadFITSKeywords() helper routine. This has been fixed.

- Fixed a bug in the Add Custom Frames dialog: When no explicit selection was done in the Image type combo box, an invalid image type was selected.

Thanks to Andrew (atheelen) for uploading a raw data sample which I have used to fix the .fts bug.

Thanks to everybody for your patience and help with this script.

A few ideas that we want to implement in the next version:

* Support for multiple objects in a single instance of the BatchPreprocessing script. Some professional astronomers have asked us to add this feature.

* Optional automatic removal of intermediate calibrated images (e.g. calibrated bias, dark and flat frames).

* Automatic calculation of the required disk space and check available space during the diagnostics process.

* When one of the files being calibrated is selected as registration reference, use the corresponding calibrated/deBayered frame instead of the original frame.

Enjoy!
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 May 11 04:45:46
Hi everybody,

I have just released version 1.05 as an update (full source code attached to this post). This is a quick bugfix release while we prepare a more substantial update during the next days:
...
Great! Has the updated cosmetic correction module also been released?
http://pixinsight.com/forum/index.php?topic=3483.msg29800#msg29800

...
* Support for multiple objects in a single instance of the BatchPreprocessing script. Some professional astronomers have asked us to add this feature.
...
Professional astronomers are using  a script intended for newbies?  >:D
Georg
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 11 05:00:07
Quote
Great! Has the updated cosmetic correction module also been released?

Yes, I have released it also this morning. Sorry I forgot to mention it.

Quote
Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift :)

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 11 09:00:11

Quote
Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift :)

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.
[/quote]

They must be asking of bias oversscan too?

Max
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 11 09:02:45
Quote
Great! Has the updated cosmetic correction module also been released?

Yes, I have released it also this morning. Sorry I forgot to mention it.

Quote
Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift :)

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.

They must asking for bias overscan too?

Max
Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 11 12:13:59
Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.

Hi
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

Harry

P.s.

I do wish you would look at the Debayer settings not right in my mind  >:D

Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 May 11 12:48:57
Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources  ;)

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 May 11 12:51:57
Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.

Hi
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

Harry

P.s.

I do wish you would look at the Debayer settings not right in my mind  >:D

Harry, I reported this same problem here:  http://pixinsight.com/forum/index.php?topic=3654.msg28997#msg28997 (http://pixinsight.com/forum/index.php?topic=3654.msg28997#msg28997)

It was fixed at one point, so I'm not sure how it has returned.

Craig
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 12 02:38:29
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

This is precisely the purpose of the "Export calibration files" option. When enabled (default state), newly created instances will contain all file paths selected in all frame groups. When this option is disabled, new instances will contain no file paths but only calibration, integration, registration and deBayering parameters.

Maybe the tool tip information is not clear enough for this check box?
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 12 03:06:58
Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources  ;)

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig

Hi Craig,

All processes currently executed by the BatchPreprocessing script are parallelized. However, not all of them are parallelized in the same way, neither all of them are using multithreading with the same efficiency for large batch tasks.

The ImageCalibration and CosmeticCorrection tasks use high-level parallelization. Basically, to process N files with P worker threads, each worker thread runs independently for a sublist of N/P files. There is an additional coordination thread for worker thread supervision and file reading/writing tasks. For large file lists, this is usually the most efficient implementation. ImageIntegration also uses a very efficient multithreading scheme.

Other processes use low-level parallelization, which in some cases is not appropriate for the BPP script. For example, some parts of the StarAlignment task are parallelized, such as the star matching and RANSAC routines. This is very efficient to process a few images (e.g. for mosaic construction, or to align less images than the number of processors available), but rather inefficient to align large sets of disk files. A future version of the StarAlignment process will use a high-level parallelization scheme, which will improve the performance of the BPP script considerably. This is near the top of the to-do list.

The whole PixInsight platform will be improved significantly for multithreaded execution in future versions. There is still a lot of work to do in this regard.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 12 03:21:13
Quote
They must asking for bias overscan too?

Not so far, but they may ask for it eventually. That will depend essentially on whether they start using PI intensively for preprocessing tasks or not.

I'm not sure if overscan support is actually a necessity for the BatchPreprocessing script, but since this doesn't add too much complexity, I'll implement it in the next version.
Title: Re: Calibration and align batch operations
Post by: mmirot on 2012 May 12 06:26:48

Thanks Juan

You rock  8)


Title: Re: Calibration and align batch operations
Post by: Harry page on 2012 May 12 11:31:31
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

This is precisely the purpose of the "Export calibration files" option. When enabled (default state), newly created instances will contain all file paths selected in all frame groups. When this option is disabled, new instances will contain no file paths but only calibration, integration, registration and deBayering parameters.

Maybe the tool tip information is not clear enough for this check box?

Hi

Ok I get it now , not what I thought it meant so A better description would be better

Regards
Harry
Title: Re: Calibration and align batch operations
Post by: NKV on 2012 May 12 18:53:43
I'm not sure if overscan support is actually a necessity for the BatchPreprocessing script
Juan, my camera 160803 FLI ProLine required overscan for proper calibration. I can't use it without overscan.
Of course, I can apply overscan to all images in ImageCalibration module. And after that use BPP script.
So i will happy to see overscan in BPP. For me enough one overscan region.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 May 13 16:44:31
Let's start the week with a smile :) Version 1.10 has been just released as an update; the source code is attached to this post, as usual.

Version 1.10 implements full support for calibration overscan with four independent overscan regions, just as the ImageCalibration tool. Please check that it is working well.

Enjoy!
Title: Re: Calibration and align batch operations
Post by: NKV on 2012 May 13 17:45:37
Let's start the week with a smile :)
:)
Title: Re: Calibration and align batch operations
Post by: atheelen on 2012 May 14 04:36:17
Let's start the week with a smile :) Version 1.10 has been just released as an update; the source code is attached to this post, as usual.

Version 1.10 implements full support for calibration overscan with four independent overscan regions, just as the ImageCalibration tool. Please check that it is working well.

Enjoy!

Juan, looks good. The issues I was having with loading images via FITS header info and binning numbering problem looks to be fully resolved with v1.10  ;D
Title: Re: Calibration and align batch operations
Post by: cs_pixinsight on 2012 May 14 11:16:23
Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources  ;)

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig

Hi Craig,

All processes currently executed by the BatchPreprocessing script are parallelized. However, not all of them are parallelized in the same way, neither all of them are using multithreading with the same efficiency for large batch tasks.

The ImageCalibration and CosmeticCorrection tasks use high-level parallelization. Basically, to process N files with P worker threads, each worker thread runs independently for a sublist of N/P files. There is an additional coordination thread for worker thread supervision and file reading/writing tasks. For large file lists, this is usually the most efficient implementation. ImageIntegration also uses a very efficient multithreading scheme.

Other processes use low-level parallelization, which in some cases is not appropriate for the BPP script. For example, some parts of the StarAlignment task are parallelized, such as the star matching and RANSAC routines. This is very efficient to process a few images (e.g. for mosaic construction, or to align less images than the number of processors available), but rather inefficient to align large sets of disk files. A future version of the StarAlignment process will use a high-level parallelization scheme, which will improve the performance of the BPP script considerably. This is near the top of the to-do list.

The whole PixInsight platform will be improved significantly for multithreaded execution in future versions. There is still a lot of work to do in this regard.

Juan, thank you for the information regarding this request.  In watching the CPU resources while running the BPP script, it's apparent that just about everything is running in parallel.  It's really the debayering (which is fairly fast even using 1 core) and the noise estimation (very slow) which slows the process down significantly.  When you get around to improving the BPP script in this regard, please keep in mind the script will run several times faster if the noise estimation utilizes the high-level parallization similar to the CosmeticCorrection module.

Thanks again,
Craig
Title: Re: Calibration and align batch operations
Post by: mcbbcn on 2012 May 25 15:44:05
Hi,

I'm quite new to Pixinsight, but I'm sure I'm doing something wrong when I try to run the script, I get this error:

C:/Program Files/PixInsight/src/scripts/BatchPreprocessing/BatchPreprocessing-helper.js

run --execute-mode=auto "C:/Program Files/PixInsight/src/scripts/BatchPreprocessing/BatchPreprocessing.js"

Processing script file: C:/Program Files/PixInsight/src/scripts/BatchPreprocessing/BatchPreprocessing.js
*** Error [000]: C:/Program Files/PixInsight/src/scripts/BatchPreprocessing/BatchPreprocessing-GUI.js, line 811: Error: ProcessInstance.iconsByProcessId(): unknown process class: 'CosmeticCorrection'

I don't have any images or anything open, I just tried to run it...do I need to have any images loaded or select anything?

Thank you,

Miquel Casas
Portland, OR
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 May 26 06:29:17
You Need to use the ladest Pi Version with all Updates, which should include an install of the cosmetic correction Module.
Title: Re: Calibration and align batch operations
Post by: mcbbcn on 2012 May 26 14:28:19
Hi Georg,

Thanks for the quick reply.  I "think" I have all the updates, when I go to "Resources->Updates->Check for updates", I get a message saying that I have all the updates.

I'm running Windows 7 Ultimate 64 bit edition and my pixinsight version is 01.07.06.0793 Starbuck (eng x86_64).

Do you have any thoughts if I have do anything else other than the standard way to update PI?

Thanks for your help,

M.
Title: Re: Calibration and align batch operations
Post by: mcbbcn on 2012 May 26 14:31:09
...also, I don't know if this has anything to do with it, but I noticed that I script called "CosmeticCorrection v1.6.1" and it seems to work just fine...
Title: Re: Calibration and align batch operations
Post by: mcbbcn on 2012 May 26 14:53:21
 :D
I found a solution:

I edited the file updates.xri (Pixinsight folder) with PI Script editor, and I deleted the last two packages (i.e. batchreprocessing & cosmeticcorrection), save the file, and I checked for the updates which force PI to re-download & re-installed both packages.  Now it works!

Cheers,

M.
Title: Re: Calibration and align batch operations
Post by: mcbbcn on 2012 May 26 17:27:16
...and yet, another question.  Using this script, is there a way to substract the master bias from the master dark (i.e. creating a thermal)?  I think you accomplish this during calibration by selecting the calibrate checkbox, but I did not see this option in the script...

Thanks for your help,

M.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 June 09 01:42:47
Hi,

I'm getting this error just after the bias integration.


Regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 June 09 01:51:25
I also get the same error with the generated master flat if I use a master bias.


V.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 June 09 02:04:54
Also I would want to know which normalization is being done for the flat frame integration.

V.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 11 02:37:04
May be of interest to the BPP script users here, I just posted a bit of a tutorial on how to create a BPP template (http://pixinsight.com/forum/index.php?topic=4359.0) that remembers things like your master bias and dark files, and CC settings etc.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 11 11:02:19
Hi Vicent,

What version of PI and the FITS module are you running? Have you applied all the updates available? This looks like an old bug that I fixed some time ago.

Quote
Also I would want to know which normalization is being done for the flat frame integration.

Multiplicative normalization for output. Flux equalization normalization for rejection.
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 June 11 11:29:31
Hi,

PI is completely updated in my computer... At least that's what it says...

My FITS SO file is from April 2012. How can I see the version??


Thx,
V.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 11 11:42:55
Your installation is up-to-date. To understand what happens, I need a sample of these files. Could you upload them to Endor?
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 June 11 11:49:43
Tomorrow. ;-)


V.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 11 16:29:59
Version 1.21 has been released as an update. The source code is attached to this post, as usual.

Main changes:

- Fixed a regression where the registration reference image was being assigned to the output directory.

- Engine changes to make use of read-only output data from ImageCalibration (output file paths).

- New dark frame exposure time tolerance feature.

- Additional error checks to make the process more robust to lack of access to working files during the calibration and registration phases.

- Fixed a potential problem with weird filter names used as directory names.

- Additional diagnostics messages to warn on invalid file name characters in filter names.

- GUI fix: Push buttons no longer will have icons on Mac OS X. This is to fix some extremely ugly interface problems we've seen recently. Note that this will change once we release PI version 1.8.0.

- Added a warning message box to say that the light frames integration feature is not intended to produce a final (that is, optimal) result.

Enjoy!
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 11 16:36:17
Version 1.21 has been released as an update.
...
Main changes:

...

- GUI fix: Push buttons no longer will have icons on Mac OS X. This is to fix some extremely ugly interface problems we've seen recently. Note that this will change once we release PI version 1.8.0.

The GUI is still not the best.  Refer attached screenshot. 
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 11 16:42:44
Strange...  ::)
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 11 16:45:13
Not sure if I'm going crazy, but I'm sure that with previous versions of BPP, if I save a process icon with master darks and biases chosen, the next time I open that process icon they are already there for reuse.  This version the process icons no longer remember the files selected at time of saving.  It does remember settings like the CC process icon, and the master bias/dark boxes are ticked etc.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 11 16:46:57
Strange...  ::)

What the?  I'm on a MacBook Pro running Lion 64bit.  Where do you reckon that difference is?
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 12 00:23:12
Quote
What the?  I'm on a MacBook Pro running Lion 64bit.  Where do you reckon that difference is?

I have no idea. What happens if you resize the script's dialog? It can be resized arbitrarily in the usual way (bottom right corner). In your screenshot it looks like the dialog's size is too small, which is really weird.

Starting from version 1.8, PixInsight will no longer be a "Mac application". Its custom style will look exactly the same on all supported platforms, pixel by pixel. The 1.8 application will not depend on Mac OS X or Windows to render a single pixel of its GUI. This should fix all of these issues once and for all.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 12 00:31:15
If I resize the window, as soon as I start dragging it the bottom kind of snaps down so that the LinearFit High line can now be seen.  But the text on some of the buttons is still chopped off, as is the ImageIntegration drop down boxes etc.

Note that this behaviour isn't unique to BPP.  I've noticed it on some other scripts as well.  Some parts are chopped off or only half readable, resize and the window snaps so they are usually visible.  I don't recall it happening in modules, just scripts I think.

Looking forward to PI 1.8  :D
Title: Re: Calibration and align batch operations
Post by: georg.viehoever on 2012 June 12 00:51:29
Troy,
did you modify the default fonts used in the GUI, or are you using some utility that is supposed to enhance the Mac GUI? Just a wild guess...
Georg
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 12 02:49:09
I'm not using any Mac GUI enhancements and I don't recall changing the fonts in PI but will check tomorrow.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 12 09:19:47
This version the process icons no longer remember the files selected at time of saving.

I have just checked this on Linux, Windows and Mac OS X, and it works perfectly. Could somebody please confirm the same issue?
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 14 20:07:59
G'day Juan, not sure if you want to move this stuff to the bugs section, or it's fine here.  Sorry.

I called a friend that uses the Win version, 32 bit Win XP.  He confirmed that if he creates a process icon, it does retain the filenames as expected.

I'll try to get in touch with someone that has a Mac 64bit similar to mine.
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 15 16:44:36
Hi all,

I have just released an update with version 1.22 of the BatchPreprocessing script (source code attached). Just a few minor changes for compatibility with the new ImageIntegration module version 1.8.1, also released today.

Enjoy!
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 June 15 18:04:22
using the older script (not the release made just above this post) i'm not having any problems with the process icons not remembering the filenames.

clicking the process icon brings up an intermediate window with metadata about the script, and applying that window globally re-opens the script UI. not sure if that's normal but that's how it works on my installation. this is on lion with all the latest updates; macbook air core i7.

maybe the only caveat is that i'm only trying this whole process after having loaded a project containing the script process icons.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 16 01:49:24
Sorry to say it's still not remembering the files for me after this update  :(
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 16 01:59:59
Quote
Sorry to say it's still not remembering the files for me after this update

This is really strange. I have checked the script exhaustively on FreeBSD, Linux, Windows and Mac OS X, and this doesn't happen.

Do you have the "Export calibration files" option disabled? In such case, the script won't include any file paths in exported instances. This is by design. This option should normally be enabled (checked).
Title: Re: Calibration and align batch operations
Post by: Juan Conejero on 2012 June 16 02:32:53
Rob,

Quote
clicking the process icon brings up an intermediate window with metadata about the script, and applying that window globally re-opens the script UI. not sure if that's normal but that's how it works on my installation

Yes, this is the normal behavior. Scripts are very different from normal processes (modules) in PixInsight. There is a Script process that works like a bridge between both worlds and can be used to execute any script instance, so when you double-click a Script icon, the Script tool is shown loaded with the icon's data.

You can also run a Script icon globally without opening the Script tool. Right-click the icon and select "Execute in the global context".

Quote
this is on lion with all the latest updates; macbook air core i7.

AHA! :) I am extremely interested in knowing how the script (and the rest of PI in general) looks on Lion. Some users are reporting screen rendering problems that we cannot reproduce (Troy is an example). Could you please post a screenshot?
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 June 16 02:42:46
Quote
Sorry to say it's still not remembering the files for me after this update

This is really strange. I have checked the script exhaustively on FreeBSD, Linux, Windows and Mac OS X, and this doesn't happen.

Do you have the "Export calibration files" option disabled? In such case, the script won't include any file paths in exported instances. This is by design. This option should normally be enabled (checked).

Aah.  Indeed.  That's it!  I assumed that I didn't need any calibration files exported, because they were masters.  Enabled that option back again, and now it is remembering the files.  Thankyou and sorry for the confusion/distraction.
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 June 16 10:44:18
Rob,

You can also run a Script icon globally without opening the Script tool. Right-click the icon and select "Execute in the global context".

thanks for the tip. the intermediary process did come in handy last night though because i had updated BatchPreprocessing and it failed to run due to a different checksum. luckily i figured out that all you have to do is update the checksum in the script process window and i was off and running again.

Quote
this is on lion with all the latest updates; macbook air core i7.

AHA! :) I am extremely interested in knowing how the script (and the rest of PI in general) looks on Lion. Some users are reporting screen rendering problems that we cannot reproduce (Troy is an example). Could you please post a screenshot?

sure. strangely i do not have the UI artifacts that Troy has reported, but i did have to grow the script window a little bit. unfortunately on Lion i do still have the problem i had on Snow Leopard - as scripts run and activate the console window, PI raises itself and steals focus. i pretty much have to stop doing other work while i'm running any script in PI.

Title: Re: Calibration and align batch operations
Post by: pfile on 2012 June 16 13:03:30
oh - one other thing. i just realized that my image has been mirror-reversed by this script. changing the FITS reader to up-bottom, or giving the up-bottom hint in ImageIntegration fixes this. but i'm wondering why this even happens in the first place. the FITS reader was already set to bottom-up, and the script had bottom-up checked by default... so where did the mirroring come from?
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 July 04 19:06:06
and one more one more thing :)

i don't know if this qualifies as a bug or as a feature request, but i see that the script always overwrites files. i have some older canon RAW files that i've been reprocessing, and at the time i captured these lights, the filenames were generated as IMG_NNNN.CR2. if i have 2 files with the same name, the last one in the list overwrites any prior ones with the same name.

i thought i might get around this by doing Add Custom and assigning a bogus filter name to each night's files. however, when i do this, the lights do not show up in the list unless i add them twice, the 2nd time with a different filter name. at that point the script says the file was already added with a different filter name and displays the file in the list. finally, i then need multiple flats. if i try to add the same master flat with different filter names in Add Custom, it is not added to the list. that's probably the correct behavior though and i can easily duplicate those files at the filesystem level...

i originally tried to get around the multiple lights with the same name problem by making symbolic links to the files, but adding the files to the script causes the symlinks to be dereferenced. this might be an OSX issue more than anything else though.

can we get an option to uniquify calibrated/debayered/registered output filenames in the script?
Title: Re: Calibration and align batch operations
Post by: vicent_peris on 2012 July 07 14:13:50
Hi,

There is a problem with this script: the flat frames are classified by exposure time. IMHO, there isn't any reason for doing this. And it's a big problem if you have skyflats of varying exposures, as the script won't make any master flat frame because each single flat frame has a different exposure time.

Could this be corrected, please?

Best regards,
Vicent.
Title: Re: Calibration and align batch operations
Post by: troypiggo on 2012 July 08 16:21:49
Hey guys, for those using Win32 and having the memory problem with ImageIntegration, I don't think they get the option of adjusting the Buffer and Stack sizes with BPP as it is.  I just posted a manual edit solution (http://pixinsight.com/forum/index.php?topic=4442.msg31104#msg31104) that I'm hoping will work for them.  Perhaps these values could be available for editing in the next update?

PS - I know the idea is that the II in BPP is just a quickie and should be done manually anyway for the best results, but for some the BPP II is enough, and for those on Win32 it doesn't even work and the script isn't usable with II enabled in it.
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 July 13 12:50:35
Hi,

There is a problem with this script: the flat frames are classified by exposure time. IMHO, there isn't any reason for doing this. And it's a big problem if you have skyflats of varying exposures, as the script won't make any master flat frame because each single flat frame has a different exposure time.

Could this be corrected, please?

Best regards,
Vicent.

Hi,

I have tested with the latest update: flats are grouped by FILTER - the script integrates sky flats with varying exposure time values. Its a problem with the user interface. The first value found for a filter name seems to be displayed in the tree view.

Best,

Kai
Title: Re: Calibration and align batch operations
Post by: pfile on 2012 July 13 21:22:43
another buglet - i collected more frames on a target; opened up my project, started up the batchpreprocessing script, deleted the old lights and added in the new ones, started the script.

script apparently deleted everything in the "registered" directory before writing the new files; only the new files are in that directory and all the old ones are gone.

 the "calibrated/light" and "calibrated/light/debayered" directories correctly contain all the files, old and new. so all is not lost, i can run StarAlignment to regenerate the deleted files.
Title: Re: Calibration and align batch operations
Post by: Eddy Timmermans on 2012 August 08 07:15:19
Hi all,

Troypiggo suggested I post my question about Bpp here.
I had some problems using this tool but after trying a couple of times, I managed to find the error I made.
Instead of using prestacked master bias and flats (from DSS), I let the tool make these master files from scratch.
But now I have another problem. While trying to figure out what went wrong the first times, I used stacks of about 10 images for bias, darks, flats and lights. And that worked fine.
Now, when I try to stack more than about 25 lights, the process ends up not finding enough matching star pairs. And after 25 retries I had to manually abort the process.
Is there a way of using some sort of parameter to end this, after say 10 tries and move on to the next image? I used several different reference images so I don't think that will be the problem.
Or is it possible to make stacks of 20 lights and stack these stacks, if you understand what I'm trying to say ?
I really want to use PI for the entire process, not only post-processing. And using the information on this forum I'm quite confident, that will be possible.

Eddy
Title: Re: Calibration and align batch operations
Post by: Eddy Timmermans on 2012 August 09 00:47:24
This is a screen shot of the problem.
Apparently it has to do with the fact that the image was not guided and wide field.
But since DSS can handle the stack, PI should also be able to. So can I tweak some parameters in order to stack them anyway ?
I tried Linear Fit instead of Winsorized Sigma an changed the sliders a bit, and that seemed to work.
But after a while the problem came back.

Eddy
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 August 26 07:51:07
Hi,

There is a problem with this script: the flat frames are classified by exposure time. IMHO, there isn't any reason for doing this. And it's a big problem if you have skyflats of varying exposures, as the script won't make any master flat frame because each single flat frame has a different exposure time.

Could this be corrected, please?

Best regards,
Vicent.

This is a small problem in the GUI code - the flat frames are internally grouped by FILTER only.

 line 2182 in BatchPreprocessing-GUI.js should be

      if ( frameGroup.exposureTime > 0 && frameGroup.imageType != ImageType.FLAT )

to fix this.

Kai
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 August 26 08:08:35
another buglet - i collected more frames on a target; opened up my project, started up the batchpreprocessing script, deleted the old lights and added in the new ones, started the script.

script apparently deleted everything in the "registered" directory before writing the new files; only the new files are in that directory and all the old ones are gone.

 the "calibrated/light" and "calibrated/light/debayered" directories correctly contain all the files, old and new. so all is not lost, i can run StarAlignment to regenerate the deleted files.

The problem may be in line 960 in BatchPreprocessing-engine.js

 SA.overwriteExistingFiles = true;

This var should be set to 'false' (which is the default of StarAlignment, and the original mono CCD script uses the default)

Kai


Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 August 26 08:11:41
This is a screen shot of the problem.
Apparently it has to do with the fact that the image was not guided and wide field.
But since DSS can handle the stack, PI should also be able to. So can I tweak some parameters in order to stack them anyway ?
I tried Linear Fit instead of Winsorized Sigma an changed the sliders a bit, and that seemed to work.
But after a while the problem came back.

Eddy

Can you please upload these files?

Kai

Title: Re: Calibration and align batch operations
Post by: pfile on 2012 August 26 11:16:04
another buglet - i collected more frames on a target; opened up my project, started up the batchpreprocessing script, deleted the old lights and added in the new ones, started the script.

script apparently deleted everything in the "registered" directory before writing the new files; only the new files are in that directory and all the old ones are gone.

 the "calibrated/light" and "calibrated/light/debayered" directories correctly contain all the files, old and new. so all is not lost, i can run StarAlignment to regenerate the deleted files.

The problem may be in line 960 in BatchPreprocessing-engine.js

 SA.overwriteExistingFiles = true;

This var should be set to 'false' (which is the default of StarAlignment, and the original mono CCD script uses the default)

Kai

thanks, i should have read the source.
Title: Re: Calibration and align batch operations
Post by: papaf on 2012 September 12 00:15:17
Hi all,
I tried to use this script with some images coming from a friend's QHY8. Since this camera doesn't need any dark and he didn't do any bias or flats, I only have lights to work with. The script fails in this condition, telling me it cannot find darks.
I know the central point of this script is to calibrate lights, so running it without calibration files may seem useless. However, what I really like about this script is the DSS likeliness, automating the whole process of alignment/stacking.
My question is: is it possible, someway, to use this script without any calibration files?

I hope this is the right place to ask for this. If not, please point me in the right direction.

Thanks!
Title: Re: Calibration and align batch operations
Post by: jeffweiss9 on 2012 September 27 13:06:33
   Can someone mention what the script is asking for when the BatchPreprocessing script v1.22 fails ("couldn't execute in global context...") because it says that it's "searching for a Master Dark of 2sec exposure" presumably to match the exposure time of my 2-sec lightbox flats.  Most of the time, it works to completion without this message and this only seems to happen when I have binned 2 RGB frames.  Are we supposed to include short-exposure darks that go with the exposure time of the flats in addition to the long-exposure darks that are matched to the exposure time of the lights?  When it works, it does so successfully using only the long-exposure darks to calibrate the flats.
   As a work-around when this happens, I create the master flats the old way with ImageCalibration and ImageIntegration (provided I supply a few missing FITS headers that the script requires to use the master flats), but I would love to  understand what is happening in the first place and be able to use the script to completion all of the time.
   Thanks for the help.
-Jeff
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 October 13 06:51:56
Hi Jeff,

can you please give me a console log or screenshot?


Kai
Title: Re: Calibration and align batch operations
Post by: Shelyak on 2012 November 13 07:46:14
Hey Astronomers :)

I'm  completely new in this hobby  so I've some questions.
The first: I've downloaded the BatchPreProcessing Script from page 29 and my PI says: A Version newer than V.1.7.6. is required. But if i check my Pi version the program says that it is on the newiest version. ???

And then the 2nd question is how do i install the scripts or where I've to copy them?

I followed this tutorial: http://pixinsight.com/tutorials/master-frames/en.html to step 1: By creating a Masterdark PI gives out 3 Images called: Gray_Integration, Gray_rejection low and Gray_rejection high. Which oh these images is my Masterdark? And analogue which of these images will be my master-bias?

Kind regards from germany :D

Martin
Title: Re: Calibration and align batch operations
Post by: kwiechen on 2012 November 16 06:05:02
To use the bpp script you have to install all updates.

The master dark frame is 'Gray_Integration'

Kai