Calibration and align batch operations

[Juan Conejero]

New version 0.7 attached:

 * Added batch cosmetic correction functionality.

 * CSV text files are now selectable as registration references -- useful to register against a synthetic star field.
 
Cosmetic correction works by selecting an existing CosmeticCorrection process icon. The script automatically finds CC icons and presents them in a combo box. Additional diagnostic checks have been also implemented to detect most related pitfalls. I am sure some of you won't agree with this system (Harry, I'm hearing you), but as I see it, this has three important pros:

- It allows fine tuning CosmeticCorrection using the nice real-time functionality that Nikolay has implemented.
- It allows for much more refined and accurate corrections than a simple defect map.
- It does not clutter the script's interface.

and just one con:

- It is not a simple one-parameter thing.

But look, for simple one-thing things there are other applications out there. We are not that way. Please check it and let me know if it works.

[Harry page]

Hi

Cool I will have a go , I am not adverse to better it was that it " seemed to be a problem "

BTW whats all this listening to users    " Tiger and spots "   

Harry 

[Juan Conejero]

I forgot to mention something important. The CosmeticCorrection icon used in the script is just a template to provide functional parameters; it does not need to be defined completely. In particular, it does not need to have a set of target frames defined---in fact, existing target frames in the icon will be ignored and the script's calibrated frames will be used instead.

The same happens with the output directory, the output postfix, and the CFA option of CC; the script will force these parameters to the corresponding subdirectory in the calibration tree, to the string "_cc" (no prefix), and to the state of its "CFA images", respectively. Also, the executed CC instance will overwrite possibly existing output files.

[Juan Conejero]

Cool I will have a go

Great, let me know if you like it.

BTW whats all this listening to users

You know we love you 

[Harry page]

Hi

Getting this , will not run

Harry

[Juan Conejero]

This happens because your light frames have the keyword FILTER=Ha1.00 in their headers, but your flat frames don't. The script tries to match sets of images acquired with the same filters.

The solution is easy though: add your flat frames as custom files, and specify: image type = flat, filter = Ha1.00. It works fine this way (I've checked).

[mmirot]

New version 0.7 attached:

 I am sure some of you won't agree with this system (Harry, I'm hearing you),

Your the boss Juan  


When is better a bad thing?  If we wanted mediocre we all would be someplace else.

Seriously, I have not tried it yet but sounds great.

Max

[Harry page]

Hi

The script did not do this before    Not sure this is an improvement

I thought add files was going to look at the fits header !!!

Harry

[Harry page]

Hi

The script falls over a lot when selecting files , I will try and repeat it

Also I hate having to make sure all files need the correct name header etc 

But the cosmetic thing works quite well 

Regards Harry

[Harry page]

Hi

If you use custom to add more than one type of file it corrupts the script !!

when you use custom to say add a flat , make sure you have the flat tab selected as it puts the file in the selected tab ?? so if you have the darks selected it puts in in there 

I understand why you want to make a script which trys and calibrate all subs in one go , but as I said ages ago let me decide which files to use , why do we need custom !!! just have a tick box to force the script to use selected files ( after a user warning if you like )

Harry

[georg.viehoever]

New version 0.7 attached:
 * Added batch cosmetic correction functionality.
 * CSV text files are now selectable as registration references -- useful to register against a synthetic star field.
...
 

This is working beautifully for me. Look at this beautiful result 

Seriously, I have never been able to produce a decent and reproducible calibration with PI, even when following all those recipes posted on this forum. I really don't know what I did wrong again and again, there must be some magic ingredient that is now encoded in those scripts. To give you an example, have a look at the screenshot: The best I had been able to do with a wide field from the Grand Canyon until 2 weeks ago was the left hand image - this was after days with PI, and I still had not figured out how to get rid of those hot pixels. I figured out how to eliminate the hot pixels last week only last week, after another 5 hours of experiments. Now the whole job including the hot pixels runs in just 1.5 hours after a couple of mouse clicks, see right hand side.

Thanks a lot for the script  .

I am probably in danger of stretching my luck, but may I propose 3 additional functionalities for the script?

- for non-fits files, add a functionality that determines the image type based on the name (bias*, flat*, dark*,light*), similar to what you do based on keywords for fit files in StackEngine.prototype.addFile . Saves a lot of mouse clicks for those poor DSLR users.
- add a functionality that does an inital integration. I know that ultimately I have to tweak the integration parameters by repeating those runs, but at least the script would provide a basic result when I wake up in the morning (and would have filled the file cache). Also, this would bring the script to the same functionality level as DSS - which I would finally retire. Maybe you can choose the same route as for CosmeticCorrection: let the user provide a process icon for integration.
- I know that the script cannot provide all the functionalities that the basic processes can. For those of us that want to fine tune the calibration after a first run of the script, it would be helpful if the script were able to produce the basic process icons it used to do its job. These could be used as a starting point for the manual tweaking process.

I am a happy man today  ,
Georg

[mmirot]

I processed a 150 images last without any problems. 

A couple notes:

Cosmetic correction works great but I don't see why this is greyed out when calibrate only is checked. It may be desirable to produce calibrated and Cal_CC images only at times.

Dark Master flat file name : Can temp be added to the name for clarity?

I think retention of paths would have helped several times last night.
I had a trucated file that caused an error. I had go back a reselect everything from scratch just to remove the single bad image file and start over.

Also, I split my work into two script runs since I had multple nights. The weather got warmer so the lasts few nights were performed at -30C  rathar than -35. I was using the same masters except for the dark. Also, I was aligning to the same image and save file to the same directory. If it retained all the paths it would have been easier.

My last request:
It would be very helpful to select a flat and light subsets. This is for nights with a different flats set or rotator meridan flips as I explained previously. It could be done through the custom button I think.

Max

[Juan Conejero]

Version 0.8 attached:

* Optional integration of registered light frames (master light frame generation). This is enabled by default, and there are specific integration parameters for lights. Take into account that this is only intended as a preview of the final image, not as a final result. A careful manual integration process is always required in order to get the best possible image out of the calibrated and registered data.

* Rejection maps are now stored with the master images in multiple image FITS files (using IMAGE FITS extensions). This option is now enabled by default. Since the ImageCalibration tool does not support multiple image files, rejection maps are only stored for master lights, not for master bias, dark and flat frames, by this version of the script. This will be fixed soon with a new version of the ImageCalibration module.

* For FITS files without IMAGETYP keywords and other file formats, the script can now infer image types from file names with 'bias', 'dark', 'flat' and 'light' substrings (case insensitive). For FITS files, keep in mind that the IMAGETYP keyword has precedence over file names, so if the keyword value is wrong the selected image type will also be wrong, irrespective of a correct file name.

* Improved FITS header generation. More complete information on image integration results is now stored as FITS keywords.

* Some GUI refactoring.

Hope you like it!

[georg.viehoever]

...Optional integration of registered light frames (master light frame generation).
...can now infer image types from file names with 'bias', 'dark', 'flat' and 'light'

Did I mention that I love you?  Will you marry me? 

* Rejection maps are now stored with the master images in multiple image FITS files (using IMAGE FITS extensions).

I am not sure that rejection maps for master images are really necessary. I believe it is sufficient to fix bad pixels in the calibrated lights only. Or what are they good for?

Fantastic. Never expected this in my wildest dreams.
Georg

[Harry page]

Hi

Before I start winging again  I would like to say thanks for this , a much needed item In Pixinsight

Now winge begin

I still can not select the same file twice ( i.e master bias instead of darks )   still hate I have to mess about with file names 

Small bug -the scroll bar on the consol disapears after running the script  resizing the consol makes it reapear  and could the final stack be left on the screen

for inspection

Much Love from your PI users

Harry