Problems with the BatchPreprocessing Script

[helgeras]

Hi,
I am have just started using PixInsight, and am trying to use the Batch Preprocessing script.

I am running on a set of 90 pictures with relatively short exposure (40 sec), which has been taken without any guiding, in short, star movement is pretty significant between the first picture taken and the last taken (looks like about 20 pixels drift).

When I try to run the script on this, I get a message box stating: "File I/O Error: Invalid or empty filename".

The log seems to indicate that the star alignment failed, and thus some files weren't created.

Is this a correct diagnosis of my problem? (piece of the log is attached below).

What can I do to get star alignment to work on these data? Which parameter should I try to change?

Regards,
Helge


Here is an extract from the process console:
.
.
.

Registering target image 11 of 20
Loading target file:
D:/Astronomy/Capture/2013-03-25/M101/Work/calibrated/light/M101-011_G_c.fit
Reading FITS: 32-bit floating point, 1 channel(s), 2749x2199 pixels: done
Structure map: done
Detecting stars: done
9645 stars found.
Matching stars ...
* Target image: Limiting to 2000 brightest stars.
337 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #2
useScaleDifferences=false
* Target image: Limiting to 2000 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #3
useScaleDifferences=true
* Reference image: Limiting to 1000 brightest stars.
* Target image: Limiting to 1000 brightest stars.
194 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #4
useScaleDifferences=false
* Reference image: Limiting to 1000 brightest stars.
* Target image: Limiting to 1000 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #5
useScaleDifferences=true
* Target image: Limiting to 4000 brightest stars.
406 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #6
useScaleDifferences=false
* Target image: Limiting to 4000 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #7
useScaleDifferences=true
* Reference image: Limiting to 500 brightest stars.
* Target image: Limiting to 500 brightest stars.
76 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #8
useScaleDifferences=false
* Reference image: Limiting to 500 brightest stars.
* Target image: Limiting to 500 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #9
useScaleDifferences=true
* Target image: Limiting to 6000 brightest stars.
376 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #10
useScaleDifferences=false
* Target image: Limiting to 6000 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #11
useScaleDifferences=true
* Reference image: Limiting to 250 brightest stars.
* Target image: Limiting to 250 brightest stars.
50 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #12
useScaleDifferences=false
* Reference image: Limiting to 250 brightest stars.
* Target image: Limiting to 250 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #13
useScaleDifferences=true
* Target image: Limiting to 8000 brightest stars.
369 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #14
useScaleDifferences=false
* Target image: Limiting to 8000 brightest stars.
*** 0 star pair matches found - need at least six matched stars.
* Previous attempt failed - this is try #15
useScaleDifferences=true
* Reference image: Limiting to 125 brightest stars.
* Target image: Limiting to 125 brightest stars.
49 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #16
useScaleDifferences=false
* Reference image: Limiting to 125 brightest stars.
* Target image: Limiting to 125 brightest stars.
7 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #17
useScaleDifferences=true
* Reference image: Limiting to 60 brightest stars.
* Target image: Limiting to 60 brightest stars.
31 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #18
useScaleDifferences=false
* Reference image: Limiting to 60 brightest stars.
* Target image: Limiting to 60 brightest stars.
16 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #19
useScaleDifferences=true
* Reference image: Limiting to 30 brightest stars.
* Target image: Limiting to 30 brightest stars.
14 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #20
useScaleDifferences=false
* Reference image: Limiting to 30 brightest stars.
* Target image: Limiting to 30 brightest stars.
15 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #21
useScaleDifferences=true
* Reference image: Limiting to 15 brightest stars.
* Target image: Limiting to 15 brightest stars.
8 putative star pair matches.
Performing RANSAC ...
* Previous attempt failed - this is try #22
useScaleDifferences=false
* Reference image: Limiting to 15 brightest stars.
* Target image: Limiting to 15 brightest stars.
10 putative star pair matches.
Performing RANSAC ...
*** Error: Unable to find an initial set of putative star pair matches

*** Applying error policy: Continue on error.
.
.
.
Lots more log after this before the I/O error

[jerryyyyy]

I will be interested in the answers here... perhaps settings changes.  I have just run my first two allignmentsand one did not fail but did a poor job on 4 globular cluster images.  The other was very sucessful on 14 galaxy images.  I am just using the defaults. 

[georg.viehoever]

...I am running on a set of 90 pictures with relatively short exposure (40 sec), which has been taken without any guiding, in short, star movement is pretty significant between the first picture taken and the last taken (looks like about 20 pixels drift).
...

The log indicates that your image could not be aligned to the reference image. Three common causes:
1. Image 11 does not have enough overlap with your reference image (you are not guding). Usually an overlap of 20% is sufficient.
2. You are doing a wide field image. In this case, lens distortions may cause that the images cannot be matched any longer
3. You stars are not round enough in image10, and StarAlignment is not properly finding stars for this reason.

StarAlignment will do 25 tries before finally failing, consuming lots of RAM and CPU time, and unfortunately sometimes crashing PI.
Try to align the problematic image by using StarAlignment directly: You can use its "Debug options" such as Working Mode=Strucure Map etc., and tune its parameters such that the images align. Then use those parameters in BatchPreProcessing.

If everything fails: Make your reference image and the failing image available as FITS files, and let the forum experts give it a try.

Georg

[jerryyyyy]

I am still using this script and get this error in diagnostics:

** Warning: No flat frames have been selected to calibrate lights for binning=1 and filter='Blue'

I run the script separately for each filter.

Can you run them together (all filters)? 

Does the script key into any (FIPS) characteristics of the lights and flats to integrate/recognitize them?

[Geoff]

You can load all the flats, lights, darks and biases in one go as long as the fits header contains info about  filters used and frame type, which will be the case if you have done the right things during acquisition.
Geoff

[jerryyyyy]

You can load all the flats, lights, darks and biases in one go as long as the fits header contains info about  filters used and frame type, which will be the case if you have done the right things during acquisition.
Geoff

Thanks.  I tried to stack 6 Lum subs of NGC2903 and get this error when I run the "diagnostics" check:

** Warning: No flat frames have been selected to calibrate lights for binning=1 and filter='Luminance'

The stack seems to run OK but I do not think the flats are subtracted off, and when I look at the FITS headers of the flats and they do not have the right entries.  I put them together folowing the tutorials and they look gorgeous to the eye. 

Perhaps I better edit the FITS entries of the flats?  In Maxim you can also accomplish this by putting the proper label in the file name.  Not much documentation of this in PI. 

[jerryyyyy]

I seemed to have resolved this problem by editing the FITS headers to include XBINNING YBINNING FILTER and IMAGETYP. 

The batch ran with all the lights and the whole mess wound up in the right designed folder...

Now with LRGB files that actually do not look too bad, I believe the next step is to register them prior to integration with the LRGBCombination tool. 

I used the Dynamic Allignment tool and tried to register everything to the L file.  the stacking was a bit off but it did integrate with the LRGBCombo tool.

SANITY CHECK  :  Is this the right sequence?  When using the Dynamic Allignment tool, I have toi do pair-wise alignment, but can register RGB all to the same L file? 

Thanks for the help... if I get this right, I can keep all my processing in PI. 

[pfile]

does the automatic star detection and alignment as part of BPP fail? the images should already be registered at the end of BPP as long as that works right. otherwise if the star quality is too poor you are right that you need to register by hand with DynamicAlignment. however, i've never had to do this. if you have that problem you probably need to work on your image aquisition...

[jerryyyyy]

does the automatic star detection and alignment as part of BPP fail? the images should already be registered at the end of BPP as long as that works right. otherwise if the star quality is too poor you are right that you need to register by hand with DynamicAlignment. however, i've never had to do this. if you have that problem you probably need to work on your image aquisition...

Thanks, was obviously easier than I thought.  Works fine this way.  Congrats to whomever wrote this...

I have been working my way through various scenarios of workflow, but it looks like color calibration is next, or I wonder if it is best to try DBE on individual images BEFORE ColorCalibration?  I can see a pretty good gradient in my L image. 

Thanks for the help! 

[pfile]

yes, i would do DBE first before anything else. then maybe deconvolution, then backgroundneutralization, then colorcalibration.

[Carlos Milovic]

I think that color calibration should be done before deconvolution, specially if you are deconvolving the lightness or luminance.

[jerryyyyy]

Thanks, I will have to read about deconvolution, but you both agree DBE before color correction?  Does it make any sense to do DBE before LRBG Combination?  Seems to make sense to me as I can have gradients going orthogonal/in different directions in different components. 

[pfile]

I think that color calibration should be done before deconvolution, specially if you are deconvolving the lightness or luminance.

good point, i have become too accustomed to working with mono images.

although, thinking about this a little more, when extracting L* in order to do the deconvolution, shouldn't you set the rgb weights to 1,1,1? in that case it seems like the color correction is 'ignored' as it were.

or are you talking about using the implicit L/L* when deconvolving the RGB image?

[pfile]

Thanks, I will have to read about deconvolution, but you both agree DBE before color correction?  Does it make any sense to do DBE before LRBG Combination?  Seems to make sense to me as I can have gradients going orthogonal/in different directions in different components.

i think if you have gradients hanging around, and they are interpreted as signal by the color correction, then you're not happy.

i usually do DBE on each of my L/R/G/B mono images. DBE actually treats each of the R,G,B planes of an image separately and computes a gradient independently for each, but the controls (tolerance, smoothing, etc.) are common. i suppose you may have an image that "wants" different tolerances or smoothing, etc. in each channel, and so it's better to do each channel separately.

[jerryyyyy]

Thanks, I will have to read about deconvolution, but you both agree DBE before color correction?  Does it make any sense to do DBE before LRBG Combination?  Seems to make sense to me as I can have gradients going orthogonal/in different directions in different components.

i think if you have gradients hanging around, and they are interpreted as signal by the color correction, then you're not happy.

i usually do DBE on each of my L/R/G/B mono images. DBE actually treats each of the R,G,B planes of an image separately and computes a gradient independently for each, but the controls (tolerance, smoothing, etc.) are common. i suppose you may have an image that "wants" different tolerances or smoothing, etc. in each channel, and so it's better to do each channel separately.

This is my suspicions confirmed.  I have a couple datasets in process and will try this.  I have to set up an Excel spreadsheet to keep track of the procedures, but at least it looks like I will be able to keep all my processing in one set of software.