Batch PreProcessing error

[tscottt]

Over the past few months, I've tried using the Batch PreProcessing script to no avail (I've been falling back to DSS).  Thinking it may be because of the path-names associated with network-stored files, I moved all my files to a local directory and tried again.

I am getting the same behavior:  The script stops after integrating the Bias frames, and throws the error "invalid view identifier".

See the attached screenshot.

I did run "diagnostics" first, and the script said everything was fine.

[tscottt]

Well... I tried to re-process my images in stages.  I ran the script to make a master dark and master bias, then ran it again to make a master flat.  Finally, I ran it with the masters selected and with my light frames.  I was careful to specify a working directory and a reference light frame.  It wrote the intermediate calibrated and then registered files to the working directory (yay!).

See the settings picture (attached) to see the... well... settings.

However, on the last step, it gave an error (see attachment).  I believe I've seen this error before when I tried to process other images.

Can somebody help me understand what I did wrong?  Also, why would the script have almost succeeded with master calibration files, but not with the original CR2 files?

Thank you!
 -Scott

[georg.viehoever]

This has been reported before (e.g. http://pixinsight.com/forum/index.php?topic=3654.msg29824#msg29824 http://pixinsight.com/forum/index.php?topic=4314.msg30303#msg30303), and I believe it has been fixed-but I could not find the corresponding forum message. Did you update PI to the latest version of scripts and modules?

[tscottt]

Thank you for the reply Georg!

I have 1.07.06.0739.  I believe that's the latest - at least I don't get any notifications about available updates.

I'll check the thread you mention and see if I can find a fix in the discussion there.

-Scott

[georg.viehoever]

I think it was related to something in the fileformats modue. Check if you have all updates (e.g. the CosmeticCorrection process). If not, see http://pixinsight.com/forum/index.php?topic=4191.msg29433#msg29433.

Georg

[tscottt]

Well... 1.8 RC7 and I still can't use Batch PreProcessing with my Canon DSLR images.  I must be doing something silly, as nobody else seems to be having this problem.  I have master bias and a master flat frame, and individual darks and lights.  I'm still getting the File I/O error.

Can somebody please help me understand what I may be doing wrong with this module?

[georg.viehoever]

My guess is that it is caused by your rather elaborate filenames. Try changing them to something like "light_1.fits", "light_2.fits" etc., and try again. The fits module of PI has a history of failing with certain file names (e.g. containing spaces or "()"),
Georg

[tscottt]

Great idea Greg!  I tried shortening the filenames of the Light frames to something like:

 "M17_30s_F1_8_61a.cr2", "M17_10s_F1_8_61g.cr2" etc.,


 and still get the same problem and File I/0 error.  The Dark, Flat and Bias frames are all PI generated and have rather short names.   

I remain at a loss.  I wish I could use PI to do my Stacking, but after a year of trying (off and on) to get PI to do the trick, I'm still using DSS.

 

[georg.viehoever]

Hmm, I am out of ideas.
Juan, maybe you can chime in. Maybe you can convince PI to produce a more helpful error message.
Georg

[tommy_nawratil]

hi,

no, Scott, you are not the only one running into problems using the preprocessing script and CR2 files.
Have 1.8 RC7 and PP script 1.30

I sometimes get the File I/O error message, and also "cannot integrate less than three files" while there are 67 selected.
Sometimes I can get through by just clearing all the files from the list, and simply re-selecting them.
Then running the script again without changing anything else.
Fortunately that happens not too often, and then the script works like a charm.
Unfortunately I am not able to specify what could be a cause.

Just noted: File I/O error occurs after registration, so the c_d_r.fit files produced by the PP script may be used with the integration module,
and that leads to a successfully integrated master light. The PP script just could not do the integration for some reason.
My filenames are all as given by the Eos ;-)

cs, Tommy

[tommy_nawratil]

hi,

this indeed is a little strange...

I have a set of 27 DSLR RAWs, and 25 bias, 20 darks, 20 flats. All loaded nicely into the PP script.

After dark integration it stops "cannot integrate more than 3 images"
After clicking ok, the script surface appears again, and I click "run" again without any change.
Then it integrates the flats.
Starts integrating the light frames.
Console stops with "Star alignment: 7 succeeded, 20 failed, 0 aborted, 0 skipped"
"End of registration"
Error message appears: "File I/O error: Invalid or empty file name"
Script stops. No integrated light.

This is an older set of pics, got them without dithering, so all the stars are on roughly the same place in all frames...
All names are unchanged as given by Eos 450d, IMG_XXXX.

Running PI 1.8 RC7  core 1.08.00.1022   PP script v1.30 - updated everything today

Played around last night, and today, no chance.
With older PP script, there was no such problem, worked prfectly. But I heard integration module also was changed.

Fed same set of pics into DSS, after 5 minutes I had the integrated frame. So data should be ok.

With fits pics from my CCD, no such problems.

clear skies, Tommy

[papaf]

I think you should stop at the first problem, for now.
Be sure that your darks are all in the same group, and no subgroups gets created. If you could post a screen capture of the dark listing from the script, it would be helpful.

[Juan Conejero]

I cannot reproduce this problem with any of our test data sets. From your screenshot, some of your images are not being registered, and this is probably the origin of the problem. This looks like a bug in the BPP script that causes it to abort when one or more images cannot be registered. I'll try to fix this.

Anyway, you should investigate why some of your images cannot be aligned. Are these images very noisy?

As always... I need a minimal sample of the data where the error can be reproduced. Otherwise I have no way to know what happens. If you can upload some images I'll take a look.

[pfile]

somewhat related - there used to be a bug where duplicate input filenames would not be uniquified in the pipeline and so if you had 100 IMG_0001.CR2 files, only one (probably the last one) would remain as the _c image. that's not happening here, i think, but who knows if it's related.

[jerryyyyy]

I have downloaded some images from the French PixInsight group and they do not align with the PreProcessing Script.  According to Ze French, this stated with version 1.30 and it worked in 1.29. 

Can I get the earlier version somewhere to try.  The images look fine to the eye. 

Here is the correspondance (in French, with the error I get):

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

la V1.29 ne plante pas, c'est un bug du script.

Nicolas Outters
Président d'AIP
www.astro-images-processing.fr


Le 30/07/2013 18:18, Jerome Yesavage a écrit :
> V 1.30
>
> -----Original Message-----
> From: president [mailto:president@astro-images-processing.fr]
> Sent: Tuesday, July 30, 2013 8:53 AM
> To: Jerome Yesavage
> Subject: Re: Bienvenue à AIP
>
> Jérome,
>
> Quelle version pour le script batch preprocessing script ?
>
> Nicolas Outters
> Président d'AIP
> www.astro-images-processing.fr
>
>
> Le 30/07/2013 16:13, Jerome Yesavage a écrit :
>> Bonjour,
>>
>> Je suis désolé pour mon française de mauvais qualité, mais le
>> problème
> est:
>>
>> J'ai téléchargé tous les fichiers mais le "Batch Preprocessing Script"
>> ne peut pas intégrer les "Red"... je trouve que tous les entres dans
>> le forum date de deux mois... y-a-til des réussites avec ces fichiers?
>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR:
>>
>> Registering target image 3 of 3
>> Loading target file:
>> C:/Users/yesavage/Desktop/Pixinsight.France/PI
>> Output/calibrated/light/NGC1333_Red03_c.fit
>> Reading FITS: 32-bit floating point, 1 channel(s), 2048x2048 pixels:
>> done Structure map: done Detecting stars: done
>> 401 stars found.
>> Matching stars: done
>> 139 putative star pair matches.
>> Performing RANSAC: done
>> *** Error: RANSAC: Unable to find a valid set of star pair matches.
>>
>> *** Applying error policy: Continue on error.
>>