crash after StarAlignement with large data sets in 01.08.04.1171 (x64, Win7)

[tommy_nawratil]

hi Mike,

I think we will get this problem solved, so I would not take too much time to find a workaround.
My friends all have been stepping back to 1.8.3 but I could not.
For some reason - I don't have the time to find out - it could not recognize its own xisf files "unknown file format".
So I could not work on my projects. I think of using fits again for future compability.
After coming back from holiday, the guys will take care I am sure.

Tommy

[Philippe B.]

hello

After vacation I installed 1.8.4.1171 version (coming from 1.8.3) on win 8.1 64bits "Pro" and I also got some kind of freeze when aligning 500 DSLR images.
After 200 images (I don't know the exact number), the aligment (star matching process) become very slow.
If I stop, PI stay freezed during a long time and I need to close the application task !
I launch PI and restart my alignment at the file where stopped and the process continues at normal speed.
The RAM is OK (I have 64GB on computer and less than 5 or 8GB is used in the task manager.
Also, when freezing, the PI processor task is arount 10% (so windows in not freezed)

I have to make further test but I cannot share 100GB of files to make the test. 

[msmythers]

Philip

Could you not align in smaller groups using the first image in the total set as the reference? Say groups of 50.

In my tests only once killed PI in the task manager. Other wise it GUI always came back to life. With the 265 image test I did it was 21 minutes after the alignment was finished. Realizing that I might have a faster machine your time might be longer. The thing I noticed was if you click inside more then a couple of time the dialog box would ask if you wanted to wait or close the program. I always said wait. My guess for my computer is if I did 500 images then the gui would not return for somewhere between 7 to 10 times longer then it did for 265 images. So 21 minutes would become 2.75 to 3.5 hours.

The other thing I noticed was the actual alignment was a good percentage faster in the 1123 vs 1171. It may be that StarAlignment is more robust and more accurate now or it could be memory/io operations are just that different now. This is probably the slowdown your seeing. I just didn't do a large enough group of images.

Like I said I think the way to approach this is to do alignments in smaller groups for now.


Mike

[Philippe B.]

Yes, Mike, I agree I could process group of 50 images with the same reference (VIEW) file for each
But as I have 2600 images to align for differents deep sky objets, between 25 and 550 images per object depending of the DSLR (A7s with 30s exposure and D810A with 5 min exposure), I launch the "batch" of alignment when I am at work or during the night and I see the result after.


But I will do 50 or 100 images. It seems to work fine in this case.




cheers

[msmythers]

To automate the job use a single process container. Make a process icon for every star alignment grouping and drop the process icons into the process container. Then do a global execute of the process container. Each star alignment grouping will run one after the next.

[Philippe B.]

Good idea ! thanks  


I always forget to use this Process container to automate some process 

[Philippe B.]

I did this method (42 images per StarAlignment process) last night but unfortunately, it slowed down after 220th image.

I saw the time of saving image (XISF and DRZ file) :

From 1 to 220, I got an image processed every 60s (around).
222th image took 7 minutes
223th took 14 minutes
224th took 12 minutes
225th took 12 minutes
226th took 20 minutes
227th took 4h25 !!!!!!!
Now I stopped but PI still frezzed since >1hour
Processor task for PI at this state is always 10-11%


I did "CTRL-ALT-SUPPR"  


And I launched the process with all image... if I can go to again 220, I will do again and again...
 
Or I will take my MBP with 1.8.3 to do this long process. It is less powerfull than my home win 8.1 computer (i7 extreme 12 core + 64Gb of RAM) but will be faster in fact 

[msmythers]

Your slow downs are interesting as I had no progressive slow down during the actual alignment process or 265 images. Each alignment took the same amount of time. I had the GUI non-responsive after the alignment finished for a period of time. There was a difference in processing time between the older version of PI and 1.8.4.1171. With the  265 images the alignment finished after 115 minutes. Then I had a period of non-responsive GUI for 21 minutes. It is this lack of GUI response that for me is disturbing. That on my machine get worse as more subs are added.

What is the dimension of the images you were using during your run? Also if I'm reading things correctly your computer is miles faster then mine. I only have an AMD X6 processor and 16GB of ram.

[Philippe B.]

In fact it seems to be more complicated than that !


I process DSLR images from Sony A7s (12Mpix) and Nikon D810A (36Mpix). So XISF calibrated images (32bits) have 142MB (A7s) and 426MB (D810A)
A7s have huge amount of 30s exp time images (200 to 500) and I am doing Drizzle
D810A have longer exposure time so 10x less images. So, let says from 20 to 50. No drizzle for them.


I processed D810A images and after the starAlignment of 30 or 50 images where the speed is fast (1-2 min per image), at the end, the GUI is freezed too.
On A7s, even if I do 50 per 50 or less, at 1 moment, I will have GUI freezed. Sometimes when finished


I use PENTAGONS (and no triangle similarity). I don't know if it has an impact (but for me the result is better because sometime it cannot find matching stars with triangle similarity .
Anyway, I will retest with triangle similarity...


Yes, my computer is "boosted" and I use also SSD. And it is Windows 8.1 where the 1.8.3 had no problem on it.


Cheers

[Juan Conejero]

If you want to help me diagnose and fix these problems, please try the following:

1. Download the following zip file:

https://pixinsight.com/update-test/20150827-corelibs.zip

2. Uncompress it on any folder.

3. Copy all files extracted in step 2 to the following folder, replacing existing files and folders:

C:\Program Files\PixInsight\bin

You'll need administrative rights to do this. If you have installed on a nonstandard location, change "C:\Program Files\" as appropriate.

Let me know if this patch fixes all of these issues with version 1.8.4 on Windows.

[Philippe B.]

Thank you Juan
I will do a full test with 300 or 500 images over the night.

[msmythers]

Juan

I'll post my findings in the next few hours. Thanks!


Mike

[msmythers]

Juan

A quick response from the updated /bin files. The issue I was having http://pixinsight.com/forum/index.php?topic=8903 has gone away. I am in the process as I type of batch debayering my 265 images. There is no perceivable mouse or keyboard interruptions. The cursor is not disappearing now.

Mike

[msmythers]

Juan

I timed the entire process externally. The align process completed in 106 minutes. The Gui was not released for another 21 minutes. Here is something that I did not notice before, the Process Console is reporting an incorrect time. It is showing 04:47.84. Sorry to say but no real difference overall.


Mike

 

[Philippe B.]

Juan
Same result for me than before
after 220 images, the process solwed down to rise 1h instead of 1min for each image. 250th took 4h 
Needed to CNTRL-ALT-SUPP PI as GUI didn't come back after 1h