PI 1.5.2 Some problems

[eddiet]

Hi Juan, all

Congratulations on the excellent evolution of PI. It is truly approaching mega status as the Star of image processing software and I am finding myself using it more and more since V1.4.

I recently tried the mosaic star alignment on Previews and it is excellent! I'm looking forward to using this more.

I have found some problems with 1.5 however. I use the Automatic background extraction a lot with 1.4, but it is no longer with 1.5 (and 1.5.2).

I have a test (300MB) RGB Tiff that I could perform ABE in 5 minutes or so with V1.4. But after upgrading to 1.5, this process now never completes. Taking about 6 hours to get halfway through determining the background samples on channel 1. This first happened to me with V1.5.

After reverting to 1.4, the same process takes only 10 minutes to complete on the same image. I did this because my needs were urgent at the time. Sorry for not reporting it sooner!

Now that 1.5.2 is released, I thought it might be fixed, but undortunately the same problem happens 1.5.2. So now I thought I better report this!

If I perform ABE on a single frame RGB FITS image, (70MB), this takes about 1 minute per channel to complete. Not unreasonable, but a lot slower than V1.4.

When I try to save my extracted result image, I cannot. I use Save-As, the filename and path are valid. I select OK, the Save-As dialog disappears, but the file has not been saved, regardless of selected output type. Fits or Tiff, I have tried.

And when I try to exit PI, it does not exit. The process Icon disappears, however the window stays open and I can still select menu options. About 5 minutes later, I get JIT error "Win32 Exception occurred in PixInsight.exe [3456]".

This happens with both PI 1.5 and 1.5.2, however if I revert to PI 1.4, all is working correctly again.

I'm using XP Pro 64. I followed the installation instructions to de-install 1.4 before installing 1.5.2.

I am really impressed with PI generally and am looking forward to getting this sorted out so I can get right into it!

All the best

Eddie Trimarchi
http://astroshed.com

[Nocturnal]

Hi Eddie,

two things:

1) never sit on problems, report them as soon as you find them. Don't hope they 'magically' get resolved in the next version. Juan takes regressions very seriously and would most likely have fixed whatever is wrong in 1.5.2.

2) please provide the image that causes this problem along with the psm file with the exact DBE process that causes it. It is the *only* way Juan will be able to reproduce the problem as DBE generally works fine. It certainly does for me and that was on a 32 bit XP Pro machine. You can use a free service like skydrive.live.com or sendspace.com or whatever is your favorite to upload large files.

Hang in there and let's get this resolved!

[eddiet]

Thanks Sander.

I use the Bug Report system ? This is a first for me.

[Juan Conejero]

Hi Eddie,

Welcome to PixInsight Forum!

The bug tracking system is still experimental. For now, please use the Bug Reports board on this forum.

I'm working right now on the problem you've reported with ABE. I have reproduced it with a large RGB image . It seems a memory block growing problem, but I'm still unsure. Of course I'll fix it for PI 1.5.3, which is almost ready for release.

Thank you and I hope you'll enjoy PixInsight. Please show us some of your wonderful works on the Gallery!

[eddiet]

Hi Juan!

It's great to be here I have to say!

I'm very glad you found the error I reported and I'm confident you will have it sorted out. No problem! You must really have your hands full maintaining this mega-software, you have my utmost respect

I have posted my crash-bug-on-close to the Bug Reports forum. Though I'm not sure how bad a problem it is, as it only appears to happen when I close PI. Unless it is related to another problem..

Anyway, I am looking forward to ABE repair and using PI a whole lot more. It is now part of my SOP!

I will post images, certainly!

All the best, great work!

[Juan Conejero]

Hi Eddie,

This problem with ABE is quite strange. It's a slowdown in the sample generation process when the number of samples to generate is very high. For some reason the slowdown gets exponential when the process is about 50%.

I'm working to find what's up, but there is a simple workaround. Don't use the default box size and box separation parameters. I've tested this with an RGB 10000x6000 image. Instead of 5 pixels, I've set both parameters to 20 pixels and everything worked fine. With these parameters set to 5 pixels, the number of sample boxes is huge for a large image, and you don't get more accuracy with such a sample density. 20 or 32 pixels seem reasonable values for an image of these dimensions, and works without flaws on PI 1.5.2.

[eddiet]

Hi Juan,
Yes indeed, your findings tally with mine. I just tried box size and seperation of 20 pixels and my test 8000x8000 RGB FITS image completed in just 250s. Excellent!

Also, agreeing with yoru findings, when using the default size and seperation of 5 pixels, the process on the same file got only ~50% (halfway through plane 1), then seemed to stall. I left it overnight and it was no further through in the morning, after 6 or more hours.

And the best thing is, that now using 20 instead of 5, I could even successfully save the resulting image ! Great!

The very odd thing, is that using the defaults of 5 on even a small image, the ABE process succeeds in around the same time ~200+ seconds, but saving the image fails without a reported error. The Fits selection box does not appear, only the save-as box disappears when Save is chosen, but no file results. Very odd!

But it is good that we now at least have a workaround. Thank you!

[Juan Conejero]

Hi Eddie,

The problem is now identified and fixed. It was a race condition between the threads used to calculate image statistics for each sample box. The problem arises when background samples are very small and each thread has only a few pixels to process. When this happens and there is a large amount of samples, some threads start spending more time blocking mutually than doing their processing work, and this leads to a degenerate situation where the whole process "freezes".

To demonstrate this, enter this command on PI's console:

Code: [Select]

parallel -disable
This command disables parallel processing for the whole PixInsight platform. Now run ABE on your image with the default 5x5 samples, and you'll see how everything works fine (and not slower than when you use your four processors, since ABE is one of the few processes that I still haven't parallelized completely).

To undo the command above:

Code: [Select]

parallel -reset
and to check that everything is again OK with parallel processing:

Code: [Select]

parallel -query
Thank you for discovering this problem.

[eddiet]

Hi Juan,
Yes indeed. Process completed with default box size/sep on my 8k x 8k image in 20 minutes with parallel disabled. Still about twice as long as PI 1.4, but quite acceptable!

So I imagine the problem did not occur at all on single processor/core machines. Well done picking this problem up Juan!

Many thanks

I still have some other instability though...Can you tell me what happens for you, if you start an ABE process on a big file, then Abort it midway. The amount of time is not relevent, after a few seconds only.

After aborting the process, can you still open new images ? I cannot. And closing PI takes several minutes, before I get the process actually closes and I get the close crash I mentioned in the other thread.