Slowness after a while

[Philip de Louraille]

I am processing over 1500 FITS files, 1392x1040, 1 channel.
I have a History Process applied to an image container.
The process is to open an image, apply a mask, perform a HistogramTransform and then a HDRMultiScaleTransform and save the resulting image of course
At first everything happens very fast; by the time I get to image 250 of 1500, things start to move slower. From 400+ and up, every thing is trapped in glue.

Since every image is independent, why wouldn't each instance be the same speed?
Memory leak? It seems related to swap but since there is no dependence from one image to another, I feel that there is some memory management problem at hands.

I have 8GB of RAM, working on a 3GHz Intel Core i7 PowerBook.Running under 10.9

[akiel]

Hi,

I have the same problem using image integration. If I integrate about 400 images, it slows down after about 200 images. The images are 3-channel RGB 32-bit float FITS files from a 16 MP camera.

I use PI 1.8 1071 on a Late 2013 Macbook Pro with 16 GB RAM and OSX 10.9.

[Juan Conejero]

This is most likely a consequence of Mavericks' App Nap "feature":

http://pixinsight.com/forum/index.php?topic=6560.0

Try disabling App Nap and let me know if this fixes the problem.

[akiel]

Hi,

I disabled App Nap but after my screen locked I found PI in the slow state. This are the last lines from Process Console:

Code: [Select]


Generating registered image
Homographic Projection / Lanczos-3 interpolation, c=0.30: done
Registration successful.
Writing output file: /Users/akiel/astro-scratch/M42-20131230/registered/131230-XE1-3713_c_r.fit
Writing FITS image: 32-bit floating point, 3 channel(s), 4952x3288 pixels: done

Registering target image 79 of 275
Loading target file:
/Users/akiel/astro-scratch/M42-20131230/calibrated/light/131230-XE1-3711_c.fit
Reading FITS: 32-bit floating point, 3 channel(s), 4952x3288 pixels:  85%

It takes about 25 seconds to increment the percent counter in the last line one percent. Normally reading such a 160 MB FITS file is less than a second on my machine.

PI uses 100% of one core of my cpu but is just dead slow.

[akiel]

Hi,

I have more information: StarAlignment gets slow after about 40 images while my Mac was never in sleep. After it gets slow, I can abort the process by pressing Pause/Abort in the Process Console. Now I can restart the StarAlignment (removing the processed frames) and it is fast again.

So I can get PI fast again without restarting it. I just have to restart the StarAlignment Process.

Can I help you with more information? How can I create a thread dump or something like this. I'm a Java programmer but I have no experience with C++.

[mschuster]

Thread synchronization has caused loss of performance. See this post as an example. Could this be a regression?

Mike

http://pixinsight.com/forum/index.php?topic=4210.msg29556#msg29556

[Juan Conejero]

I've got good news and bad news regarding this problem.

The good news is that I have fixed a somewhat important bug in the PixInsight Core application. Yesterday I conducted a series of tests on Linux, Mac OS X 10.8 (*not* 10.9) and Windows. Working with batches of 200 and 400 images, I discovered a bug that was causing a slowly increasing but measurable performance degradation. The bug was that finished threads were not being destroyed immediately, leading to accumulation of unused data structures in memory. The unused threads were only destroyed at the end of the process. During long batch processes, the amounts of unused data items and memory fragmentation were causing noticeable delays, especially on machines with many processor cores, and working with tasks that create many threads. The performance degradation was quite similar on all platforms. This problem is now completely fixed. In fact, the thread creation and destruction tasks are now much faster, so we've got a significant performance improvement along with a bug fix.

The bad news is that this bug cannot explain, by itself, the kind of slowness problems that you are experiencing. The bug only manifested after relatively long sequences of processes. For example, to get a relevant delay with StarAlignment I needed to run batch tasks of at least 100 images. With 400 images the process was clearly slower (about a 25%), but nothing remotely similar to what you are reporting with just 40 or 80 images.

So I suspect that the most important part of the problem is still in OS X 10.9. We haven't migrated our two Mac machines to 10.9 yet, mainly because the new version is still, IMO, of beta quality. Let's see if the new version of PixInsight, which I'm building and preparing for release right now, improves this situation.

[akiel]

Hi Juan,

I have this sample for you. I created it with the OSX Activity Monitor. I took it during a slow star detection process. During this time one CPU core was at 100% but nothing really happened (perhaps 1% star detection progress per minute).

I repeated StarAlignment several times now. Every time with different images as I try to do real work. Every time it gets slow after about 40 images.

Hope that sample helps you.

[Philip de Louraille]

I am very happy to report (since I started the thread) that my flow process is now very fast. Each image gets processed within 2-3 seconds from start to end so it takes about  one hour to process 1400-1500 images.

Many thanks Juan for fixing that bug!

[pfile]

very strange, i'm seeing some awful performance on osx and PI 1.8.1

subframeselector is taking over 200 seconds per image, and there are only 15 images in the batch.

also i've seen PI "go away" for a long time at the beginning of ImageIntegration.

rob

[pfile]

here are 2 samples taken about a minute apart while PI was 'hanging' in SFS.

rob

[mschuster]

Rob,

Wild guess: your sampling appears to show issues related to image statistics, resampling, memory management and maybe threading. Also SFS may also not conform well to recent PI PJSR updates. Can't really tell. I will be remote for a while imaging, will look again when I return.

Thanks,
Mike

[pfile]

this morning i tried it again and noticed that the per-subframe execution times were increasing, first 10s, then 20s, then 60s… not a good trend

i also had some very, very slooooow StarAlignment and top showed that the processor was stuck at 100% throughout; some phases of SA should run in parallel. i restarted PI and was unable to reproduce the problem.


rob