Out Of Memory Error during Weighted Batch Preprocessing script

gnewell

Well-known member
1592154019445.png


Latest of everything. Windows 10 64bit. 32GB RAM.

21.5 GB available after quitting Pixinsight.

Log ends with:

[2020-06-14 16:48:08] Computing descriptors for image 123 of 207
[2020-06-14 16:48:08] G:/Astro-Pics backup/SGP/Koi Fish Galaxy/calibrated/light/2020-06-08_234436_Light_Blue_60sec_1x1__frame126_c.xisf
[2020-06-14 16:48:08] Loading image: w=4656 h=3520 n=1 Gray Float32
[2020-06-14 16:48:09] 74 FITS keyword(s) extracted.
[2020-06-14 16:48:09]
[2020-06-14 16:48:09] StarDetector: Processing view: Image01
[2020-06-14 16:48:11] 25 star(s) found
[2020-06-14 16:48:11]
[2020-06-14 16:48:11] DynamicPSF: Global context
[2020-06-14 16:48:12] 1 view(s)
[2020-06-14 16:48:12] 25 star(s)
[2020-06-14 16:48:12] 25 PSF fittings
[2020-06-14 16:48:12] 142.764 ms
 
last night when we were talking about this i thought your problem happened during ImageIntegration but clearly it is doing something else at the time.
is this a weighting phase of the script? looks like it is computing FWHMs. i wonder why it would be holding on to so much memory at this point, unless it keeps all the FWHM info in memory and thus does not release any of the image objects until it's done looking at all of them.

unfortunately the only workaround i can think of is to run WBPP 3 times, once per filter, which should get you down to ~66 images per run.

@robyx any ideas here?

rob
 
I'll try the non wieghted version, just to see if it's specific to the weighted version.

I can also try per filter, as you suggest.
 
By the way I have over 200 subs per filter...

Anyway, yeah it looks like a memory leak in StarDetector or DynamicPSF because the memory usage just keeps increasing during the "computing descriptors" phase until it runs out.
 
Actually with only the red subs I was able to get past that point.

I saw 4 "garbage collection" reductions in memory in the phase that previously ran out of memory. In other words the memory kept increasing, but then at a certain point reduced a bunch, then again started increasing with each sub. As I said I saw that happen 4 times before finishing the 210 subs.
 
interesting. well the scripts do run in a javascript engine and memory management in java dynamic as far as i understand it.
 
FYI I seem to be on the hairy edge with around 205 subs from an asi1600. Red and Blue channels when through by themselves. Green channel failed once, but made it through the 2nd time with no changes in parameters or anything.
 
Hi @gnewell, @pfile,
sorry, I just noticed this thread!

Indeed there are objects that are dynamically allocated while computing the image descriptors as FWHM, eccentricity, and SNR. Javascript garbage collector should take care of them properly and free memory once needed.
By the way, if leaving the memory management completely in garbage collecor's hands seem to produce this issue in some cases then I can update the WBPP in order to manually deallocate objects that are no more needed after the computation.

Will do on a next release., thanks for reporting!
Robyx
 
I'm trying to do WBPP on 500 subs - seem to run out of memory around 322nd sub when its doing star detection/dynamic PSF. This is OSC data.

Windows 10 2004, 32GB ram

I closed a bunch of apps that were running, will try again.
 
I closed out of everything and got a bit further

1594309507076.png


Going to try and manually select an image...
 
So it turns out star alignment breaks if the file begins with __. - I had a bad $VAR$ in my nina profile so the file wasn't getting var substitution and had a __. in file name. that broke star alignment. Using powershell to regex the filename and rename seemed to have fixed the start alignment.

as for memory - i see it go way up and drop at times but shutting down everything and i mean everything but PI seems to be getting me further - i'm on 414 of 500 - wish me luck
 
spoke too soon. same error. doh.

I have automatic registration reference image... maybe i'll try a manual one.

[2020-07-09 20:19:49]
[2020-07-09 20:19:49] Best reference frame for registration: undefined
[2020-07-09 20:19:49]
 
Hi @robyx
Hi @gnewell, @pfile,
By the way, if leaving the memory management completely in garbage collecor's hands seem to produce this issue in some cases then I can update the WBPP in order to manually deallocate objects that are no more needed after the computation.

Will do on a next release., thanks for reporting!
Robyx
Did you implement and release that manually deallocate mechanism ?
I use trial PI. The same issue is reproducted yet.
 
Last edited:
Hi @robyx
Did you implement and release that manually deallocate mechanism ?
I use trial PI. The same issue is reproducted yet.

Hi, not implemented yet. There is an update that will be released regarding the friendly management of the errors mentioned above but nothing related to memory management for the moment.

I will investigate asap.
Robyx
 
FYI I hit this error again today. Out Of Memory Error during computing descriptors for image 115 our of 313. Windows 10, PI 1.8.8-7, Weighted Batch Preprocessing v2.0.2.

I guess I will try splitting the subs into 4 groups and then running the master lights from those groups as inputs to a final master light.
 
Back
Top