Out Of Memory Error during Weighted Batch Preprocessing script

gnewell

Well-known member
Aug 19, 2018
60
2
1592154019445.png

Latest of everything. Windows 10 64bit. 32GB RAM.

21.5 GB available after quitting Pixinsight.

Log ends with:

[2020-06-14 16:48:08] Computing descriptors for image 123 of 207
[2020-06-14 16:48:08] G:/Astro-Pics backup/SGP/Koi Fish Galaxy/calibrated/light/2020-06-08_234436_Light_Blue_60sec_1x1__frame126_c.xisf
[2020-06-14 16:48:08] Loading image: w=4656 h=3520 n=1 Gray Float32
[2020-06-14 16:48:09] 74 FITS keyword(s) extracted.
[2020-06-14 16:48:09]
[2020-06-14 16:48:09] StarDetector: Processing view: Image01
[2020-06-14 16:48:11] 25 star(s) found
[2020-06-14 16:48:11]
[2020-06-14 16:48:11] DynamicPSF: Global context
[2020-06-14 16:48:12] 1 view(s)
[2020-06-14 16:48:12] 25 star(s)
[2020-06-14 16:48:12] 25 PSF fittings
[2020-06-14 16:48:12] 142.764 ms
 

pfile

PTeam Member
Nov 23, 2009
5,298
63
last night when we were talking about this i thought your problem happened during ImageIntegration but clearly it is doing something else at the time.
is this a weighting phase of the script? looks like it is computing FWHMs. i wonder why it would be holding on to so much memory at this point, unless it keeps all the FWHM info in memory and thus does not release any of the image objects until it's done looking at all of them.

unfortunately the only workaround i can think of is to run WBPP 3 times, once per filter, which should get you down to ~66 images per run.

@robyx any ideas here?

rob
 

gnewell

Well-known member
Aug 19, 2018
60
2
I'll try the non wieghted version, just to see if it's specific to the weighted version.

I can also try per filter, as you suggest.
 

gnewell

Well-known member
Aug 19, 2018
60
2
By the way I have over 200 subs per filter...

Anyway, yeah it looks like a memory leak in StarDetector or DynamicPSF because the memory usage just keeps increasing during the "computing descriptors" phase until it runs out.
 

gnewell

Well-known member
Aug 19, 2018
60
2
Actually with only the red subs I was able to get past that point.

I saw 4 "garbage collection" reductions in memory in the phase that previously ran out of memory. In other words the memory kept increasing, but then at a certain point reduced a bunch, then again started increasing with each sub. As I said I saw that happen 4 times before finishing the 210 subs.
 

pfile

PTeam Member
Nov 23, 2009
5,298
63
interesting. well the scripts do run in a javascript engine and memory management in java dynamic as far as i understand it.
 

gnewell

Well-known member
Aug 19, 2018
60
2
FYI I seem to be on the hairy edge with around 205 subs from an asi1600. Red and Blue channels when through by themselves. Green channel failed once, but made it through the 2nd time with no changes in parameters or anything.
 

robyx

Well-known member
Jul 17, 2019
51
1
43
Zurich
Hi @gnewell, @pfile,
sorry, I just noticed this thread!

Indeed there are objects that are dynamically allocated while computing the image descriptors as FWHM, eccentricity, and SNR. Javascript garbage collector should take care of them properly and free memory once needed.
By the way, if leaving the memory management completely in garbage collecor's hands seem to produce this issue in some cases then I can update the WBPP in order to manually deallocate objects that are no more needed after the computation.

Will do on a next release., thanks for reporting!
Robyx
 

sn2006gy

Member
Jun 10, 2020
15
1
I'm trying to do WBPP on 500 subs - seem to run out of memory around 322nd sub when its doing star detection/dynamic PSF. This is OSC data.

Windows 10 2004, 32GB ram

I closed a bunch of apps that were running, will try again.
 

sn2006gy

Member
Jun 10, 2020
15
1
I closed out of everything and got a bit further

1594309507076.png

Going to try and manually select an image...
 

sn2006gy

Member
Jun 10, 2020
15
1
So it turns out star alignment breaks if the file begins with __. - I had a bad $VAR$ in my nina profile so the file wasn't getting var substitution and had a __. in file name. that broke star alignment. Using powershell to regex the filename and rename seemed to have fixed the start alignment.

as for memory - i see it go way up and drop at times but shutting down everything and i mean everything but PI seems to be getting me further - i'm on 414 of 500 - wish me luck
 

sn2006gy

Member
Jun 10, 2020
15
1
spoke too soon. same error. doh.

I have automatic registration reference image... maybe i'll try a manual one.

[2020-07-09 20:19:49]
[2020-07-09 20:19:49] Best reference frame for registration: undefined
[2020-07-09 20:19:49]