ImageIntegration

[edd]

I've been running an ImageIntegration on single channel 16 bit 4k x 5k (roughly) images FITS format, 450-odd of them. I've been repeatedly reducing my stack and buffer sizes due to it using up nearly all memory as it goes, and I'm on a 32G system. I've done far bigger image integrations in the past without trouble (relatively speaking) - it seems like when it starts integrating a new set of pixel rows memory isn't being freed or swapped out efficiently.

Are things bugged currently? Do I need to try to revert things to solve this for now? Before, memory usage would go up and get capped at about the level I expected, but no matter what stack and buffer sizes I use I seem to get stuck currently. I'm doing a 16k stack 16k buffer right now and have on a 32G Ubuntu system top reporting %MEM of 97 and VIRT of 40g and I'm not even at 70% of pixel rows 1058->2115.

It's agonising.

[Juan Conejero]

Hi,

I cannot reproduce this problem on any platform. I've just run a test with 125 4Kx4K images on a machine with Fedora 20 and 8 GB of RAM, default parameters, and it has worked without problems with a memory consumption of ~3 GB, as expected.

The latest version of ImageIntegration has not changed anything besides optional generation of drizzle files. Anybody else is having this issue with ImageIntegration?

[edd]

I had enabled generation of drizzle data. I'll try it without.

Unfortunately the process got killed overnight probably due to its memory use. It's on Ubuntu for what it's worth. I'll run further tests today.

edit to add: Of course last night I meant 16M stack and buffer...

[edd]

It does appear to be the drizzle data generation that's using all the memory. It completed fine with it disabled.

I suspect this isn't great data for drizzling anyway so I'll not worry about it too much for now but I guess it might be good to try to write out drizzle data during the process to temporary files rather than hold it all till the end, which is what I guess is happening now?

[Juan Conejero]

This is very strange, and I can't reproduce it either. Drizzle file generation has a very small memory footprint. Drizzle files are updated by ImageIntegration as a final step, after all images have been integrated (it has to be this way because of the way drizzle works), but this should not cause memory problems even for large images with high percentages of rejected pixels.

How have you generated the .drz files that you are using with ImageIntegration?

[edd]

Standard star alignment as in the previous thread. I've got a roughly 3x2 mosaic pattern (individual frames are not precisely aligned) so any individual sub has a fair amount of 0 pixels, but other than that I don't think there's anything very unusual about it.

I'm redoing things with a bit of extra calibration on the lights at the moment, so once that's done I'll see if I can replicate it again.

[edd]

I'm working on a smaller scale drizzle this time. Memory usage is very significant (top showing 47.9g VIRT, 29g RES, 93.6% overall on 32G system, 4G stack, 16M buffer, ~550 files) at the stage of writing out drizzle data to the .drz files. It's continuing to increase during this writing phase, which seems strange given the integration itself apparently finished.

edit to add: it does periodically release a small percentage so isn't crashing in this case, but is struggling nonetheless.