Drizzle Speed

[dhb2206]

I'm not sure if this is a bug (I very much doubt it) and could well be my setup. I have a 253 frame integrated image that I'm drizzling a 2779 x 2002 patch, the drizzle has been running for 13 hours now (and is almost there). Each frame is a debayered 6000 x 4000 originally raw image from a Canon EOS 760D.

I have a tootin' fallutin' machine running on Windows 10 with 8 processors and 32GB of RAM so that shouldn't be an issue. I'm running my swap location from an external USB drive (USB 3) and have the base files on the same drive, which may well be the issue. Is such speed normal? I have a feeling it used to be a bit faster.

Thanks,

David

[pfile]

use the task manager to look at the memory footprint of PI if it is still running the drizzle integration. even with a 2k by 2k patch it's possible that PI has needed to allocate > 32GB and so it's paging stuff on and off the disk, which will slow everything to a crawl.

rob

[dhb2206]

On a second pass (last was the Flame Nebula) and this the Orion Nebula. 258 frames in this one and the drizzle is running at around the same speed. 21GB of memory free. PIs working set sits just under 3GB, all 8 CPUs in use at an average 82% for PI alone so I guess it must be my images! :-)

[pfile]

this sounds like you are running an x86 version of windows? if PI can only allocate 3GB of memory then your OS must be 32-bit...

rob

[dhb2206]

Nope 64 bit - running PI v 1.8.8-4 (x64).

[pfile]

well, if it turns out that PI is under-allocating memory for this task (which is a possibility since i think ImageIntegration is set up to automatically calculate the memory usage, and prior to that people were having problems running out of memory on windows), you can look at the Buffer Size and Stack Size tooltips in ImageIntegration and perhaps increase those, following the hints in the tooltips.

since you have a 64 bit OS, it's not likely that the OS is paging stuff in and out of memory, so that wouldn't be the cause of the slowdown.

are these calibrated images? meaning are they in XISF format? XISF and fits support incremental reading. CR2 files do not support incremental reading so they have to be loaded into memory wholesale. since you say they are debayered i assume they are in XISF format and should be OK.

rob

[dhb2206]

Rob,

Thanks for the reply. Yes, all the files are in XISF format. I have noticed a behaviour change with the machine from recent updates, drizzle used to kill the thing, try doing anything else with the box and it was hopeless. Now, however, I can happily multi-task whilst drizzling.

This current run looks like it'll be heading closer to 32 hours to complete (on a much larger 4k x 3k patch of the integrated image). It is taking about 7½ minutes on each frame - the drizzle files are between 7 and 10MB in size and the XISFs having come from raws are 283MB, which is the same as it has always been. I don't see anything on the drizzle setup that would allow me to change memory management and in global preferences the only thing that would have any effect (given my limited knowledge) would be the parallel processing and threads, all of which are checked with thread priority on Time Critical (read from a second instance of PI, I'll check this when the drizzle terminates on the primary instance).

Cheers,

David

[pfile]

whoops - sorry yeah, forgot we were talking about Drizzle when i wrote that reply about ImageIntegration's parameters... duh.

well i am out of ideas, not sure what to suggest. although since they are CFA images you might try Bayer Drizzle against the calibrated but still bayered xisf files. i don't know if there is actually any performance benefit here but at least the input files will be 1/3 the size vs. using debayered files. you can see how bayer drizzle works by looking at the "Enable CFA Drizzle" tooltip in DrizzleIntegration.

rob