PI 1.7 ImageIntegration cache

[cs_pixinsight]

Over past few days I have been integration some images and noticed the following behavior on a Windows 7 x64 computer.

• Integrate several sub frames with eval noise turned on
• The cache should now contain the noise estimates for each sub frame
• PI crashes or kill the process in Task Manager
• Restart PI and attempt to integrate the same images
• PI will once again have to eval the noise for each sub frame

It appears that the noise estimates are not saved to the cache until PI is exited cleanly.  This seems like a bug as I would expect the information to be saved to a permanent location as the processing is taking place thus preventing unnecessary extra work should an unexpected crash occur.  The current behavior would require exiting PI every time ImageIntegration is run to ensure all the noise estimates are preserved.

Craig

[Juan Conejero]

ImageIntegration keeps and maintains its data cache in memory, and dumps it to disk upon application exit. This is not a bug, but an optimization feature, and in fact it works nicely under normal conditions. We use ImageIntegration with its cache enabled all the time.

The current behavior would require exiting PI every time ImageIntegration is run to ensure all the noise estimates are preserved.

That would be true if PixInsight were an extremely unstable application. AFAIK, it is not. If you are experiencing crashes, then there are other problems (probably unrelated to ImageIntegration's cache) that need to be fixed. Are you running version 1.7 with all the updates applied? Have you tried 1.8.0 RC3?

[cs_pixinsight]

For the most part PI 1.7 is very stable (I run it for days without restarting at times), but lately while using the BatchPreprocessing script it will randomly crash while determining all the noise estimates before the final integration.  I'm integrating several hundred Canon XTi CR2 images, so maybe it's just the number of subs that isn't being handled gracefully.  Smaller sets never seem to be an issue however.

I'm curious; how much noise data is actually stored in the cache for a single image?  Given the speed of modern day disks it would have to be megabytes of data to really be a performance concern in writing the data to permanent storage after each evaluation.  Perhaps it's the writing out of the entire cache that is the concern...

I'm not a file format expert, but does the FITS format allow storing data in the header of the file?  Would it be beneficial to store the noise evaluation in the file itself, so if the cache doesn't contain the necessary data, the file header is checked and finally fall back to image evaluation if the header doesn't contain the data?  Reading file headers should be fairly quick - much faster than reading the entire file and evaluating the image again.

I don't feel comfortable using PI 1.8 at this time due to the issues still being resolved and the number for scripts/modules that are still being updated by their authors.  I'll wait until at least a month after GA before really deciding if I should test those waters.  Hope you understand.

In any case, thanks for taking look.
Craig