PixInsight Big Files

[TinySpeck]

As I work on an image, many mystery files build up in my operating folder.  They range in size from a few dozen kB to 100 MB, and often total about a GB after a while.

I back up my computer daily, including my current image processing folder, and I get many GB of accumulated mystery files even using incremental backups. 

What are these files for?  Do I need to restore them to restart a project after a computer crash?  Can I reduce or eliminate them?

[pfile]

they are likely the PI "swap" files. what these are is the entire processing history for every view open on your workspace.

the way projects work is that when you save a project, PI copies all of these swap files into the folder that's created when you save the project. and when you load a project, PI copies all of those files from the project folder back into the swap folder (the location(s) of which are set in the global preferences.)

when PI quits, it should delete everything in the swap folder. sometimes if PI crashes, it can orphan those swap files in the swap folder. so if you find some swap files in the swap area(s) when PI has been shut down and is not running, you can delete them.

having said all this, you say "my operating folder." i don't know what that exactly means but your swap area should be somewhere that windows/linux/macos expects to use for temporary files, not any folder in your home directory. but if you have somehow set the swap location to your operating folder, then that would explain it.

rob

[TinySpeck]

Thanks, Rob, you're always a great help and I appreciate it.

By "operating folder" I mean the folder where my project is stored.  The .seal and all image files are kept there.  I specify it the first time I Save Project.

I looked at my Global Preferences with my project open and the Swap Storage Directories shows only one: a Temp folder in my App Data / Local path (which I believe is the default).  I don't see a path to the current project folder anywhere in the Preferences.

I'm attaching a snapshot of my current project folder, with the mystery files building up, if that helps.  It looks like maybe the mystery files are written when PI closes, judging by the file times from yesterday.

As an experiment, I moved all the mystery files (only) to a temp folder, started PI, and loaded the project.  It showed this error stream on the console:

Reading image: Image_a
*** Error: Parsing ImageWindow MainView child element (line=12264 offset=6): No such file: D:/gb/Astro Image Processing/NGC6946 Fireworks Gal/Image_2.data/ZGUCXV1ZM-000001
* Loading images
Reading image: bkgd1
*** Error: Parsing ImageWindow MainView child element (line=44871 offset=6): No such file: D:/gb/Astro Image Processing/NGC6946 Fireworks Gal/Image_2.data/ZGUCXV1ZM-000008
* Loading images
Reading image: range_mask
*** Error: Parsing ImageWindow MainView child element (line=44954 offset=6): No such file: D:/gb/Astro Image Processing/NGC6946 Fireworks Gal/Image_2.data/ZGUCXV1ZM-000009
* Loading interface states
* Loading embedded XPSM
Reading icon: Process01
Reading icon: Process02
* Regenerating masking relations
* Updating workspaces
** Warning: The project has been loaded partially with 3 error(s).
1.209 s

The .xisf image files don't even load without the mystery files, so some of them at least seem to be critical.

[pfile]

ok - i see... i think you should not save any of your own files into the project directory that PI has created. PI is probably getting confused since it does not expect to find those extra files inside the project folder.

rob

[TinySpeck]

Oh, that's interesting!  So you save your project files to a different place than you Save Project?  That seems odd and counterintuitive to me, but I will give it a try as soon as PI is done with the lengthy process it's involved in now and report back.

[pfile]

yes, a project folder is created by PI when you save a project, and in fact in newer versions of PI juan takes advantage of the operating systems ability to make a container (if that capability exists on an OS), so on OSX for instance it doesnt even look like a folder, it looks like a file. thus further discouraging saving random files inside the project folder.

anyway besides saving a final image in some lossy format or maybe saving an image to disk that is needed by a process (like GradientsMergeMosaic), theres no reason to save individual files... just save the project and all the views (and their history) is saved.

rob

[TinySpeck]

anyway besides saving a final image in some lossy format or maybe saving an image to disk that is needed by a process (like GradientsMergeMosaic), theres no reason to save individual files... just save the project and all the views (and their history) is saved.

That sounds just like it ought to be.  But when I Save Project and then exit PI I get dire warnings of pixel information being permanently lost unless I save whatever working images I have open.  Images are labeled <*new*> until they are saved, also, which makes me think they need saving, and if I close an image which hasn't been saved I'm told pixel information will be permanently lost.  It sure doesn't sound like images & histories are being preserved in the project!

[pfile]

anyway besides saving a final image in some lossy format or maybe saving an image to disk that is needed by a process (like GradientsMergeMosaic), theres no reason to save individual files... just save the project and all the views (and their history) is saved.

That sounds just like it ought to be.  But when I Save Project and then exit PI I get dire warnings of pixel information being permanently lost unless I save whatever working images I have open.  Images are labeled <*new*> until they are saved, also, which makes me think they need saving, and if I close an image which hasn't been saved I'm told pixel information will be permanently lost.  It sure doesn't sound like images & histories are being preserved in the project!

yes i think this has been raised before and juan had a reason for still giving this warning after a project save. however, it's perfectly safe to ignore the warning and quit immediately after saving a project. everything is saved and it will all be reloaded when you load the project. the only exception is (and you would get a warning about this) is that the state of any active dynamic process would not be saved. you can of course save a process icon of the dynamic process and then close the dynamic process and you won't get the warning.

rob


rob

[TinySpeck]

OK, thanks again Rob.  I'll experiment with this this evening.

[TinySpeck]

Here's what I've found.  I can indeed create & change an image, save only the project, quit PI, ignore the warnings, and restart the project without a hitch.  All images and history are preserved whether they were individually saved or not.  So what in the world are those dire warnings for?  When you save your project you DO NOT risk permanent pixel information or process history loss.  Those warnings are a complete red herring.

I've also found that all the mystery files which accumulate in my project folder do seem to be necessary to recreate the project.  I can't delete all but the files from the most recent session, for example.  All mystery files (all the way back to the oldest) are also renamed to different random names each time the project is started (or exited?).

I'm not using bundled projects, so I bet that explains the files.  The aggregate of all the files seems to represent the whole project.  Next project I will do bundled, so at least I'll only have one file to deal with and to back up.

Thanks for the tips, Rob, I learned an important lesson about PI.

[pfile]

indeed and turfing any one of those files could really mess things up when restoring the project.

the files would exist bundles or not. they are the undo history for every view that's open in your workspace, so there's one for each process step that's been applied to every view. they would just be mostly hidden by the OS if you use bundles (at least OSX will let you open a bundle if you really want to. i don't know about other OSs.)

anyway only juan can really address the warning message. i agree that it is confusing. on my end i was using PI before projects existed so i guess maybe when juan introduced them he explained the warning. i can't remember anymore, it was too long ago...

rob