Batchpreprocessing and image integration file i/o error.

[focaldepth]

Hi This is not a new bug but it has been arround ever since I started using Pi a number of years ago. It has been reported on here several times but has not changed at all. I am on the latest version as of this date.

I get random(ish) failures due to FILE I/O error, file in use by another user when using BPP or Image integration. It seems worse when the files are on a network drive (Linux SMB in my case) but also occurs on local files. I have tried changing afinitiy to 1 CPU but it still occurs.

It occurs on windows 10 and 8.1. My images are 8MP Starlight eXpress fits. Pixinsight Standard Edition 01.08.04.1195 Ripley (x64)

There must be a path through the code where files are created and not closed then reopened again.
I have noticed that on this run there were some images where no stars were found in some images when alining. I will remove them and try again....

Please look into this as it spoils an otherwise amazing piece of software.

[msmythers]

I have been running PI for the last 4 years on Windows 7 and have never run into this. The error seems to indicate that at that exact time some other program was accessing the file. I have found over the years(decades now) that this is most of the time some software or multiple software like Anti-Virus or Malware detection or both at the same time checking the file. I've even had automatic defragmentation software behave this way. What I do on my computers is only run anti-virus software that allows me to selectively excluded folder paths and exclude programs or files of my choice. I then exclude the main PI folder and the swap folder or folders. I also exclude the main PI executable. All the files were checked while being downloaded and installed. That is just how I deal with the possibility of anti-virus/anti-malware programs messing up a long processing session. I never allow automatic defragmentation and of course you never defrag a SSD drive.



Mike

[focaldepth]

I am aware of virus checkers and even more likely windows (pointless) file indexing. I have tried disabling both ( on W8.1) and I still have the same problem from time to time. Even though it affects random files, once it occurs it hits the same file over and over, if I delete the file it moves onto the next file in sequence. I have been in computer hardware and software for decades and this just FEELS like an unhandled error path to me. I run all sorts of file heavy processing on many computers, compiling, photo processing, video processing, data scraping, hard real time........ on Linux and windows and others and this is not a common occurrence (except with windows file explorer). There is something that PI is doing in the image integration that is not handled quite right.


I am trying to find a repeatable, testable scenario, and will of course post it if I mange to do so. Unfortunately it is the nature of stacking that we have quite a few files in a stack so pinpointing the offending combination is tricky.

[focaldepth]

Well, I think I have a work round for now.

The bug shows up when I have duplicate file names in different directories. This causes things to get confused. All I had to do was rename all source files to have unique file names.

I still think that this is a bug and BPP should append/prepend a unique number to processed files.
It must be common for file names to duplicate from night to night.
A bug/feature request already posted asks for this. https://pixinsight.com/forum/index.php?topic=7381.msg49483#msg49483

I have encountered this effect when using image integration but have no evidence of the problem. I will start a new thread if/when I have something useful to say about it.

If this could be resolved I would be a happy imager. If not then maybe this thread will help others.