Updating drizzle files extremely slow on network NAS drive

[ecoles]

Recently I installed a Synology NAS system to house all my image files. The image folder was taking up more and more of my desktop drive and I didn't think adding more large USB drives was the answer. The NAS was installed along with three 6 TB drives. All my original and processing files were then transferred to the NAS. Everything was going well with processing until i did a drizzle integration.

Part of the drizzle process is updating the drizzle files. Usually that takes only a few seconds. With the files located on the NAS, the process grinds to a virtual halt. Ultimately they will be updated, but that takes so long that I end up trying to Cancel the operation. That crashes PixInsight. The bottom line is that the drizzle file updates work fine on my desktop or an attached USB drive, but not on the networked NAS.

I have been told by an expert that this is a problem with windows doing many writes on small files. Apparently that is exactly what the drizzle file update does. My question is, can the drizzle file update be changed so that  this is not an issue. Perhaps all the updates can be done in memory and the file written all at once to the external drive.

Any other suggestions would be welcome.

Eric

[ecoles]

I tried out running the drizzle process from another PC but with the same data on the NAS. The results were the same. When it came time to update the drizzle files, the process ground to a virtual halt. IMHO, this is something that can only be addressed by the PI developers. I think PI is developed in Linux, so this problem would not be apparent.

I would appreciate some kind of response from the development crew.

Thanks.

[Juan Conejero]

Hi Eric,

That happens because the NAS system that you are using does not implement file buffering, or has it disabled. Unbuffered file I/O is extremely inefficient in general, but especially on a network file system. I would investigate your NAS configuration options, since it should implement some settings to control file buffering and caching. With file buffering enabled and properly configured, NAS will always be slower than local storage, but reasonably efficient, and definitely not as slow as you are reporting.

can the drizzle file update be changed so that  this is not an issue. Perhaps all the updates can be done in memory and the file written all at once to the external drive.

This could be implemented as an option, but please realize that we have many urgent priorities before this. The current drizzle data generation implementation will eventually be improved, but I strongly recommend you try to configure your NAS file system to work with appropriate file buffering enabled.

[ecoles]

Thanks Juan. I am passing your comments to Synology. Let's see what they can do. Eric

[multispeed]

Hello is i want to save may files from BPP  on my NAS  it schows me an error 
is there an setting in pixinsight where to allow writing to the nas...
frindly regards
werner

[RadOD]

Sorry to resurrect an old thread, but I discovered this problem the hard way.  This clearly seems to be a bug rather than a  limitation of NAS's. 

When the files were located on a NAS (QNAP TS-253A - relatively new with an Intel N3150 CPU) it took 20-30 minutes per DRZ file.  It is inconceivable that a 25-30MB file was taking 20-30 minutes due to network speed constraints. 

PC CPU use via NAS was consistently 1-2%.  Network use was steady at 3MB/s.  (the network is easily able to sustain 10X that in real world use.)

Moving the files directly to the PC results in ~30% CPU use with 2.5-3MB/s disk use -- similar to the network transfer rate.

Moving files off the NAS is an awkward but easy workaround, but I don't believe this is a NAS limitation and should be looked into at some point.

Hi Eric,

That happens because the NAS system that you are using does not implement file buffering, or has it disabled. Unbuffered file I/O is extremely inefficient in general, but especially on a network file system. I would investigate your NAS configuration options, since it should implement some settings to control file buffering and caching. With file buffering enabled and properly configured, NAS will always be slower than local storage, but reasonably efficient, and definitely not as slow as you are reporting.

can the drizzle file update be changed so that  this is not an issue. Perhaps all the updates can be done in memory and the file written all at once to the external drive.

This could be implemented as an option, but please realize that we have many urgent priorities before this. The current drizzle data generation implementation will eventually be improved, but I strongly recommend you try to configure your NAS file system to work with appropriate file buffering enabled.

[danoid]

I'm not using a NAS, it's a plain jane slow WD Velociraptor.    One processor core out of six utilized, drive utilization around 1 MB/s.

At least iTunes doesn't skip while I'm waiting, there's lots of headroom...

[pfile]

it's not necessarily network speed constraints - the problem might devolve into being seek-bound rather than network or disk-stream bound depending on how many file handles PI has open and what it is doing (if it is seeking all over the place in 10 files at once, the NAS will probably fall over.)

rob