Hiya.
This is indeed a fascinating topic, and very timely for me. I recently acquired (hand me down) a new PC with quad core 3.07GHz with 24GByte of RAM and a stack of WD Raptor drives. Very lucky and very grateful am I. I'm (obviously, as there is no greater need) dedicating this for PI image processing.
So, I'm following these threads and understanding cpu, memory and disk tradeoff. It seems that a really good idea (if you have a bucketload of RAM) is to keep the files in RAM, or at least the temp/scratch files. I use Linux (ubuntu 12.04), and linux has some 'knobs' to tweak in this regards.
I haven't looked at setting up a RAM disk - that seems a bit coarse, but entirely do-able, I wanted something a bit smarter than a ram disk.
What I have done is investigate some kernel tuning parameters in /etc/sysctl.conf. There is a lot of information on this around the net, but
http://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ is a good description.
WARNING: This can be dangerous - do your own research, your mileage may vary, and no-one else will be accountable for any negative results.
I have set these as follows;
vm.dirty_background_ratio = 80
vm.dirty_ratio = 80
vm.dirty_writeback_centisecs = 1500
Descriptions;
vm.dirty_background_ratio is the percentage of system memory that can be filled with “dirty” pages — memory pages that still need to be written to disk
vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk
vm.dirty_expire_centisecs is how long something can be in cache before it needs to be written
So, what this actually means is that the kernel will set aside up to 80% of RAM (in my case = ~19GBytes) for disk cache. When it gets full, it WILL be written to disk, files can reside in memory for up to 15 seconds before the WILL be written to disk.
Obviously, there is a huge risk with this - a power outage will cause potential large loss of files. This isn't an issue (generally) for PI temp files, but these settings are system wide, so if your system is doing other stuff...... Caveat Emptor as my economics teacher used to say.
Anyway. I don't profess to be an expert at all, it has been 15 years since I've attempted to tune a kernel to this degree, but the results are quite stunning in my case, and the PI benchmark is exceptionally useful for understanding benefits of system tuning.
Now I have 3 x 146GByte WD Raptor drives (10,000 rpm) each dedicated for temp storage, with ext4 filesystem, with journals, dirtime, atime and something else turned off, so they're pretty quick by themselves. The benchmark results are;
Standard system;
Execution Times
Total time ............. 01:46.11
CPU time ............... 01:24.50
Swap time .............. 00:21.57
Swap transfer rate ..... 768.381 MiB/s
Performance Indices
Total performance ...... 4433
CPU performance ........ 4480
Swap performance ....... 4256With Kernel tuning (as above);
Execution Times
Total time ............. 01:27.97
CPU time ............... 01:23.02
Swap time .............. 00:04.90
Swap transfer rate ..... 3385.359 MiB/s
Performance Indices
Total performance ...... 5347
CPU performance ........ 4559
Swap performance ....... 18750So, that's a massive improvement in performance by letting the system use much of RAM for disk cache. (From memory, if the system requires the RAM for applications, it will take precedence over this disk caching...)
Some notes:
* Interestingly, despite putting these in /etc/sysctl.conf, they do not persist across a reboot - I need to issue sysctl -p for them to take effect. Must investigate that.
* Whilst this does start to look like tuning to beat a benchmark, I have tested this config in some real world tests. Running BatchPreProcessing on 25 x .cr2 files (12.2MP), including integration (but using master bias/dark etc.) saw a massive improvement. Monitoring actual disk i/o at the time showed that there was NO disk i/o during this process. This is a typical set of my inputs, so I call this close to 'real world' for me.
* When the cache does get full, or the system does decide to actually flush the stuff, you may be in for a wait - that's the trade-off/risk here.
* This config seems to beat using SSD's in my case, although not that many desktops would support 24GByte of RAM, so maybe this is academic/theoretical?
* ext4 without journalling and mounted with defaults,data=writeback,noatime,nodiratime also seems to make a pretty decent difference
* The results can be a little inconsistent, sometimes the benchmark will result in ZERO actual disk i/o, other times there will be a little bit here and there
* Oh, I commissioned an old UPS I had lying around - shoudl give between 30 seconds and 5 minutes of protection ;-)
* Also acquired some caching SATA/SAS controllers which should help with speed when something actually needs to hit a disk platter, but haven't bothered yet - why would I, given these results?
Is anyone else doing anything similar? How dangerous is this? Am I mad? (OK, don't answer the last one...)
Cheers -
