Enhanced Parallel Swap File Storage: Again

[sreilly]

I'm in the process of building a new processing computer and am moving up to an i7-5820K 6 core processor. I have 32GBS DDR4 RAM and a 1TB boot SSD (Samsung 850Pro) and a Samsung 840 500GB that will be used for a swap drive. What I'm unsure of is the process of creating these swap locations. What I had done before is simply point to E:/PI_Swap as the location. From reading thread http://pixinsight.com/forum/index.php?topic=7644.0 it appears I might improve my performance by adding that again 3 more times. Is that correct or do I need to create these folders and give them different names? Actually as I type this I'm beginning to think I need to create partitions and not folders? Is there a simple step by step procedure for the correct way to do this? I'm more confused now than I was when I started this thread....

Steve

[jkmorse]

Steve,

I am assuming you are using Windows and so will just let you know what I found to give me the best performance with a similar high end desktop that I built.  I read in one of the posts from Juan that it is best to have the swap files be at least 65Gb in size so I built a 65Gb partition that I use for nothing other than as a swap folder location.  At the front end I also use a ramdisk of 16Gb (you should keep your ramdisk to no bigger than half of your Ram).  I then point to the ramdisk 4 times and then the 65Gb swap folder twice, though you should test different setups (i.e., pointing to the swap file 4 times).  I tested it each time using the benchmark script as my measure of success. 

Btw, you'll love the Samsung SDDs.  They are great upgrades.

Welcome thoughts from others, but that's worked for me.

Best,

Jim   

[sreilly]

Jim,

That's correct, Windows 10 Pro 64 to be exact. My current computer is the same except I have an i7-2600K CPU which has performed well considering its specs. It's time for this computer to be repurposed for the observatory to run the multiple programs and equipment required for automated, unattended imaging. I have put SSD drives in all my computers over the past few years and have enjoyed very fast boots and program executions. MY WD Black platter drives are mostly for storage. I don't throw away any good raw data and also use a USB3/eSATA dual docking unit for the bare storage drives. This allows me to backup all the data to drives that are only active during backup and then stored back in their static bags. I use Beyond Compare 4 for data comparison/transfers.

I haven't looked at creating a ramdisk but it may be worthwhile. It's been many years since I used one. Now I kinda wished I had gone with 64GBs RAM.

The latest benchmark test after increasing the swap files was:

CPU Identification

CPU vendor ............. GenuineIntel

CPU model .............. Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz



System Information

Platform ............... Windows

Operating system ....... Microsoft Windows 10 Pro

Core version ........... PixInsight Core 01.08.04.1195 (x64)

Logical processors ..... 8

Total memory size ...... 31.968 GiB



Execution Times

Total time ............. 01:31.42

CPU time ............... 01:09.24

Swap time .............. 00:22.16

Swap transfer rate ..... 747.974 MiB/s



Performance Indices

Total performance ...... 5145

CPU performance ........ 5467

Swap performance ....... 4143



Additional Information

Swap disks ............. Samsung EVO 840 500GB

Motherboard ............ Asus P8Z68-V Pro


Thanks Again,

Steve

[jkmorse]

Steve,

Just delivered my equipment to a remote site and will soon be embarking on unattended imaging myself, once the bugs get worked out at the new location.  What control software are you using?  I only ask in hoping its AutoPilot 5 since that what I will be using to allow me to continue to use the SkyX Camera Add-on as my acquisition software.  If you are using AP5, would love to hear your thoughts (or even if not, anything I should be particularly attentive to).

Also, I plan to use RadMin as the remote server/client software and Dropbox Pro for file transfers but would welcome hearing your method.

Thanks,

Jim

[sreilly]

The remote setup I have been using belongs to a friend that expects me to keep it busy. So far we're still getting everything up running properly. I build the computer to dual boot as the system is a TOA-150/STL-11000M and a 12.5" RC/STX-16803. It was easier for him to use having each system boot into which ever system he wanted to use and already have the software already configured, easier for me as well. Both sit side by side on an AP3600GTO with the optional encoders and limit switches. I think that mount can handle a 24" scope! The software is the same I use at home, ACP/MaxIm. To be exact, ACP Expert. This allows me to list a ton of projects and they get run depending on how I set up their constraints as far as minimum altitude, Moon avoidance, time before/after crossing the Meridian and a ton more. I set the series up by filters for LRGB, RGB, HaRGB and so on. I have targets that won't get done until next fall and that's OK. I just add them as I come across them. Using a rotator I have my FOVi setup in TheSkyX and use the J2000 coordinates along with the rotator PA. Sit back and wait for the data to pour in. For efficiency I calibrate the images on site via remote connection. I use Remote Desktop (secure firewall connection onsite), TeamViewer free, RealVNC, and LogMeIn. But I can also use the ftp server in ACP to transfer the raw data.

Make sure you take very careful attention to cable management because a snag there is way different than at home. While I do have Sam at SRO to do things that are needed he's not there 24/7 and he also has all the other systems to maintain as needed. If something comes loose or tangled it could be a day or two before its dealt with. A single setup should be much simpler.

HTH,

Steve

[sreilly]

Doing some additional testing with the new system:

i7-5820K
MSI X99-SLI MB
32 GB G. Skills 2400 DDR4 RAM
Primary Drive C; Samsung SSD 850 EVO
Secondary Drive: Samsung SSD EVO 840

I had setup the second SSD as a swap drive using 4 folders for swap as PI-1, PI-2, PI-3,and PI-4 and listed in the Global Settings as such. Max swap speed was listed as 909.13 MiB/s, not to bad
I set up and added another group in the default location for a single drive, User/Steve/AppData/Local as PI-1 and PI-2 which yielded 1165.705 MiB/s, even better
Lastly, I deleted the entries for the secondary drive and just used two folders on the root of the primary drive as PI-1 and PI-2 as well as the two User/Steve/AppData/Local as PI-1 and PI-2  which was even better at 3207.128 MiB/s

What I can gather from this is that using Samsung Magician software that came with the drives allows me to configure the drives for performance, reliability, and maximum capacity. I have mine set for performance. The software also allows you to set one drive up in Rapid mode. I had this set for the primary drive leaving the secondary as standard. Using the secondary drive as the swap drive actually slowed down the swap drive performance as it wasn't running in Turbo mode. Eliminating the use of the secondary drive for swap allowed the faster drive to increase swap performance significantly due to it being in Rapid mode.

So the question would beg to be asked if this is the right way to approach this or should I change the Rapid mode to the secondary SSD and let everything else, Programs and such, run at the normal SSD speed?

CPU performance went up just by have a 6 core CPU versus the older 4 core. Both systems had 32 GB RAM and the difference between DDR3 and DDR4 is questionable from what I've been able read. The new CPU/MB required DDR4 and that's why I have it.

Any thoughts?

[sreilly]

Using the same tests as before and only using the secondary SSD in Rapid mode the results were:

Swap transfer rate: 2081.697 MiB/s

So better than the first two methods but still 50% slower than using the Primary drive in Rapid mode. That was interesting to do but I'll have to compile the overall data and post for a more detailed report. More on this later.

[sreilly]

With the assistance of my better half I managed to compile the data and come up with this report. I've attached as a PDF file so hopefully it will be easily opened. The conclusion is I'm much better off with having one of the SSD drives in Rapid Mode (SamSung Magician Software for Samsung SSD). What I'm not clear about is which drive is best served in this respect. Which of the numbers gets highest regard? Total Performance or other? I'm guessing that the higher averaged Total Performance rating would be best but that's unclear. I see various CPU rates in the same series (settings) tests as well as Swap transfer rates. I guess the real question is what is the real world difference? I know I'm running faster from the older system but I couldn't quantify that if I had to.

Your Thoughts? Am I over thinking this?

Thanks,

Steve

[jkmorse]

Steve,

Re overthinking, for me the key real world test is drizzle integration.  On my laptop that can literally take hours.  On my special built desktop it only takes a couple of minutes for a 50 sub run.  I also see real improvement in all my other runs that take substantial processing power, I.e., star alignment, image integration, etc.  As I see it, that's real and matters.  All the rest is just numbers.  Depends on your level of patience.  But trying to go from an integration run of 45 seconds to one of 40 seconds feels like chasing my tail.  I'd rather be processing data.

IMHO

Best,

Jim