PixInsight Benchmark

[Juan Conejero]

Hi all,

I am glad to inform you that today we have started a new project: The PixInsight Benchmark.

The PixInsight benchmark consists of a set of scripts and data to evaluate the performance of a machine for execution of intermediate and advanced image processing tasks with PixInsight. This process has been carefully designed to provide our users with reliable estimates of quality for their actual user experience on the PixInsight platform. These estimates have been standardized so that all benchmark results obtained with different hardware and/or PixInsight versions are directly comparable.

The PixInsight benchmark is a community-driven project. By submitting their benchmark results, our users help us build a comprehensive database of benchmark data. This is an invaluable resource, both for us to improve our software and for our users to make informed hardware-related decisions.

For more information on benchmark data, usage instructions and the list of current benchmark results, please visit:

http://pixinsight.com/benchmark/

Benchmark results are updated automatically as new benchmarks are submitted (you'll have to refresh your browser).

The PixInsight Benchmark Script

The PixInsight Benchmark script, which we have just released as a regular update on all supported platforms, is a JavaScript implementation of the PixInsight benchmark. It requires PixInsight Core version 1.8.1.1092 or later, and will form part of the standard PixInsight distribution from now on. To run the benchmark on your machine, you just have to execute the script by selecting SCRIPT > Benchmarks > PixInsight Benchmark from the main menu. For detailed instructions and tips on how to perform and submit reliable benchmarks, see the information given on the webpage I have linked above.

I hope you'll find this new project useful for all PixInsight users.

[dzso.bacsi]

Hi,

I would be glad to provide the benchmarks, but life is never easy for the "unenlightened" like me...
The description says: "You will be asked to provide optional information about your swap disks..." Could anybody explain what are those Swap disks in PI terminology or what is the concept behind it? Should I have set my disks up according to an optimal PI way?

My disks are:
Samsung SSD 840PRO SCSI Disk (256GB) as System disk
Intel C600 SATA RAID Controller
RAID10 (4TB) - 4*2TB WD WD2002FAEX as Data disk

Just to make it more complicated I usually use a virtual memory disk (58GB) out of the 64GB system memory. It is sometimes enough for PI processing, sometimes not. If not I just simply use the Data disk.

So, what would be the PI ideal disk setup for the benchmark (and my processing exercises) with those disks?

Cheers,
Joseph

[Alejandro Tombolini]

Hi, see this

[Nocturnal]

Nice! I remember talking about this a long long long time ago

I ran it on my work lappie but the results were so slow I won't report them. Will try my other computers soon.

[sreilly]

One suggestion would be to have a notice before hand of the information wanted, scratch disk locations and make and model of those hard drives, motherboard make and model, and any comments on custom configuration you may want to send as well. After running at least three tests and hitting submit opens a dialog asking for that information and you can't access PI Global Prefs while the script is running. My first run is saved as a text file but I don't see a way to submit without running more tests. I ran a second time and submitted those. The downloaded pictures I think are about 300 MBs and are download for each test.

Also, just a note for those who don't know much about their computers, there is a free program called Blearc Advisor http://www.belarc.com/free_download.htmlthat is free for personal use and will tell you most everything you could want to know about the hardware, software licenses,network, and state of your security updates. Not sure if there are any versions other than for Windows but is a great tool to have. I run it about every several months and save the html file for each system. You can also print it out for your records.

[sreilly]

Actually to check myself, and I should have done that before posting, the downloaded pictures seem to be about 10 MBs. I guess I saw the download speed and got confused, getting old I guess.

[Sean]

Interesting results. My little MacBook Air with 4 logical processors and 8GB RAM outperforms my iMac with 8 logical processors and 12 GB RAM. The reason - 500GB Flash Drive in the Air is very fast, leading to much improved swap performance.

[Alejandro Tombolini]

Help!, I have downloaded and run the Benchmark script and have the following massage:

Error: File I/O Error: Unable to open file: No such file or directory: /tmp/pixinsight_benchmark_v1.fit

It seems that do not have image in /tmp/, and the script do not download the input image from this server. What am I missing?

Linux 3.2.0-31 generic
KDE SC 4.8.5

Saludos, Alejandro.

[slang]

Hi.

I noticed this as well. I managed to solve it by unchecking the 'secure connections' option (as well as selecting the force image input download).

I suspect that the file may be available over http, but not https, or something like that  ;-)

Cheers -

[Ignacio]

This is very useful. I already learned about multiple swap files and its effect on performance, and reconfigured PI accordingly, doubling the swap performance.

Thanks!
Ignacio

[Leandro VARGAS]

Hello,

When I try to send the Benchmark I have the next error:

*******************************************************************************
Benchmark Results
*******************************************************************************

Benchmark version ...... 1.00.07
Input checksum ......... 2cd72b67e12fff2812ef5b5da054ab2a70a25e23
Serial number .......... Y8O3D920O6OM5CLG02QF6LV070318X95

CPU Identification
CPU vendor ............. GenuineIntel
CPU model .............. Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz

System Information
Platform ............... Windows
Operating system ....... Microsoft Windows 8.1 Single Language
Core version ........... PixInsight Core 01.08.01.1092 (x64)
Logical processors ..... 8
Total memory size ...... 15.892 GiB

Execution Times
Total time ............. 02:49.81
CPU time ............... 01:21.72
Swap time .............. 01:28.06
Swap transfer rate ..... 188.217 MiB/s

Performance Indices
Total performance ...... 2770
CPU performance ........ 4632
Swap performance ....... 1042

Sending benchmark data to server...
*** Error: Could not resolve host: pixinsight.com
Error: Could not resolve host: pixinsight.com: https://pixinsight.com/benchmark/cgi-bin/v1/benchmark-submit.php?serialNumber=F9ZIZFYD8W37932US1G88K86N4LP7MDLversion=1.00.07cpuVendor=GenuineIntelcpuModel=Intel%28R%29%20Core%28TM%29%20i7-3630QM%20CPU%20%40%202.40GHzplatform=WindowsplatformBrief=WindowsoperatingSystem=Microsoft%20Windows%208.1%20Single%20LanguagecoreVersion=PixInsight%20Core%2001.08.01.1092%20%28x64%29numberOfProcessors=8totalMemorySizeKiB=16663832totalTime=166.571cpuTime=80.013swapTime=86.522swapTransferRate=191.567totalIndex=2824cpuIndex=4730swapIndex=1061machineInfo=HP%20ENVY%20dv6%20Notebook%20PC%0AC7B48LA%23ABMmotherboardInfo=Intel%28R%29%20Core%28TM%29%20i7-3630QM%20CPU%20%40%202.40GHz%0A181E%2052.24diskInfo=ST750LX003-1AC154%2C%20750%20GB%2C%20SATA%2C%207.200rpmcomments=16GB%20%282%20%2A%208GB%20%20Samsung%201600MHz%29%0ANVIDIA%20GeForce%20GT%20650M


Thank you,

Leandro

[Pierre]

This test answers a few questions. Many thanks for setting this up.

The swap file tip was not something I had really paid attention to. Boy it does have a HUGE effect on the Benchmark speed! 8)
It was 142.5 sec without the extra swap file location, dropped down to 83.94 seconds with .
I am now much more curious about doubling the RAM from 16 to 32, to see what that would do. This is on a Ubuntu 12.04 LTS using i7-3770K, Asus Sabertooth Z77, SSD 60g for the system and a WD 500G data disk.
Pierre

[eneilson]

I have a triple-booting system (Win7/Linux/OSX) so I thought it would be fun to benchmark PixInsight on the same hardware with each OS.  If you ever needed an incentive to move to Linux, this is it.  Summary of total performance by OS:

Windows 7 x64 Ultimate: 2541
OSX Hackintosh with Mavericks v9.2 x64:  2827
Linux Fedora x64: 5833  <--Damn!

The only variables in the benchmarks were the OS and the hard drives and the hard drives are all effectively 7200 rpm SATA drives,

I've uploaded the Windows and OSX benchmarks to the server, however, the Linux upload fails with an error 406 so I've attached the Linux report here.  With the more than 2x speed increase on my less than new i7-950, I'm definitely moving my processing over to Linux.

OSX:
http://pixinsight.com/benchmark/benchmark-report.php?sn=KI48JT2SLT6KJM0J6UIN35QYX3AYD6C1

Win7:
http://pixinsight.com/benchmark/benchmark-report.php?sn=50HVE72ZX658K4VZI8EE4L5TRTS21228

I hope this is helpful,
Eric

[GaryP]

Readers should be aware that the benchmark is not recommended for computers with less than 6 GB ram. I received the attached message. I would be more annoyed if the installation process weren't so quick and easy. But now I have an extra unneeded script taking up space. How do you remove a script?

[Bob Andersson]

Good fun but I think one has to interpret the results with care. There are some very high scores due to use of RAM disks as swap disks but that only works if the swap files are smaller than the (probably) limited size of the RAM disk.

My own result, lke many others, benefited from using an SSD array as the swap disk. That seems to be a good way to go although even SSDs with a high headline transfer rate can bog down under a heavy workload or at times when they are performing internal housekeeping. There's a good analysis of such issues (performance consistency) as part of this review over at Anandtech.

I'd be very wary about using the benchmark to drive the design of a machine and no, that isn't sour grapes as I (currently) have the leading score for my choice of processor. 

Update: I've just checked the size of the "fits" file supplied with the benchmark. At 10.6 MB it's tiny for an RGB image - my monochrome files are 32 MB straight from the camera and the RGB of my recent M31 image sits at 384 MB. None of us would be happy with a benchmark that required such a large file before it could be run but maybe the supplied 10.6 MB file is so small that it comfortably fits in virtually all RAM disks, even the tiny ones, without any need for paging out to more permanent storage. If so then I think that significantly distorts the benchmark results.

Bob.