Dedicated machine for Batch Preprocessing

[Earl_UK]

Working from one PC its becoming increasingly a pain when running a large number of Subs through the Script as it does make the PC unusable especially during star alignment.

If I use a 4 socket server with quad core opterons will PI utilise all 16 cores? and have 32gb ram? running windows 2008 r2 server (to support the sockets i can get a server like this on the cheap far cheaper than an i7 machine)?

I would access this via a remote session and transfer the results to my desktop pc for working with after?

Any thoughts?

[mschuster]

If by unusable you mean in Win7 for other, concurrent processes, try this:

Preferences > Parallel processing and threads
   Uncheck Allow using all available processors
   Set Maximum number of processors used to one less than all

Mike

[Earl_UK]

Windows 7 only supports 2 socket CPUS but as many cores as thouse two CPUs have.

Im looking at having 4 quad core CPUs this is only supported with certain Server editions of windows.

[mschuster]

OK. Would make sense to farm the work out to multiple cheap machines? Then you can scale up easily.

[cfranks]

There is a simple way of increasing the through-put on your single PC if you have a lot of RAM.  I use Dataram's RAMDisk and have allocated 16GB to a RAM-Disk.  PI's Swap file is directed to this 'disk' and the BatchProcessing script now runs in 1/5 of the time previously. 

memory.dataram.com/products-and-services/software/ramdisk

Charles

[Earl_UK]

a Ram drive would be nice, however this quad core i use as my man PC only supports 8gb which im already at. for everything else I do its perfectly fine.

before i go clicking buy I just need to confirm that PI will run on 4 CPU's for a total of 16 cores.

[Earl_UK]

Ah ha !!

RTM

The PixInsight platform will use all processors and processor cores available on your machine. We strongly recommend using it on a quad-core or six-core processor.

[georg.viehoever]

If StarAlignment is your bottleneck, try tuning its parameters. See http://pixinsight.com/forum/index.php?topic=4540.msg31669#msg31669 for more.

I did some tests with PI on a windows server machine that had 24 cores. It did run the test not much faster than a retail 6 core machine. The speed you get from a machine with many cores strongly depends on the actual problems. For most image processing operations a system is limted by memory bandwidth, not processing power.

That said: i dont think PI limits the number of cores you are using. But the speedup you get from additional cores may be limted. Do test before buying a large machine. Also consider renting a cluster (eg. Amazon EC2), and run in a distributed fashion.

Georg

[Earl_UK]

Hi Georg

thanks for your reply, its interesting to see that throwing core in does not make that much of a change, ill have to look into the ram drive route instead, either way its a new pc

Cheers

[Juan Conejero]

If StarAlignment is your bottleneck...

I did some tests with PI on a windows server machine that had 24 cores. It did run the test not much faster than a retail 6 core machine.

This will change with the next version of the StarAlignment tool. Right now StarAlignment is the main bottleneck for preprocessing in PixInsight. Although SA uses all available processors and cores, it implements a low-level parallelization scheme that does not scale well. In the next version we'll implement high-level parallelization in SA: Each core will work to register an image, so if you have 16 cores you'll be registering 16 images at the same time. The ImageCalibration tool already works this way.

[Earl_UK]

This will change with the next version of the StarAlignment tool. Right now StarAlignment is the main bottleneck for preprocessing in PixInsight. Although SA uses all available processors and cores, it implements a low-level parallelization scheme that does not scale well. In the next version we'll implement high-level parallelization in SA: Each core will work to register an image, so if you have 16 cores you'll be registering 16 images at the same time. The ImageCalibration tool already works this way.

Very interesting is there an eta on that?

[Andres.Pozo]

If StarAlignment is your bottleneck...

I did some tests with PI on a windows server machine that had 24 cores. It did run the test not much faster than a retail 6 core machine.

This will change with the next version of the StarAlignment tool. Right now StarAlignment is the main bottleneck for preprocessing in PixInsight. Although SA uses all available processors and cores, it implements a low-level parallelization scheme that does not scale well. In the next version we'll implement high-level parallelization in SA: Each core will work to register an image, so if you have 16 cores you'll be registering 16 images at the same time. The ImageCalibration tool already works this way.

I would like to ask you to implement ALL the processes with the capability of executing on a view not only on a file. For example, neither ImageCalibration nor CosmeticCorrection can be executed on a view. The problem with this is that this processes can not be used in a ProcessContainer. The ProcessContainer allows to automatize sequences in a very easy way.