Author Topic: parallel processing  (Read 753 times)

Offline ChoJin

  • PixInsight Addict
  • ***
  • Posts: 106
    • View Profile
parallel processing
« on: 2018 February 23 11:35:45 »
Hello,

I'd like to implement a new process module, and for that I'd like to know which process module would be the best example regarding parallel processing and how it should be best done (best practice/coding-wise within the PCL framework).

NB: I know how to do parallel algorithms, my question is specific to how to do it within PCL for a process module.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: parallel processing
« Reply #1 on: 2018 February 27 07:30:58 »
Hi,

This depends on the kind of parallel processing you're going to implement. Basically, at a high level we have two big types of parallel implementations: tools that process pixels of a single image, and tools that process a set of disk image files. Both types can be mixed, that is, you can process several files in parallel and each file can be loaded in a thread and processed in parallel by several sub-threads.

The first thing you should do, irrespective of how you are going to implement your module, is reading the reference documentation for the pcl::Thread class:

http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1Thread.html

Although this class still lacks a general description, all member functions are well documented and provide you with important information that you must know to write a module implementing parallel processing. Also read the documentation for the AbstractImage::RunThreads() template static member function:

http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1AbstractImage.html#a6f12dbf60c5f32ac57961a8e35ceca17

This member function shows you how to run a set of threads with synchronized status monitoring and the possibility of aborting the task asynchronously.

A good example of the first type is the ColorSaturation tool. See the implementation in ColorSaturationInstance.cpp. The ColorSaturationEngine::ColorSaturationThread template class implements a parallelized color saturation transformation, and the ColorSaturationEngine::Apply() template member function generates and runs a set of threads to process an image in parallel chunks of pixels. This process implements an embarrassingly parallel task, that is, each chunk of pixels can be processed independently without any significant thread contention. Another excellent example of this type is the CurvesTransformation process, but in this case the code is much more complex because a CurvesTransformation instance can apply several curves in a single execution.

A complete example of high-level parallelism, that is, a process that can load and transform a set of disk files in parallel, is LocalNormalization. The LocalNormalizationThread class can work either on a disk file or on an existing image (note that this class has two constructors that differ in how the target image is specified: either a file path or a reference to an ImageVariant object). This class is interesting because it shows you some advanced features in PCL's multihreading implementation, such as using Console objects to generate thread-specific feedback and error information, or handling PCL exceptions in threads.

There are much more examples in our open-source code base, including internal PCL code (see for example the implementations of the pcl::Convolution and pcl::Resample classes), but I think you'll have enough material to start working with these tools. Let me know if you have more doubts or need more information on any PCL programming topics.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline ChoJin

  • PixInsight Addict
  • ***
  • Posts: 106
    • View Profile
Re: parallel processing
« Reply #2 on: 2018 February 27 09:58:09 »
thank you very much for the pointers! :-)