Creating and compiling a new PCL module

[jmurphy]

I wish to port NormalizeScaleGradient to C++. So far, I have refreshed my knowledge of C++, cloned the PCL repository onto my Windows 10 PC, and started to look at source code to see how it all works. I have installed Microsoft Visual Studio Community 2019. I am making a start by looking at a simple module: 'FastRotation'. I will then probably start by copying LocalNormalization, using it as a template as I transfer the NormalizeScaleGradient JavaScript code to C++.

So far I have not been able to figure out how to compile and create a dll for a module. I can load a visual studio project (for example, 'E:\PixInsight Dev\PCL\src\modules\processes\Geometry\windows\vc16\Geometry.vcxproj', or 'E:\PixInsight Dev\PCL\src\modules\processes\ImageCalibration.vcxproj' but this leaves red squiggles underneath many definitions, so I assume this is not the correct way to compile a module.

I see that PixInsight provides a 'Makefile Generator Script'. I assume that I will need this once I start coding the new NSG module. I have set up the necessary environment variables. However, I have not yet figured out what to put in the various text boxes.

Any help would be very useful.
Thanks
John Murphy

 

[pfile]

more than likely your project directory is going to be:

$(PCLDIR)/src/modules/processes/contrib/jmurphy/NormalizeScaleGradient/

the project identifier you can leave blank and the ID will become NormalizeScaleGradient, taken from the path above.

if you set the target to windows instead of all platforms that might save some headache, but i don't think anything bad will happen as the generator will just generate makefiles which you end up not using.

beyond that, you may not need any of the -I or -L or -l stuff (last 3 boxes) unless you do have external dependencies for this project, but it seems like NSG might be self contained. when i built StarNet for the mac i had to at least point to the tensorflow header directory, but i think that was it. everything is dynamically linked on OSX so no tensorflow stuff was compiled in to the StarNet module itself. windows might be the same? (does DLL stand for dynamically linked library?)

once you have the makefile, at least on unix-like systems all i've ever had to do is just "make all" in the project directory and the module library file is eventually created. however, before this i've always had to go back up higher into the PCL tree and build various stuff. generally the errors produced when running "make all" point the way to what .o files the module needs, and for whatever reason most of this stuff has been under $(PCLDIR)/src/3rdparty. once the module successfully builds the library file is in $(PCLDIR)/bin.

maybe it makes sense to try this whole flow out on LocalNormalization just to pipe-clean everything...

rob

 

[jmurphy]

more than likely your project directory is going to be:

$(PCLDIR)/src/modules/processes/contrib/jmurphy/NormalizeScaleGradient/

the project identifier you can leave blank and the ID will become NormalizeScaleGradient, taken from the path above.

if you set the target to windows instead of all platforms that might save some headache, but i don't think anything bad will happen as the generator will just generate makefiles which you end up not using.

beyond that, you may not need any of the -I or -L or -l stuff (last 3 boxes) unless you do have external dependencies for this project, but it seems like NSG might be self contained. when i built StarNet for the mac i had to at least point to the tensorflow header directory, but i think that was it. everything is dynamically linked on OSX so no tensorflow stuff was compiled in to the StarNet module itself. windows might be the same? (does DLL stand for dynamically linked library?)

once you have the makefile, at least on unix-like systems all i've ever had to do is just "make all" in the project directory and the module library file is eventually created. however, before this i've always had to go back up higher into the PCL tree and build various stuff. generally the errors produced when running "make all" point the way to what .o files the module needs, and for whatever reason most of this stuff has been under $(PCLDIR)/src/3rdparty. once the module successfully builds the library file is in $(PCLDIR)/bin.

maybe it makes sense to try this whole flow out on LocalNormalization just to pipe-clean everything...

rob

That does get me a bit further Thanks.

 

[jmurphy]

However, when I run the MakefileGenerator, I get an error:
*** Error [000]: C:/Program Files/PixInsight/src/scripts/MakefileGenerator/MakGenUtility.js, line 137: Error: removeDirectory(): Relative directory.

I think this might be due to my pathnames starting with "E:/" instead of "/". If this is the case, how can I get it to run on Windows?

 

[pfile]

i'm out of my element with windows... never did a windows build, hopefully someone else knows...

rob

 

[jmurphy]

I am making some progress ...
To fix compile errors, I added:
#define __x86_64
to line 225 of include/pcl/Defs.h

I am not sure if this was the correct thing to do, but most of the source will now compile.
Windows 10, i7-4770 CPU, 64-bit operating system, x64-based processor

 

[jmurphy]

(1) In the PCL\src\modules\processes\ImageCalibration\windows\ folder there are two folders:
vc16 and vc16c
What is the difference, and which one should I be using?
(2) What order do I need to compile things in? Can I just compile a single module, or do I need to compile 3rdparty, installer, pcl and utils first?
(3) How do I start the compile? I assume I start it from Visual Studio 2019, using Build -> Build Solution ?

I have set the following environment variables:

To fix compile errors, I added:
#define __x86_64
to line 225 of include/pcl/Defs.h
I am not sure if this was the correct thing to do, but most of the source will now compile.
Windows 10, i7-4770 CPU, 64-bit operating system, x64-based processor

My current attempts at compiling a module fail with the following errors:
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppBuild.targets(1363,5): warning MSB8012: TargetPath(C:\ImageCalibration-pxm.dll) does not match the Linker's OutputFile property value (E:\ImageCalibration-pxm.dll). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Link.OutputFile).
1>LINK : fatal error LNK1104: cannot open file 'PCL-pxi.lib'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1075,5): error MSB6006: "link.exe" exited with code 1104.
1>Done building project "ImageCalibration.vcxproj" -- FAILED.

 

[Juan Conejero]

You must always use MakefileGenerator for PCL development.

*** Error [000]: C:/Program Files/PixInsight/src/scripts/MakefileGenerator/MakGenUtility.js, line 137: Error: removeDirectory(): Relative directory.

Honestly, I haven't used the MakefileGenerator script on Windows for many years now. To fix this problem, replace line 135 in MakGenUtility.js:

   if ( dirPath.indexOf( '/' ) != 0 )

by the following:

#ifeq __PI_PLATFORM__ MSWINDOWS
   if ( dirPath.indexOf( '/' ) != 2 || dirPath.indexOf( ':' ) != 1 )
#else
   if ( dirPath.indexOf( '/' ) != 0 )
#endif

A good option for PCL development on Windows is using a virtual Linux machine running Kubuntu 20.04 LTS, which you can install on VMware or VirtualBox. Anyway, if you encounter further problems with MakefileGenerator, let me know and we'll fix them.

(1) In the PCL\src\modules\processes\ImageCalibration\windows\ folder there are two folders:
vc16 and vc16c
What is the difference, and which one should I be using?

vc16c is for hardware compatibility builds, which we are not using in current versions of PixInsight. Simply ignore this folder.

(2) What order do I need to compile things in? Can I just compile a single module, or do I need to compile 3rdparty, installer, pcl and utils first?

installer and utils are not needed at all. In principle you don't need 3rd-party libraries either, but if you are using PSF fitting routines you'll need cminpack. You must always build the PCL library in first place.

(3) How do I start the compile? I assume I start it from Visual Studio 2019, using Build -> Build Solution ?

We never use VS's IDE. We use the msbuild command-line utility exclusively for all of our Windows development.

To fix compile errors, I added:
#define __x86_64
to line 225 of include/pcl/Defs.h

Never modify PCL headers. You won't need to do that if you generate your makefiles and project files correctly with MakefileGenerator.

 

[jmurphy]

You must always use MakefileGenerator for PCL development.



Honestly, I haven't used the MakefileGenerator script on Windows for many years now. To fix this problem, replace line 135 in MakGenUtility.js:

   if ( dirPath.indexOf( '/' ) != 0 )

by the following:

#ifeq __PI_PLATFORM__ MSWINDOWS
   if ( dirPath.indexOf( '/' ) != 2 || dirPath.indexOf( ':' ) != 1 )
#else
   if ( dirPath.indexOf( '/' ) != 0 )
#endif

A good option for PCL development on Windows is using a virtual Linux machine running Kubuntu 20.04 LTS, which you can install on VMware or VirtualBox. Anyway, if you encounter further problems with MakefileGenerator, let me know and we'll fix them.



vc16c is for hardware compatibility builds, which we are not using in current versions of PixInsight. Simply ignore this folder.

installer and utils are not needed at all. In principle you don't need 3rd-party libraries either, but if you are using PSF fitting routines you'll need cminpack. You must always build the PCL library in first place.

We never use VS's IDE. We use the msbuild command-line utility exclusively for all of our Windows development.

Never modify PCL headers. You won't need to do that if you generate your makefiles and project files correctly with MakefileGenerator.

The modified MakefileGenerator appears to work for the PixInsight Clas Library, finishing with:
==> Generating VC++ 2019 project file:
E:/PixInsight Dev/PCL/src/pcl/windows/vc16/PCL.vcxproj
* Directory created: E:/PixInsight Dev/PCL/src/pcl/windows/vc16/Win32/Release
* Directory created: E:/PixInsight Dev/PCL/src/pcl/windows/vc16/x64/Release
* Directory created: E:/PixInsight Dev/PCL/src/pcl/windows/vc16/Win32/Debug
* Directory created: E:/PixInsight Dev/PCL/src/pcl/windows/vc16/x64/Debug

Thanks, John Murphy

 

[jmurphy]

I have successfully built PCL.vcxproj from within Visual Studio.
Thanks, John Murphy