[1.8RC1] Possible hacking attempt: intermodule object detachment

[georg.viehoever]

Even after upgrading to PCL2.0, I still get the error message already described in http://pixinsight.com/forum/index.php?topic=4870.msg34086#msg34086:

Code: [Select]

Python: Processing view: Mond_combined_3_very_small
Writing swap files...
0.78 MB/s
Profile function loaded
Traceback (most recent call last):
  File "python3DPlot.py", line 25, in execute_on
    lumImage=image.get_luminance()
Exception: <* Warning *> Possible hacking attempt: intermodule object detachment
*** Error: Problem while executing execute_on() (see console for details)
Reading swap files...
0.03 MB/s
<* failed *>
The terminal window shows that the problem happens inside the call to GetLuminance():

Code: [Select]

PixInsight Core 01.08.00.0932 RC1 Ripley (x86_64)
Copyright (c) 2003-2012 Pleiades Astrophoto

Before New
Before GetLuminance
The code I am trying to execute is this:

Code: [Select]

/// get the luminance of self. Note that ownership of the returned pointer is with the caller
ImageVariant * imagevariant_get_luminance(ImageVariant const & self)
{
std::cout<<"Before New"<<std::endl;
ImageVariant &res= *new ImageVariant();
std::cout<<"Before GetLuminance"<<std::endl;
self.GetLuminance(res);
std::cout<<"Before Return"<<std::endl;
return &res;
}
To be honest, I still dont understand what causes this problem. Is is a property of the self object? Is it a property of res? What can I do to get to the root cause of this?
Georg

[georg.viehoever]

When stepping through the call with a debugger, the problem appears to happen when

Code: [Select]

template <class P1>
   void GetLuminance( GenericImage<P1>& Y, const Rect& rect = Rect( 0 ), bool parallel = true ) constimage.h, line 12704 is left. I see that the destructors of ThreadData being called, and it appears somewhere there an exception is thrown. I dont see where. The screenshot show the last stack I see before the system suddenly jumps elsewhere, and the second screenshot shows the location afterwards. No idea what happens.

Georg

[georg.viehoever]

Here is the stack trace of the location where the throw happens. I am sure Juan will easily see what's going on here.
Georg

[Juan Conejero]

Hi Georg,

Just a note to let you know that I am working on this bug. I hope it will be fixed in the RC2 version that we'll release in a couple days.

Happy 2013 to everybody!

[georg.viehoever]

Good 2013 to you and the community as well!
Georg

[Juan Conejero]

Hi Georg,

I have investigated this problem. We have a good beast of a bug here From your backtrace, it is obvious that the problem happens when the threads used to compute the luminance (in GenericImage<P>::GetLuminance()) are destroyed. I can trace back the exception easily within the core application, and I know where and why it is being thrown.

An instance of pcl::UIObject (or of a derived class, such as pcl::Thread) must be created and destroyed by the same module. You cannot create an UIObject in "module A" and destroy it from "module B". When the core application detects an attempt to dereference an object from a module that didn't create it, it throws this exception. In addition, these illegal attempts are regarded as "hacking attempts" to alert the user about possible security exploits.

Something strange is happening with your Python module. Neglecting the likelihood of you being involved in hacking activities (  ), the only way for this problem to happen is if the global Module variable is being overwritten or corrupted in your module. This would explain why the core application detects different modules when the GetLuminance() threads are created and destroyed. To add more oddity to the situation, both modules are valid, or an "invalid module handle" error would be thrown before the hacking attempt error.

So..., how are you linking your module? I'll download and build it to see if I can discover something, but I bet this is a linkage issue. For example, I have seen similar problems on Linux when two or more shared objects loaded by the same application load the same shared object(s). Doing that overwrites static data.

[georg.viehoever]

Hmm, I dont know, I think I am linking correctly.The output from Eclipse when recompiling everything is attached below (note that it contains the gcc commands executed). Looks to me like what happens in the Makefiles as well).

Another possibility: It may be the case the ImageVariant is a copy (via copy constructor) of the original view.GetImage() that PI gives to execute_on(). Boost::Python does a lot behind the scenes. Would that be a possible source of problems?

ImageVariant is not a UI object. Can this be a cause of problems?

The exception gets thrown when the GetLuminanceThread object is destroyed. This object is created in GetLuminance(), which would mean that the module value changed while GetLuminance() is executed ?!?

Any hints are welcome.

Georg

Log of compiler:

Code: [Select]

03:04:24 **** Build of configuration Debug for project PiPython ****
make all
Building file: ../PythonBindings.cpp
Invoking: GCC C++ Compiler
g++ -D__PCL_LINUX -D_REENTRANT -I"/home/georg/PCL_2.0/include" -I/usr/include/python2.7 -O0 -g3 -Wall -c -fmessage-length=0 -Wno-parentheses -fPIC -MMD -MP -MF"PythonBindings.d" -MT"PythonBindings.d" -o "PythonBindings.o" "../PythonBindings.cpp"
Finished building: ../PythonBindings.cpp
 
Building file: ../PythonInstance.cpp
Invoking: GCC C++ Compiler
g++ -D__PCL_LINUX -D_REENTRANT -I"/home/georg/PCL_2.0/include" -I/usr/include/python2.7 -O0 -g3 -Wall -c -fmessage-length=0 -Wno-parentheses -fPIC -MMD -MP -MF"PythonInstance.d" -MT"PythonInstance.d" -o "PythonInstance.o" "../PythonInstance.cpp"
Finished building: ../PythonInstance.cpp
 
Building file: ../PythonInterface.cpp
Invoking: GCC C++ Compiler
g++ -D__PCL_LINUX -D_REENTRANT -I"/home/georg/PCL_2.0/include" -I/usr/include/python2.7 -O0 -g3 -Wall -c -fmessage-length=0 -Wno-parentheses -fPIC -MMD -MP -MF"PythonInterface.d" -MT"PythonInterface.d" -o "PythonInterface.o" "../PythonInterface.cpp"
Finished building: ../PythonInterface.cpp
 
Building file: ../PythonModule.cpp
Invoking: GCC C++ Compiler
g++ -D__PCL_LINUX -D_REENTRANT -I"/home/georg/PCL_2.0/include" -I/usr/include/python2.7 -O0 -g3 -Wall -c -fmessage-length=0 -Wno-parentheses -fPIC -MMD -MP -MF"PythonModule.d" -MT"PythonModule.d" -o "PythonModule.o" "../PythonModule.cpp"
Finished building: ../PythonModule.cpp
 
Building file: ../PythonParameters.cpp
Invoking: GCC C++ Compiler
g++ -D__PCL_LINUX -D_REENTRANT -I"/home/georg/PCL_2.0/include" -I/usr/include/python2.7 -O0 -g3 -Wall -c -fmessage-length=0 -Wno-parentheses -fPIC -MMD -MP -MF"PythonParameters.d" -MT"PythonParameters.d" -o "PythonParameters.o" "../PythonParameters.cpp"
Finished building: ../PythonParameters.cpp
 
Building file: ../PythonProcess.cpp
Invoking: GCC C++ Compiler
g++ -D__PCL_LINUX -D_REENTRANT -I"/home/georg/PCL_2.0/include" -I/usr/include/python2.7 -O0 -g3 -Wall -c -fmessage-length=0 -Wno-parentheses -fPIC -MMD -MP -MF"PythonProcess.d" -MT"PythonProcess.d" -o "PythonProcess.o" "../PythonProcess.cpp"
Finished building: ../PythonProcess.cpp
 
Building target: Python-pxm.so
Invoking: GCC C++ Linker
g++ -L"/home/georg/PCL_2.0/lib/linux/x86_64" -shared -Wl,-soname=Python-pxm.so -o "Python-pxm.so"  ./PythonBindings.o ./PythonInstance.o ./PythonInterface.o ./PythonModule.o ./PythonParameters.o ./PythonProcess.o   -lPCL-pxi -lboost_python -lpython2.7
Finished building target: Python-pxm.so
 

03:06:18 Build Finished (took 1m:53s.258ms)

[georg.viehoever]

Juan,

can you check if you see different definitions of "Module" from within the Python module and from within the core application? I am not really an expert for linking/scope problems, but I wonder if somehow the declaration of Module in Metamodule.h confuses the system. I checked if there are symbols called Module in libpython, libboost_pytho or any of the .o files that I generate, but of course there are none (especially not in namespace pcl), so this is not the problem.

A test code for the python module to provoke the error:
- load image into PI
- execute the following from within the Python module (e.g. paste the text to the script window):

Code: [Select]

import pixinsight as pi
print "getting luminance"
lum=pi.View.all_views()[0].image.get_luminance()
print "got luminance"
The text "got luminance never appears, and either in the PI console or on your xterm the error message about hacking appears.

Georg

[georg.viehoever]

Here is the solution to this mystery. The problem can be reproduced with the Sandbox Module as follows:
- Change the following code to SandboxInstance.cpp so that it tries to do the operation that I wanted to do:

Code: [Select]

bool SandboxInstance::ExecuteOn( View& view )
{
ImageVariant rVariant=view.Image();
ImageVariant &res= *new ImageVariant();
rVariant.GetLuminance(res);
return true;
}- From linux/g++/makefile-x86_64 remove option "-fvisibility=hidden"
- Load the newly compiled sandbox module into PI and drag blue triangle of Sandbox module on image.

The ""-fvisibility=hidden" flag was missing in my Eclipse configuration (as can be seen in the logs that I posted earlier). I must admit I do not fully understand how this flag changes the behaviour of the executable - it most likely has something to do with the way that PI loads modules. Maybe Juan can shed some light on this.

Thanks for the support in resolving this.
Georg

[Juan Conejero]

The -fvisibility=hidden and -fvisibility-inlines-hidden options make all symbols hidden by default on GNU (and compatible) C++ compilers, respectively for regular symbols and inline functions and class member functions. These options are _required_ for all PixInsight modules and applications.

By default, all nonstatic symbols are made public by the standard compilers on UNIX/Linux. This includes FreeBSD, Mac OS X and Linux. This default behavior is very unfortunate because it leads to conflicts between symbols exported by multiple shared objects loaded by the same application, degrades execution performance due to the excessive overhead caused by huge symbol tables, and blocks important compiler optimizations. This is especially problematic for heavily modular applications like PixInsight. You have just experienced how obscure and complex these problems can be.

By contrast, the default symbol visibility has always been hidden on Windows. This is why you need to explicitly tell the linker which symbols are to be exposed publicly by a DLL, either through a module definition file (.DEF), or by using special symbol attributes such as __declspec(dllexport) and __declspec(dllimport) in Visual C/C++. There are a few things---very few---where Windows development has been superior to UNIX, and this is clearly one of them. Fortunately, since version 4.0 GCC has the -fvisibility=hidden and -fvisibility-inlines-hidden options, which solve this problem efficiently. I suggest you pay a read at this article on GCC's Wiki for more information on these options. If you really want to learn how to write good shared libraries on UNIX, reading this document by Ulrich Drepper is necessary.

Besides the standard loadable module entry points (_fini, _info, _init), a PixInsight module on UNIX/Linux must export exclusively the following three symbols:

IdentifyPixInsightModule
InitializePixInsightModule
InstallPixInsightModule

These are the names of the public functions that a module provides for interaction with the PixInsight Core application. Test your module with the following command:

nm -D --defined-only <your-module-name>-pxm.so

(replace <your-module-name> with your module's actual name). If it shows more defined symbols (prefixed with 'T') than the ones listed above, then you have a linkage problem and your module is not valid. For example, the latest version of the PixelMath module:

nm -D --defined-only PixelMath-pxm.so

produces the following output:

0000000000194df0 T IdentifyPixInsightModule
00000000001954d0 T InitializePixInsightModule
000000000009f1c0 T InstallPixInsightModule
0000000000449b14 A __bss_start
000000000044a968 B __gnu_lto_v1
0000000000449b14 A _edata
000000000044a970 A _end
00000000001b4df0 T _fini
0000000000035898 T _init

My general recommendation is to always generate makefiles and project files using the latest version of the Makefile Generator script in PixInsight. This ensures that your module will install and interact optimally with the PixInsight Core application. The script also applies the best set of compiler optimizations that we have tested thoroughly with the current PCL versions. For example, the version that comes with 1.8.0 RC2 (Makefile Generator 1.73) enables -Ofast optimization by default with GCC 4.6 and later for PCL 2.0.1. It also generates project files for Visual C++ 2012, which we support officially from now on.