PCL: creating new pixels out of existing ones

[Nocturnal]

Hi,

so I've been playing (fighting is more like it) with PCL for a few days. Made a little progress but now I'm thoroughly stuck in template land.

Here's my code:

Code: [Select]


   template <class P>
   static void Apply( Generic2DImage<P>& img, const SampleInstance& instance )
   {
  int c = 0;
       int src_w = img.Width();
       int src_h = img.Height();

        String id = String().Format( "_RGB%02d", c+1 );
ImageWindow window( src_w/2, src_h/2, 3, 32, true, true, true, id );
View view = window.MainView();
ImageVariant imgvar = view.GetImage();
Image target = imgvar.AnyImage();
int row, column;
double color_val[3];
typename Image::sample* f;
for (int row = 0; row < src_w/2; row++) {
for (int col = 0; col < src_h/2; col++) {
//for (int p = 0; p < 3; p++) {
f = target.PixelAddress( col, row, 0 );

*f = * P::PixelAddress(col * 2, row * 2, 0); // red pixel
}
}
window.Show();
window.ZoomToFit(true);
   }
};

bool SampleInstance::ExecuteOn( View& view )
{
   try
   {
      view.Lock();

      ImageVariant v;
      v = view.Image();

      StandardStatus status;
      v.AnyImage()->SetStatusCallback( &status );

      Console().EnableAbort();

      if ( !v.IsComplexSample() )
         if ( v.IsFloatSample() )
            switch ( v.BitsPerSample() )
            {
            case 32: SampleEngine::Apply( *static_cast<pcl::Image*>( v.AnyImage() ), *this ); break;
            case 64: SampleEngine::Apply( *static_cast<pcl::DImage*>( v.AnyImage() ), *this ); break;
            }
         else
            switch ( v.BitsPerSample() )
            {
            case  8: SampleEngine::Apply( *static_cast<pcl::UInt8Image*>( v.AnyImage() ), *this ); break;
            case 16: SampleEngine::Apply( *static_cast<pcl::UInt16Image*>( v.AnyImage() ), *this ); break;
            case 32: SampleEngine::Apply( *static_cast<pcl::UInt32Image*>( v.AnyImage() ), *this ); break;
            }

      view.Unlock();

      return true;
   }

   catch ( ... )
   {
      view.Unlock(); // Never leave a view locked!
      throw;
   }
}



I'm trying to create a simple superpixel debayer module. I've got a basic GUI working but the important stuff, actually painting pixels is a bit complicated.

You can see that I create a standard 32 bit float color image with half the resolution of the source image. Elsewhere I make sure only monochrome images can be used as the source so let's not worry about the ExecuteOn dispatcher. I'm tempted to get rid of the templates and restrict this to only 16b integer images but I'd like to understand this better.

Why does VC++ complain as follows?

Error   1   error C2039: 'PixelAddress' : is not a member of 'pcl::FloatPixelTraits'   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   2   error C3861: 'PixelAddress': identifier not found   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   3   error C2039: 'PixelAddress' : is not a member of 'pcl::DoublePixelTraits'   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   4   error C3861: 'PixelAddress': identifier not found   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   5   error C2039: 'PixelAddress' : is not a member of 'pcl::UInt8PixelTraits'   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   6   error C3861: 'PixelAddress': identifier not found   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   7   error C2039: 'PixelAddress' : is not a member of 'pcl::UInt16PixelTraits'   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   8   error C3861: 'PixelAddress': identifier not found   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   9   error C2039: 'PixelAddress' : is not a member of 'pcl::UInt32PixelTraits'   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer
Error   10   error C3861: 'PixelAddress': identifier not found   c:\pcl\src\pcl-dev\debayer\debayerinstance.cpp   107   Debayer

The PixelTraits classes appear to be incorrect. Why isn't P an Image class which has the PixelAddress method?

Thanks!

[Nocturnal]

I should add that I perused the samples of course. The problem is that most code uses PCL primitives to actually modify the pixel values. If you can point me at an example that 'paints' pixel values at arbitrary locations that would help a lot.

[Juan Conejero]

Hi Sander,

One problem with your code is here:

Code: [Select]

Image target = imgvar.AnyImage();

It should be:

Code: [Select]

Image* target = static_cast<Image*>( imgvar.AnyImage() );

since ImageVariant::AnyImage() returns a pointer to an AbstractImage.

Normally, you must use ImageVariant::IsFloatSample() and ImageVariant::BitsPerSample() to learn the exact type of image transported by an ImageVariant instance. In this case, however, you know for sure that imgvar transports a 32-bit floating point image, since you have just created a new image window with that format, and imgvar is a shared image that refers to its main view's image.

Another problem is:

Code: [Select]

*f = * P::PixelAddress(col * 2, row * 2, 0);

because PixelAddress() isn't a static member function (this explains the 'is not a member of' error from VC++). I assume that your source pixels come from the img argument to your Apply() routine, and that you want to iterate the three R, G, B channels of img to form your target image. Then, a good way to do this would be:

Code: [Select]

for ( int c = 0; c < 3; ++c )
{
   float* f = target->PixelData( c );
   for ( int row = 0; row < src_w/2; ++row )
      for ( int col = 0; col < src_h/2; ++col )
         P::FromSample( *f++, img.Pixel( col*2, row*2, c ) );
}

FromSample() is a static member function of all pixel traits classes, and is particularly optimized for this kind of transfer loops. The pixel traits class is customarily identified as the P template argument throughout PCL code. P::FromSample() allows you to convert from P::sample to any other numeric type. In this case, we are converting from typename P::sample to float.

Note also that the code above uses a single pointer variable, namely f, to access all pixels of target in a linear fashion. In the PCL, all pixels in an image channel are guaranteed to be accessible as a contiguous block. When applicable, this technique is much faster than calling PixelAddress() for each pixel.

Other than that, instead of PixelData::PixelAddress() you can use PixelData::Pixel(), as above, which is more friendly (and yields easier to understand code) since it allows you to use references to pixels samples instead of pointers. PixelData::Pixel() is a convenience inline function that simply expands to (code taken from pcl/PixelData.h):

Code: [Select]

sample Pixel( int x, int y, int c = 0 ) const
{
   return *PixelAddress( x, y, c );
}

Of course, there is also the non-const version of this function:

Code: [Select]

sample& Pixel( int x, int y, int c = 0 )
{
   return *PixelAddress( x, y, c );
}

so you can assign a value directly to Pixel() (provided that the value is of the P::sample type). (side note: In a future version of the PCL these functions will be obsoleted and replaced with identical functions called SampleData(), SampleAddress() and Sample(), respectively. Of course, existing code won't be broken).

Code: [Select]

I'm tempted to get rid of the templates and restrict this to only 16b integer images

Oh, please don't give up. The whole PixInsight/PCL is based on template constructs similar to these, so they can't be such a bad thing :-)

Come on, you're close to have your first module working (and I'm really happy with that)

[Nocturnal]

Great, thanks Juan. Let me see if I can get it working now

[Nocturnal]

Hi,

ok, making some progress here. Because this is a debayer routine I can't just sequentially step through the source image, I need to address individual pixels and copy them into the target. I figured the easiest way to get started is to create a super pixel debayer as there's no interpolation except for averaging the green pixels.

I'm probably blowing over the image size limit and an 'unknown exception' is getting raised that is not getting caught by the catch(...) that is supposed to unlock the view. So after this happens PI has to be closed as there's no way to recover.

Of course I need to fix this bug but I also need to make sure future exceptions get caught properly. Any ideas what might be going on?

[Juan Conejero]

Hi Sander,

At this point I need to see your code to help you. An unknown exception usually happens in the core application, after a module causes some severe corruption. It must be a wrong access to image data, or something wrong with view handling. If I see what you're doing I'll be able to fix it.

[Nocturnal]

Hi Juan,

the basic code hasn't changed since the first post so the exception handling shouldn't be affected. I didn't want to ask you to fix my bug, I should be able to do this myself. It's working 'almost' in that I can see the debayered image getting created but for small images the bottom part isn't debayered and large ones cause the crash. So I'm stomping on some memory somewhere.

I was just wondering why the exception handling as written doesn't seem to work in this case?

Anyway, I'll post my current Apply code here:

Code: [Select]


    static void Apply( Generic2DImage<P>& img, const SampleInstance& instance )
   {

       int src_w = img.Width();
       int src_h = img.Height();

        String id = String().Format( "_RGB%02d", 1 );
ImageWindow window( src_w/2, src_h/2, 3, 32, true, true, true, id );
View view = window.MainView();
ImageVariant imgvar = view.GetImage();
Image* target = static_cast<Image*>( imgvar.AnyImage() );
int s_row, s_col, s_row2, s_col2, row, col;
for (row = 0; row < (src_w/2 - 1); row++) {
for (col = 0; col < (src_h/2 - 1); col++) {
s_col = col * 2;
s_row = row * 2;
P::FromSample( target->Pixel( col, row, 0), img.Pixel( s_col, s_row) );
s_col = (col * 2) + 1;
s_row = row * 2;
s_col2 = col * 2;
s_row2 = (row * 2) + 1;
P::FromSample( target->Pixel( col, row, 1), (img.Pixel( s_col, s_row) + img.Pixel( s_col2, s_row2))/2);
s_col = (col * 2) + 1;
s_row = (row * 2) + 1;
P::FromSample( target->Pixel( col, row, 2), img.Pixel(s_col, s_row) );
}
}
window.Show();
window.ZoomToFit(true);
   }
};


I calculate the source columns and rows separately for easier debugging. Clearly this is not optimized. When I stop on the window.Show statement the s_row and s_col values look good, they are less than the size of the source. Row and col look good too, right up to the edge of the target image. For some reason the target isn't fully finished though.

See a screenshot here:

http://gallery.tungstentech.com/main.php?g2_itemId=1355

Sure would be nice if we could attach small images to our posts in this forum

[Juan Conejero]

Hi Sander,

Very nice first PCL routine!

OK, the problem is funny. I have needed half an hour or so to detect it; it definitely is one of those bugs that can provide fun for a week

Code: [Select]

for (row = 0; row < (src_w/2 - 1); row++) {
   for (col = 0; col < (src_h/2 - 1); col++) {

Replace it by:

Code: [Select]

for (row = 0; row < (src_h/2 - 1); row++) {
   for (col = 0; col < (src_w/2 - 1); col++) {

and all will be fine. Apart from that, the for loops should be bounded by src_h/2 and src_w/2, instead of (src_h/2 - 1) and (src_w/2 - 1), since you're retrieving source pixels following this scheme:

R G
G B

so the target image should have exactly half the source dimensions (if the source has odd dimensions, then an integer division by two will smoothly ignore the last row or column).

As for the exception, it was happening within the core application, not in your module, so you have no control over it. For the sake of performance, the PCL does very minimal checks to ensure proper addressing (for example, it does no coordinate bounds checks), so if you do things like swapping rows by columns, a severe crash is served (unless your images are perfectly square, of course ).

[Nocturnal]

Hi Juan,

I think a 'doh!' is in order here. That's what I get for mixing columns and width. Should have used max_col and max_row or something like that.

You are right about the bounding issue. Because I'm already doing a /2 I'm discarding the last row/column if the size is odd. I thought I was still going too far.

Thanks for the help! Now to make the bayer matrix configurable and to add more advanced debayering. And multi threading of course.

[Juan Conejero]

Hi Sander,

There is another potential problem with the code:

Code: [Select]

P::FromSample( target->Pixel( col, row, 1 ), (img.Pixel( s_col, s_row ) + img.Pixel( s_col2, s_row2 ))/2);

If img is a floating point image (pcl::Image or pcl::DImage), the above code works just fine. However, if img is an integer image (pcl::UInt8Image, pcl::UInt16Image or pcl::UInt32Image), the division by two above is an integer division, which leads to truncation. Since most likely you're going to apply this routine to pcl::UInt16Image, this is relevant.

There are two possible solutions: one is somewhat tricky and the other is more formal. The tricky one first:

Code: [Select]

double g = (img.Pixel( s_col, s_row ) + img.Pixel( s_col2, s_row2 ))/2.0/P::MaxSampleValue();
DoublePixelTraits::FromSample( target->Pixel( col, row, 1 ), g );

The key is in the '2.0', which forces a double division result, and in dividing by P::MaxSampleValue(), which rescales the result to [0,1]. DoublePixelTraits does the rest. Note that since your target image is a pcl::Image, the above code simplifies to:

Code: [Select]

target->Pixel( col, row, 1 ) = (img.Pixel( s_col, s_row ) + img.Pixel( s_col2, s_row2 ))/2.0/P::MaxSampleValue();

The not-so-tricky way is:

Code: [Select]

double g1, g2;
P::FromSample( g1, img.Pixel( s_col, s_row ) );
P::FromSample( g2, img.Pixel( s_col2, s_row2 ) );
target->Pixel( col, row, 1 ) = (g1 + g2)/2;

which is less interesting but perhaps easier to follow.

[Nocturnal]

Good catch Juan and great solutions both!

[Nocturnal]

So can I simplify:

Code: [Select]

s_col = (col * 2) + 1;
s_row = (row * 2) + 1;
P::FromSample( target->Pixel( col, row, 2), img.Pixel(s_col, s_row) );


to:

Code: [Select]


s_col = (col * 2) + 1;
s_row = (row * 2) + 1;
target->Pixel( col, row, 2) =  img.Pixel(s_col, s_row) ;


Even though the formats are not necessarily the same? Or is the FromSample required to do the int to float cast?

[Nocturnal]

Right, so now the problem is that PI loads images differently than other software. I'm sure Juan will say PI does it correctly and others do it wrong but the fact is that images in PI appear horizontally flipped compared to DSS, Fitswork and probably Nebulosity. So straight up RGGB debayer doesn't work. The image becomes blue instead of green.

So what do I do? People will expect an RGGB debayer to work the same regardless of which software they use. So I should adapt internally, right?

[Juan Conejero]

Hi Sander,

target->Pixel( col, row, 2 ) =  img.Pixel( s_col, s_row );

That only works if *target and img are instances of the same template instantiation (i.e., they share the same PixelTraits, or P template argument).

In your case, target is a pointer to pcl::Image, but img can be an instance of any image type (and anyway, it isn't likely to be pcl::Image), so P::FromSample is mandatory.

Right, so now the problem is that PI loads images differently than other software.

This is the default behavior. It can be changed via Format Explorer > FITS > Preferences > Miscellaneous Options > Coordinate Origin.

Juan will say PI does it correctly and others do it wrong

Of course not. No orientation is more correct than any other since the FITS standard says nothing (once more) about it. Since PI is being used in professional/academic contexts, and most professional telescopes generate FITS images using the lower-left coordinate origin, we decided to follow it by default. When necessary, it's easy to apply a vertical mirror to a set of images, and we provide a way to change the default FITS orientation, as I've explained above.

Now the actual problem is that your module has no way to know if FITS files are being loaded with a particular orientation. To be sincere, this is the first time I have to face this problem. And it is indeed a problem

As you probably know, I really don't like the FITS format. It is supposed to be a "flexible" format, but flexibility, as I understand it, doesn't consist of "absence of rules". The FITS format is something that we have inherited from the magnetic tape times. The problem is that FITS is *still* a good format for magnetic tapes. We definitely need something more contemporary. To make things worse, many CCD makers have decided to use FITS to store their raw data in proprietary formats, (ab)using nonstandard keywords and extensions to store hardware-dependent data into an abstract image transport system. The result is a big mess.

Well, enough digression for this morning, so let's return to the real problem. We need a way to let a module learn the default orientation of FITS images. Right now this is impossible. The solution that I've figured out is to add the following static member functions (see pcl/GlobalSettings.h):

Code: [Select]

bool PixInsightSettings::DefineGlobalVariable( const IsoString& globalId, PixInsightSettings::variable_type type, bool permanent = false );

bool PixInsightSettings::UndefineGlobalVariable( const IsoString& globalId )
{
   return DefineGlobalVariable( globalId, GlobalVariableType::Undefined );
}

In this way a module will be able to create and delete global variables. Right now a module can only read global variables, but can't generate and manage new ones. Note that this functionality requires changes to the low-level C API used for communication between the core application and modules, and significant changes also to internal core routines. Of course, only the module that created a global variable can redefine (or undefine) it.

With this new functionality, the FITS module will define a permanent global variable, namely:

Code: [Select]

/FileFormats/FITS/CoordinateOrigin

whose type will be GlobalVariableType::String, with possible values "bottom-left" and "top-left". A module will then be able to use PixInsightSettings to retrieve the current value of this global variable. All this will be possible in version 1.5. Right now, you have to hope that the user has set the top-left orientation manually via preferences.

What do you think? I'm open to hear other ideas or potential pitfalls associated to these new functions.

[Nocturnal]

Hi Juan,

indeed FITS is a mess but we're kind of stuck with it for now

I thought of solving this with persistent settings for the debayer module rather than for the FITS module. After all someone can convert a FITS to TIFF and they'd expect to use the same debayer settings to convert the image.

I haven't tried yet to create persistent settings for my module. I'll look in the file format sample code for pointers on how to do that.

For now I first want to convert my image to the correct color. Even when I flipped the bayer matrix I still got a blue image rather than a green one so I'm not doing it correctly yet. On the plus side I was able to rip out all the controls I didn't need anymore, add an icon and I changed the bayer setting to a combo box with presets rather than the 4 combo boxes.

As an aside the performance difference between debug and release code was rather startling. A 3000x2000 image takes about 0.15 seconds to debayer in release mode, 3.15 in debug. In general this is so fast (my machine is only a p4D@3G) that I won't worry about multi threading right now. First want to get a workable module ready for testing.

Thanks for clarifying the Pixel thing. I was afraid it wouldn't work in the general case. I'm slowly starting to grasp the structure of PCL. I'm more used to polymorphism than templates and it takes a while to get used to it.