Creating 'Cosmetic Correction' files - a suggestion

[Niall Saunders]

Hi all,

Further to my questions on BiasOffset frames in the BugReports section here on the Forum, let me try and esplain what I was trying to achieve.

My reasoning was that, because BiasOffset frames are based on extremely (infinitely) short exposures, they ONLY contain the 'bias' or 'offset' level that has been imposed by the readout electronics in the camera hardware.

There is little or no thermal noise, because the exposure is too short to collect enough thermal photons
There is little or no 'photon' information, for the same reason, and for the fact that they should have the lens covered anyway
There is no 'flat frame' calibration to worry about - because there is no optical path involved
There is no 'real' chance of cosmic ray strikes, again because of the extremely short exposures (and, in any case, because it is so easy to collect large quantities of these exposures, subsequent statistical analysis should be more than robust enough to easily eliminate such rare possibilities)

So, all that is really acquired are ADU values corresponding to the offset voltage applied by the digitising electronics in the camera.

However, I would expect that the BiasOffset frames would also contain information regarding 'faulty' pixels within the CCD. Specifically, the following pixel faults should be present irrespective of the length of exposure, or the temperature of the CCD :-

DEAD PIXELS - pixels that always present as an ADU value of '0'
STUCK PIXELS - pixels that always present as a 'maximum' ADU value. say 65535 for a 16-bit A/D converter
FIXED PIXELS - pixels that always present as a 'fixed' ADU value, neither 'zero' nor 'max', and also not anything close to the Bias ADU level

My idea was to take my BiasOffset frames and ImageIntegrate, using Average Combine, with NO NORMALISATION (anywhere), with NO NOISE WEIGHTING, and with fine-tuned WinsorizedSigmaClipping.

I have already tried this for a random set of subs, about 60 images, varying in temperature from 4C to 6C, all taken at 0.0001s using my DSI-IIPro. I tweaked the Winsorization Sigma values to eliminate the same number of pixels for High and Low (special note here Juan - I would like to be able to set a 'target' number, or percentage, of High or Low clipped pixels, and have ImageIntegration adjust Sigma to achieve this level of clipping, but I'll present this in more detail as a new thread in the Wish List section).

At this point I end up with an image that seems to have an almost perfect Gaussian Distribution curve in the Histogram window. So, I reckon that I have been successful so far.

However, my argument now is that this 'integrated' image has pixels that are both above and below the Mean (or Median) ADU value - which is to be expected, naturally. But, at the very 'edges' of the distribution curve are ADU values that are present NOT because of the normal Gaussian distribution, but are present because they are one of the 'faulty' types of pixel, described above.

Now I wanted to be able to 'clip' the image data again - to extract those pixels at the 'edges' of the distribution curve. The pseudo-code I chose (easily implementable as a single statement in PixelMath) is as follows:-

Code: [Select]


// $T is the 'target image'

upper_sigma = 3
lower_sigma = 4

image_mean = mean($T)
image_stdev = sdev($T)

lower_limit = image_mean - (lower_sigma * image_stdev)
upper_limit = image_mean + (upper_sigma * image_stdev)

if ( adu_value($T,x,y) <  lower_limit ) then adu_value($T,x,y) = 0

if ( adu_value($T,x,y) >  upper_limit ) then adu_value($T,x,y) = 1

if ( adu_value($T,x,y) >=  lower_limit AND adu_value($T,x,y) <=  upper_limit ) then adu_value($T,x,y) = image_mean

The new image will therefore contain only three possible ADU values:
0.000 for pixels below the lower clipping point, corresponding to DEAD PIXELS
1.000 for pixels above the upper clipping point, corresponding to STUCK PIXELS
'mean value' for all other pixels (FIXED pixels being ignored, or identified at STUCK pixels by tweaking the upper_sigma multiplier

In reality, all we need to do is identify EITHER of these faulty pixels, because the next step would be identical anyway, i.e. to use some form of 'nearest neighbour' interpolation to recreate the faulty data. In fact, I envisage a simple 'Faulty Pixel' mask being created, directly, from a (modified) version of the above pseudo-code, where all 'good' pixels are set to zero (i.e. masked-out) and all faulty pixels are set to 1.000 (i.e. un-masked).

Then a typical 'blur' process could be applied (as a batch process using the ImageContainer) to the CalibratedLights, through the 'fault mask', just prior to the calibrated lights being StarAligned. Obviously, after StarAlignment, the spatial registration is lost, and the cosmetic correction can no longer be applied.

Am I missing something?

Is there an easier way?

Does anybody care 

Cheers,

[NKV]

Then a typical 'blur' process

Or like this:

Code: [Select]

convolve (
[1,1,1,
 1,0,1,
 1,1,1 ] )

could be applied (as a batch process using the ImageContainer) to the CalibratedLights

My first script for ImageContainer

Code: [Select]

/* DarkPixelRemoval v1.0
   The script replase value of dark pixels on averaged value from near pixels.
*/

#feature-id    Sample Scripts > DarkPixelRemoval

#feature-info  The script replase value of dark pixels on averaged value from near pixels

#include <pjsr/Sizer.jsh>
#include <pjsr/FrameStyle.jsh>
#include <pjsr/TextAlign.jsh>
#include <pjsr/StdButton.jsh>
#include <pjsr/StdIcon.jsh>
#include <pjsr/NumericControl.jsh>
#include <pjsr/UndoFlag.jsh>

#define VERSION   1.0
#define TITLE     DarkPixelRemoval

/**
 * The BadPixelRemovalEngine object defines and implements the BadPixelRemoval
 * routine and its functional parameters.
 */
function DarkPixelRemovalEngine()
{
   this.initialize = function()
   {
      // Default parameters
      this.ShadowClipping = 0.001;

      if ( Parameters.isViewTarget || Parameters.isGlobalTarget )
      {
         // Our script is being executed as a Script instance.
         // Retrieve instance parameters
         if ( Parameters.has( "ShadowClipping" ) )
            this.ShadowClipping = Parameters.getReal( "ShadowClipping" );
      }

      if ( Parameters.isViewTarget )
         // View context: use the target view.
         this.targetView = Parameters.targetView;
      else
      {
         // Direct or global contexts: use the active view.
         var window = ImageWindow.activeWindow;
         if ( !window.isNull )
            this.targetView = window.currentView;
      }

   };

   this.apply = function()
   {
      with ( this )
      {
         // Export script parameters. We must carry out this here, *before* applying
         // our routine to targetView, so that a newly created Script instance will
         // encapsulate our current set of working parameters.
         exportParameters();

         // Tell the core application that we are going to change this view.
         // Without doing this, we'd have just read-only access to the view's image.
         targetView.beginProcess();

         // Prepare temp image
         var img = targetView.image;
         var TempWin = new ImageWindow( img.width, img.height, img.numberOfChannels,
                        img.bitsPerSample, img.isReal, img.isColor, "temp_"+#TITLE);
         var TempView = TempWin.mainView;
         with (TempView)
         {
            beginProcess( UndoFlag_NoSwapFile );
            image.apply( img );
            image.convolve([ 1,1,1,
                             1,0,1,
                             1,1,1 ]);
            endProcess();
         }

         // PixelMath iif ( targetView < Shadow , temp , targetView )
         var mergeThem = new PixelMath;
         with (mergeThem)
         {
            //expression = "iif($T<"+ ShadowClipping + "," + TempView.id +",$T)";
            expression = "iif("+targetView.id+"<"+ ShadowClipping + "," + TempView.id +","+targetView.id+")";
            useSingleExpression = true;
            rescale = false;
            executeOn( this.targetView );
         }
         TempWin.close();

         // Done with view.
         targetView.endProcess();
      }
   };

   this.exportParameters = function()
   {
      with ( this )
      {
         Parameters.set( "ShadowClipping", ShadowClipping );
      }
   };


   this.initialize();
}

// Global DrawSignature parameters.
var engine = new DarkPixelRemovalEngine;

/**
 * DarkPixelRemovalDialog is a graphical user interface to define
 * DarkPixelRemoval parameters.
 */
function DarkPixelRemovalDialog()
{
   this.__base__ = Dialog;
   this.__base__();
   this.windowTitle = #TITLE + " Script";

   //

   var emWidth = this.font.width( 'M' );
   var labelWidth1 = this.font.width( "Target image:" );

   //

   this.helpLabel = new Label( this );
   with ( this.helpLabel )
   {
      frameStyle = FrameStyle_Box;
      margin = 4;
      wordWrapping = true;
      useRichText = true;
      text = "<p><b>" + #TITLE + " v" + #VERSION +
             "</b> &mdash; This script replase pixels which value les then " +
             "shadow slider. New value is averag of near 8 pixels.";
   }

   //

   this.targetImage_Label = new Label( this );
   with ( this.targetImage_Label )
   {
      text = "Target image:";
      textAlignment = TextAlign_VertCenter;
      minWidth = labelWidth1;
   }

   this.targetImage_ViewList = new ViewList( this );
   with ( this.targetImage_ViewList )
   {
      getAll();
      currentView = engine.targetView;
      toolTip = "Select the image to cosmetic correction";
      onViewSelected = function( view ) engine.targetView = view;
   }

   this.targetImage_Sizer = new HorizontalSizer;
   with ( this.targetImage_Sizer )
   {
      spacing = 4;
      add( this.targetImage_Label );
      add( this.targetImage_ViewList,1 );
   }
   //

// Shadow slider
    this.ShadowClipping = new NumericControl(this);
    with (this.ShadowClipping)
    {
      label.text = "Shadow:";
      label.minWidth = labelWidth1;
      toolTip = "Define shadow clipping value";
      setRange( 0, 1 );
      slider.setRange(0,1000000 );
      slider.minWidth = 500;
      setPrecision( 8 );
      setValue (engine.ShadowClipping);
      onValueUpdated = function( value ) engine.ShadowClipping = value;
    }

// Standart buttons

   this.newInstance_Button = new ToolButton( this );
   with ( this.newInstance_Button )
   {
      icon = new Bitmap( ":/images/interface/dragObject.png" );
      toolTip = "New Instance";
      onMousePress = function()
      {
         this.hasFocus = true;
         engine.exportParameters();
         this.pushed = false;
         this.dialog.newInstance();
      };
   }

/*   this.ok_Button = new PushButton( this );
   with ( this.ok_Button )
   {
      text = "OK";
      onClick = function()
      {
         this.dialog.ok();
      };
   }
*/
   this.buttons_Sizer = new HorizontalSizer;
   with ( this.buttons_Sizer )
   {
      add( this.newInstance_Button );
      addStretch();
//      add( this.ok_Button );
   }

   //

   this.sizer = new VerticalSizer;
   with ( this.sizer )
   {
      margin = 6;
      spacing = 4;
      add( this.helpLabel );
      add( this.targetImage_Sizer );
      add( this.ShadowClipping );
      add( this.buttons_Sizer );
   }

   this.adjustToContents();
   this.setFixedSize();
}

// Our dialog inherits all properties and methods from the core Dialog object.
DarkPixelRemovalDialog.prototype = new Dialog;

/*
 * Script entry point.
 */
function main()
{
   // If the script is being executed as a Script instance on a view context,
   // then apply it and exit, without showing any graphical user interface.
   // This allows us to run a script just as a regular (module-defined) process
   // instance.
   if ( Parameters.isViewTarget )
   {
      engine.apply();
      return;
   }

#ifndef __DEBUG__
   console.hide();
#endif

   // If the script is being executed either directly or in the global context,
   // then we need a target view, so an image window must be available.
   if ( !engine.targetView )
   {
      var msg = new MessageBox( "There is no active image window!",
                                (#TITLE + " Script"), StdIcon_Error, StdButton_Ok );
      msg.execute();
      return;
   }

   var dialog = new DarkPixelRemovalDialog();
   for ( ;; )
   {
      // Execute the DarkPixelRemoval dialog.
      if ( !dialog.execute() )
         break;

      // A view must be selected.
      if ( engine.targetView.isNull )
      {
         var msg = new MessageBox( "You must select a view to apply this script.",
                                   (#TITLE + " Script"), StdIcon_Error, StdButton_Ok );
         msg.execute();
         continue;
      }

      // Perform the DarkPixelRemova routine.
      engine.apply();

      // Quit after successful execution.
      break;
   }
}

main();


[dhalliday]

Yikes !!!
Niall;go for a LONG walk,immediately !
Maybe a double Latte and a good movie after....

Dave

[Niall Saunders]

Hi Dave,

Well, I've been for a long walk - actually a rather unpleasant cross-country drive in the freshly-fallen and blizzard-blown snow this morning.

And now I can sit down and try to get to grips with Nikolay's excellent script.

By the way - don't let 'scripts' frighten you. If they have been (well) written by someone here on the Forum, then they should be as simple to use as any other process in PixInsight. Mostly they are used to automate a series of tasks that you could otherwise do yourself, only more tediously.

Somethimes though, they can work a magic all of their own - and are well worth looking at, if they solve a particular problem that you might be experiencing.

Cheers,

[dhalliday]

Niall
Sorry to here the UK is not improving...
To me "writing a script" is synonymous with computer programming.
I just don't do that stuff...
Maybe I could learn...but maybe I could learn to cook great pies as well...!
Its a question of limits,and time....
I see a lot of techno types ( ) interested in this astro stuff...and hats off to you all.
Software,drivers,CPU's etc are no biggie,I guess.Pixel math...?
More programming !!

Me I finally had to get my 30 yr old nephew over to get Pixinsight installed onto a second computer...
So for me,...if something is so useful that I am going to use it ALL the time,then I hope,and expect dear Juan ( )
to help me out.
How is your Crab coming along ?
Check out my LRGB combo M57 (version 12) in the other thread !

Dave

[Niall Saunders]

Hopefully Dave - if a script proves useful, or is required, then we can show you how to 'execute' it. You won't have to 'write' anything - somebody else has done all that hard work for you

Even PixelMath - you WILL (eventually) be able to get some of the simplest PixelMath expressions executed. You may not need to 'invent' them, you will just need to know how to 'enter' the expression, and to press the right buttons thereafter.

It's no more difficult than chewing gum and breathing at the same time - and, apart from one or two 'celebrities', most of the human race can manage that (with practice!)

Cheers,