Author Topic: Removing Canon Banding with PJSR  (Read 58038 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
New Release Re: Removing Canon Banding with PJSR
« Reply #45 on: 2009 September 20 11:44:33 »
Dear all,

my vacation permitted to develop another incarnation of the Canon Banding Script, now giving an (almost) realtime preview with the capability to use the images STF for the preview, asynchronous operation on GUI actions, and progress reporting. These capabilities have been made possible by a number of bug fixes that Juan incorporated in PI 1.5.6, so the script will not work with older versions. It still has a couple of issues with full size images, mostly related to memory consumption, and I sincerely hope that I will be able to resolve these in later versions. Still, it is very useful, and most likely the only real cure for the Canon Banding problem available, as recent discussions in the Digital Astro Yahoo Group (e.g. http://tech.groups.yahoo.com/group/digital_astro/message/130842) show.

The attached screenshot shows the GUI. Use "Amount" to adjust the strength of the correction. Use "Activate Preview" to see the result, optionally with "STF". "Protect from Highlights" excludes bright spots from the statistical analysis of the image, and is helpful if you have bright stars in the field. Sometimes, this also helps to improve the result, since the banding sometimes seems to be related to the brightness of objects. Higher values cause a stronger effect. Unfortunately, using this slider can be slow, depending on the size of your image.


An example using the Eagle Nebula (M16, NGC 6611) is also shown as part the screenshot. The raw data (lights, darks, flats, biases) have been processed with DeepSkyStacker, and PI has been used for a DBE and STF only. The original is on the top right, the processed version is on the bottom right. The preview on the bottom left has an orange cast due to the different effects of corrections for the different channels. You will probably agree that this is a real live saver for this image...

Juan, I would like to work with you on resolving the remaining issues, in particular the high memory consumption. Please see the comments at the head of the script file. If you like, you can include the script with the next PI releases... after a careful code review.

Enjoy,
Georg
« Last Edit: 2009 September 20 11:50:07 by georg.viehoever »
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Removing Canon Banding with PJSR
« Reply #46 on: 2009 September 21 03:07:22 »
Hi Georg,

You should have more vacations more often  :laugh:

What a great work. Most of the problems you are facing with this script will be fixed in the new PI 1.5.8 version. This version has lots of bugs fixed, including of course the infamous memory consumption problem: now when an image window is closed, its whole image data are deallocated immediately.

Those "walking sliders" should be fixed too, although I still have to verify this. I have fixed several bugs caused by reentrant event handlers. Basically, the problem is that multiple events are triggered while a previous event is still being processed. For example, your code (actually, internal PI Core code) receives a mouse move event while it is working in response to a previous move event. Most of PixInsight Core is fully reentrant, so this doesn't break anything internally, but of course your script doesn't expect this odd behavior. These problems happen on Windows and Mac OS X, not on Linux/X11. I have fixed all of them and I guess they are the cause of your script's magical behavior :)

Other than this, the script works extremely well. I'll be glad to include it in the standard set of PI 1.5.8 :)

I'll take a deeper look at the script to see if we can improve it a bit more before release. As soon as I have a working Windows version of 1.5.8, I'll upload a special prerelease version for you, so you can test the script on the new PJSR.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Removing Canon Banding with PJSR
« Reply #47 on: 2009 September 21 03:42:17 »
Hi Juan,

this is great news! I am looking forward to the next PI version.

Would it be possible to make scripts executable as part of an Image container (see http://pixinsight.com/forum/index.php?topic=1266.0)? I guess you would need to add the capability to output the script parameters, and the capability to distinguish between batch and GUI mode calls of the script.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Removing Canon Banding with PJSR
« Reply #48 on: 2009 September 21 04:54:48 »
Hi Georg,

A script doesn't have the necessary interfaces to work with ImageContainer. Basically, this is because ImageContainer has been designed to work with process instances, which are objects created by module-defined code.

However, implementing a similar functionality should be relatively easy in PJSR. For example, the basic interface and tasks implemented by the BatchFormatConversion script could be easily adapted to apply a script (with a predefined set of parameters) to a set of images, either open images or disk files. I think we could design a generic script capable to run another script in batch mode. This would need a standard interface for communication between the "server" and "client" scripts, so to say. Not accurately defined but that's a basic idea. What do you think?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Removing Canon Banding with PJSR
« Reply #49 on: 2009 September 21 05:14:33 »
Juan,

if possible I would prefer if the necessary interfaces to create process instances from PJSR scripts would be added. This is just because scripts would then seamlessly fit into the remainder of the PI architecture. That's not only Image Containers, but also I would be able to look at the process history to see what I have done at a certain point in processing the image, and I would be able to store the process.

Would something like an "emitHistoryCommand()" call as part of PJSR help? I could use it to output a command line with the actually used parameters, and could become part of a scripting process instance. Later when executing this instance (i.e. starting the script), I could check for those parameters and run it in non-GUI mode. Something along this line...

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline David Serrano

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 503
Re: Removing Canon Banding with PJSR
« Reply #50 on: 2009 September 21 06:10:15 »
I would bet that's what the Script process is for ;). Somebody asked recently about it, if memory serves me well.

Or a different paradigm to scripts could be designed, in which we programmers would define a series of callbacks that PI core would call when necessary, just like in binary modules.
--
 David Serrano

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Removing Canon Banding with PJSR
« Reply #51 on: 2009 September 22 00:02:23 »
David,

I think that's exactly the right idea. We would need some way for a script to manipulate the state of its corresponding script object. Later, it could retrieve the settings again. What do you think, Juan?

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Removing Canon Banding with PJSR
« Reply #52 on: 2009 September 23 13:24:21 »
Hi all,

At first glance, the notion of being able to use a ProcessContainer to 'hold' all of the parameters of a PJSR script seems straightforward.

However, if your script reads and writes to disk, then how does the container know that the paths that have been stored are still applicable? In fact, I have the same question about the 'core' StarAlignmnet and ImageIntegration processes. Yes, these do fit in with normal container usage methods - but they must, obviously, 'fail' if the requested data is no longer available, or an output path is no longer valid, or would cause possible data loss due to uncontrolled 'overwrites'.

The whole notion of recording your image processing 'history' is a complex one, and it is also not always 'intuitive'. Whenever you start to deal with 'multiple images' in your processing path, you then have to - very carefully - record the processing history associated with ALL of those parallel paths if you hope to be able to duplicate, or document, a final image.

Certainly, some 'multi-image' processes are 'trivial' when it comes to recording the process - all you really need to record are the 'actual parameters' used by a process - the (multiple) images that were actually processed are more easily identified by careful folder naming and general good housekeeping on your hard drive. That is how I deal with the stages from RAW data, through calibration, debayering, alignment and integration.

But, once you start 'getting clever, and doing stuff like extracting the Rd channel and blending it with Ha data, and then re-introducing it to the RGB mix, THEN you need to start saving individual images, with clever filenames and descriptions, and with carefully named process icons.

Which is also why I had asked for a facility to 'save' my COMPLETE workspace right in the middle of a PI session. I honestly wouldn't care if this DID require 10Gb of disk space - I would still prefer that to the option of just leaving the PC 'on' (or in Standby, or Hibernate) all the time - hoping that spiky gremlins didn't eat all my hard work in the meantime  :'(

Having access to a 'replayable' Process History is one of the huge advantages of PI - you just have to see how Harry makes use of this in his tutorials to see a good demonstration of that. It is a great way to allow you to 'tweak' and 'twiddle' with an image - never too sure where you are going, or what you have done to get there - but with the confidence of being able to retrace your steps to find out 'how', or to quickly recover from a 'sea of mud' scenario. (I use this all the time - especially the red button with the white cross in it, at the top right of the PI window  :yell: )

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline David Serrano

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 503
Re: Removing Canon Banding with PJSR
« Reply #53 on: 2009 September 24 00:52:03 »
The whole notion of recording your image processing 'history' is a complex one, and it is also not always 'intuitive'. Whenever you start to deal with 'multiple images' in your processing path, you then have to - very carefully - record the processing history associated with ALL of those parallel paths if you hope to be able to duplicate, or document, a final image.

That limitation of .psm files has the bright side that histories can be associated with any image, and not only images with given identifiers.

For recording whole processing histories, including helper images (masks and so on) and opening/closing of files, something more powerful than process history is needed. Something that would integrate in one place everything we can do with PixInsight, a sort of script that would call ProcessingConsole commands (like open), processes and JS scripts. All with the obvious limitation that it wouldn't be reusable against other images.

OR, improve the JS engine to add the missing pieces of functionality.
--
 David Serrano

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Removing Canon Banding with PJSR
« Reply #54 on: 2009 September 24 04:25:30 »
Hi,

I must admit that my immediate goal is much more humble than having the capability to document the whole process, or saving the whole PI state (which would be nice indeed. I am currently using the Windows Hibernate option... which works nice until you have to reboot). My minimum requirement would be do be able to embed a process that includes a PJSR script into a process that I want to apply to a mass of images ( which to me sounds like Image Container).

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Removing Canon Banding with PJSR
« Reply #55 on: 2009 September 24 04:28:01 »
I think we must face this problem and try to solve it as soon as possible. As Georg and others have pointed out, this is becoming an obstacle for PJSR developers since not being able to integrate scripts within the PI platform is frustrating. I continue being amazed by how far PI scripting is going.

I have been thinking on a way to integrate JavaScript scripts into PixInsight's object-oriented architecture. Presently scripts can be executed but they can't be properly encapsulated. This departs from PI's OO design, and here is the problem. So let's provide some efficient encapsulation for these rebels :)

The natural and obvious way to encapsulate scripts is by means of a Script process. This process already exists. Its purpose is to allow integration of executed scripts in processing histories. A processing history is a read-only ProcessContainer, which is a sorted list of process instances. A Script instance includes a script's source code (as a block of plain UTF-8 text) and provides just the basic functionality to interact with ProcessContainer. The problem is that we need Script to interact with the script it encapsulates, just as a process class knows how to play with its own instances. In particular, we need a way of communication between a script and a Script instance to allow for persistent storage of script parameters.

Persistence of instance parameters is an automatic process in PixInsight. The core application provides all the required resources. This is why a newly developed process (a C++/PCL module) integrates so smoothly without the developer's intervention (other than defining process parameters --a task that can also be automated, and it will be, with an upcoming code generation script).  

Now we must define exactly the same interface for the Script process, both internally (between Script and its encapsulated script) and externally (between Script and Core):

* The external interface is relatively simple. It reduces to adding a unique String parameter to Script. A formal description for this parameter would be:

Script::parameters:
   <script-parameter-list>opt
<script-parameter-list>:
   <parameter-id>=<parameter-value>(<separator><script-parameter-list>)opt
<separator>:
   one of: \n ;

Note that the above description fits strictly within JavaScript syntax.

Why a unique string parameter, instead of a set of parameters, each with its own type, as happens with regular process instances? Because we have no way to know in advance what a script wants as its parameters. Each script will define and use a completely different (perhaps empty) set of parameters. The only way to store this is as a string with an internal syntax. Fortunately, JavaScript provides us with all the syntax we need :)

We could implement something more sophisticated, as a second parameter indicating the type of each parameter. IMO, this wouldn't add more functionality, given the dynamic nature of javaScript, and it could rather be restricting or limiting in the sense that such a fixed system wouldn't be able to support script-defined objects. Let's allow the JavaScript engine do all the nitty-gritty.

* The internal interface is somewhat more complicated. We need a bidirectional communication:

- A script must broadcast its own working parameters during execution, and Core must be hearing to take note of them. Upon script termination, Core will generate a new Script instance with the corresponding value for Script::parameters.

- During execution, a script has access to several read-only properties of a special JavaScript object. As you probably figure out at this point, that properties are just the result of interpreting and executing Script::parameters as JavaScript code.

Let's put a practical example. A script could do the following:

Code: [Select]
function MyParameters
{
   // Set default parameter values.
   this.setDefault = function()
   {
      this.radius = 100;
      this.lineWidth = 3;
      this.helloText = "Hello, PI!";
   };

   // Retrieve working parameters.
   this.retrieve = function()
   {
      if ( Parameters.has( "radius" ) )
         this.radius = Parameters.get( "radius" );
      if ( Parameters.has( "lineWidth" ) )
         this.lineWidth = Parameters.get( "lineWidth" );
      if ( Parameters.has( "helloText" ) )
         this.helloText = Parameters.get( "helloText" );
   };

   // Send parameters and values to Core.
   this.broadcast = function()
   {
      Parameters.set( "radius", this.radius );
      Parameters.set( "lineWidth", this.lineWidth );
      Parameters.set( "helloText", this.helloText );
   };

   // Create and initialize properties.
   this.setDefault();
}

function main()
{
   var P = new MyParameters;
   P.retrieve(); // acquire Script instance parameters
   // ...
}

The above snippet exemplifies our internal communication system. The Parameters global object provides a bidirectional interface between our script and the Script process. Parameters has three main methods:

Code: [Select]
Boolean Parameters.has( String id )
Returns true if the specified parameter id has been defined. In general, that will be true if the script is being executed from an existing instance of Script, false otherwise.

Code: [Select]
void Parameters.set( String id, Object value )
Broadcasts existence of a parameter whose name is id and has the specified value. If the specified parameter already exists, its current value is replaced with the new one. If the parameter does not exist, it is newly created.

Code: [Select]
Object Parameters.get( String id )
Acquires the specified parameter id and returns its value, if it exists, or undefined otherwise.


I can try to implement this for the upcoming version 1.5.8, or postpone it for 1.6, depending on the difficulties that I find, but this definitely has to be done. What do you think?
« Last Edit: 2009 September 24 04:35:21 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Removing Canon Banding with PJSR
« Reply #56 on: 2009 September 24 04:50:29 »
Quote
Which is also why I had asked for a facility to 'save' my COMPLETE workspace right in the middle of a PI session. I honestly wouldn't care if this DID require 10Gb of disk space - I would still prefer that to the option of just leaving the PC 'on' (or in Standby, or Hibernate) all the time - hoping that spiky gremlins didn't eat all my hard work in the meantime  Cry

Of course, I feel your pain :) But this is a different topic. You are talking about a project functionality for PixInsight. We have this scheduled for implementation in version 1.6. PixInsight projects will be special XML files that will allow you to store and recover the state of the entire PixInsight platform. This will include virtually everything: from the current console text to all icons, all processing histories and the current states of all tools. Images will also be stored, of course, either as external linked files or as embedded objects (in the latter case though, a PI project can easily grow up to the multi-gigabyte level). We are working on this, and you can be sure it will be implemented.

As always, I'll be glad to hear all comments and suggestions from all users. PixInsight's project functionality is extremely important and we definitely need the opinions and contributions of everybody.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Removing Canon Banding with PJSR
« Reply #57 on: 2009 September 24 05:00:02 »
Hi David,

(and my apologies for posting this without realising that both Juan and Georg had posted during my editting session !!)

Yes, I agree, PSM files are now, more or less, a 'stable entity' - they do what they have to do, and do this reasonably well. Their incarnation should now only be 'tweaked', not fully reinvented.

The same goes for the current implementation of the Process History (which is really a list of Process Icons) and Process Containers (which 'contain' a 'Process History' - part, or full).

The Processing Console can maintain a record of EVERYTHING that has happened since the current instance of PI was started, but it cannot be used (currently) to 'replay' all of these steps. Perhaps we need a simple, clickable, 'Save' button, that captures some, or all, of the Console steps onto disk. Perhaps we also then need a 'Replay' facility (with a 'Step Through / Jump Over') set of controls that can be used to FULLY recreate a 'set point' in any workflow. That way it would seem that - providing ALL source data was still available (and in the same location on disk) - a 'Save Workspace' scenario COULD be created (you simply save the Console History, and replay it to suit). OK, you may have to 'veto' a replayed history - just in case some steps are no longer appropriate, and some 'PI working environment' changes that might be recorded by the Console may have to be (automagically) 'ignored' during replayed steps.

A PJSR script can have its last used 'SETTINGS' data recorded by PI - but the script needs to be written to support this feature. It is not that difficult to achieve (he says, not yet having ACTUALLY written such code !!). So it might be possible to implement some form of 'clickable icon' that would trigger a PJSR script from the on-screen workspace, that icon having been either loaded from disk, or 'dropped' on a workspace from a PJSR GUI (in the same way as the little 'arrow' icon on a Core Process can be dropped on the desktop to create a PSM icon).

These icons would be saved and reloaded, and grouped and organised just like their PSM counterparts. If acceptable (by virtue of some notional flag within the PJSR script itself) they could even be 'dragged and dropped' onto an image in order to work their magic on that image. If the script was designed to work on multiple images, then 'drag and drop' could trigger an error condition - in which case the user could be offered the choice of opening the GUI for the script, in order to verify that the pre-existing file or image list was still applicable.

Icons suitable for the PJSR could be created by the author (within PI, of course !!!) appropriate for use when being displayed on the PI workspace.

And, finally, I can see a situation whereby Process Container and PJSR icons could be added to the graphical selection system that we already have - initially into appropriate groups on the display, and then, optionally, into the 'Favourites' selection as desired. I even see a situation where more than one 'Favourites' group should be creatable - named according to the user's desire, to suit the way that person most regularly uses PI.



Perhaps, Juan (or some other PTeam member), we need to 'break off' or 'copy' this developing thread into its more appropriate home under 'Wish List' ?



Cheers,
(and I hope you didn't have any plans for a 'family life' for the next few months Juan  ::) )
« Last Edit: 2009 September 24 05:05:14 by Niall Saunders »
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Removing Canon Banding with PJSR
« Reply #58 on: 2009 September 24 05:18:38 »
....
I can try to implement this for the upcoming version 1.5.8, or postpone it for 1.6, depending on the difficulties that I find, but this definitely has to be done. What do you think?

Sounds good to me. While I certainly would like to have this ASAP, I would prefer to have a 1.5.8 without it if implementing this would delay the release.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Removing Canon Banding with PJSR
« Reply #59 on: 2009 September 24 05:54:22 »
Niall, don't worry. Juan doesn't know what the words "life" or "vacation" means ;)

Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com