Author Topic: New Script: Batch Debayer  (Read 24367 times)

Offline Ken Pendlebury

  • Newcomer
  • Posts: 47
New Script: Batch Debayer
« on: 2010 March 07 14:20:19 »
Hi All,

I am still on my trial license, but I am very impressed so far.  As part of my trial, it was important to me to experience scripting at least at a very basic level.  As such, I noticed a module named Debayer and while certainly useful, requires opening each file individually in order to apply the module.  Since I have many short exposure images to debayer, I have devised a script to apply this module over multiple images at once.

This script doesn't do anything fancy and I do not claim much of anything here.  The interface is blatantly stolen from Niall Saunders (CMYG Batch Debayer) and the actual Debayer code is from the PCL debayer module.  As such, you will find the same options (Debayer method and Bayer Pattern) in the script dialog.  The chosen method and pattern will be applied to all images in the tree selector.

This is admittedly not tested to an acceptable point, but I only have access to a Mac (and have not even tested all permutations here).  Anyone interested in testing is welcome to do so.  Also on the todo list is better commenting around certain assumptions and methodologies chosen...




Regards,
Ken
« Last Edit: 2010 March 07 14:44:15 by Ken Pendlebury »
Regards,
Ken
My Astro Photo Stream
http://www.flickr.com/photos/kenastro/

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: New Script: Batch Debayer
« Reply #1 on: 2010 March 08 11:53:52 »
Hi Ken,

EXCELLENT

This is exactly the kind of 'newbie' that we are after - obviously you have found our 'deep and cold water' to be just to your liking. There are not many 'newbies' who jump in at the deep end, and then go straight down to the bottom of our 'murky pool' to see what old pennies might be lying around there ;D

But, my code was partly plagiarised from other's efforts, and helped along the way by Juan, and offered for all to use however they wish. So, I am glad that you found it (a) useful and (b) useable enough to allow you to adapt it for you own needs (that actually is my reward for the hundreds of hours I put into writing it in the first place - for someone to be able to use it as the skeleton for their own work - and is why Juan was happy to include it as part of the standard distribution of PI ever since)

Serious congratulations are due to you !

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New Script: Batch Debayer
« Reply #2 on: 2010 March 08 13:38:17 »
Crickey

Most newbies are happy to get to the opening screen and load a image  >:D


Harry
Harry Page

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: New Script: Batch Debayer
« Reply #3 on: 2010 March 08 13:50:26 »
Hi Harry (et Bonjour from la deepest France, un fois encore !!),

Yes - it can certainly be used as an example for all those who claim that the PI interface is 'too quirky' to use  ::)

Perhaps we have a future Jedi in the making !!

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New Script: Batch Debayer
« Reply #4 on: 2010 March 09 01:33:00 »
Hi Ken,

Welcome to PixInsight.

Excellent work. I support everything that Niall has said. Please feel like at home on this forum. Don't hesitate to post your work here, and ask us for everything you need regarding development on PixInsight.

By the way, Niall, I think you underestimate the importance of your script. Besides a useful tool, it is a reference work for how you wrote it and the way you documented it.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Ken Pendlebury

  • Newcomer
  • Posts: 47
Re: New Script: Batch Debayer
« Reply #5 on: 2010 March 09 09:41:16 »
Thanks for the encouragement and feedback.  I am very excited about PI and still wondering how I have never heard of it until recently.  It is possible that I just assumed I could not use it on a Mac (too often in AP, this is a valid assumption).  Its usefulness is mind-boggling and the speed at which I can process data blows me away.

Niall, I certainly agree with Juan.  Without your script (and the others included) this script would have been a long time in the making.  Thank you for your very thoughtful commenting.  I already feel like a have a pretty firm grasp around the UI and general architecture.

I have had a few questions regarding the difference between bilinear interpolation vs. superpixel so I will try to update the "tooltip" boxes to cover this as briefly as I can.

Again, thanks for the compliments and warm welcome.  I look forward to becoming an active PI contributor.

Regards,
Ken
Regards,
Ken
My Astro Photo Stream
http://www.flickr.com/photos/kenastro/

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New Script: Batch Debayer
« Reply #6 on: 2010 March 09 09:58:59 »
Hi Ken

I have had a go with my sxvf m25 OSC camera and the script works brill  8)

Would you know how to introduce weights to the colour channels  ???


Regards Dumb Harry
Harry Page

Offline Ken Pendlebury

  • Newcomer
  • Posts: 47
Re: New Script: Batch Debayer
« Reply #7 on: 2010 March 09 10:16:14 »
Harry,

I am pretty sure that I do know how to weight the color channels independently.  Unfortunately, I am using the PCL Debayer module to accomplish all the "image" work so I don't have access to the image pixels directly during this process.  I believe, in my limited knowledge of the scripting model, that I would be able to grab the output of the debayer module and apply a "post" weighting to all data via script.  Because of this, the script might turn into a huge time hog.

This is definitely a great idea and is surely worth exploring.  I do have one question though... Would a color weighting during color reconstruction provide any distinct advantage over the Color Calibration module that ships with PI?  This is an honest question... I really don't know.

Regards,
Ken
Regards,
Ken
My Astro Photo Stream
http://www.flickr.com/photos/kenastro/

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New Script: Batch Debayer
« Reply #8 on: 2010 March 09 10:54:01 »
Hi

As you say colour calibration does sort out a lot of problems , but I like to get things as close as possible as not all images have a lot of stars and or
a nice galaxy  ;) for CC

I have used other programs to do this  :'(

Thanks for your time Harry
Harry Page

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: New Script: Batch Debayer
« Reply #9 on: 2010 March 09 15:41:34 »
Hi guys,

Ken - as you will have seen from my (DSI-IIC, CMYG) deBayer PJSR script, I used several methods to try and get my data 'colour perfect' at the individual image stage - long before they were aligned and integrated.

However, that script was conceived 'at the same time' as the introduction of the BackgroundNeutralisation and ColourEqualisation modules were being developed, and I would certainly, now, not even waste my time trying to compensate colours until well AFTER I had used ImageIntegration (with the stunning power of Winsorized Sigma Clipping for 'crud' removal) to sort out all of my data calibration, and I would then be invoking DynamicCrop and DBE (perhaps twice) before I felt that my master image was ready for the BackgroundNeutralisation and ColourEqualisation.

I don't know how well I have ever described the ColourCurve adjustment steps that I include in my script, but I figured out a process that just 'worked' - and stunningly well at that, given that my deBayer was of a CMYG 2x4 CFA matrix - which is a devilishly difficult beast to tame when it comes to extracting RGB from the RAW data!

Basically, what i did was to print out a very simple colour chart (included in a post elsewhere on the Forum, but whose link may still be broken pending Juan's final 'tidy up' needed subsequent to his server swap) - essentially three swatches of R, G and B, and three swatches of Cy, Mg and Gn - created as 'perfect colours' and printed out on 'proper' photographic paper by the likes of Costco, WalMart, etc. I then take a LONG EXPOSURE photograph of this (several actually) using EXACTLY the same steps (and equipment) as I would if it were an astronomical target - using natural daylight to illuminate the print, and a simple 50mm lens attached to whichever camera I am using. This will involve SIGNIFICANT 'workarounds' to filter the INTENSITY of the daylight, but NOT at the expense of the SPECTRUM of the incident light. You will have to work this out to suit your available equipment.

Once I have calibrated, aligned and stacked the RAW data, and deBayered where necessary, I have an image that has 'colour' - but not 'pure' colour.

Then I look at the 'Red' swatch, for example, and figure out where I need to place a 'modifying' point on the ColourCurve transformation to 'change' what 'should be' Red (and isn't) into 'actual' Red. I repeat this for Bu and Gn, and then add intermediate points for Cy, Mg and Ye. The LivePreview helps you to see what is happening as you prepare this horribly complex, twisty little curve. Then you apply the curve to the image, see what the result was - and consider repeating the exerciswe again - fine tuning, if you will.

Very importantly - you need to save ProcessIcons for each of the ColourCurve transformations - the idea being that these curves actually 'define' the 'colour response' of your camera, given the specific deBayer routine that you have used. My theory is that the camera has NO IDEA that it wasn't looking at a DSO - it just gathered photons, over a specific period of time. If the Calibration and deBayer process remains (effectively) 'constant', then the application of a 'ColourCurve' correction should also be a 'constant'

What you WILL need though - I forgot to mention this - is a swatch of 'Pure Black' and one of 'Pure White' (effectively the photographic paper itself, of course) in the image as well - to allow you to accurately define the Black and White points. Interestingly, you will notice that the 'greyscale' is NOT affected by your ColourCurve transformation.

You might be able to make sense of the process if you break down the post-deBayering steps that I have placed into my script.

Certainly, when I defined a double-correcting pair of ColourCurves from my 'simulated' colour test-card, and then applied the same Curve processes to RAW data acquired of a general countryside landscape, and of some highly colourful 'still-life' samples, the effect was incredible. And when I then applied the same process to ACTUAL data collected for M81 / M82 (using my DSI-IIC on a Moonfish ED80 APO) the effort had DEFINITELY been worthwhile.

I just haven't really made a big thing about it, because myself and Alex next door are probably the ONLY PixInsight users left who are still trying to image with CMYG one-shot CCDs - and even I have since turned back to my DSI-IIPro mono imager, to try and collect the best photons that I can (Alex may thus be the VERY LAST surviving DSI-IIC imager using PixInsight for post-processing - and he currently doesn't bother with Flats or FlatDarks, and relies entirely on Nebulosity's deBayer routine to create his 'master image', and then just throws very simple BackgroundNeutralisation and ColourEqualisation processes at the data as 'automatic' steps in his initial openeing gambit - and his data ends up quite nicely colour-balanced, thank-you very much - with the minimum of fuss, bother or effort).

It would be worth your while examining different strategies - one will suit you best, and you will know that it may need to be changed as your skill, and the power of PI, improve in the future.

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New Script: Batch Debayer
« Reply #10 on: 2010 March 10 12:41:25 »
Hi

My problem with a fixed rgb weight is that is does not help when using say Light pollution filters and imaging at different dec

As I said before CC works wonders most of the time , but take this image I took at the weekend  Full high res at http://www.harrysastroshed.com/Image%20html/CONE.HTML

I had a devil of a time trying to get the colour right , mainley due to the lack of background and again its nice to start off as close as possible

Of course what I / we want is a total package  calibration inc debayer , registration and stack All wrapped up in a easy to use interface  :P  Well I can hope


Harry
Harry Page

Offline sleshin

  • PixInsight Old Hand
  • ****
  • Posts: 431
Re: New Script: Batch Debayer
« Reply #11 on: 2010 March 10 13:50:21 »
Harry,

That's a very nice image.

Steve
Steve Leshin

Stargazer Observatory
Sedona, Arizona

Offline Ken Pendlebury

  • Newcomer
  • Posts: 47
Re: New Script: Batch Debayer
« Reply #12 on: 2010 April 19 20:55:07 »
Hi Folks,

I was going to spend a little time to make debayer decisions easier for folks by including actual camera types in the "Bayer Pattern" drop down (in addition to the 4 patterns).  This should help ease the string of "how do I verify my debayer is correct?" and "What bayer pattern should I use?" questions.

I am in the process of collecting a list of OSC models that use RGB debayer pattern.  If you would like to contribute your "verified by experience" patterns for your camera, I will include them in the next revision.

Also... the debayer module currently supports 2 types of debayer algorithms.  I will try to create a tooltip that summarizes the pros and cons of each type.

While I realize this script is not overwhelmigly popular (with 17 downloads  ;)), I think it offers value with the introduction of the Image Calibration Module.  This makes it easy to do ALL of your OSC pre-processing and calibration without leaving PI.

Thanks for your help,
Ken
Regards,
Ken
My Astro Photo Stream
http://www.flickr.com/photos/kenastro/

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New Script: Batch Debayer
« Reply #13 on: 2010 April 20 12:10:34 »
Hi Ken

As a official script inc ver 1.6 you will not get any further downloads , but I use it all the time and are very grateful of your efforts

As for fixed camera settings , I know you will struggle with my sxvf m25 because depending on firmware versions the settings are different  ???

Harry

Harry Page

Offline Ken Pendlebury

  • Newcomer
  • Posts: 47
Re: New Script: Batch Debayer
« Reply #14 on: 2010 April 20 21:42:58 »
Thanks Harry.  I think I have a misunderstanding about OSC cameras.  Is the bayer film applied to CCD camera not a property of the chip?  I didn't think the firmware could change the bayer pattern of a CCD chip.

Ken
Regards,
Ken
My Astro Photo Stream
http://www.flickr.com/photos/kenastro/