New Script: BayerDrizzlePrep

[IanL]

Hi,

I've put together a new script for:

- Converting Mono CFA images produced by OSC/DSLR cameras to RGB Bayer images suitable for processing by DrizzleIntegration as part of a Bayer drizzle workflow.
- Copying and updating the accompanying Drizzle Data files (.drz) to point at the new RGB Bayer Images.

This process is covered extensively in other threads such as http://pixinsight.com/forum/index.php?topic=7184.0 and http://pixinsight.com/forum/index.php?topic=7151.0.

Until now the only automated way to undertake this task has been via the BatchPreprocessing script, which limits flexibility in achieving ideal calibration and integration settings.  This script greatly simplifies the tedious image file conversion and renaming tasks and should make stand-alone Bayer drizzle much easier.

Note that there is also the CFAToRGB module available here: http://pixinsight.com/forum/index.php?topic=8319.0, which can convert the image files manually or via an ImageContainer.  Unfortunately the compiled versions are currently not working with the current release of PI and I don't have the ability to recompile from source.  It isn't yet part of the standard PI release either so instead my script relies on PixelMath to perform the image file conversion tasks (shamelessly borrowed from BPP of course, along with bits from other scripts - full credits in the source and copyright files).

The theory and instructions for use are available in the included PIDOC documentation so I won't reproduce them here.  Again parts shamelessly borrowed from Juan's various posts on drizzle.

Please test and report back any issues via this thread.  Once I am happy I will create a pull request in the PI GitHub repository and hopefully it will then make it in to the next official release, assuming Juan is happy with my work.

ChangeLog

1.0: First release.

Script Installation Instructions

Note that BayerDrizzlePrep is now part of the standard PixInsight  release as of version 1.8.4.1193. If you would like to add the script to an older version, please follow the instructions below.

1. In the attached .zip file you'll find a folder called "BayerDrizzlePrep".  Place the unzipped folder and its contents in your <PixInsight Folder>/src/scripts folder. (Note: Do not place the contents of the "BayerDrizzlePrep" folder directly in to /src/scripts but rather ensure that it is inside the scripts folder in a sub-folder called BayerDrizzlePrep).

2. In PixInsight select "Script" menu -> "Feature Scripts...".

3. Now click the "Add" button from the "Feature Scripts" dialog. Make sure the <PixInsight Folder>/src folder is selected and click "OK".

4. You should get a message saying that "1 additional script(s) were found on directory ...". Click "OK" and then "Done".

5. The BayerDrizzlePrep Script should now appear under the  "Script" menu -> "Batch Processing" -> "BayerDrizzlePrep".

If you need more help, please see this video: http://www.pixinsight.com.ar/en/docs/2/pixinsight-add-script.html.

Documentation Installation Instructions

7. To install the documentation unzip the "pidoc" folder to a temporary location.

8. In PixInsight select "Script" menu -> "Development" -> "Documentation Compiler".

9. Click "Add Files" and browse to the pidoc folder that you just unzipped.

10. Choose the "BatchStatistics.pidoc" file and click "Open" to add it to the "PIDoc Source Files" list.

11. Click the "Run" button.

12. Click "No" when asked if you want to perform another compilation.

13. The console should display "PIDocCompiler: 1 succeeded, 0 failed, 30 warning(s)".

14. You can read the documentation by clicking the "Documentation" button at the bottom left of the "BayerDrizzlePrep" dialog, or by finding it in the "Process Explorer" under "<Scripts>" -> "Batch Processing" -> "BayerDrizzlePrep".

[jkmorse]

Ian,

That is great stuff.  I am a monochrome CCD guy but I have a cribsheet that a number of folks use and I know some are using DSLRs.  Would you mind if I put a link to this post in my next revision?

Thanks,

Jim

[IanL]

Ian,

That is great stuff.  I am a monochrome CCD guy but I have a cribsheet that a number of folks use and I know some are using DSLRs.  Would you mind if I put a link to this post in my next revision?

Thanks,

Jim

Of course Jim, it's all about sharing!

[NKV]

Note that there is also the CFAToRGB module available here: http://pixinsight.com/forum/index.php?topic=8319.0, which can convert the image files manually or via an ImageContainer.  Unfortunately the compiled versions are currently not working with the current release of PI and I don't have the ability to recompile from source. 

CFA2RGB module updated for PI1.8.4.1195

[johnpane]

It would be useful if this script would allow the user to specify an output file format.

I have CR2 RAW CFA images from a Canon camera. If I turn off debayering in DSLR_RAW, PixInsight will open the file as a CFA image. However, if it does this within the BayerDrizzlePrep script, it wants to write out the file in the same format, and generates the error: "No installed file format can write .CR2 files". Thus, I must first do a BatchFormatConversion to .xisf before running BayerDrizzlePrep.

If I could specify to BayerDrizzlePrep that I would like it to write out .xisf file, this extra step (and the extra data files generated) could be avoided.

[johnpane]

Something is not working correctly in my attempts to perform Bayer Drizzle assisted by BayerDrizzlePrep.

I am starting with 160 RAW .CR2 files from a Canon camera. I follow the workflow described in the documentation for BayerDrizzlePrep. The tracking drifted slightly throughout the acquisition process, so there is plenty of natural dithering.

Sample images are in this folder:
    https://www.dropbox.com/sh/ykmckzv15pjvrnx/AAAYFid-c9FdN-rO-iuJmJx5a?dl=0
These are 5x crops from near the center of the image, format converted from .xisf to .png. When from a subframe, I used the reference (for alignment) subframe.

1.2.1 Step 1. Calibration
Not applicable. These test images do not have calibration frames.

1.2.2 Step 2. Debayering
    Source .CR2 images processed with BatchDebayer to produce 32-bit floating .xisf color images.
        1_debayer_JFP_4475_1.png

1.2.3 Step 3. Registration
    Images from Step 2 run through ImageRegistration; reference frame does not change.

1.2.4 Step 4. Integration
    ImageIntegration run with settings: iterative k-sigma/biweight midvariance; windsorized  sigma clipping; generate drizzle data.
        2_integration_normal_1.png

1.2.6 Step 6. Bayer Drizzle
    As I posted yesterday, I needed to first run BatchFormatConversion to create .xisf CFA images. To do this, I set the DSLR_RAW preference to "create raw bayer CFA image", converting the .CR2 images to 16-bit unsigned integer greyscale CFA images.
        3_JFP_4475_1.png

    Then I ran BayerDrizzlePrep, and it created a Bayer rgb image (32-bit float .xisf) and updated the drz files.
        4_JFP_4475_brgb_1.png

    Then I ran DrizzleIntegration and obtained the following integrated image,
        5_drizzle_integration_1.png
    and drizzle weights.
        6_drizzle_weights_1.png

Everything seems fine until the DrizzleIntegration step. The integrated image has a strong green tint and has alternating vertical lines with blue-green and red-green tints. The stars have black artifacts at diagonal corners. The resulting integrated image is inferior to the one produced through the normal process of debayering first.

I tried DrizzleIntegration with scale=1, dropshrink=1, and with scale=2, dropshrink=1. Results are similar in both cases, here I show the one with scale=1.

Any ideas what I am doing wrong?

Thanks,
John

[IanL]

It can only be that the wrong source images are being substituted in to the .drz files by the BayerDrizzlePrep script or something else related to using .CR2 files at the start. I don't have access to a machine to test your workflow right now but the fact that you are starting with uncalibrated .CR2 files seems most likely to be the culprit for some reason.

I suggest that you start with the BatchFormatConversion script to create undebayered FITS or XSIF files and proceed from there. That should be functionally equivalent to the calibration process by taking the .CR2 files out of the loop rather than doing the format conversion midway through.

[johnpane]

Thank you Ian.

I discovered tha the change in image color is due to differences in how white balance is handled for deBayer vs. CFA conversions from CR2, and filed this bug report: http://pixinsight.com/forum/index.php?topic=10342.0

The remainder of the problem must have been as you stated, the source images getting mixed up. I ran again and got good results.

Following your advice to only do one conversion from .CR2, I adopted this series of conversions as more efficient:

CR2 -> RGB Bayer -> DeBayered -> Registered -> ImageIntegration
                 -> BayerDrizzle

This saves a conversion step versus:

CR2 -> CFA -> DeBayered -> Registered -> ImageIntegration
           -> RGB Bayer -> BayerDrizzle


To accomplish the file swap for the Drizzle integration, I used a unix shell script instead of BayerDrizzlePrep:
      
    mkdir debayer
    foreach f ( IMG*.xisf )
        mv debayer_$f debayer/ 
        mv $f debayer_$f
    end

(IMG*.xisf uniquely identifies the RGB Bayer images; prefixing this with debayer_ identifies the deBayered images).