Hi all,
Today I have released a first version of the new RAW PixInsight module. RAW implements support for digital camera raw formats based on the LibRaw open-source library (https://www.libraw.org/). RAW replaces the old DSLR_RAW module, which was based on a custom adaptation of the (now discontinued) dcraw (https://www.cybercom.net/~dcoffin/dcraw/) program by Dave Coffin. The use of LibRaw provides us with new demosaicing methods, interesting post-processing features, such as new noise reduction algorithms, and the guarantee that we'll support the latest camera models now and in the foreseeable future.
Availability of the RAW module
The new module is stable and has been thoroughly tested. However, since this module provides critical functionality, and given the complexity and difficulty to test it on a large amount of different raw formats, I prefer to have it extensively tested in real-world scenarios before releasing it as an official update. For this reason you have to use our development update repository to install it. Do the following:
1. In PixInsight, select RESOURCES > Updates > Manage Repositories from the main menu.
2. On the Manage Update Repositories dialog box, click the Add button.
3. This will show the Add PixInsight Repository dialog. Copy and paste the following in the URL field:
https://pixinsight.com/update-devel/
4. Click the OK button on the Add PixInsight Repository dialog.
5. Click OK on the Manage Update Repositories dialog.
6. From the main menu, select RESOURCES > Updates > Check for Updates.
7. A new update with the RAW module should appear. Accept the update, then exit PixInsight to install it, and wait for the application to restart, as usual.
This update will replace the old DSLR_RAW module with the new RAW module. You'll get full support for the latest camera models, such as the Canon EOS-1D X Mark II, EOS 5D Mark IV, Fujifilm X-T20, etc., and added features for management of raw images.
Here is a screenshot of the RAW Format Preferences dialog box, showing all features available in the new module:
(http://forum-images.pixinsight.com/20180205/RAW-preferences.png)
New Demosaicing Algorithms
Besides the superpixel, half-size, bilinear, VNG, PPG and AHD demosaicing methods, which were already available in the old DSLR_RAW module, the new RAW module introduces the following demosaicing algorithms:
* DCB demosaicing algorithm (http://www.linuxphoto.org/html/dcb.html) by Jacek Gozdz.
* DHT demosaicing algorithm by Anton Petrusevich.
* Modified AHD demosaicing algorithm (AAHD) by Anton Petrusevich.
I suggest you review the In-Depth Demosaicing Algorithm Analysis (https://www.libraw.org/node/2306) article on LibRaw's website. Keep in mind that not all algorithms available in LibRaw have been included in the new RAW module. Unfortunately, I cannot include algorithm implementations released under the GPL license, since GPL (any version) forbids inclusion in proprietary applications, even when the relevant parts of the application using those implementations are released as open-source products (as is the case with the RAW module).
From the tests I have done, I have to say that the new algorithms, especially DCB, are great for daylight images. However, my personal view is that VNG remains the best option for underexposed data, including deep-sky astronomical images. I'd love to know your opinions and personal experiences in this regard.
FBDD Noise Reduction
After quite a few years in this image processing business, I don't get impressed easily. But I have to say that the FBDD noise reduction algorithm (http://www.linuxphoto.org/html/fbdd.html), by Jacek Gozdz, has been a nice surprise. I am still asking myself how on earth I didn't discover it before. This algorithm is simply fantastic to remove small-scale bright artifacts from mosaiced data without altering significant image structures. In the following example you can see how efficient FBDD noise reduction is to suppress hot pixels in an uncalibrated raw frame acquired with a Canon EOS 450D camera:
(http://forum-images.pixinsight.com/20180205/FBDD-comparison-450D-tn.jpg) (http://forum-images.pixinsight.com/20180205/FBDD-comparison-450D.png)
Click on the image for a full-size version (http://forum-images.pixinsight.com/20180205/FBDD-comparison-450D.png)
I still have to figure out how FBDD can be optimally integrated with our image preprocessing pipelines. For now, this noise reduction algorithm is available for postprocessing (demosaicing) of raw frames, and is enabled by default (with three FBDD iterations). The good news is that this algorithm works on mosaiced data, that is, before interpolation of CFA frames. This opens the door to integrating FBDD in the complete preprocessing flow, all the way from calibration up to drizzle integration. More news on this topic soon.
X-Trans Sensor support
The new RAW module fully supports Fujifilm X-Trans CFA frames. See how the X-Trans CFA pattern can now be loaded as pure raw data (in raw RGB mode in this example):
(http://forum-images.pixinsight.com/20180205/x-trans.png)
This means that once you install the new RAW module, X-Trans raw data can be calibrated. However, for a complete support of X-Trans CFA frames, we need to implement X-Trans CFA interpolation in the Debayer module (which, by the way, should not be called Debayer anymore after such update), so the StarAlignment and ImageIntegration tools can work to register and integrate the demosaiced data, respectively. The DrizzleIntegration module requires no change at all, since it is already able to work with X-Trans CFA patterns (with arbitrary mosaic patterns, actually). I will be working on this very soon.
Open-Source Module
As promised, the new RAW module has been released as an open-source product under PCL license. The entire source code is now available at our official GitHub repositories:
https://github.com/PixInsight/PCL/tree/master/src/modules/file-formats/RAW
A custom adaptation of LibRaw is also part of our PCL development framework:
https://github.com/PixInsight/PCL/tree/master/src/3rdparty/libraw
That's all for now. The RAW module, along with an updated Debayer tool with X-Trans support, will be included in the next version of PixInsight, which will be released soon for all platforms, including FreeBSD ;) I really need to know how the new RAW module works in the real world, so *please* let me know anything you find out as soon as possible, be it positive or not. Enjoy!
Hi Carlos,
Nice to see you here! ;) Yes, with this new module, we support Canon dual pixel raw frames. For example, you can download this sample file:
http://www.imaging-resource.com/PRODS/canon-5d-iv/E5D4hSLI000100DPRaw_FINE.CR2.HTM
and open it with the new RAW module without problems. I am not sure about how LibRaw manages these dual image frames (we can investigate this). Note that the resolution of the interpolated image is indeed very high. On a side note, compare the results with the VNG and DCB demosaicing algorithms. DCB produces an interpolated image with virtually no color or ringing artifacts:
(http://forum-images.pixinsight.com/20180207/VNG-DCB-tn.jpg) (http://forum-images.pixinsight.com/20180207/VNG-DCB.png)
Click on the image for a full-size preview (http://forum-images.pixinsight.com/20180207/VNG-DCB.png)