Compatibility Update: RAW Module

Juan Conejero

PTeam Member
Sep 2, 2004
7,555
96
56
Valencia, Spain
pixinsight.com
Updated 2020 March 6, core version 1.8.8-5

Hi all,

This announcement is relevant for PixInsight users working with raw data acquired by some DSLR cameras. It describes a serious incompatibility introduced by recent versions of LibRaw, an open-source software library which our RAW module uses as a back-end to support raw formats of digital cameras.

Description of the Problem

The RAW module included in the standard PixInsight distribution since core version 1.8.8 uses LibRaw 201910 snapshot. Since this version, LibRaw has changed the way some raw formats are decoded. The changes we are interested in here refer to the coordinates of the visible region of the raw image with respect to the entire (uncropped) raw frame. As you probably know, many (most?) DSLR raw formats define unused areas of the raw frame, which correspond to rows and columns of pixels that are never exposed to light. Obviously, these unused regions are located at the edges of the raw frame, and their dimensions and location depend on the particular formats used by each camera model. Our RAW module provides a preferences option to enable/disable automatic cropping of unused areas, which is enabled by default.

Unfortunately, the changes introduced by LibRaw since the 201910 snapshot version for some cameras create an incompatibility with master calibration frames generated from raw frames of the affected formats with PixInsight 1.8.7 and earlier versions. This problem affects mainly to some Canon cameras, where the visible region of the image begins at odd vertical pixel coordinates from the upper left corner of the entire raw frame. This includes, probably among others, the following cameras:

EOS 1D Mark IV
EOS 5D Mark II
EOS 7D
EOS 60D
EOS 550D
EOS 600D
EOS 1200D

See the original bug report where this problem has been first described (to my knowledge) on this forum. The report refers to the Canon EOS 600D, which is quite popular among PixInsight users working with DSLR cameras. As you can see, in this camera the visible region of the frame starts at vertical pixel coordinate y=51, where x=y=0 corresponds to the pixel at the upper left corner of the uncropped raw frame.

LibRaw versions before the 201910 snapshot version reported frame coordinates correctly for all Canon cameras, that is, the actual coordinates of the visible image within the raw frame, as well as the actual Bayer pattern, which is always RGGB for these cameras.

Since the 201910 snapshot version, LibRaw intentionally removes the first row of pixels from the visible image for these raw formats. In the case of EOS 600D for example, LibRaw reports now a visible region starting at y=52 with a height of 3464 pixels, instead of the actual 3465 pixels. Since the cropped frame starts at y=52 instead of y=51, the reported Bayer pattern is now GBRG instead of RGGB.

The authors of LibRaw have good reasons to implement this change. As a software developer, I can understand their reasons from a technical point of view. The repercussions of this change for daylight or 'normal' photography applications are as minimal as a single row of pixels cropped at the top of the image. However, the problems generated are severe for applications requiring calibration of raw data.

LibRaw is not going to revert these changes. Since there is no alternative to LibRaw for a multiplatform application, we have to find a suitable solution.

Our Solution

The solution we have implemented is quite simple: a special update-compat repository, where you can opt-out of the new RAW module in PixInsight 1.8.8-1 and newer versions, replacing it with a module that is compatible with existing master calibration frames generated with PixInsight 1.8.7 and older versions for the affected cameras (see a partial list above).

If you really need to use master calibration frames incompatible with the latest version of LibRaw, here is the procedure to opt-out of the current RAW module:

1. Make sure that you are using PixInsight core version 1.8.8-5 or later.

2. Make sure that you have applied all available updates for your current PixInsight version. Note that this might include an update to the standard RAW module at the time you are reading this. In such case you must apply that update before continuing.

3. From the main menu, select Resources > Updates > Manage Repositories.

4. On the Manage Update Repositories dialog, click the Add button.

5. Enter the following repository URL:


6. Click OK, Then OK again on the Manage Update Repositories dialog.

7. From the main menu, select Resources > Updates > Check for Updates.

8. The Select and Download Updates dialog should now be open. Make sure that the RAW_compat update is available, then click Apply.

9. After downloading and extracting the update, as usual, exit the PixInsight core application and apply the update.

10. Check that you have the RAW_compat module in Format Explorer.

Here is a small video showing the above procedure:


After the update, the standard RAW module will be replaced by a special RAW_compat module that uses an improved version of LibRaw 0.19.5-Release. The RAW_compat module is basically the same RAW module included in PixInsight 1.8.7. You can use it while you have to use old master calibration frames for Canon cameras suffering from this incompatibility. Our recommendation, however, is to return to the standard RAW module once you are ready to work with newly acquired data.

To return to the standard RAW module, remove the update-compat repository enabled with the above procedure. Then add the following repository URL:


and repeat the check for updates process, etc. This will reinstall the standard RAW module using the latest LibRaw version available.

I hope this solution will be useful for all users that have to suffer from this serious compatibility issue. Please realize that we are not causing this problem. For obvious reasons, we cannot stick to an old version of one of our fundamental third-party support libraries. Thank you for your patience and understanding.
 

GeBs

New member
Sep 10, 2019
3
0
Thanks for the Info. I have a large Collection of Bias and Darks and wonder why I could not integrate them anymore und also wonderd why the debayer matrix is chagned. So I have to reintegrate all Bias and Darks again or is it possible to delete the one pixel row?
 

Felipe

Member
Jan 24, 2013
11
0
Hi
Please just confirm no problem with arw files from Sony A7S ?
thank you
Philippe LAURENT
 

segger

New member
Jan 21, 2018
1
0
Hi
Please just comfirm that if I rebuild my dark and bias library im fine?
And my new Bayer pattern is now GBRG for my Canon 60D? 
 

dghent

Member
Feb 12, 2019
16
3
daleghent.com
Hi, Juan

Have you been in conversation with Alex of LibRAW regarding this? I don't see a relevant thread on the LibRAW forums so I am wondering if he is aware of the issue and has any opinion concerning it.
 

dghent

Member
Feb 12, 2019
16
3
daleghent.com
bulrichl said:
See https://pixinsight.com/forum/index.php?topic=14206.0 , reply #5.
Ah, thanks for the pointer, Bernd. Alex's description the in the GHI makes sense. I'm in the process of adapting libRAW in another app so this issue made me perk up in my chair a bit ;)
 

Jim_

Active member
Sep 26, 2012
31
0
Hi all,
I've searched but can't find an answer.
Do I still need to pick GBRG for my Canon 60da?

Jim