Compatibility Update: RAW Module

Juan Conejero

PixInsight Staff
Sep 2, 2004
8,719
714
58
Valencia, Spain
pixinsight.com
Updated 2020 March 6, core version 1.8.8-5

Hi all,

This announcement is relevant for PixInsight users working with raw data acquired by some DSLR cameras. It describes a serious incompatibility introduced by recent versions of LibRaw, an open-source software library which our RAW module uses as a back-end to support raw formats of digital cameras.

Description of the Problem

The RAW module included in the standard PixInsight distribution since core version 1.8.8 uses LibRaw 201910 snapshot. Since this version, LibRaw has changed the way some raw formats are decoded. The changes we are interested in here refer to the coordinates of the visible region of the raw image with respect to the entire (uncropped) raw frame. As you probably know, many (most?) DSLR raw formats define unused areas of the raw frame, which correspond to rows and columns of pixels that are never exposed to light. Obviously, these unused regions are located at the edges of the raw frame, and their dimensions and location depend on the particular formats used by each camera model. Our RAW module provides a preferences option to enable/disable automatic cropping of unused areas, which is enabled by default.

Unfortunately, the changes introduced by LibRaw since the 201910 snapshot version for some cameras create an incompatibility with master calibration frames generated from raw frames of the affected formats with PixInsight 1.8.7 and earlier versions. This problem affects mainly to some Canon cameras, where the visible region of the image begins at odd vertical pixel coordinates from the upper left corner of the entire raw frame. This includes, probably among others, the following cameras:

EOS 1D Mark IV
EOS 5D Mark II
EOS 7D
EOS 60D
EOS 550D
EOS 600D
EOS 1200D

See the original bug report where this problem has been first described (to my knowledge) on this forum. The report refers to the Canon EOS 600D, which is quite popular among PixInsight users working with DSLR cameras. As you can see, in this camera the visible region of the frame starts at vertical pixel coordinate y=51, where x=y=0 corresponds to the pixel at the upper left corner of the uncropped raw frame.

LibRaw versions before the 201910 snapshot version reported frame coordinates correctly for all Canon cameras, that is, the actual coordinates of the visible image within the raw frame, as well as the actual Bayer pattern, which is always RGGB for these cameras.

Since the 201910 snapshot version, LibRaw intentionally removes the first row of pixels from the visible image for these raw formats. In the case of EOS 600D for example, LibRaw reports now a visible region starting at y=52 with a height of 3464 pixels, instead of the actual 3465 pixels. Since the cropped frame starts at y=52 instead of y=51, the reported Bayer pattern is now GBRG instead of RGGB.

The authors of LibRaw have good reasons to implement this change. As a software developer, I can understand their reasons from a technical point of view. The repercussions of this change for daylight or 'normal' photography applications are as minimal as a single row of pixels cropped at the top of the image. However, the problems generated are severe for applications requiring calibration of raw data.

LibRaw is not going to revert these changes. Since there is no alternative to LibRaw for a multiplatform application, we have to find a suitable solution.

Our Solution

The solution we have implemented is quite simple: a special update-compat repository, where you can opt-out of the new RAW module in PixInsight 1.8.8-1 and newer versions, replacing it with a module that is compatible with existing master calibration frames generated with PixInsight 1.8.7 and older versions for the affected cameras (see a partial list above).

If you really need to use master calibration frames incompatible with the latest version of LibRaw, here is the procedure to opt-out of the current RAW module:

1. Make sure that you are using PixInsight core version 1.8.8-5 or later.

2. Make sure that you have applied all available updates for your current PixInsight version. Note that this might include an update to the standard RAW module at the time you are reading this. In such case you must apply that update before continuing.

3. From the main menu, select Resources > Updates > Manage Repositories.

4. On the Manage Update Repositories dialog, click the Add button.

5. Enter the following repository URL:


6. Click OK, Then OK again on the Manage Update Repositories dialog.

7. From the main menu, select Resources > Updates > Check for Updates.

8. The Select and Download Updates dialog should now be open. Make sure that the RAW_compat update is available, then click Apply.

9. After downloading and extracting the update, as usual, exit the PixInsight core application and apply the update.

10. Check that you have the RAW_compat module in Format Explorer.

Here is a small video showing the above procedure:


After the update, the standard RAW module will be replaced by a special RAW_compat module that uses an improved version of LibRaw 0.19.5-Release. The RAW_compat module is basically the same RAW module included in PixInsight 1.8.7. You can use it while you have to use old master calibration frames for Canon cameras suffering from this incompatibility. Our recommendation, however, is to return to the standard RAW module once you are ready to work with newly acquired data.

To return to the standard RAW module, remove the update-compat repository enabled with the above procedure. Then add the following repository URL:


and repeat the check for updates process, etc. This will reinstall the standard RAW module using the latest LibRaw version available.

I hope this solution will be useful for all users that have to suffer from this serious compatibility issue. Please realize that we are not causing this problem. For obvious reasons, we cannot stick to an old version of one of our fundamental third-party support libraries. Thank you for your patience and understanding.
 

GeBs

Member
Sep 10, 2019
9
0
Thanks for the Info. I have a large Collection of Bias and Darks and wonder why I could not integrate them anymore und also wonderd why the debayer matrix is chagned. So I have to reintegrate all Bias and Darks again or is it possible to delete the one pixel row?
 

Felipe

Member
Jan 24, 2013
13
0
Hi
Please just confirm no problem with arw files from Sony A7S ?
thank you
Philippe LAURENT
 

segger

New member
Jan 21, 2018
1
0
Hi
Please just comfirm that if I rebuild my dark and bias library im fine?
And my new Bayer pattern is now GBRG for my Canon 60D? 
 

dghent

Active member
Feb 12, 2019
26
19
daleghent.com
Hi, Juan

Have you been in conversation with Alex of LibRAW regarding this? I don't see a relevant thread on the LibRAW forums so I am wondering if he is aware of the issue and has any opinion concerning it.
 

dghent

Active member
Feb 12, 2019
26
19
daleghent.com
bulrichl said:
See https://pixinsight.com/forum/index.php?topic=14206.0 , reply #5.
Ah, thanks for the pointer, Bernd. Alex's description the in the GHI makes sense. I'm in the process of adapting libRAW in another app so this issue made me perk up in my chair a bit ;)
 

Jim_

Well-known member
Sep 26, 2012
50
1
Hi all,
I've searched but can't find an answer.
Do I still need to pick GBRG for my Canon 60da?

Jim
 

astroalbo

New member
Mar 16, 2020
3
0
Hi:
Excuse me for hanging on an old thread, but I have the following problem: I have photos in CR2 format (Canon 600D) and photos in FIT format (QHY 168C). I follow the instructions given by Juan, to register both groups of photos, each one with the corresponding calibration routine.
However, once the registered files are generated, when I try to integrate both groups of photos, it throws me the message "Incompatible image geometry" . I did this process in 2020, with another object, without any problem, but now PixInsight does not allow it.

I would appreciate it if someone could tell me if anyone has had this problem and if they have been able to solve it.

Thanks,
Pablo
 

pfile

PTeam Member
Nov 23, 2009
7,792
488
were both sets of images registered to the same reference image? sounds like they were not, as all the images should have the same dimensions as the reference image.

rob
 

astroalbo

New member
Mar 16, 2020
3
0
Thanks for your answer Robar. No, I registered each group of photos separately, with a reference image from its own group, just as I had done on previous occasions.

I stacked the CR2s with BatchPreprocessing, choosing an image from their own group. And I stacked the FITs with WeightedBatchPreprocessing, with Manual mode to choose the reference image.

Pablo
 

pfile

PTeam Member
Nov 23, 2009
7,792
488
well, no matter the type of file, if you are going to integrate images together they all need to be aligned to the same reference image. even if they started out as the same size, if the reference is different, you'll get garbage if you integrate images together that have different references.

you can give WBPP the same reference image for both runs if you want to make sure all the images are aligned to one another. another possibility is to take the two masters you have, align one of them to the other with StarAlignment, then load the aligned master and the alignment reference master into ImageIntegration, but since II requires a minimum of 3 files you will need to load each of your masters twice, so that there are 4 input files to ImageIntegration.

rob