image file size in 1.8.8 different from 1.8.7 for RAW files

[jbaetz]

Hi,

after upgrading to 1.8.8 I found an issue while calibrating my CR2 files from my Canon 600D with already processed files (flats/master darks) in older PI version.
I'm using the "pure RAW" config set.
By investigating the issue I stumbled over the output of the RAW decoder in 1.8.8:

Code: [Select]

Reading 1 image(s):
D:/data/Astro/PI/rework/Vega_260mm.CR2
Reading metadata: done

Camera ........... Canon EOS 600D
Timestamp ........ 2018-07-01T21:34:39Z
Exposure ......... 30s
ISO speed ........ 12800
Focal length ..... 50 mm
CFA pattern ...... Bayer GBRG
Raw dimensions ... w=5344 h=3516
Image geometry ... x=142 y=52 w=5202 h=3464
Image rotation ... 90 deg

Raw decoding parameters:
Output mode ............... cfa
Auto crop ................. enabled
Wavelet noise threshold ... 0
White balancing ........... disabled
Black point correction .... disabled
Highlights clipping ....... disabled
Auto rotate ............... disabled
Output image .............. w=5202 h=3464 n=1 Gray

Reading RAW data: done

As you can see the resulting Image geometry and the CFA pattern is wrong.
The reported image geometry high is cut of by 1 px. When I open the image with the RAW module without cropping the overscan area, I see the missing top most row in the imported image.

I think that's the reason for the pattern mixup.

For comparison: I could open the file correctly in full patched 1.8.7:

Code: [Select]

Reading 1 image(s):
D:/data/Astro/PI/rework/Vega_260mm.CR2
Reading metadata: done

Camera ........... Canon EOS 600D
Timestamp ........ 2018-07-01T21:34:39Z
Exposure ......... 30s
ISO speed ........ 12800
Focal length ..... 50 mm
CFA pattern ...... Bayer RGGB
Raw dimensions ... w=5344 h=3516
Image geometry ... x=142 y=51 w=5202 h=3465
Image rotation ... 90 deg

Raw decoding parameters:
Output mode ............... cfa
Auto crop ................. enabled
Wavelet noise threshold ... 0
White balancing ........... disabled
Black point correction .... disabled
Highlights clipping ....... disabled
Auto rotate ............... disabled
Output image .............. w=5202 h=3465 n=1 Gray

Reading RAW data: done


addentum: This occurs for all RAW files, older or newer ones. Canon 600D.

Thank you,
Jürgen

[bulrichl]

Hi Jürgen,

this seems to be a bug in the new LibRaw version, 201910 snapshot, that is used by PixInisght 1.8.8. Since cropping the first row changes the CFA pattern:

R G          G B
G B          R G

the displayed CFA pattern is not wrong. However, the main problem is that these light frames cannot be calibrated with old Master Calibration files (different image geometry).

A workaround would be: load the CR2 files and crop them with parameters: left margin 0, top margin 1, right margin 0, bottom margin 0. This adds an empty first row to the image. Thereby the CFA pattern becomes RGGB again and the image geometry is consistent with the previously generated Master calibration files. The cropping can be done for an Image Container.

Note that in this case the Bayer/mosaic pattern has to be set manually for the Debayer process, 'Auto' doesn't work with this workaround.

Bernd

[jbaetz]

Thank you Bernd for your explaining and the workaround.
Now I'm able to continue to (pre-)process my lights with PI  .

Hope the guys from libraw will fix it soon and bring the 600D support back into a release-snapshot.

Best Regards!
Jürgen

[Juan Conejero]

Hi Jürgen,

I am aware of this problem with the latest LibRaw 201910 snapshot version, which has been included in PixInsight 1.8.8. I am considering the possibility to revert this version to the previous stable LibRaw 0.19.5-Release.

With this change we'll lose CR3 format support, as well as support for the raw formats of some of the newest camera models. However, stability and correctness are more important. I'll decide about this change before 48 hours, and will release an update consequently.

Sorry for the inconvenience.

[Juan Conejero]

I have created a new issue on LibRaw's GitHub repository:

https://github.com/LibRaw/LibRaw/issues/252

Let's see if this is a confirmed bug and we can have a solution soon.

[Juan Conejero]

As you can see, we have an answer from LibRaw's development team:

This is not bug, but intentional (so won't be fixed).
The problem:
LibRaw::COLOR(row,col) operates in active area coordinates, (0,0) relates to (left_margin,top_margin) of entire sensor.
To make this call valid for both active area and entire sensor area, left_margin/top_margin are always made even for bayer images (and multiple of 6 for X-Trans images)

This makes sense because the visible image region cannot start in the middle of a Bayer pattern, so its starting coordinates cannot be odd. The latest version is doing it correctly; there is no bug here. My recommendation is to repeat all preprocessing tasks, when possible, with PixInsight 1.8.8.

[jbaetz]

Juan, thank you for your explanations and getting in contact with the libraw people.

My calibration files are integrated already and stored in a database, so I have to adjust them properly to align them with the newer light frames after the 1.8.8.

Thank you
Jürgen

[bulrichl]

It seems that LibRaw will maintain the changed cropping behavior of version 201910 snapshot. Thus the workaround that I suggested above (expecting that they revert to the original cropping behavior) is not reasonable.

In order to use old MasterCalibration files that were prepared with PixInsight versions < 1.8.8, a better workaround would be: crop (remove) the first row of the Master Calibration files (MasterDark and MasterFlat) and save them as new files. These modified Master Calibration files could then be used furthermore with PixInsight versions starting from 1.8.8.

---

The 600D is not the only camera model that is concerned; so far as I know the following Canon models had an uneven number of rows as well, and I guess it will be similar:

EOS 1D Mark IV
EOS 5D Mark II
EOS 7D
EOS 60D
EOS 550D
EOS 600D
EOS 1200D

Bernd

[Juan Conejero]

You are not the only user suffering from this problem. We are receiving a lot of complaints. As Bernd has said, there are many Canon cameras affected by this incompatibility introduced by LibRaw. Basically, the same problem happens with any raw format where the visible region of the frame begins at odd distances from the upper left corner of the entire (uncropped) raw frame on the X and/or Y axes.

I am working on a solution to this problem. It is basically ready now, so I'll make an official announcement later today. The solution is a special update repository where you can opt-out of the current version of the RAW module, and return to a RAW_compat module that uses the same LibRaw version used in version 1.8.7 of PixInsight. This allows you to continue working with your existing master calibration frames generated before 1.8.8. You can opt-in to the current RAW module as soon as you are ready to process newly acquired raw data.

Edit: Of course, other solutions based on cropping existing master frames, as Bernd has described, are also possible. However, many users don't feel comfortable altering master calibration frames, or simply prefer the previous behavior and/or a more integrated solution. The special compatibility update repository is the best option for most of these users IMO.

[bulrichl]

That's great. And sorry, Juan, I did not intend to confuse the forum with my suggestions.

Bernd

[Juan Conejero]

Please see our solution to this serious problem:

https://pixinsight.com/forum/index.php?topic=14219.0

I hope this helps. Please let me know what you think.

[jbaetz]

Hi Juan,

the final solution is a good way to handle the issue. The existing calibration files can now easily adapted to the new (and correct) interpretation in libRAW. For faster translating the old files I tested yesterday the method of cropping the first row from the files without problems.

Thank you very much for your fast action and providing the workaround.

Jürgen