Author Topic: Incompatible image geometry between masters  (Read 5362 times)

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Incompatible image geometry between masters
« on: 2013 April 27 23:37:31 »
Hi, I don't know if this is a bug...

It seems that BatchPreprocessing in PI 1.7 generates masters different of  BatchPreprocessing in PI 1.8, as I have the message "incompatible image geometry"  when mixing masters from both sources.
Indeed, while reading the images to generate the masters, I noticed that some values are different:

for Canon 5DIII' images, PI 1.7 on windows say:

Thumb size: 5760x3840
Full size: 5920x3950
Image size: 5920x3950
Output size: 5920x3950

while in PI 1.8 on Linux say:

Thumb size: 5760x3840
Full size: 5920x3950
Image size: 5796x3870
Output size: 5796x3870

Saludos, Alejandro

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Incompatible image geometry between masters
« Reply #1 on: 2013 April 28 10:09:46 »
is it just batchpreprocessing, or do you think the version of DCRAW is different between 1.7 and 1.8 and it's converting the CR2s differently?

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Incompatible image geometry between masters
« Reply #2 on: 2013 April 28 11:45:45 »
Hi Rob, Maybe you are right and the version of DCRAW is different. I opened a single image and PixInsight is reading it in different ways.

PI 1.8 RC7
Code: [Select]
open "/home/alejandro/Datos/Pictures/A PROCESAR/2013_01_07 M93 5 DIII TRENEL/2013_01_08 LT 02_41_54 ISO 800 010018_0125.CR2"
Reading 1 file(s):
/home/alejandro/Datos/Pictures/A PROCESAR/2013_01_07 M93 5 DIII TRENEL/2013_01_08 LT 02_41_54 ISO 800 010018_0125.CR2

Timestamp: Tue Jan  8 02:41:54 2013
Camera: Canon EOS 5D Mark III
Owner: Alejandro Tombolini
ISO speed: 800
Shutter: 60.4 sec
Aperture: f/inf
Focal length: 0.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  5760 x 3840
Full size:   5920 x 3950
Image size:  5796 x 3870
Output size: 5796 x 3870
Raw colors: 3
Filter pattern: RGGBRGGBRGGBRGGB
Daylight multipliers: 2.125175 0.943985 1.338680
Camera multipliers: 1680.000000 1024.000000 1971.000000 1024.000000

Invoking: dcraw -D -k 0 -t 0 -o 0 -4
Decoding Canon EOS 5D Mark III file (5796x3870 pixels, ISO=800, Exposure=60.40s): done
Loading raw image: done

PI 1,7
Code: [Select]
Reading 1 image(s):
/home/alejandro/Datos/Pictures/A PROCESAR/2013_01_07 M93 5 DIII TRENEL/2013_01_08 LT 02_41_54 ISO 800 010018_0125.CR2

Timestamp: Tue Jan  8 02:41:54 2013
Camera: Canon EOS 5D Mark III
Owner: Alejandro Tombolini
ISO speed: 800
Shutter: 60.4 sec
Aperture: f/inf
Focal length: 0.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  5760 x 3840
Full size:   5920 x 3950
Image size:  5920 x 3950
Output size: 5920 x 3950
Raw colors: 3
Filter pattern: RGGBRGGBRGGBRGGB
Daylight multipliers: 2.391381 0.929156 1.289254
Camera multipliers: 1680.000000 1024.000000 1971.000000 1024.000000

Invoking: dcraw -q 1 -k 0 -t 0 -o 0 -4
Decoding Canon EOS 5D Mark III file (5920x3950 pixels, ISO=800, Exposure=60.40s): 100%
Loading raw image: 100%

Saludos, Alejandro.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Incompatible image geometry between masters
« Reply #3 on: 2013 April 29 02:01:21 »
Hi Alejandro,

The Canon EOS 5D Mark III camera has been supported since version 9.15 of dcraw, which was released in June 2012. PI 1.7 uses an older version, so it doesn't support this camera. PI 1.8 includes dcraw 9.17 (December 2012). This explains why you get wrong image geometries with PI 1.7, while PI 1.8 fully supports your camera.

One solution is cropping your old masters to the correct output size (5796 x 3870), but this is tricky because you may have alignment issues. The best option is recalibrating everything with PI 1.8.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Incompatible image geometry between masters
« Reply #4 on: 2013 April 29 04:29:00 »
Thank you Juan, I will recalibrate with 1.8 then.
Saludos, Alejandro

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Incompatible image geometry between masters
« Reply #5 on: 2013 May 01 16:56:34 »
Hi,

I am trying to re doing the masters with 1.8RC7 due to geometry issues but without luck. :-[

First I tried it in KDE, and I have reproducible crashes when BPP is reading bias in number 159. I thought that was a memory related problem but I got more free space and the crash is always when reading the same image.
Anyway I tried again in Ubuntu where I have 5 times more memory and the resutl is the same, but here PI do not crashed and stop. See message in the first screenshot.

I repeated the process several time, removing some bias, etc. until in one ocasion a new messages with a "out of memory" warning appeared. See message in the second screenshot.

I don't know if could be a bug in BPP or a real problem with memory. I have 412 GB free space in home where PI is located and 16 GB of ram

Any help?

BTW, I also noticed that in ubuntu the "output size" is inverted respet ot KDE, could it be a later geometry problem between masters?

Saludos, Alejandro.

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Incompatible image geometry between masters
« Reply #6 on: 2013 May 01 20:17:04 »
Hi,
One more try with image integration and it fail, which means that is obviously a memory problem. The question is what I need to change in order to integrate it? More RAM or more free space on disk?
Saludos, Alejandro.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Incompatible image geometry between masters
« Reply #7 on: 2013 May 03 02:48:06 »
Hi Alejandro,

In first place, let's diagnose the problem. DSLR raw file formats don't allow incremental file reading operations. This means that to integrate a set of these files, all of them have to be read and loaded in RAM first. Each one of your 5D Mark III raw images requires the following space in memory:

5796 * 3870 * 4 = 89,722,080 bytes = 85.57 MiB

Each pixel requires 4 bytes because ImageIntegration stores raw pixel values in 32-bit floating point format. Now if we multiply this by 159:

5796 * 3870 * 4 * 159 = 14,265,810,720 bytes = 13.29 GiB

which is still manageable on Linux with your 16 GiB. However, take into account that each image requires more space temporarily, especially to compute some statistics and to perform noise evaluation (by the way, you should disable it to integrate bias frames). There are also a lot of internal data structures that ImageIntegration creates for its internal use. All of these items are contributing to exhaust your RAM after loading 159 images.

A brute force solution to this problem is obviously installing more RAM in your machine. However, a better solution in this case is converting the DSLR raw files to a format that allows incremental file reading, such as FITS. I'd use the BatchFormatConversion script to convert all of your bias frames to FITS. With incremental file reading, you can integrate thousands of images without problems because the files are read by small pixel strips (the practical limit would be the number of open files allowed by the operating system, which is 1024 by default on x64 Linux, and you can increase it if necessary with the ulimit command).
« Last Edit: 2013 May 03 02:54:19 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Incompatible image geometry between masters
« Reply #8 on: 2013 May 04 08:11:14 »
Hi Juan and thanks!!! :), converting to .fit did the trick to integrate the 300 bias. Anyway I am going for the brute force solution and will install more RAM.
Now I remember that it was what I usually did before when used XP with low RAM. O:)
Thanks again!
Saludos. Alejandro.