Conversion of RAW dark and flat files to .fit for IC - is it essential?

[astropixel]

This one is for the developers - possibly obvious, but I am going to ask it anyway.

For DSLR users creating master frames, conversion of RAW bias frames to .fit for integration is essential, because II only accepts .fit files.

With that in mind, is it essential to convert RAW dark and flat files to .fit before processing with ImageCalibration. I suspect not, however, if there are any subtle, behind the scenes processing issues, it would be nice to know whether these are trivial or not.

That is, should RAW dark and flat files always be converted to .fit as a first step?

[zvrastil]

Or better (at least from my point of view) - would it be possible to extend ImageIntegration to accept RAWs?
I see several benefits:
 - you do not need to use or even to know the BatchFormatConversion script (so it's more intuitive)
 - you do not need to remember to set up DSLR_RAW format module correctly (assuming II and IC use correct way to load RAW)
 - you save a disk space
 - you save a time

Zbynek

[Juan Conejero]

Or better (at least from my point of view) - would it be possible to extend ImageIntegration to accept RAWs?

The ImageIntegration tool performs incremental file reading operations during integration. Basically, it reads a strip of pixel rows and integrates them, then proceeds with the next strip, and so on. That's why ImageIntegration is able to integrate thousands of CCD frames with relatively small memory requirements. The number of pixel rows in a strip can range from one to the number of rows that fit in the specified buffer size (16 MB by default for each image).

The FITS format supports incremental reading natively. However, ImageIntegration refuses working with DSLR raw images because the DSLR_RAW module doesn't support incremental reading (it uses dcraw.c as its file format backend).

It's true that ImageIntegration could be modified to perform standard block reading operations as an option, although this would limit the number of integrated images to the size of the physical memory. In my opinion, however, the upcoming master frame generation script will simplify things to the point that the RAW -> FITS conversion will be almost transparent (BatchFormatConversion will be integrated into the script, so no user actions will be necessary). It's true that this extra conversion step requires additional disk space and some time. The time and disk space are relatively small anyway because this conversion is only necessary for master bias and darks (each individual flat frame is calibrated separately).

- you do not need to remember to set up DSLR_RAW format module correctly (assuming II and IC use correct way to load RAW)

II and IC know nothing about image formats, including DSLR raw or FITS. They request loading the images through intermodule communication, but they see the images as abstract entities. For example, when the upcoming PixInsight native image format is implemented, II will be able to use it without additional requirements, as this format will support incremental reading. This is the object-oriented design of PixInsight.

I'm thinking on changing the default DSLR_RAW settings so they are valid for astronomical use out-of-the-box. Right now the default settings are more appropriate for daylight photographers. I was concerned about scaring them too much when we first published this format module. Now I think this was a wrong decision.

Feed for my thought for sure, thanks for the insights

[zvrastil]

Hi Juan,

thanks for your explanation.

The ImageIntegration tool performs incremental file reading operations during integration. Basically, it reads a strip of pixel rows and integrates them, then proceeds with the next strip, and so on. That's why ImageIntegration is able to integrate thousands of CCD frames with relatively small memory requirements. The number of pixel rows in a strip can range from one to the number of rows that fit in the specified buffer size (16 MB by default for each image).

The FITS format supports incremental reading natively. However, ImageIntegration refuses working with DSLR raw images because the DSLR_RAW module doesn't support incremental reading (it uses dcraw.c as its file format backend).

I understand this and agree that incremental reading is the only meaningful option. However, DeepSkyStacker can do it somehow - it provides constant-memory integration for both RAW and FITS formats. And it also reads RAWs using dcraw. So I assume there is a way around.

the upcoming master frame generation script will simplify things to the point that the RAW -> FITS conversion will be almost transparent (BatchFormatConversion will be integrated into the script, so no user actions will be necessary).

Great, I'm looking forward to this script.

II and IC know nothing about image formats, including DSLR raw or FITS. They request loading the images through intermodule communication, but they see the images as abstract entities. For example, when the upcoming PixInsight native image format is implemented, II will be able to use it without additional requirements, as this format will support incremental reading. This is the object-oriented design of PixInsight.

Wouldn't it be possible to pass some parameter through intermodule communication, to override DSLR_RAW configuration when loading image on demand?

regards, Zbynek

[Juan Conejero]

Wouldn't it be possible to pass some parameter through intermodule communication, to override DSLR_RAW configuration when loading image on demand?

The II or IC modules must know nothing about any particular format module. This preserves isolation between processes and data (images). This is a fundamental design principle of PixInsight, and one of the main keys of its strength.

The functionality that you're proposing, that is the ability to override the default settings of a particular format, must be implemented without violating the principles of modularity and encapsulation. A possibility is a system based on "hints".

For example, on most GUI environments, one usually passes one or more hints to the windowing system to create a widget with some particular features. A top-level window can have a maximization button, if the system or a particular combination of features support it. The application requests the maximization feature through a hint flag passed to the window constructor:

foo = CreateWindow( ... , MAXIMIZE_BUTTON );

We could have a similar system to create format instances in PCL (a format instance is the abstraction that PCL uses to manage image files). This could be an additional argument passed to the FileFormatImplementation::Open member function. The prototype of this function is now:

   virtual ImageDescriptionArray FileFormatImplementation::Open( const String& filePath );

It could be:

   virtual ImageDescriptionArray FileFormatImplementation::Open( const String& filePath, FileFormatHints hints = FileFormatHints::NoHints );

where FileFormatHints would be a collection of flags requesting specific format features. One of these flags could be:

   FileFormatHint::RawData

which in the case of the DSLR_RAW module would select a set of settings to load pure raw data (no debayerization, no black pedestal, no white balance, etc.).

This is interesting indeed. What do you think? Guess I'm gonna draw it on a paper sheet

[zvrastil]

Hi Juan,

yes, I something like that could be the way. Till now, I had no idea what are the possibilities of intermodule communication in PixInsight, so I was not imagining any particular solution.
In the software I work on in my daily job (electron microscope controlling software), we need more flexibility in communication between different modules, so we use kind of central service (singleton class) with well defined interface, where modules can post messages and register for particular messages. Communicating modules has to know the same message type (and its format), but they do not need to know each other - it is loosely coupled event system.  However, something like this would be an overkill for this use case. Hints are simple and solve the problem.

BTW, with current limitations in mind, how does ImageIntegration module do partial FITS image loading?

One more thought - even if it's not possible to load RAWs partially, it may be ok to just load whole image each time, extracting only required part and then discarding it. RAWs are usually smaller than FITS and if the computer has enough memory, they would be kept in system cache anyway. And loading RAW without any debayering should be quite fast, I believe. I expect the time overhead would be marginal compared to whole integration process time.

best regards, Zbynek

[astropixel]

Thanks Juan. That is what I thought. The new tools sound great.