Understand "DARKFLAT" image type in WBPP/BPP

[dghent]

Hi all,

This is an ask/proposal that the WBPP and BPP scripts gain the ability to understand and sort files accordingly when a DARKFLAT value for the IMAGETYP FITS keyword/XISF image type property is encountered.

Some background and justification:

One of NINA's features is a flat frame creation wizard, which automates the flat and dark flat frame acquisition process. When "dark flat" frames are created, we have always given the FITS and XISF image type properties a value of DARKFLAT[1] to make them readily identifiable as those types of frames, as they serve a similar yet functionally different purpose than that of classic DARK-type frames.

The issue that prompts my request here is that, when using the "Add Files..." button in WBPP or BPP, the dark flat frames will be automatically, and errantly, sorted into the "Flats" tab and category of images, when they should actually be sorted into the the "Darks" tab and category. The unaware user might not catch this or know that these certain types of files should be actually treated as darks instead, leading to situations such as the kind that Adam Block explains in a recent video.

I understand that NINA (and other applications) could tag these as DARK type images and this issue would be solved, however given what dark flats are and how they are acquired, I think there is value in maintaining their own image type designator for categorization and awareness purposes so that they are not accidentally intermingled with actual DARK frames in the user's own frame type sorting methods.


[1]

 

[dghent]

I realize that this proposal also bleeds over in to XISF specification territory, as "DarkFlat" and "MasterDarkFlat" is not one of the image types that is specified under 11.5.1 Mandatory Image Attributes and so specifying such a type in an XISF file (NINA writes XISF files) would not conform to the XISF 1.0 specification. This leaves us in kind of an odd situation as we wish to uniquely identify images of this type, but there is no appropriate, standards-conformant definition that is suitable.

I don't know what the formal process is for proposing extensions or changes to the XISF specification, but if such a mechanism does exist, I'd like to engage in it to put this image type forward for consideration.

 

[bulrichl]

The term 'dark flat' is completely wrong. It is generally accepted that the last word in a compound noun is the component characterizing the main meaning and the other component is characterizing it more specifically, e.g. the purpose of use.

Examples:
A 'bus stop' is a station that is used to let passengers get in and get off, it is not a kind of bus,
a 'school bus' is a bus that is used to transport pupils, it is not a kind of school.

Accordingly the term 'dark flat' is totally wrong, because there is no flat frame that is used for the calibration of dark frames. The right term is 'flat dark' which means a dark frame that is applied for the calibration of flat frames.

So IF a new 'type of frame' category is to be created, the term 'flat dark' (and for IMAGETYP the category 'FLATDARK') should be used, and NOT 'dark flat' (and category 'DARKFLAT').

Bernd

 

[szymon]

The term 'dark flat' is completely wrong. It is generally accepted that the last word in a compound noun is the component characterizing the main meaning and the other component is characterizing it more specifically, e.g. the purpose of use.

Examples:
A 'bus stop' is a station that is used to let passengers get in and get off, it is not a kind of bus,
a 'school bus' is a bus that is used to transport pupils, it is not a kind of school.

Accordingly the term 'dark flat' is totally wrong, because there is no flat frame that is used for the calibration of dark frames. The right term is 'flat dark' which means a dark frame that is applied for the calibration of flat frames.

So IF a new 'type of frame' category is to be created, the term 'flat dark' (and for IMAGETYP the category 'FLATDARK') should be used, and NOT 'dark flat' (and category 'DARKFLAT').

Bernd

Can we please keep religious debates off the board?

More seriously, while personally I prefer DARKFLAT (it's a DARK do be used for FLAT frames, and not a FLAT to be used for DARK frames -- having the word DARK first stresses the fact that this is a dark frame, not a flat frame), but there are many people on both sides of the debate (I am sure that we are wrong and you are right, if that kills the debate).

I think one thing we can all agree on is that these frames should not be recognised and sorted into the FLATs category, as they are a kind of DARK and not a kind of FLAT, and that it would be nice if there were a new category created to handle them.

If the developers agree to that much, perhaps then they could open a poll to settle the religious discussion ;-)

-simon

 

[szymon]

(As an impertinent and completely irrelevant inflammatory aside, I blame the flat-darkers for the fact that these get sorted into the FLATs category by default -- if it wasn't for their flat-dark cult, they would be sorted into the DARKs category by default which would be much more appropriate!)

 

[Juan Conejero]

The term 'dark flat' is completely wrong.

... unless we are speaking of flat frames that are encrypted and cannot be used by conventional image calibration procedures - sorry, I couldn't help myself ?

Seriously now, the problem here is not the XISF specification. A format specification exists (or should exist) to be useful besides beautiful, so it should be able to incorporate any changes necessary to improve its functionality in a structured and efficient way. This is the case of XISF, and the main reason for the word 'extensible' to be part of its name.

The problem that I see here is the fact that, in my opinion, there are no differences at all between the proposed flat darks and any dark frames. Our WeightedBatchPreprocessing script will automatically select the best dark frame(s) for calibration of specific light or flat frames, based on exposure times acquired from image metadata. For this to work correctly, of course, the dark frames in question must be identifiable as dark frames (i.e. IMAGETYP='Dark Frame' or imageType="Dark").

So in my opinion, the proposed new frame type is redundant. I understand the possible convenience of what you are proposing, but I see no valid reason to modify the current XISF standard, or to include a new special case in the way we handle FITS files on a regular basis, especially considering that FITS is already enough loosely defined and prone to interoperability problems of any kind.

That said, I see no major issue to include the 'DARKFLAT' and 'FLATDARK' special cases (both, just to be prepared for all possible present and future variants ) in the particular case of the WeightedBatchPreprocessing script. Let's see what Roberto thinks about this.

 

[szymon]

That said, I see no major issue to include the 'DARKFLAT' and 'FLATDARK' special cases (both, just to be prepared for all possible present and future variants ) in the particular case of the WeightedBatchPreprocessing script. Let's see what Roberto thinks about this.

You're supposed to choose one religion or the other, not believe in both sides! What are you, some sort of DARK-FRAME-TIMED-TO-PRECISELY-MATCH-FLAT-FRAME-EXPOSURE agnostic freak??!

I'm enjoying the control of doing it all manually so far, but I can see how some may prefer a simple "hit this button and it gets acceptable results" approach, so glad to hear this might be made to work with WBPP and BPP in future.

 

[bulrichl]

Can we please keep religious debates off the board?

No, I think there is no similarity to a religious debate at all.

The reason why I picked the denomination up is: We get a bunch of questions regarding image calibration in this forum. I have written a guide to explain what is going on during the ImageCalibration process in PixInsight in order to clarify some aspects. Because of important differences between sensors the topic is considerably complicated. However, if we don't use the same terms, we can expect to talk at cross purposes. So the name also concerns discussions and tutorials. 'Dark flat' is counterituitive and unclear for a beginner, and it has not become a standard merely because it's currently used in one acquisition software.

I agree to Juan that it is not necessary to create a new category for IMAGETYP and neither to modify the XISF standard.

Bernd

 

[szymon]

The reason why I picked the denomination up is: We get a bunch of questions regarding image calibration in this forum. I have written a guide to explain what is going on during the ImageCalibration process in PixInsight in order to clarify some aspects. Because of important differences between sensors the topic is considerably complicated. However, if we don't use the same terms, we can expect to talk at cross purposes. So the name also concerns discussions and tutorials. 'Dark flat' is counterituitive and unclear for a beginner, and it has not become a standard merely because it's currently used in one acquisition software.

Consistency doesn't exist for this subject unfortunately. It's not just in "one piece of acquisition software" but all over the net. I agree it would be nice to have one consistent term, but even in this PixInsight product tutorial on ImageCalibration:

PixInsight — Master Calibration Frames: Acquisition and Processing

PixInsight - Advanced Image Processing. PixInsight is a modular, open-architecture, portable image processing platform available for FreeBSD, Linux, Mac OS X and Windows.

www.pixinsight.com

we see "dark-flat" used in preference to "flat-dark":

Don't waste time acquiring dark-flat frames: IC will rescale the master dark to match the dark noise in your flat frames.

(As an aside, it would be nice if this made clear that it's talking exclusively about CCD cameras where scaling the dark frames is feasible, and not CMOS cameras which exhibit amp glow and starburst patterns; but that's another denomination).

-simon

PS. Do you have a link to your guide? I'm reading all I can at the moment, and would love to read your explanations.

 

[bulrichl]

My guide is here: https://pixinsight.com/forum/index.php?threads/for-beginners-guide-to-pis-imagecalibration.11547/ .

Bernd

 

[dghent]

The problem that I see here is the fact that, in my opinion, there are no differences at all between the proposed flat darks and any dark frames. Our WeightedBatchPreprocessing script will automatically select the best dark frame(s) for calibration of specific light or flat frames, based on exposure times acquired from image metadata. For this to work correctly, of course, the dark frames in question must be identifiable as dark frames (i.e. IMAGETYP='Dark Frame' or imageType="Dark").

So in my opinion, the proposed new frame type is redundant. I understand the possible convenience of what you are proposing, but I see no valid reason to modify the current XISF standard, or to include a new special case in the way we handle FITS files on a regular basis, especially considering that FITS is already enough loosely defined and prone to interoperability problems of any kind.

Well, it seems that this thread grew some legs while I was asleep! I'm not going to engage in Lilliputian debates until it's necessary so, for now, I'll stick to talking about and the broader concept.

I understand and agree that Dark Flats are, when you strip off all the clothes, just another kind of Dark Frame. This is why I wanted to bring this concept up for discussion rather than outright demand its inclusion (which would be bad in other ways as well, of couse.) What I think makes dark flats (or flat darks, if you're of that persuasion) stand out from the typical dark frame are a few key factors, and these factors weigh in on general file organization methodologies outside of PI.

Importantly, Dark Flats, like Flats, are generally ephemeral in nature. With regular Dark Frames, one can build (semi)permanent libraries of Master Dark Frames and use them over and over and over again for quite a long time. On the contrary, Dark Flats, with their corresponding Flats, are very specific to a session due to the factors that govern the creation of the Flats themselves. We've found that because of this ephemeral nature and their tight and intrinsic association with Flat Frames, our users strongly wish to identify Dark Flats in a unique way, separately from that of a classic (and more permanent) Dark Frame. Otherwise, there would be no outward distinction between a Dark Flat frame and a Dark Frame unless one attempted to separate the two groups of files based on attributes such as exposure time, which can quickly get arbitrary and specific to each user's own habits.

Regardless of the technical aspects that might define differences between file types, it can be argued that Dark Flats (or Flat Darks) have already been well-established in the minds of CMOS users and in published methodologies as unique file types that replace the function of Bias Frames. These are to be created at a particular stage in the process of data acquisition and for a purpose and role that is specific. So, perhaps the distinction can be "meta" as well as technical.

Lastly, there is the minor question concerning how WBPP/BPP handles "unknown" image types - that is, image types that are not Light, Dark, Flat, or Bias. It seems that the default mode is to shuffle these into the Flats tab. I have not hunted down the WBPP source, but perhaps it is doing a substring match and the test for Flats comes before Darks, and so an image with an IMAGETYP/imageType of "DARKFLAT" gets moved into the Flat category, instead of Darks?

 

[dld]

Importantly, Dark Flats, like Flats, are generally ephemeral in nature. With regular Dark Frames, one can build (semi)permanent libraries of Master Dark Frames and use them over and over and over again for quite a long time. On the contrary, Dark Flats, with their corresponding Flats, are very specific to a session due to the factors that govern the creation of the Flats themselves. We've found that because of this ephemeral nature and their tight and intrinsic association with Flat Frames, our users strongly wish to identify Dark Flats in a unique way, separately from that of a classic (and more permanent) Dark Frame.

Why are Flat Darks ephemeral? If one has temperature control, why not using the same temperature for flats and build a library of Flat Darks? Dust over optics/sensor is ephemeral while the noise characteristics of the sensor is not that ephemeral, if not at all. (The time scale of changes of the noise characteristics of a sensor is probably much larger than that of the dust movement on the optical surfaces).

IMHO Flat Darks are not a special kind of darks in the same manner as the 15 sec darks accompanying a set of 15sec light exposures for achieving an HDR image of Orion Nebula are... Darks and not... "Highlight Darks".

 

[dghent]

Why are Flat Darks ephemeral? If one has temperature control, why not using the same temperature for flats and build a library of Flat Darks? Dust over optics/sensor is ephemeral while the noise characteristics of the sensor is not that ephemeral, if not at all. (The time scale of changes of the noise characteristics of a sensor is probably much larger than that of the dust movement on the optical surfaces).

It's simple - variances in flat light sources that cause inconsistencies of Flat Frames from one session to another will cause Flat Frame exposure times to vary, and thus cause Dark Flat frames to vary with them. Master Flat Darks might be possible in theory, but they aren't always practical. Think sky flats, or just any light source that is not reliably consistent from session to session. Not everyone operates with calibrated flat panels inside observatories, and it's also common that flat panels and those artists' tracing pads that people tend to use will also have varied light output at the same setting due to variances in ambient temperature or input voltage.

It's like how it's not practical to build a master dark library with a DSLR or other camera with unregulated sensor temperatures only, in the case of Flat Frames, it's "unregulated" sources of light which causes variances, sometimes substantial, in exposure times which has knock-on effects on the corresponding Dark Flat frames. This makes having libraries of Dark Flats impractical because you cannot always predict 1 of the 3 components (exposure time, sensor temperature, gain) that are required to make one.

 

[pfile]

tracing pads that people tend to use will also have varied light output at the same setting due to variances in ambient temperature or input voltage.

that could be true but whatever the variance is it's probably not going to cause the user to change flat exposure durations. therefore the darks are still going to match even if the ADUs in the flat are off by 100 or 200 or whatever.

i dont know about NINA - does it do auto sky flats like ACP or Voyager? if so, realistically what range of autoflat exposures are you really going to see? 1-30s? you should be able to handle this with just a few darks of differing duration. after all most of the signal in a dark that short is bias signal; with a cooled sensor there's not a lot of dark current building up over that duration.

rob

 

[dghent]

that could be true but whatever the variance is it's probably not going to cause the user to change flat exposure durations. therefore the darks are still going to match even if the ADUs in the flat are off by 100 or 200 or whatever.

i dont know about NINA - does it do auto sky flats like ACP or Voyager? if so, realistically what range of autoflat exposures are you really going to see? 1-30s? you should be able to handle this with just a few darks of differing duration. after all most of the signal in a dark that short is bias signal; with a cooled sensor there's not a lot of dark current building up over that duration.

I think the danger here is that we would be making a fairly large assumption that concerns a wide range of hardware and user situations. We're dealing with users who have a wide range of equipment, from uncooled DSLRs and fan-"cooled" cameras, to cooled cameras with 50C deltaT's. Flat panel quality also varies from calibrated output to "varies with season Amazon Prime special." So while it's true that small departures in matching ADU could, objectively, be tolerated, these situations and outcomes will never be guaranteed. And if a range of ADUs could be tolerated, how much is too much? A lot of users aren't prepared to get down in the weeds enough to be able answer that. In the world of mobile and amateur imaging, consistency is never a given, especially when there are factors that are clearly outside the user's control (such as, how bright the sky happens to be at that moment.)

From our perspective, It's far simpler from an automation standpoint to say that the exposure time of the dark flats equals the exposure time of the flats, as one would with regular light and dark frames. It's far less ambiguous for that purpose and for the user that way. They take seconds to make at any rate, so their only real cost is the disk space they consume. Accordingly, Dark Flats then live and die with the Flats they are associated with. It's simple and straight-forward, and there's no guesswork required. We would just like to see the categorization glue exist to be able to programmatically associate the two in the post-processing stages.

I'd like to redirect attention, at least for the moment, back to the video from Adam Block that I linked in my original post. Specifically, starting at around the 6:45 mark, and again at 9:23, where this problem that we're trying to solve is quite plainly illustrated. I've seen this same problem repeat itself in my users' questions, and I think this is a situation that can be addressed in a more formal way through being able to grok a new image type, as well as codifying in an addition to the XISF specification.

 

[bulrichl]

I want to explain my point of view in some more detail.

Flat frames have to be calibrated in order to
1. subtract the constant bias offset (if you don't do that, the flat field correction will not work properly and will result in undercorrection) and
2. remove "fixed pattern noise" inherent in bias frames.

The procedure for the calibration of flat frames that was recommended initially in Vicent's tutorial ( https://www.pixinsight.com/tutorials/master-frames/ , chapter 'Master Flat Generation (I)') was: use the normal MasterDark and MasterFlat and apply dark frame optimization (dark scaling). It turned out that this approach doesn't work well for most cameras equipped with a CMOS sensor. For this type of sensors, it is better to use either bias frames or 'flat darks' that are captured under exclusion of light, matching the flat frames in all other aspects (gain + offset / ISO setting, exposure time, temperature).

However, there are some cameras that produce weird bias frames (with deviating median intensity) when the lowest possible exposure time is used. Moreover, some cameras have different readout modes, one video and one still image mode, and the camera switches the readout mode depending on exposure time. These are the reasons that I recommend to generally use 'flat darks' in the definition described in the preceding paragraph.

In most cases flat darks will be indistinguishable from bias frames, because they contain no or almost no dark current, e.g:
- because the exposure time is very short (use of a LED flat panel or electroluminescent foil) or
- because a cooled camera is used (lowering the sensor temperature drastically decreases the dark current).

In rare other cases (due to e.g. very long exposure time or usage of a not cooled camera) flat darks may contain a small amount of dark current. Anyway, the transition from bias to dark frames is a smooth one. Regarding the "ephemeral" character of flat darks: they are just as little ephemeral like bias or dark frames. That means: they probably will not work forever, but you definitely need not take them in each capturing session. Neither I have observed bad reproducibility of a flat panel to be an issue. Even after months, the same exposure time could be used to produce well-exposed flat frames.

I don't consider it necessary to introduce a new 'type of frame' category. From a practical point of view, I think it is best to categorize flat darks as dark frames in both data acquisition and preprocessing (although in most cases they are in fact bias frames). This approach that already is used in the Weighted Batch Preprocessing script is working well.


Bernd

 

[dghent]

In rare other cases (due to e.g. very long exposure time or usage of a not cooled camera)

What I'm detecting throughout this thread is a bit of a CCD-centric mindset to things combined with a bit of Parkinson's Law of Triviality. Uncooled camera usage in astrophotography is not at all "rare" and is exceedingly common. I'm actually a little taken aback that you'd assert that. You only need to swing through the DSLR forum on CN or tap any planetary imager on the shoulder to know that the situation is quite the opposite of "rare." You might not be exposed on a regular basis to users who do not have "observatory-class" light-spewing apparatus, but we are. Sky flats with t-shirts, old tablets with cracked screens and layers of paper to diffuse light, and all.

Also the opposite of "ephemeral" isn't necessarily "forever." When I rotate my camera to 84.6 degrees because that's how I wish to frame something, and the resulting sky flats the next morning have 3.201 second shutter speeds because that's what it took to hit a 50% histogram on a cloudy morning, the resulting flats and their attendant dark flats are pretty much going to be useful for those light frames and that's it. However I'll be pretty sure to use the Master Dark library I built 8 months prior and used with each of the 25 images I collected data for and finished between then and this latest one. Will I remake that Master Dark library in a year or two? Yeah probably. But that's hardly "ephemeral" compared to the lack of longevity that flats have.

But I really don't want to get sucked into these kinds semantic debates that you seem so raring to engage in. I've laid out an argument explaining the use cases and how a new image classification can help improve the user experience when it comes to organizing data and using it in existing tools. I have not heard how any new image classification would be detrimental, other than "I don't like it."

 

[pfile]

Flat panel quality also varies from calibrated output to "varies with season Amazon Prime special." So while it's true that small departures in matching ADU could, objectively, be tolerated, these situations and outcomes will never be guaranteed. And if a range of ADUs could be tolerated, how much is too much? A lot of users aren't prepared to get down in the weeds enough to be able answer that. In the world of mobile and amateur imaging, consistency is never a given, especially when there are factors that are clearly outside the user's control (such as, how bright the sky happens to be at that moment.)

i don't really disagree that WBPP should be enhanced to do exact match of darks to flats (or for that matter, lights), i just don't agree that panel irregularities are an ongoing source of flat duration variability. the same users that don't know how much ADU tolerance is OK are the same users that aren't going to be adjusting their panel brightness or flat duration to fix the problem. and anyway as long as you reach the "million electron" point in your master flat and your flat subs are still in the linear range of the sensor, the exact flat ADU is not terribly important.

rob

 

[Juan Conejero]

This is now implemented in WeightedBatchPreprocessing version 1.4.5, which I have released as an update today.

WBPP supports now the following values of the IMAGETYP FITS keyword (case insensitive):

'dark frame'
'dark'
'master dark'
'flatdark'
'flat dark'
'darkflat'
'dark flat'

All of them are used to identify dark frames.

 

[szymon]

This is now implemented in WeightedBatchPreprocessing version 1.4.5, which I have released as an update today.

WBPP supports now the following values of the IMAGETYP FITS keyword (case insensitive):

'dark frame'
'dark'
'master dark'
'flatdark'
'flat dark'
'darkflat'
'dark flat'

All of them are used to identify dark frames.

You rock