WBPP 2.2.0 grouping keyword don't match bias

dvolny1

Member
Hi, I try to use the grouping keywords to automatically select correct calibration images (bias, lights, flats). I am using keyword FSEQ to group together correct flats and lights (FSEQ=flat sequence=group of light frames that belong to specific master flat) and keyword CCD-TEMP2 to chose correct darks, lights and bias frames (CCD-TEMP2=basically the same as CCD-TEMP but CCD-TEMP changes and my bias frames differ with temperature and years). I plan to use YEAR keyword to chose a correct bias, darks to lights as bias and darks are changing during years. I tried this (see image), but I got error " Master Dark contains the bias and will be optimized but no Master Bias matches...." even if there is a bias with corresponding keywords match.
 

Attachments

  • WBPS_no_match.jpg
    WBPS_no_match.jpg
    468.7 KB · Views: 80
Hi @dvolny1,

well this is a tricky case, the keywords are NOT exactly matching, light and flats indicates "-25.00" while the bias indicates "-25.0".
WBPP is agnostic with respect the keywords values, this means that all these values are simply "strings", it does not try to parse them as numbers, it does not try to interpret any data as a real "date".

Being strings, "-25.00" and "-25.0" are two different values so, as expected, WBPP can't match the bias with any flat or light.

You should fix your bias keyword value to be "-25.00" or your light/flats keyword value to be "-25.0".
To match, each single character shown as value must be the same!

Robyx
 
I don't understand this stance. There is nothing to "fix" on the user's side of things - there shouldn't even be a question regarding any given keyword and its value being treated in accordance to its data type. -25.0 and -25.00 are both floats and represent the same number. Feed them into Double.parseDouble() and be done.
 
No. In this context they are both strings (that can, as it happens, be interpreted as numbers - but the strings "frog" and "X101" would be equally valid).

And that's my point. It's a mistake to blindly interpret all keyword values as strings. That is the bug. The bug is not a given keyword's data type being what it is.
 
Interestingly I ran into the same issue recently with SET-TEMP in FITS being one decimal and as written by an application in the file name being two. I have no ability to change either one, so my choice is to make darn sure I only use one or the other, which ... well, stuff happens.

There's two reasonable solutions I see: one is that when declaring grouping tokens, to specify "number" or "text". The other is to check against standard FITS headers that do have a standard. Well, and the third option is to ensure that the humans are 100% consistent in use all the time, but that sounds hard.
 
And that's my point. It's a mistake to blindly interpret all keyword values as strings. That is the bug. The bug is not a given keyword's data type being what it is.
Thanks for sharing your concerns.

This topic less trivial than it may seem. I perfectly understand that you consider this as a bug since you assume that the string "-12.5" must be interpreted as a number, this looks quite obvious and simply the way to go.

WBPP is most probably the most used script of PixInsight, used by tens of thousands of astrophotographers around the world that adopt a wide variety of conventions and custom ways to name files and embed/encode data from different acquisition and processing softwares. For this reason, keywords feature grew slowly and carefully, trying to accomodate as much users as possible while trying not to take choices that reveal to be awkward or even wrong to many users. The reason why I was always skeptic in terms of internally convert keyword values into defined data types is that assuming the type may work for some but not for others and this may not always be the correct way to proceed in general, it's very easy to end up satisfying a set of users but displease others.

Let me share my concerns regarding the interpretation of numerical values. First of all, you cannot simply apply the parseFloat() JS function to keyword values since it succesfully converts non-numerical strings like "12:00" into the numbers, making "12:00" and "12:10" both 12 resulting equal each other while they are not.
But let's also assume that we're smart and we use a regexp to test the string format before parsing the values, like the following:

"if this string matches a valid numerical format (as the FITS standard defines for float numbers, for example) then parse it as a float, use the string value otherwise"

Now, if "12.1" should be taken as number, the consequence is that the value "12.10" is the same. But here we're just reasonably guessing, I'm quite sure that someone will come back saying that WBPP is buggy because it matches these values while, in their case, they have to be considered different because they didn't meant a float number. It could be month/day code for the session, as an example, but it could be whatever information you decide to encode into a xxxx.xxxx format., being x a any digit.

Now, we need to assume some conventions on top of which WBPP behaviors are developed, the "agnostic" way is as reasonable as the "attempt numeric conversion" way, as long as the user is responsible of storing the metadata accordingly.

By the way, don't get me wrong, I welcome your concern and it's not something new on the horizon, indeed a pending feature that I didn't implement yet is to specify the type of a keyword amongs a very short list "auto" (the default), "string" and "numeric". Of course, you need to be aware that "auto" interpret values as numeric first and string otherwise, so "12.1" and "12.10" will be both interpreted as 12.1.

I guess times are mature enough to plan its implementation :)
 
Last edited:
@robyx, one thing you might consider as you move forward into the world of numbers (;)) is providing a tolerance. I'd love to be able to use actual sensor temperature as a keyword (vs set-temp) and then have it match to the "best" dark frame temperature. That way if ambient went up, and my cooler can't keep up, WBPP will just pick the best one (within a specified tolerance).

In other word, if you allow keyword type choice of numeric (or auto=>numeric) allow a tolerance to be specified as well. I'm not sure how many keywords might benefit from that, but temperature is one. Maybe gain.
 
@robyx, one thing you might consider as you move forward into the world of numbers (;)) is providing a tolerance. I'd love to be able to use actual sensor temperature as a keyword (vs set-temp) and then have it match to the "best" dark frame temperature. That way if ambient went up, and my cooler can't keep up, WBPP will just pick the best one (within a specified tolerance).

In other word, if you allow keyword type choice of numeric (or auto=>numeric) allow a tolerance to be specified as well. I'm not sure how many keywords might benefit from that, but temperature is one. Maybe gain.
yes this is one of the things that stopped me from implementing it haha :)
More than the tolerance I would ask for a precision, like parse integer numbers or float with a fixed number fo digits, (1/2/3/4 etc)... but then the UI needs to accomodate this information and the keywords control was not designed for such amount of information at the beginning, so it could require a redesign.

Let's see...
 
The standard answer for when function exceeds real estate: popups. o_O

At least until the ReadUsersMind() method is fully implemented.
 
Let me preface by saying that I'm completely unaware of what functions and facilities are available to scripts such as WBPP regarding general metadata munching. I'm making the assumption that there's not much, based on what I'm inferring here.

The FITS value fields are pretty well codified in their form and format. This makes it somewhat straight-forward to take an arbitrary keyword and determine its value's data type for proper parsing. This can be done through various means. I personally have a background steeped in unix OSes, perl, and python, so I immediately reach for regexes; but there's more than one way to skin this cat.

Consider this off-the-cuff pseudocode regarding parsing keyword values:

Code:
if string begins and ends with '  // It's a text string
else
if string is "T" or "F" // It's a boolean
else
if string is all digits // It's an integer
else
if string is accepted by parseDouble() // It's a float - it should grok all accepted float/double/decimal formats
else
throw an error or just accept the indeterminable crud as a string

If you determine things to be a string type, you can delve further into that for more benefit, such as seeing if the string is accepted by DateTime as a valid ISO 8601 date string, such as those found in DATE-OBS and friends. This would, for example, allow people to group by any granularity of date/time as you then have a generic DateTime object to play with. If it throws an exception then it's clearly not a string that DateTime can grok, so at that point it would just be a generic string of random content.

In the end, the keyword values can be used and are usable in accordance with their data type, with some possibilities of enhanced use of them (datetime values, etc.) as a direct side effect. You also avoid weird things such as the collation rules of the user's locale imparting an inconsistent effect on ordering and sorting. I'm not sure how locale collation rules are handled now, but it could lead to differing experiences and expectations across all keyword values in its current "everything is a string" form.

Speaking of metadata formats, I think that XISF's FITSKeyword image property element could be further extended to include an attribute that defines the data type of its value. This would eliminate the need to divine what a given value really is and put the responsibility on the software that is writing out that information and, presumably, already knows what it is. That's a discussion for another thread, though.
 
You are going to break something very nice.
Filenames (and paths) are 100% strings.
Matching criteria as invoked in filenames take precedence over FITs header keywords. (This is a big benefit.)
This will break if value type requires interpretation.
In terms of usage, a user will be very confused and find that some value in a file name is regarded differently in the FITs header.
(For those paying attention, this already exists in a special case..and guess what, not everyone "gets it.")

-adam
 
I was speaking in the context of actual FITS headers and XISF FITSKeyword elements themselves. Encoding metadata in filenames is, to be blunt, a shit show, and a user deserves what they get (a string!) when it comes to encoding exposure data into entirely arbitrary entities such as paths and filenames. The fact that this is a common and tacitly approved practice is a real tragedy.

But even in that terrible world and practice, basic determination numeric values can be attempted. If it produces a consumable result then great. If not, then enjoy your string.
 
@robyx thanks for explanation. Auto type conversion (guess) is not good idea, but it would be great to have the possibility to define for each keyword its type manually (default string, number, date, ...). Then for number, date or where it makes sense define precision (e.g. +-1 degree Celsius, +- 7 days or "nearest" to compute the nearest match). But this is rather a wish than request ;)
 
FITs headers do not come out smelling fresh. FITs writers do not always subscribe to conventions..and has been shown here over and over and over, the information in the headers can be *wrong*. You might place the blame on the FITs writers- but nothing will change and it is a losing proposition. No one is forcing anyone to use this dimension of keyword matching. If you can live by the FITs headers then fine. But don't think life there is any easier. Units, sign conventions, decimal places, strings (yeah, these are in the headers too)- the need for the outer precedence comes from the inner turmoil. When the user simply wants to override metadata in the header- or wants to characterize/group things in ways that are different/unavailable by means of the data found within the header- filenames can be very useful. It is very common, especially with automation, to create degenerate data (in attributes and metadata). This limits the numbers of degrees of freedom one has to organize the data.

So, if FITs headers have data of a type- and you want to override that information with a string- this will break things or add complexity to accommodate it. That is all I was trying to bring attention to- there are strings attached.
 
By the way (for Roberto @robyx )- we have the ability to manually select darks. I do not think we have the same for biases.
I wonder if the manual selection could simply list both darks and biases? I don't see the harm in it personally and in this particular case- it would also solve the problem without any coding. Instead of fight over values- we can all see the issue- and manually forcing the issue seems like a good compromise so we can all just get on with life. lol

adam
 
Last edited:
I tried today to match the bias, but it still won't match. Maybe the best way would be manual selection as @ngc1535 suggested.
 

Attachments

  • WBPS_no_match2.jpg
    WBPS_no_match2.jpg
    505.2 KB · Views: 57
Sorry, there might be some confusion here. If you are trying to use the bias frame to calibrate your FLATs... *turn off dark frame optimization* and your flats will be calibrated with the bias. (You will get a kind "Warning" that tells you there is no matching Dark so the bias will be used.)

I kinda wish there was a "use bias" choice or something... people seem to be confused on this point. This would turn off auto-Dark matching and just configure for bias subtraction.

You do not need to optimize the dark to calibrate your flats assuming you have a well-behaved sensor.

-adam
 
Back
Top