PI1.8-1054 - Auto STD on rejection map do not help

[bitli]

This problem may just be a side effect of the new behavior rather than a bug.
The rejection maps produced by an integration are shown all black (as usual). So normally do an STF with automatic STF (the radioactive button). On PI 1.8.RC7 and previously, the rejected points where immediately visible. With PI 1.8 1054 (Windows), the black point is placed at the same location as the mid point and nothing is visible.
The work around is to manually reset the black point to the left (which somewhat defeat the purpose of an automatic STF).
I understand that rejection maps are not typical images, but it would be convenient to be able to 'auto-STF' them in one click, as in previous versions of PI.
-- bitli

[Juan Conejero]

Hi Bitli,

Yes, this is a consequence of the new STF AutoStretch implementation. I understand the convenience of a single-click solution to make rejection maps well visible. Let me think on a solution to this problem (and other similar). I have some ideas...

[mschuster]

Here is an idea to consider:

Scale is often zero on my high rejection maps (MAD == 0, Sn == 0) with zero and one pixels ignored. The STF in 1054 sets MTF shadows == median and midtones == 0. This gives a very contrasty result and also does not satisfy the STF target background specification.

With no information on scale, the two parameter system is underdetermined, and so a choice must be made. How to choose?

How about the least contrasty solution that satisfies the target background specification?

Least contrasty implies shadows == 0. Target background satisfaction implies solving the system

targetBackground = (midtones - 1) * median / ((2 * midtones - 1) * median - midtones)

for midtones. This follows from the HistogramTranformation process midtones transfer function (MTF) definition. An algebraic solution is easy.

This gives a much better presentation on my high rejections maps.

Pixels whose stacks in the integration had only one rejection are dimmest. Pixels with more rejections are brighter. This is a better default STF IMO.

Here is an example.

https://dl.dropboxusercontent.com/u/109232477/PixInsight/DegenerateSTF.zip

Input median == 0.0357143. Requirement target background == 0.25. Solution shadows == 0, midtones == 0.1.  Output median == 0.25.

Thanks,
Mike

Comment: I have a custom integration process that generates a pair of rejection maps for each subframe. All pixels in these maps are either zero or one. The Statistics process of course reports Count == 0 and the same for the other values. I think in this case STF should do nothing and just use its Reset values.

[Juan Conejero]

Hi Mike,

Nice reasoning. Actually, the only problem is with the shadows clipping point. As you have pointed out, it should always be zero when the image (excluding zero and saturated pixels) has zero or insignificant dispersion. Then the rest of the process works unchanged thanks to the invertible property of MTF (see my comment of August 12th in this post).

I have this change already implemented in the STF tool now, and it works remarkably well. However, for rejection maps and other similar control images, the default 0.25 target background is in general too low IMO. For rejection maps, 0.5 or higher works much better. See for example how the STF tool works now with the new AutoStretch and a target background of 0.75:

Click for full size image

I am considering adding a second radioactive button for an aggressive AutoStretch function. The one that we have now with a default target background of 0.25, and the new one in the range 0.5 - 0.75. What do you think? Should I use a "pixel on fire" icon for this one?

[Juan Conejero]

Comment: I have a custom integration process that generates a pair of rejection maps for each subframe. All pixels in these maps are either zero or one. The Statistics process of course reports Count == 0 and the same for the other values. I think in this case STF should do nothing and just use its Reset values.

I plan on implementing this separate rejection maps functionality as an option in the next version of ImageIntegration. However, the maps will be saved to disk, instead of being created as multiple images. This is also convenient for blinking purposes.

For binary images the STF makes no sense at all, so the user shouldn't use it. Detecting a binary image automatically in STF is doable, but the extra task can degrade its performance, so I'm not sure about this.

[mschuster]

Thanks Juan.

I am considering adding a second radioactive button for an aggressive AutoStretch function. The one that we have now with a default target background of 0.25, and the new one in the range 0.5 - 0.75.

On my high maps 0.5 is better than 0.75. IMO lower values help make the differences easier to see when zoomed in, at least on my maps. On my low maps, 0.5 is a little dark. I work with high maps more, so I think closer to 0.5 is better.

Do linear fit slope maps look OK?

The invertible property of MTF

I noticed this also and still find it surprising. A sort of self-solving function.

Regarding performance, I have noticed that applying an STF gives a median close to target background but not exactly, with maybe a 1% error. I assume you are not iterating to find a solution. If you know the input's median and MAD exactly, a result with an exactly correct median is possible of course. Maybe you estimate the input's median and MAD, and this accounts for the small error? Or maybe a performance issue? But I think exact can be quick too.

The maps will be saved to disk.

Good.

Mike

[pfile]

i'm confused, i'm using 1054 and the STF on rejection maps is working okay… as long as i zoom into the image, i can see pixels.

is it the inability to see any pixels when the image is zoomed out what we are talking about here?

rob

[mschuster]

Hi Rob,

On my high maps with the default 1054 STF, I can see pixels but the per pixel rejection percentage gray level encoding is not evident. If fact, you might even forget that this data is there. Everything is full black and white, no shades of gray.

On my low map, the borders are visible, and if I look very closely the interior pixels are barely visible, sometimes not. I think the 1054 STF made visibility worse on low maps. bitli reported this in the OP.

Of course you can tweak the STF controls and make the map data more visible. But a better default STF for this data would be nice to have.

Mike

[pfile]

fair enough, maybe i'm just seeing part of the data. to be honest despite years and years of using PI i did not realize that the pixel intensity in the STF corresponded with the # of images that a pixel was rejected from - thought it was simply binary.

rob

[bitli]

I did not realize that the result was proportional to the number of rejected pixels before this problem either.
Is there case were it may be useful? Could you have many pixels rejected at the same place and would this signal some problem which would not otherwise be obvious? Otherwise the result could be simply binarized.

-- bitli

[mschuster]

Hi bitli,

Integration rejection parameter tuning can be done by looking at the noise evaluation and SNR statistics. I think this is the recommended approach. But these statistics tend to be measurements of smoother and/or darker areas of the integration, i.e. not near stars and other structure edges. (The noise evaluation process actually uses an internal map to partition pixels into two sets: those measured for noise and those ignored.)

Also, noise evaluation tends to be confused by unresolved stars and galaxies, and very dim nebula in the background. This is especially a problem for me on my undersampled H-alpha frames.

IMO, noise and SNR statistics are really helpful for tuning but not exclusively. The rejection maps and their gray level encoding on the other hand can help too. For example, the maps can help identify subframes with poor FWHM and guiding problems. This is one of the reasons I like to see individual rejection maps for each subframe, to help pinpoint which ones are generating the rejections.

Regards,
Mike

[bitli]

Thanks for the info, Mike,

I often do an integration with various parameters (using VaryParams, writing the file to disk), so I can compare them with Blink.
To compare the rejection map I must apply the STF to all images using the appropriate Blink button.

IMHO, if a new button is used to apply STF to rejection map, Blink should also provide that button.  Alternatively the STF method should be able to find the appropriate values as discussed in this thread.
 -- bitli