THELI vs PI

[mschuster]

ImageIntegration does full-frame noise weighting, not pixel-based noise weighting (not counting outlier rejection), hence this thread.

I am thinking of testing a combination of both full-frame and pixel-based weighting. Full-frame accounts for things not seen by a flat (e.g observing conditions, sky transparency and brightness) and the flat accounts for things not seen by full-frame statistics (e.g. pixel gain variations). Your thread probably says as much, and I just am having trouble understanding it.

Thanks,
Mike

[Ignacio]

I am thinking of testing a combination of both full-frame and pixel-based weighting. Full-frame accounts for things not seen by a flat (e.g observing conditions, sky transparency and brightness) and the flat accounts for things not seen by full-frame statistics (e.g. pixel gain variations).

Conceptually, I totally agree, and it sounds great, but I wouldn't know how to do it in a statistically consistent way. I am not saying there isn't a way (pretty sure there is), but it is not obvious to me.

Ignacio

[mschuster]

Conceptually, I totally agree, and it sounds great, but I wouldn't know how to do it in a statistically consistent way. I am not saying there isn't a way (pretty sure there is), but it is not obvious to me.

I don't know either. What I hope to try first: Pixel values get scaled twice, once by flat fielding (F) and once by scale normalization (S). Noise (N) scales likewise. Integration weight is inverse squared scaled noise. So per pixel weight is (S / (F * N))^2. Also, IMO F need to be based on per frame registered flats.

Thanks,
Mike

[Ignacio]

I don't know either. What I hope to try first: Pixel values get scaled twice, once by flat fielding (F) and once by scale normalization (S). Noise (N) scales likewise. Integration weight is inverse squared scaled noise. So per pixel weight is (S / (F * N))^2. Also, IMO F need to be based on per frame registered flats.

I am not sure I follow your notation, but it seems that you are taking care of pixel scaling only, and not full-frame noise. It is not clear to me how to estimate frame noise in this approach.

Anyway, do you know how to implement II with pixel weighting in PI? I couldn't figure out a way.

Ignacio

[mschuster]

I am not sure I follow your notation, but it seems that you are taking care of pixel scaling only, and not full-frame noise. It is not clear to me how to estimate frame noise in this approach.

Here is my thinking in more detail:

S and N are exactly the full-frame values used by ImageIntegration currently. In other words, ImageIntegration currently weights all pixels in a frame by (S / N)^2. S is the full-frame measure of scale or dispersion. N is the full-frame measure of noise. Actually II normalized these (S / N)^2 values by reference frame measurements, but this is exactly the same. So this is the full-frame part. In other words, pixels are multiplied by (1 / S) to normalize their dispersions, so noise becomes N / S, and when inverted and squared you get (S / N)^ 2.

The per pixel part is due to F = mean(flat) / flat. This is the scale factor applied to pixels when flat fielding. Because it is multiplicative, per pixel noise becomes F * N after flat fielding.

Combining full-frame and per pixel parts the net scaled noise is F * N / S. And hence the integration weight is (S / (F * N)))^2. Basically ImageIntegration currently uses this formula with F set equal to 1. So to include the per pixel scaling replace the 1 by the flat field information.

Finally, because pixels get registered and interpolated by SA, the F numbers must be processed likewise with the same matrix and the same interpolation scheme, at least I believe so.

My gut feeling on this is that if this works at all, the improvement will be minor, at least on my subs.

Regards,
Mike

[Ignacio]

[
Here is my thinking in more detail:

S and N are exactly the full-frame values used by ImageIntegration currently. In other words, ImageIntegration currently weights all pixels in a frame by (S / N)^2. S is the full-frame measure of scale or dispersion. N is the full-frame measure of noise. Actually II normalized these (S / N)^2 values by reference frame measurements, but this is exactly the same. So this is the full-frame part. In other words, pixels are multiplied by (1 / S) to normalize their dispersions, so noise becomes N / S, and when inverted and squared you get (S / N)^ 2.

The per pixel part is due to F = mean(flat) / flat. This is the scale factor applied to pixels when flat fielding. Because it is multiplicative, per pixel noise becomes F * N after flat fielding.

Combining full-frame and per pixel parts the net scaled noise is F * N / S. And hence the integration weight is (S / (F * N)))^2. Basically ImageIntegration currently uses this formula with F set equal to 1. So to include the per pixel scaling replace the 1 by the flat field information.

Finally, because pixels get registered and interpolated by SA, the F numbers must be processed likewise with the same matrix and the same interpolation scheme, at least I believe so.

My gut feeling on this is that if this works at all, the improvement will be minor, at least on my subs.

Regards,
Mike

Got it, thanks. So, in your notation, S and N are scalars for each sub, and F is a matrix (loosely speaking). It seems that you have to normalize each element of F across the stack too, for it to make sense.

I also think that this could have a minor effect, but that will depend on the quality/uniformity/cleanness of the sensor. And we fight so hard to get up to the last bit of performance, that I feel this is worth exploring.

So, have you thought of a way of implementing the pixel-level weighting in PI's II?

Thanks again,
Ignacio