ABE does not seem to apply the model

[bitli]

Hello,
I use ABE to automatically remove a strong red background in a APN image (defiltered 350D, maybe due to light polution). Extremely useful and works fine except for the follwing point which I think is a bug:

I select Apply Model, Sustraction in the Target Image Correction tab, the log indicates that the operation is done, the new image is created (if so requested), but it is indentical to the current image. I have to use a separate pixelmath operation.

Here is the script of the operation

// Start time: 2009/02/08 14:47:14 UTC
// Execution time: 45.755 s
var p = new AutomaticBackgroundExtractor;
with ( p )
{
   tolerance = 1.000;
   deviation = 0.800;
   unbalance = 1.800;
   minBoxFraction = 0.050;
   maxBackground = 1.0000;
   minBackground = 0.0000;
   useLuminanceLimits = false;
   polyDegree = 4;
   boxSize = 5;
   boxSeparation = 5;
   abeFloatSample = false;
   abeBitsPerSample = 16;
   abeDownsample = 1.00;
   writeSampleBoxes = false;
   justTrySamples = false;
   applyModel = true;
   applySubtraction = true;
   correctedFloatSample = true;
   correctedBitsPerSample = 32;
   replace = true;
   verboseCoefficients = false;
   compareModel = false;
   compareFactor = 10.00;
}


The version is 1.4.5.0467 eng(x86) on Vista (It would be useful to have a text with the version and maybe the current OS, memory and so on as a copiable text in the About window, so we could report it to you exactly).

Have a nice week
bitli

[Simon Hicks]

I can report that I have had exactly the same problem. The generated background image looks correct (i.e. it seems to have all the colour and curvature that the background should have), but even though the Apply Model and Subtraction boxes were both ticked, the output image was exactly the same as the starting image. I too resorted to using PixelMath.....which is fine with me because I like to add a very small pedastal to the image before background subtraction.

I'm using latest Win32 version of PI on WinXP Pro if that helps.

Cheers
             Simon

[bitli]

Thanks for the confirmation Simon.

I have a question about your addition of pedestal (I had also the feeling that the image is a little too dark).
Assuming you are working with images in f32 format, in that case it seems to me that:

- If the option to rescale between 0 and 1 is set, then it should not really change anything (the zero level is set down back to to zero after the rescale). This assume that intermediate image allow 'negative' values, which I am not sure (maybe this is why adding the pedestal is useful after all).
- If the options to rescale is not set, then removing the background would also desaturate the stars, which I think is no good.

What did I miss ?

bitli

[Simon Hicks]

Hi Bitli,

It took me a while to work this out by trial and error....but I use something like the following in PixelMath;

(Image+0.05) - Image_Background

The 0.05 is a 5% pedastal. 5% is about the maximum I ever use. Its often in the region 1 to 3 %. You just need to try it and see which one works, but its very easy to change the value and run off another subtracted image.

I always uncheck the Rescale box.

I find that if I don't use a pedastal then when I look at the histogram of the resultant image, the peaks are clipped on the darker (left) side....i.e. I am losing useful data. So adding a small pedastal allows me to subtract the background model but still have three sharp histogram peaks (one for each colour) all tightly ontop of each other and not clipped on the lower side.

However.....maybe I am only needing to do this because I have the Rescale box unchecked. Maybe you are right about the negative numbers and rescaling.  Maybe Juan can give some advice?

Cheers
             Simon

[Juan Conejero]

Hi Bitli and Simon,

I cannot reproduce this problem. ABE works well on all of my machines.

I think the problem is that ABE is generating a black (or nearly black) model. In this case it's obvious that subtracting it won't help. Can you confirm this?

ABE has a special working mode that allows you to know where it generates background samples. Enable the "Draw sample boxes" and "Just try samples" options to select this mode. If you find that samples are being generated incorrectly (too few / too many / wrongly placed), then you need to tweak parameters.

ABE is a relatively complex beast. Fortunately, I have a backup copy of ABE's old site, which we created when Carlos Milovic wrote the first version of ABE as a command-line utility:

http://pteam.pixinsight.com/carlos/abe/

The thing is old (pretty old indeed!), but the documentation and examples are valid. I hope this helps you.

[Juan Conejero]

Hi Simon,

I always uncheck the Rescale box.

Not a good idea! During expression execution, PixelMath always works with the entire range of 32-bit IEEE floating point numbers (64-bit if the corresponding option is selected). This means that there is no problem at all with values outside the [0,1] range, including positive and negative numbers with huge absolute values.

When PixelMath terminates execution:

- If the Rescale option is enabled, it rescales the resulting image to the [0,1] range.

- If the Rescale option is disabled, it truncates the resulting image to the [0,1] range.

Disabling the Rescale option is only acceptable if you are sure that your expression is naturally bounded to within the [0,1] range. In all other cases Rescale should be enabled, except when your intention is precisely to perform a final truncation (which is rare).

To subtract DBE background models, there is no need to apply a pedestal, and of course Rescale must be enabled --negative values would kill your image otherwise!

[bitli]

Hola Juan,

Thanks for the info.  I was able to reproduce the problem quickly last time. I will try again this week-end, and if successful, I will send you the image and sequence privately for test (it was just two steps).

The background to subtract was very significant to me (at least with the STF I had), and subtracting it made a difference, so I do not see why automatically subtracting would not.  But I will check again and look at the values you mentioned. Maybe an issue in the code that decide that it is not worthwhile to do the subtract ?  The image was from an APN with a very strong red dominance (the other channels may have had no significant background), maybe this fooled PI).

Regarding the rescale, it may be nice if after each operation that may produce an unscaled image (rescale no check), there would be a message in the log file about the min/max value of the image, and possibly a warning like - Image not rescaled, pixel out of range 0-1.0. Maybe the msg is already present in fact, I did not try to generate image with outside values :-)

Thanks again and hasta la próxima.

bitli

[Juan Conejero]

Hi Bitli,

The version is 1.4.5.0467 eng(x86) on Vista (It would be useful to have a text with the version and maybe the current OS, memory and so on as a copiable text in the About window, so we could report it to you exactly).

I forgot to comment this. You can use the ver command from the processing console (just type ver and press Enter, no parameters needed). This command will show the PI and PCL versions you're running.

Maybe an issue in the code that decide that it is not worthwhile to do the subtract ?

I don't think so, but I look forward to see what comes out from your tests. If necessary, I may ask you to upload the image so I can reproduce the exact problem you're having.

Regarding the rescale, it may be nice if after each operation that may produce an unscaled image (rescale no check), there would be a message in the log file about the min/max value of the image, and possibly a warning like - Image not rescaled, pixel out of range 0-1.0.

Currently PixelMath just says whether it's rescaling or truncating, but printing the actual ranges is a good idea (and other users have also asked for it); count with this implemented in the next version


Edited by David Serrano: fixed broken quoting.

[bitli]

Hi,
I sent the link to the images and log to Juan via private message, as on my system I can reproduce the problem systematically. I hope this will help.
bitli

[Simon Hicks]

Hi Juan,

Thanks for the explanation of the RESCALE tick box...that makes great sense and I will use it properly now and throw my pedastals in the bin!

But I can confirm that I have used ABE with a bright enough background to see it easily (i.e. without any STF) and it definitely wasn't being subtracted. But when I take the background model and use PixelMath it really does get subtracted....there's a very clear difference.

I'll let you concentrate on Bitli's data first, maybe that weill get to the bottom of it.

Cheers
             Simon

[David Serrano]

Are you sure that the original image and the one generated from ABE are exactly identical? It must be nice to have 32-bit sight!! .

I can confirm that the result from ABE doesn't have anything to do with the expected image, but it isn't identical to the original either, try checking it with PixelMath (abe_corrected - original). I tried it with a diurnal image, which generated quite a bright flat (minimum 0.22, median 0.44, maximum 0.60). PixInsight Core 01.04.05.0467 (x86) on Linux (Juan, the 'ver' console command omits the operating system ).

The differences between the obtained image and the original are greater where the flat is brighter (as expected).

The images obtained by applying an ABE auto-division and auto-subtraction are different.

[Juan Conejero]

I've just downloaded Bitli's data; sorry for the delay. Tomorrow I'll check them and will see if I can find an explanation for this problem. Thank you all for your help!

[Juan Conejero]

Hi again,

Here we go. There's no bug in ABE (at least not in the background subtraction routine).

This is a screen shot taken while working with Bitli's image:

http://forum-images.pixinsight.com/legacy/abe-bug-20090220/01.jpg

Top-left (1): original image
Top-right (2): ABE background model
Bottom-left (3): ABE corrected image (by subtraction)
Bottom-right (4): Rescaled absolute difference between 1 and 3.

As you can see, the original image is quite flat. This can be seen also on the generated background model. Please note that flat here doesn't mean that the image has a neutral background. Flatness refers here to the illumination distribution's profile.

When ABE subtracts a background model, it adds the original background to the corrected image. This is ABE's subtraction correction:

v' = v - b + M

where v' is a corrected pixel, v is the corresponding original pixel, b is the corresponding background model pixel, and M is the median of the original image.

Note that this is correct in the sense that the original mean background is preserved in the expression above. This means that after background subtraction, the image doesn't have to have a neutral background necessarily (unless all components of M are identical).

From the absolute difference image (4) you can see that the model is in fact applying a slight correction on the four corners of the image. It seems to be overcorrecting near the right border; this can be improved by tunning ABE parameters.

If you subtract the background model with a PixelMath expression, as we usually do with DBE models:

ngc_457_DSS_mean - ngc_457_DSS_mean_ABE_background

then the resulting image will have a nice neutral background, provided that the background model is accurate and the rescaling PixelMath option is enabled. This is probably what you were expecting from ABE. To reproduce ABE's subtraction behavior with PixelMath, apply this expression:

ngc_457_DSS_mean - ngc_457_DSS_mean_ABE_background + Med( ngc_457_DSS_mean )

with rescaling disabled.

Note that you can neutralize the background very easily with a PixelMath expression like this one:

$T - Med( $T )

applied with rescaling enabled. You can also add a small pedestal to prevent negative numbers, which allows you to apply the subtraction with rescaling disabled:

$T - Med( $T ) + 0.025

The constant above is just an example; its validity should be verified for each particular image.

Here is a screenshot showing what you have after background neutralization of your original image:

http://forum-images.pixinsight.com/legacy/abe-bug-20090220/02.jpg

Note that if you want to stretch the image (and you want, of course ), then subtracting a good background model is mandatory. This is an easy target for ABE, but its parameters need to be tweaked.

[Simon Hicks]

Hi Juan,

Thanks for the explanation. That would certainly explain exactly what I saw....which was not what I was expecting.

I wonder if there should be two tick boxes...one to subtract (i.e. image-background) and one to add the background mean back in (Image-background+Mean)? This would enable users to get the best of both worlds, and be clearer about what is going on. Sometimes neutralising the background by a straight subtraction is a useful thing.

Just a thought. But thanks again for the explanation.

Cheers
              Simon

[bitli]

Thanks Juan, this is now clear.

I really appreciate the time you take to understand our problems and explain them, this is extremely valuable.
Now I can process better a couple of images I have taken these days.
Having the STF window extend along the bottom of the screen is also great. The small 'tricks'  I see in your tutorials or examples help me getting up to speed  with PI.

Have a nice week-end,
bitli

PS: I still hope that some day I will find a real bug, but it is harder than I thought -)