ImageIntegration Problem - Automatic 'Rescale' destroying 'pedestal'

[Niall Saunders]

This is a bug that I have been fighting since ImageIntegration was first implemented.

If, as a result of the input images having 'full' data ranges (i.e. either down to [0.0] or up to [1.0], or both) then ImageIntegration, at the end of the integration phase (inclusing any clipping algorithm that may have been requested), will AUTOMATICALLY apply a 'rescale' call.

Now, this 'rescale' call will not only multiplicatively 'scale back' the data such that the maximum is brought back down to [1.0], but it will also (automatically, and ill-advisedly) 'clip' the bottom end of the data such that the minimum ADU value is taken back down to [0.0].

Whilst this may 'seem' to be an acceptable implementation of a 'rescale' algorithm, it is TOTALLY WRONG for Calibration frames, where (to me, anyway) the notion of an 'offset pedestal' remains CRUCIAL throughout the calibration process.

I have no problem when it comes to 'scaling back' at the top end of the dynamic range. After all, the errors are probably wery small anyway - especially when you take into consideration that the 'source' data for the Integration process must have all been confined to the normal [0.0, 1.0] range in the first place. In fact, it would even be acceptable to 'truncate' the data that lies above the [0.0, 1.0] range, and the remaining data would still be viable for further calibration stages.

In my experience today, I noticed this issue EXCLUSIVELY with my 60-second Darks - which were now long enough to permit single 'hot pixels' to overload (to 65535 in the 16-bit domain). When I then ran these through ImageIntegration I ended up 'losing' the 'pedestal' (of around an ADU of 3700), leaving the minimum ADU at '0' (instead of ~3700).

In order to 'prove' my understanding, I ran the Darks through an ImageContainer/PixelMath process, applying the following formula to each image:
iif($T>(65530/65535),(65530/65535),$T)

This has the effect of 'truncating' the maximum ADU value in each Dark sub to '65530'. When the IDENTICAL ImageIntegration process is then used on this modified collection of images, I end up with a MasterDarkframe that RETAINS the 'pedestal' that I believe should still be there. I haven't tried 'raising the bar' yet, to see if the result is the same if I push the threshold up from [65530] to, for example, [65534].

I also have not yet fully tried all further combinations in the ImageCalibration process (the next logical step in the calibration sequence), but I just 'know' (gut feeling, I suppose) that if I invoke ANY kind of Bias frame along with a MasterDark that has 'lost its pedestal', then I will effectively be applying Bias correction 'twice' - because the 'missing pedestal' IS the 'Bias'.

I hope I am not making some fundamental mistake in my analysis - and would really appreciate insight from others. Failing that, it would be noce to have the ability (as is the case with PixelMath, after all) to either permit 'Rescale' or to inhibit this in favour of 'Truncate' instead.

Anybody else have any thoughts on the matter?

Cheers,

[Niall Saunders]

Well, although nobody has picked up on this thread, I have been able to confirm my 'suspicions' over the weekend.

If you use ImageIntegration to try and create a MasterDark, you WILL end up with 'corrupted data'. The MasterDark that will be created will have 'lost' the 'pedestal', and it is this pedestal that contains the 'Bias' information that is scheduled to be processed out if you subsequently use ImageCalibration and specify a MasterBias frame and, at the same time, click the 'Calibrate' tick-box in the (enabled) Master Dark section.

The problem exists because the MasterBias frame (which you may have created using ImageIntegration) does NOT contain values which will 'trigger' the ImageIntegration process into applying a 'rescale' - and therefore the MasterBias frame does NOT lose the 'pedestal'.

But, because (or if) your Dark frames have ADU values anywhere near the 65535 mark, and you run these through ImageIntegration, then there is a HIGH likelyhood that your MasterDark WILL have been rescaled by ImageIntegration, and you WILL lose the pedestal.

Now, of course, when you are trying to do great things in ImageCalibration, when the MasterBias (which includes the 'pedestal) is subtracted from the MasterDark (which no longer includes the pedestal), you end up with the pedestal being subtracted from the MasterDark 'twice' (effectively) - rendering your MasterDark less able to properly calibrate any further data (Flats or Lights).

The work-around I proposed worked perfectly - well it did provided I capped at the '65533' level, it made NO difference if I capped at the '65534' level (the 'clipping' wasn't enough I assume). Now, capping at '65533' hardly affect the Darks at all - after all '65533' represents ~ 99.997% of full-range, so you are only 'losing' about 3/1000ths of your data, and they were already 'ultra-HOT' pixels anyway, so no real loss there.

What I would therefore like to know is whether my understanding of the 'rescaling' action in ImageIntegration is actually 'correct'. And, if it isn't, then I need to know why my MasterDark gets scrambled beyond use.

And, if nothing else, can we at least have an OPTION to 'rescale' within ImageIntegration - just like we have in PixelMath (and like we used to have in PixLE, if I remember correctly).

Juan, can you (or anybody else) shed light on this?

Cheers,

[Juan Conejero]

Hi Niall,

ImageIntegration is indeed normalizing the result. Normalization is a conditional rescaling operation. The algorithm is as follows:

procedure Normalize( I )
begin
   if MinimumValueOf( I ) < 0 or MaximumValueOf( I ) > 1
      Rescale( I )
   endif
end

So the only way for this to happen is if your integrated data exceeds the [0,1] range. This should never happen if you integrate your calibration data correctly. Can you tell me what are your ImageIntegration parameters? For example, a screenshot of the ImageIntegration interface would help.

Besides that, I can add a new parameter to ImageIntegration to perform a truncation instead of a normalization at the end of the process. This is easy and I could publish an updated ImageIntegration module, but I'd like to understand why is this happening with your data first.

[Niall Saunders]

Thanks for that Juan.

Let me have a look at the ImageIntegration setup when I get back to my main PC. I'll send you a screenshot, and the associated PSM file. I will also see if the problem materialises when integrating, say, 'just three' of my raw Darks - in which case I could post you the three Darks as well. (I may even be able to reproduce the problem with 'cropped' images - so that will save 'transfer bandwidth)

Hopefully I will get this done this evening.

In the meantime, perhaps the 'Truncate/Rescale' option may be of use anyway? Does anyone else feel that this would be useful?

Cheers,

[georg.viehoever]

Hi Niall,

I found "normalization" to have irritating effects in another context, the CanonBanding script. When running it, I occasionally get pixels outside of [0,1]. These were rescaled, making the resulting frames useless for integration. Ultimately, I changed the CanonBanding script to do a truncation instead. Normalization can be very disturbing if any of your process steps rely on the absolute values of pixels.

For ImageIntegration, a "truncate" or "dont normalize" switch may be appropriate as well. But I suggest  first to find out why you get denormalized values.

Georg

[Juan Conejero]

Hi Georg,

When you write an image processing tool in PixInsight, you must keep in mind that you're working on a modular platform. The platform has to enforce a set of rules to ensure that the data are compatible with the different actors playing. In particular, when a process receives an input floating point image the platform guarantees that the input pixel values are referred to the [0,1] range.

A processing tool or script is in fact free to yield an image outside the [0,1] range. For example, try this script:

Code: [Select]

var p = new PixelMath;
with ( p )
{
   expression = "10";
   useSingleExpression = true;
   rescale = false;
   truncate = false;
}

p.executeOn( ImageWindow.activeWindow.currentView );
with a floating point image selected. With both rescale and truncate equal to false, PixelMath does not perform any normalization of its resulting image. In this way the above script will fill your current view with 10.0. The resulting screen rendition will be white in this case --it could be anything, in fact, as the screen rendition routines expect [0,1]. PixelMath allows you to do this because in this way you can use it in a script to generate intermediate working images. However, you are responsible to guarantee that your script's final image is within [0,1].

BTW, if you are wondering why the PixelMath interface does not provide access to the truncate parameter, it doesn't to prevent normal users from generating incorrect images.

Enough secrets unveiled for today 8)

[georg.viehoever]

Juan,

thanks for those details. The point I wanted to make with my comment is that you need to be really careful with the current normalization default. While it is O.K. for many processes, it is a problem for those that need absolute values or a preservation of the relative magnitude of values. ImageIntegration is one of those processes. We just need to be careful about normalization. It took me quite a while to find out that normalization was a problem when feeding CanonBanding-script results (that were normalized instead of truncated in previous versions of the script) to ImageIntegration.

Georg

[Niall Saunders]

Juan, you refer to

normal users

in your reply to Georg.

Surely that is the whole point of PixInsight - we are, none of us, 'normal' users - we are all 'special'

However, in between tasks here at the coalface, I have been thinking what might be triggering the rescale action.

My darks all start life inside Meade's Envisage software, which - for those who know me all too well - takes a nice 16-bit integer ADU value and converts it into a 32-bit Float value for saving in it FITS formatted images

No matter, I now use the pre-processor routine to read these images into PI, based on a "rescale if outwith the [0.0, 65535.0] range, and then apply a format conversion to immediately re-save the data in 16-bit UInt format.

This is where I now have my first concern. "Is the conversion proceeding normally, or should I actually have specified the trigger range to be [0.0, 65536.0]?" I will have to try that experiment and see what happens?

In any case, after the conversion to 16-bit UInt, I HAVE determined that my Dark frames DO have 'some' pixels with 'maxed out' ADU values. What I do not understand is why, in the presence of some images in a group having a given pixel with 'maxed out ADU' data (which, by the time PI works on them, can only actually have a value of 1.0), the likes of ImageIntegration eventually triggers the rescale function.

If I apply my PixelMath 'capping' function, the ImageIntegration rescale algorithm semms NOT to get triggered - and I do not lose the 'pedestal'.

Certainly, tonight, I will try a 'plain jane' ImageIntegration process (i.e. 'average' the values only), and will do this with and without 'capping' and see if it is some 'sub-process' in ImageIntegration that ends up creating an 'intermediate' (i.e. 'in-process, working') image which then triggers the rescale routine.

Sorry to be a PITA, but I am sure that if there is a residual anomally that needs to be addressed, we are as well to catch it now, whilst the whole process of Calibration and Integration in PI is still in its early days.

Cheers,

[Niall Saunders]

OK Guys, I think I am now totally confused.

So, maybe someone can have a look at my data, and see if they can duplicate the problem. Then maybe we can figure out what is happening.

Firstly, I have uploaded the first ten of my raw Darks to the following locations (sorry about the numeric URL - Juan, you couldn't 'migrate' my FTP space over to the new server could you?)

ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0001.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0002.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0003.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0004.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0005.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0006.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0007.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0008.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 0009.fts
ftp://74.50.19.20/ImageIntegration_ReScale_problem/M27 IIP Dark 7-0C 60s C 00010.fts

In fact, you might be able to reproduce the problem using only Darks 01 to 04. This is because I observe the problem when any of Darks 04, 07, 09 or 10 are included in the dataset. The following description uses 01, 02, 03, 05, 06 and 08 - and, optionally (to show the 'fail mode') 04 as well.

Firstly I use a 'default reset' instance of ImageIntegration as follows:-


In this you can see which of the ten files HAVE been included in the ImageIntegration process. Don't worry about the other ImgInt parameters - they are all effectively 'off', as seen by the following Console output screenshot:-


When this process is run, the following image is created (shown here with an STF applied):-


The Histogram of this image is:-


and the associated Statistics information is:-



Now, when Dark 04 is included in the ImageIntegration process, things seem to go wrong, as follows:-

Running this process

produces

with this image being generated

which has this Histogram

and these Statistics


Crucially, in the Histogram window it can be seen that the 'Pedestal' is missing, and this is confirmed by the 'Minimum' value for the integrated image now being '0'

I have NO IDEA why Dark Frame 04 should be causing this - there doesn't seem to be any significant difference between the the two Darks, 03 and 04, as shown:-


and this would seem to be confirmed by comparing the two Statistics windows, as is shown here:-


So, over to you guys. My brain hurts

Cheers,

[NKV]

Niall, look at Median.
Median of "Frame 04" a bit lower (3 ADU less ), so during Normalization (PixelMath:Dark04 - Med(Dark04) + Med(Dark01)) high value going out of range ( 65535+3 ).
Just select "No Normalization" in "Image Integration" section.

PS Juan, maybe better to add switcher "Truncate/Rescale" in next version?

[Niall Saunders]

Thanks for that - I really had thought that I had set everything 'off' other than 'Average' - it just goes to show how 'tunnel vision' can stop your brain seeing something. I must have been seeing the word 'Additive' and associating it with 'Average', immediately above it, and thinking "yep, that's all OK"  

Anyhow, I did a quick check, and in fact the WHOLE batch of 30 Darks now Integrates without losing the pedestal.

Tonight, I will have to think about what the implications are by NOT using 'Normalisation'

Thanks for taking the time to look at this.

Cheers,

[Juan Conejero]

PS Juan, maybe better to add switcher "Truncate/Rescale" in next version?

I have already implemented it. The next version also issues a warning when the integration result exceeds the available numeric range.