Rescale Question

[Jack Harvey]

OK I need to clear up my thinking on rescale tool.  WHat exactly happens when you apply the rescale tool (say RGB button checked) to an image?  Thanks in advance

[dhalliday]

Do you mean the up/down sampling,in "geometry"...?

[Jack Harvey]

Actually neither<G>.  I understand that.  WHat happens when you use the rescale tool is the question  The one that only says rescale pixel values and you select either RGB or a Lum, no other options.  My assumption is this is for applying to images outside the floating point bounds of PixInsight and rescale to within bounds? 

[Carlos Milovic]

Hi Jack

The purpose of Rescale is to use the whole normalized dynamical range with your data. This is done by a simple rescalation and shift, so the brighter pixel in your image becomes 1, and the darker 0.
At first I wrote it to be somewhat of a shorcut of the histogram black/white points for the joint channels, but soon it turned that it was very helpfull to deal with out of limits data that is generated by other processes (mainly some convolutions).

[Jack Harvey]

SO if I understand this Carlos when getting ready to combine your master frames of say R,G,B it would be best top rescale then combine?  Or in fact is this done automatically by image integration tool when you select additive and scale in normalization?

[Carlos Milovic]

You should not need to rescale the image at that stage. If you are working with 32bits outputs is very unlikelly that information is lost due to the lack of use of the whole range.
The only reason I see to use Rescale in a common workflow is to quickly set the black and white points without clipping, before a histogram transform (or instead one), or to fill again all the range after a dynamic range extension (if used in wavelets or other process to avoid clipping).
Other processes, like PixelMath, perform their own rescalations, wich perform the same task as this process.

[Jack Harvey]

Thanks for the education Carlos.

[Carlos Milovic]

You are wellcome

[Niall Saunders]

The problem exists that there are some processes (notably ImageIntegration) that (seem to) perform an 'automatic' ReScale during the process workflow. Until this feature becomes an option, ImageIntegration has now become a more or less 'useless' tool for me - leaving me having to leave the realm of PixInsight in order to calibrate my data. Which is a shame.

This might be pefectly acceptable for normal 'Light' frames, but it totally DESTROYS any calibration frame (Dark, Flat, FlatDark, etc.). Calibration frames need to maintain their data in the vorrect 'statistical' relationship. In other words the 'peak' of the Histogram for a Dark snould NOT be allowed to shift - as it will if a ReScale process is applied.

Use ReScale with care (or with understanding!)

Cheers,

[Juan Conejero]

there are some processes (notably ImageIntegration) that (seem to) perform an 'automatic' ReScale during the process workflow

Please do the following:

1. Be sure that you have selected the [0,65535] floating point input range in FITS Format Preferences. Use the "dirty trick" of editing the configuration file while necessary (magic wireless keyboard).

2. Perform a batch conversion of all your floating point FITS files to 16-bit unsigned integer format. This includes all your light, bias, dark and flat frames. Use the SampleFormatConversion and ImageContainer tools. I have attached a screenshot with an example.

3. Use ImageIntegration with your 16-bit data in the usual way. All will be fine.

[Niall Saunders]

Hi Juan,

Thanks for coming back so quickly - I can't run the test just now, but will look at the results this evening.

However, looking at your three-stage suggestion:-
1.) Using the 'manual edit' of the INI file has worked fine - all of my FITS files are handles in the [0, 65535] range, and ALL of them (including BiasOffests, etc.) are correctly rescaled to the [0, 1] range - without the 'median' of the dataset being shifted (as was the case). So - no problems here.

2.) I am not clear in my mind how this process is going to work. I will already have a file on HDD that has the data stored in 32-bit Float. The Min Value will always be 0.0 and the Max Value will always be 65535.0 (obviously, not every image will necessarily attain either of these two 'limits', but these are the 16-bit 'limits' of the CCD digitisation process, with the data then 'translated' into a 32-bit Float range, even though this is '100% wasteful' in terms fo disk space).
If PI is going to then 'translate' the 32-bit Float [0.0, 65535.0] range into 16-bit Unsigned Integer [0, 65535] range, then I assume that PI must 'open' each image - under the rescale rule implemented in step (1) above, and then make the conversion before saving the image in the new, integer, format.

Does this mean that PI takes the (internally converted) [0.0, 1.0] range and 'multiplies' it by '65535' to create the Integer [0, 65535] range before saving it?

Does this also mean that, when PI re-reads the 16-bit Unsigned Integer [0, 65535] format, it does NOT rescale the data, it just applies a simple translation to the internal (working) [0.0, 1.0] range - so ImageIntegration doesn't 'shift the median' (which is the 'effect' that I am unable to overcome at present)?

===========

If the process you describe does solve the problem, then I think that we need to consider which CCDs are affected, and which capture software is involved.

As I have said before - there is NOTHING WRONG with the way that Meade's Envisage software actually stores its FITS data. Yes, I agree that 32-bit Float format is perhaps NOT the most efficient method of storing [0, 65535] 16-bit Unsigned Integer data - but at least Envisage can actually 'get that process right' (the programmers have NEVER addressed the 'fatal' bug associated with the "FITSINT" storage format that they offer - if you use THAT format, all bets are OFF, because it simply just does NOT work).

What I just do not understand is why, when PI sees 'my' 32-but Float file, with data having been stored 'from' a 16-bit unsigned integer 'source', it just doesn't accept the data for what it is. OK, yes, I have to tell PI that the data is 'from' a [0.0, 65535.0] range, and that it should use THAT range to rescale the results to a [0.0, 1.0] range. If that was all that happened, then I would be happy.

And, as I have said, that is certainly what happens if I use PI to 'open' any of my images with these requests in place.

However, if I use ImageIntegration on data (that presumably get translated to thj [0.0, 1.0] range) why is there then this EXTRA step, where the incoming data ALSO gets "ReScaled"?

And, by "ReScaled" I mean that the image Black and White points are examined, such that the 'new' [0.0, 1.0] range actually becomes the image [Black, White] range.

I accept that this is actually a WORTHWHILE step to implement - PROVIDING THE DATA HAS ALREADY BEEN CALIBRATED.

But, you CANNOT implement this [Black, White] rescale step to any 'calibration' data, or even to un-calibrated Lights. Not in my mind anyway - it means that all pending 'data subtraction' steps (Darks from Lights, etc.) will have a meaningless result.

And, I therefore do not see why this problem is associated ONLY with 'my' (DSI) data - surely this is fundamental to other CCD data as well?

My basic question still stands - I absolutely KNOW that the [Black, White] "ReScale" function call MUST be implemented 'somewhere' in the ImageIntegration process - because I see the result in the output image. So, the question is "Where (or When) does it take place?"

And, why will it therefore NOT take place if I have pre-converted my source data set from 32-bit Float to 16-bit Unsigned Integer, as you suggest?

Curioser, and curioser 

Cheers,

[Juan Conejero]

Hi Niall,

Does this mean that PI takes the (internally converted) [0.0, 1.0] range and 'multiplies' it by '65535' to create the Integer [0, 65535] range before saving it?

Yes.

Does this also mean that, when PI re-reads the 16-bit Unsigned Integer [0, 65535] format, it does NOT rescale the data, it just applies a simple translation to the internal (working) [0.0, 1.0] range - so ImageIntegration doesn't 'shift the median' (which is the 'effect' that I am unable to overcome at present)?

Of course. When PixInsight loads an *integer* image, it does not alter the original values in any way. Integer images are loaded unaltered because PixInsight supports five numerical data formats transparently, including 8, 16 and 32-bit integers. This means that a 16-bit integer image, for example, will be stored as a matrix (or data cube) of 16-bit integer numbers in RAM.

If the process you describe does solve the problem, then I think that we need to consider which CCDs are affected, and which capture software is involved.

The problem is exclusively a software problem. Easy: all softwares that store 16-bit raw data as floating point numbers in a range different from [0,1] will need this extra adaptation step before loading their images in PixInsight.

there is NOTHING WRONG with the way that Meade's Envisage software actually stores its FITS data.

I disagree here. Yes, the way Envisage stores raw data is extremely bad. It writes standards-compliant FITS files, but this is only a necessary condition, not a sufficient condition. Raw observational data must be stored in their native data format, that is 16-bit integers in this case.

By storing 16-bit raw data as 32-bit floating point numbers in FITS files Envisage is causing problems unnecessarily. The FITS standard does not provide a way to specify the actual data range to which floating point pixel values are referred. In other words, we have no way to know which values correspond to black and white in a floating point FITS file. For this reason PixInsight's FITS format module allows you to specify an input range for floating point images manually: because it has no other (standard) way to know what the input range is.

So Envisage is, at least:

- Wasting a 50% of storage space and I/O bandwidth unnecessarily.
- Causing compatibility problems with other applications.
- Giving a false feeling of accuracy to their users.

the programmers have NEVER addressed the 'fatal' bug associated with the "FITSINT" storage format that they offer - if you use THAT format, all bets are OFF, because it simply just does NOT work

This gives you a clear idea of how their programmers work, and how the company backs its products and treats its users.

OK, yes, I have to tell PI that the data is 'from' a [0.0, 65535.0] range, and that it should use THAT range to rescale the results to a [0.0, 1.0] range. If that was all that happened, then I would be happy.

That's all that happens, so you can be happy

However, if I use ImageIntegration on data (that presumably get translated to thj [0.0, 1.0] range) why is there then this EXTRA step, where the incoming data ALSO gets "ReScaled"?

Because ImageIntegration does not open images in the "usual" way. For performance reasons, II uses some features of the FITS format support module that are not available to GUI users, but only through the PCL framework. In particular, II reads FITS files incrementally by loading successive pixel row strips.

When a floating point image is read incrementally, pixels are not translated to [0,1] by the FITS module, but by the calling process. The calling process is II in this case, and II is not applying the FITS floating point input range, but just assuming that real-valued data are already in [0,1]. This is a limitation (not a bug, IMO) that I'll try to overcome in the next version.

For now, you have to convert your files to 16-bit integer format before using them with II.

And, I therefore do not see why this problem is associated ONLY with 'my' (DSI) data

Yes, this only happens with images like FITS files generated by Envisage. As I've said above this is exclusively a software problem.

surely this is fundamental to other CCD data as well?

Raw CCD data stored as 16-bit integer FITS files (Maxim DL, Nebulosity, AstroArt, professional observatories, etc.) works without problems with ImageIntegration. For example, the DSA folks (Vicent Peris, Jack Harvey, etc.) are working with Calar Alto Observatory and they calibrate (manually), register and process all the data entirely in PixInsight.

As noted, the *only* problem is that ImageIntegration does not know where black and white are in your Envisage FITS files.

And, why will it therefore NOT take place if I have pre-converted my source data set from 32-bit Float to 16-bit Unsigned Integer, as you suggest?

Because, as I've said, when ImageIntegration reads n-bit integer data it knows where black and white are: 0 and 2ⁿ-1, respectively.

[dhalliday]

Yikes
you all lost me at page 1....I am going to "default/reset/rescale"  by pretending I never saw this thread...
Great hockey game last nite...those Swede's gave us a scare
Dave

[Niall Saunders]

Juan,

Once again - you are a 'star' - thanks for the absolutely clear and detailed explanations. I can now understand 'why' the Envisage-created FITS files cause the problems that they do. I must admit, I have not looked at the same image data from a DSI camera when captured by Craig Stark's Nebulosity - but I would doubt that he has made the same mistakes as were made by the team behind Envisage (which, in reality, is a software package that should NEVER have been released into the public domain, a comment that I have made before, and a comment that has led to me being 'banned' from certain discussion groups because it was unacceptable to publicly denounce a Meade product on that Forum - think :police: 'Heretic' here)

So, it certainly looks as if I WILL finally be able to attempt a full PixInsight calibration process on my data - I'll try and 'video' the result (no promises though )

Dave - don't let discussions like this worry you, not at this stage anyway. Just keep in mind that, as your needs in PI grow, you will ALWAYS have the likes of GrandJediMeister Juan to furnish the information, or workaround, that you need. That fact, alone, would stop me ever moving away from PI.

Cheers,

[Niall Saunders]

Hi again Juan,

Just to confirm that the 'workaround' process that you described does work. If I manage to set the FITS input transform to [0.0, 65535.0] by editing the .ini file (which I can) and then convert my data sets, in batch mode using the ImageContainer and SampleFormatConversion (which I have done, and will now do hereafter), and then run ImageIntegration, everything works again - I end up with an integrated image that has the same effective Median as the original images.

So, thanks, and I am going to try to complete a full calibration of my M1 Lum dataset - thistime entirely in PixInsight

Cheers,