Updated ImageIntegration modules

[Juan Conejero]

Hi all,

New versions of the ImageIntegration module are now available for download:

Linux 64-bit:
http://pixinsight.com/export/ImageIntegration-x11-x86_64-20090609.tar.gz

Mac OS X 32-bit:
http://pixinsight.com/export/ImageIntegration-osx-x86-20090609.zip

Windows 32-bit:
http://pixinsight.com/export/ImageIntegration-win-x86-20090609.zip

Windows 64-bit:
http://pixinsight.com/export/ImageIntegration-win-x86_64-20090609.zip

This new version fixes the critical bug in the Winsorized sigma clipping rejection algorithm, and also comes with a nice bonus.

Vicent Peris and I have been working to design and implement a new automatic image weighting algorithm. Basically, the new method uses wavelet-based techniques to compute relative SNR estimates for each image. Relative SNR values are used to derive accurate combination weights, based on the fundamental principle of proportionality between SNR and the square root of integration time.

In all tests we have made so far, this method consistently yields the best SNR in the final integrated image. To help in evaluating SNR, we have added a new automatic noise evaluation feature.

To use these features, do the following:

- Select Weights = Noise evaluation.

- Check the Evaluate noise option.

- For large sets of images (say more than 8 or 10 images), select sigma clipping or Winsorized sigma clipping rejection. For small sets (less than 6 images), select percentile clipping or averaged sigma clipping. For intermediate sets (between 5 and 10 images) select averaged sigma clipping. These are general recommendations; you should try out different methods to see which one is the best option for your images. Bear in mind that sigma clipping methods are very bad choices if the standard deviation isn't a good estimator of dispersion, so don't use them if you have just a few images.

- Watch noise estimates and average SNR improvements. Compare the results obtained with different image normalization and weighting methods, for a given pixel rejection algorithm and rejection parameters. Try to achieve the best numbers - your so hardly acquired data deserves no less than that!

I'll be glad to hear your opinions and experiences with all this new stuff. Your help is essential to achieve the state-of-the-art image integration tool that we want.

Enjoy!

[Jack Harvey]

My system says the Mac version is a Linux version?
Last login: Tue Jun  9 18:11:34 on ttys000
/Users/coyyote/Desktop/ImageIntegration-pxm.dylib ; exit;
jack-harveys-macbook-pro:~ coyyote$ /Users/coyyote/Desktop/ImageIntegration-pxm.dylib ; exit;
-bash: /Users/coyyote/Desktop/ImageIntegration-pxm.dylib: cannot execute binary file
logout

[Process completed]

[Jordi Gallego]

Hi Juan,

Thank you for the new tool. I was very happy yesterday when I saw it as I was thinking in doing something similar but manually, using the new noise evaluation script.

These are the results of a few tests with the new tool on a set of 14 images from STL11k camera. They give significant differences in final noise levels. It is really very easy to check different setting thanks to the use of the file cache. It is really a wonderful tool!!



A question or doubt: as I commented at the beginning I was going to do a manual evaluation, so I had prepared a set of 14 small crops of the images containing only sky background. I have used the image integration on the crops and I noticed that noise evaluation of the individual crops, which I recorded, is quite different from measures taken from the whole images. Is that right? Are the whole image noise evaluations more accurate even considering that they may include low noise (high signal) areas?

Best regards
Jordi

[Harry page]

Hi Jordi
Nice work there
I have carried out some trials and find similar to you
Juan says that rejection norm should normally be checked but get better result with it unchecked
Lastly is there a good / quick way to pick a good sigma number rather than trial and error

Harry

[Juan Conejero]

Hi Jordi and Harry,

Sorry for being unresponsive, but yesterday I broke my eyeglasses and this has kept me out of business. I'm here again and I must say everything looks much clearer now  8)

Jordi, this is a very nice comparison work, but only the four last rows of the table make sense. This is the reason:

Juan says that rejection norm should normally be checked but get better result with it unchecked

Rejection normalization should always be enabled (I might even consider removing this parameter from ImageIntegration's interface). You may achieve higher SNR in the final integrated image without rejection normalization, but in such case, the reason for the apparent benefit is that rejection is not working at all. Without normalization, rejection simply cannot work, in general. The reason is that in order to carry out a valid rejection, the data must be statistically compatible. For example, imagine that one of the images has a slightly brighter background. In that case, most pixels from the "bright" image will probably be rejected, since they always will fall at one end of the histogram formed with all pixels in a stack. In fact, without normalization, the situation is normally much worse than that. Varying pixel values resulting from incompatible data usually form a sparse distribution where no statistical moment makes sense to describe the data. For example, for the median to be a valid estimator of the central value of a distribution, the distribution must be unimodal (with a single peak) and have a strong central tendency (a relatively sharp histogram peak).

Other than that, Jordi's comparison shows that Winsorized sigma clipping is in general better than sigma clipping to preserve significant data. I have achieved similar results in my tests. Of course, a relatively large set of images is necessary. With less than 8 or 10 images, no form of sigma clipping works in general, because the standard deviation isn't a good estimator of dispersion with such a reduced data set.

[Juan Conejero]

My system says the Mac version is a Linux version?

Sorry for not having explained this well. To use the new module on Mac OS X, do the following:

- Right-click the PixInsight.app application file.

- Select "Show Package Contents"

- A new Finder window opens. Select the Contents/MacOS folder inside PixInsight.app.

- Copy the zip file to PixInsight.app/Contents/MacOS and decompress it. This will create a .dylib file, which will replace the old module (say Yes when prompted for replacement). You can delete the zip file once extracted.

That's all. Now PI should load the new module.

P.S.: Yes, I am aware that this procedure is not very "friendly" for Mac users. I'll try to improve Mac updates, but please be patient with a guy that builds and manages a large application on three platforms at the same time

[Jack Harvey]

Got it and not a problem.  I did not feel this was particularly cumbersome.  Thanks

[Jordi Gallego]

Jordi, this is a very nice comparison work, but only the four last rows of the table make sense.

OK it is clear for me now

I might even consider removing this parameter from ImageIntegration's interface

provided your explanation, that might be a reasonable idea in order to avoid the same mistake from new users.

Thanks and regards
Jordi
 

[Simon Hicks]

Hi Juan,

Great work...another great module. It seems to only read FITS files. Is there a plan to let it read TIFs as well?

Cheers
         Simon

[Juan Conejero]

Hi Simon,

Thanks!

It seems to only read FITS files. Is there a plan to let it read TIFs as well?

For performance reasons, ImageIntegration has been designed to work using incremental reading operations. Incremental reading allows loading a strip of pixel rows, instead of the entire image at once. This is why ImageIntegration has no practical limits for image sizes or the number of input images.

Currently the only format able to do incremental file operations in PixInsight is FITS. Planar TIFF files could also be read incrementally, but since no other software besides PixInsight supports them, I decided not to implement incremental file operations in the TIFF module.

You can use the standard BatchFormatConversion utility script to convert a set of TIFF files into FITS. It's easy and fast. let me know if you need help using that script.

[Simon Hicks]

Hi Juan,

Thanks for the explanation. I've just tried the BatchFormatConversion script....very easy to use. Up until now I have never seen the need for FITS format files, I've just been using 32bit TIFFs. Now I have a reason to use FITS as well.

Cheers
         Simon

[Harry page]

Hi Juan

Have been playing with intergration modual and are getting some very good results and is excellent in removing outliners

As I asked before is there a technical way of selecting a good sigma or is it just trial and error

also I have some black lines on my images ( Caused by usb download ) is there a way to remove these and yes I did dither

Regards Harry

[Nocturnal]

Harry,

do you have a slow download mode? There really shouldn't be such severe corruptions in your data. What's your camera again?

You'd have to dither lots of pixels in the vertical to get rid of those streaks.

  Sander

[Niall Saunders]

Hi Sander,

From Harry's website it would appear that he is using a Starlight Express SXVF-M25C 6Mp OSC camera - so I suppose the 'USB Download' overhead might actually be quite significant if anything on his PC 'takes over' during the download phase.

Correct me if my maths is wrong, but I would consider a 6Mp 'RAW' (32-bit data byte) frame from an OSC imager to be of the order of ( 6 x 1024 x 1024 x 4 x 8 ) 'bits' long. So, at the maximum theoretical USB 2.0 transfer rate (of 480Mbps), that would result in a download time of ( 6 x 1024 x 1024 x 4 x 8 ) / ( 480 x 1024 x 1024 ) seconds - i.e. 1/10th of a second. Given that the 'maximum' rate is not really attainable, and allowing for, say, a maximum 'actual' rate of about 33% of 'theoretical', the transfer time should be around 0.3s - assuming that the main CPU and hard-drive can maintain this rate.

Surely therefore there IS scope for an imaging camera that is NOT fitted with a true mechanical shutter to collect, or create, 'noise' during this 0.3 to 0.5 seconds after actual 'photon collection' ought to have ceased?

Perhaps what you need to try Harry, is to run your camera 'flat out', as if you were collecting 'zero exposure time' bias/offset frames. How many frames can you collect in, say, a ten-second period. Run a few trials and see what the average number is. Divide this by '10' (seconds) to get your 'average USB transfer time'. If your imager does NOT have a true mechanical shutter, then this USB transfer time is a period during which your imager will still be collecting photons AND 'noise', on those parts of the pixel array that have still to be 'read out' during the 'USB transfer time' window.

Unless my understanding of how an imager (without a mechanical shutter, or on-board high-speed 'frame-store') works is flawed, then I have always believed that you need to really get the USB transfer time down as low as possible - certainly MUCH lower than the time above which you would be looking to apply Dark Frame correction to your normal subs (which I have always taken to be around 1 second, or so).

Can anyone confirm my understanding - or, better, tell me where I might be going wrong?

Cheers,

[Harry page]

Hi Sander

The camera that causes this problem is a starlight xpress H16 ( mono ) and the streaks are caused by the usb2 bus dumping info every 500k
My old computer and usb bus did not cause a problem ( shame it caught fire ) so I got a newer faster laptop ( or not that new ) and I can not seem to get
the H16 to download without the streaks ( probably a slow usb2 )
So short of changing computer I am stuck with them at the moment

Harry