PixInsight Forum (historical)
PixInsight => Announcements => Topic started by: Juan Conejero on 2009 June 09 14:20:38
-
Hi all,
New versions of the ImageIntegration module are now available for download:
Linux 64-bit:
http://pixinsight.com/export/ImageIntegration-x11-x86_64-20090609.tar.gz
Mac OS X 32-bit:
http://pixinsight.com/export/ImageIntegration-osx-x86-20090609.zip
Windows 32-bit:
http://pixinsight.com/export/ImageIntegration-win-x86-20090609.zip
Windows 64-bit:
http://pixinsight.com/export/ImageIntegration-win-x86_64-20090609.zip
This new version fixes the critical bug in the Winsorized sigma clipping rejection algorithm, and also comes with a nice bonus.
Vicent Peris and I have been working to design and implement a new automatic image weighting algorithm. Basically, the new method uses wavelet-based techniques to compute relative SNR estimates for each image. Relative SNR values are used to derive accurate combination weights, based on the fundamental principle of proportionality between SNR and the square root of integration time.
In all tests we have made so far, this method consistently yields the best SNR in the final integrated image. To help in evaluating SNR, we have added a new automatic noise evaluation feature.
To use these features, do the following:
- Select Weights = Noise evaluation.
- Check the Evaluate noise option.
- For large sets of images (say more than 8 or 10 images), select sigma clipping or Winsorized sigma clipping rejection. For small sets (less than 6 images), select percentile clipping or averaged sigma clipping. For intermediate sets (between 5 and 10 images) select averaged sigma clipping. These are general recommendations; you should try out different methods to see which one is the best option for your images. Bear in mind that sigma clipping methods are very bad choices if the standard deviation isn't a good estimator of dispersion, so don't use them if you have just a few images.
- Watch noise estimates and average SNR improvements. Compare the results obtained with different image normalization and weighting methods, for a given pixel rejection algorithm and rejection parameters. Try to achieve the best numbers - your so hardly acquired data deserves no less than that!
I'll be glad to hear your opinions and experiences with all this new stuff. Your help is essential to achieve the state-of-the-art image integration tool that we want.
Enjoy!
-
My system says the Mac version is a Linux version????
Last login: Tue Jun 9 18:11:34 on ttys000
/Users/coyyote/Desktop/ImageIntegration-pxm.dylib ; exit;
jack-harveys-macbook-pro:~ coyyote$ /Users/coyyote/Desktop/ImageIntegration-pxm.dylib ; exit;
-bash: /Users/coyyote/Desktop/ImageIntegration-pxm.dylib: cannot execute binary file
logout
[Process completed]
-
Hi Juan,
Thank you for the new tool. I was very happy yesterday when I saw it as I was thinking in doing something similar but manually, using the new noise evaluation script.
These are the results of a few tests with the new tool on a set of 14 images from STL11k camera. They give significant differences in final noise levels. It is really very easy to check different setting thanks to the use of the file cache. It is really a wonderful tool!! ;)
(http://img140.imageshack.us/img140/1733/a1snr.jpg)
A question or doubt: as I commented at the beginning I was going to do a manual evaluation, so I had prepared a set of 14 small crops of the images containing only sky background. I have used the image integration on the crops and I noticed that noise evaluation of the individual crops, which I recorded, is quite different from measures taken from the whole images. Is that right? Are the whole image noise evaluations more accurate even considering that they may include low noise (high signal) areas?
Best regards
Jordi
-
Hi Jordi
Nice work there
I have carried out some trials and find similar to you ;D
Juan says that rejection norm should normally be checked but get better result with it unchecked ???
Lastly is there a good / quick way to pick a good sigma number rather than trial and error
Harry
-
Hi Jordi and Harry,
Sorry for being unresponsive, but yesterday I broke my eyeglasses and this has kept me out of business. I'm here again and I must say everything looks much clearer now 8)
Jordi, this is a very nice comparison work, but only the four last rows of the table make sense. This is the reason:
Juan says that rejection norm should normally be checked but get better result with it unchecked
Rejection normalization should always be enabled (I might even consider removing this parameter from ImageIntegration's interface). You may achieve higher SNR in the final integrated image without rejection normalization, but in such case, the reason for the apparent benefit is that rejection is not working at all. Without normalization, rejection simply cannot work, in general. The reason is that in order to carry out a valid rejection, the data must be statistically compatible. For example, imagine that one of the images has a slightly brighter background. In that case, most pixels from the "bright" image will probably be rejected, since they always will fall at one end of the histogram formed with all pixels in a stack. In fact, without normalization, the situation is normally much worse than that. Varying pixel values resulting from incompatible data usually form a sparse distribution where no statistical moment makes sense to describe the data. For example, for the median to be a valid estimator of the central value of a distribution, the distribution must be unimodal (with a single peak) and have a strong central tendency (a relatively sharp histogram peak).
Other than that, Jordi's comparison shows that Winsorized sigma clipping is in general better than sigma clipping to preserve significant data. I have achieved similar results in my tests. Of course, a relatively large set of images is necessary. With less than 8 or 10 images, no form of sigma clipping works in general, because the standard deviation isn't a good estimator of dispersion with such a reduced data set.
-
My system says the Mac version is a Linux version????
Sorry for not having explained this well. To use the new module on Mac OS X, do the following:
- Right-click the PixInsight.app application file.
- Select "Show Package Contents"
- A new Finder window opens. Select the Contents/MacOS folder inside PixInsight.app.
- Copy the zip file to PixInsight.app/Contents/MacOS and decompress it. This will create a .dylib file, which will replace the old module (say Yes when prompted for replacement). You can delete the zip file once extracted.
That's all. Now PI should load the new module.
P.S.: Yes, I am aware that this procedure is not very "friendly" for Mac users. I'll try to improve Mac updates, but please be patient with a guy that builds and manages a large application on three platforms at the same time :)
-
Got it and not a problem. I did not feel this was particularly cumbersome. Thanks
-
Jordi, this is a very nice comparison work, but only the four last rows of the table make sense.
OK it is clear for me now
I might even consider removing this parameter from ImageIntegration's interface
provided your explanation, that might be a reasonable idea in order to avoid the same mistake from new users. ;)
Thanks and regards
Jordi
-
Hi Juan,
Great work...another great module. It seems to only read FITS files. Is there a plan to let it read TIFs as well?
Cheers
Simon
-
Hi Simon,
Thanks!
It seems to only read FITS files. Is there a plan to let it read TIFs as well?
For performance reasons, ImageIntegration has been designed to work using incremental reading operations. Incremental reading allows loading a strip of pixel rows, instead of the entire image at once. This is why ImageIntegration has no practical limits for image sizes or the number of input images.
Currently the only format able to do incremental file operations in PixInsight is FITS. Planar TIFF files could also be read incrementally, but since no other software besides PixInsight supports them, I decided not to implement incremental file operations in the TIFF module.
You can use the standard BatchFormatConversion utility script to convert a set of TIFF files into FITS. It's easy and fast. let me know if you need help using that script.
-
Hi Juan,
Thanks for the explanation. I've just tried the BatchFormatConversion script....very easy to use. Up until now I have never seen the need for FITS format files, I've just been using 32bit TIFFs. Now I have a reason to use FITS as well.
Cheers
Simon
-
Hi Juan
Have been playing with intergration modual and are getting some very good results ;D and is excellent in removing outliners
As I asked before is there a technical way of selecting a good sigma or is it just trial and error ???
also I have some black lines on my images ( Caused by usb download ) is there a way to remove these and yes I did dither :P
Regards Harry
-
Harry,
do you have a slow download mode? There really shouldn't be such severe corruptions in your data. What's your camera again?
You'd have to dither lots of pixels in the vertical to get rid of those streaks.
Sander
-
Hi Sander,
From Harry's website it would appear that he is using a Starlight Express SXVF-M25C 6Mp OSC camera - so I suppose the 'USB Download' overhead might actually be quite significant if anything on his PC 'takes over' during the download phase.
Correct me if my maths is wrong, but I would consider a 6Mp 'RAW' (32-bit data byte) frame from an OSC imager to be of the order of ( 6 x 1024 x 1024 x 4 x 8 ) 'bits' long. So, at the maximum theoretical USB 2.0 transfer rate (of 480Mbps), that would result in a download time of ( 6 x 1024 x 1024 x 4 x 8 ) / ( 480 x 1024 x 1024 ) seconds - i.e. 1/10th of a second. Given that the 'maximum' rate is not really attainable, and allowing for, say, a maximum 'actual' rate of about 33% of 'theoretical', the transfer time should be around 0.3s - assuming that the main CPU and hard-drive can maintain this rate.
Surely therefore there IS scope for an imaging camera that is NOT fitted with a true mechanical shutter to collect, or create, 'noise' during this 0.3 to 0.5 seconds after actual 'photon collection' ought to have ceased?
Perhaps what you need to try Harry, is to run your camera 'flat out', as if you were collecting 'zero exposure time' bias/offset frames. How many frames can you collect in, say, a ten-second period. Run a few trials and see what the average number is. Divide this by '10' (seconds) to get your 'average USB transfer time'. If your imager does NOT have a true mechanical shutter, then this USB transfer time is a period during which your imager will still be collecting photons AND 'noise', on those parts of the pixel array that have still to be 'read out' during the 'USB transfer time' window.
Unless my understanding of how an imager (without a mechanical shutter, or on-board high-speed 'frame-store') works is flawed, then I have always believed that you need to really get the USB transfer time down as low as possible - certainly MUCH lower than the time above which you would be looking to apply Dark Frame correction to your normal subs (which I have always taken to be around 1 second, or so).
Can anyone confirm my understanding - or, better, tell me where I might be going wrong?
Cheers,
-
Hi Sander
The camera that causes this problem is a starlight xpress H16 ( mono ) and the streaks are caused by the usb2 bus dumping info every 500k
My old computer and usb bus did not cause a problem ( shame it caught fire ) so I got a newer faster laptop ( or not that new ) and I can not seem to get
the H16 to download without the streaks ( probably a slow usb2 )
So short of changing computer I am stuck with them at the moment :'(
Harry
-
Hi Harry,
that really sucks! No other words for it. Taking good pictures is hard enough and when such artifacts are introduced with the most basic of operations (image download) then you start way back in the line before you get started.
If your laptop has a PCMCIA (or similar) card you could buy a USB card for that slot and see if it's better than the built-in ones.
My QHY8 had similar problems but then I got a better driver for the camera and that took care of it.
Good luck!
-
Hi Sander
Tried the PCMCIA usb thingy and that did not work and SX have given me a new driver and seems that its my mardy
laptop >:(
Probabley will have to change computer but as it took me several days to rebuild it all its a last resort
Harry
-
Hi Niall
Now that's a bit tech ;)
My h16 creates file of around 8 meg and takes 2.5 secs to download and does not have a shutter as its a interline chip and I understand ( Might be wrong) that at the end of exposure the image is moved to another area of the chip stopping any further light gathering ;D
The lines are caused by a usb2 computer problem
Harry
-
Hi Harry,
well that's certainly perplexing. I use a cheap desktop machine with my scope. Something like a P4@2.8G with less than 1GB RAM and XP Pro. I got it from my mother in law but they're available online for $80-$150. The other day I saw a dual Xeon desktop from Dell for $200. If you have a semi permanent setup it might be worth considering getting one of those instead of another laptop.
I hope you get it sorted!
-
Hi Juan
I have been playing with your integration module and I was trying to find the optimal sigma setting to use
and found even at a sigma setting of 10 all of the hot pixels were excluded ( impressive ) and felt perhaps the optimal might be 11 !
But of course you can not use 11 :(
I was stacking 80 images very easily , fantastic :)
Do you have any words of wisdom
Harry
-
Hi Juan
Further news on the intergration modules!
Using your modual I was able to achieve FWHM 10% better than your favorite program ( Yes CCD stack)
The 10 % figure was consistant on smaller non saturated stars but did not have such large effect on large saturated stars
this makes my skys 10% more stable than I thought :laugh: :laugh: :laugh:
Regards Happy Harry
-
Hello.
I also have been playing with Image Integration (and registration). The results are clearly better than what I got with DSS. I don't know if the algorithms are superior, or if it is the fact that I can essentially experiment almost in realtime when I work on a small segment of the image. It is sooo fast! You can just play around with sigma etc. and see what you get. 8)
Great tool!
Georg
-
Thanks! :) I'm glad you like ImageIntegration. The new version in PI 1.5.3 works better.
Do you have any words of wisdom
No, but I certainly have a lot of wines of wisdom. Must be that ;D