Need Help on Processing Comet

[twade]

To all,

I had a pretty good night last night.  I took five sets of images guiding on the comet.  Unfortunately, given the variation in the sky (some extremely thin cirrus) and a few other problems, I'm unable to remove all the stars from the image.  Does anybody have any idea on what algorithm is best and what values to choose?  Everything I've tried so far has failed.  Is there a way to manual remove the remaining stars?  I would really hate to give up on this data as it appears to be very good.  Here's an example of the problem:

http://www.northwest-landscapes.com/images/deepsky/wide-field/2009R1_06182010_lum.jpg

Any help and/or suggestions would be greatly appreciated.

Thanks,

Wade

[Juan Conejero]

Hi Wade,

How are you aligning the five images? If you are doing it in PixInsight, I assume you're using DynamicAlignment with a single alignment point set on the comet's head. Provided you don't have any field rotation, that should work well.

Once you have all your images aligned on the comet, ImageIntegration should do a fine job removing all the stars. They will be treated as outliers by the rejection algorithms. With five images you could try percentile rejection, or averaged sigma clipping. Basically any rejection method will work.

Does this make any sense for you, or am I royally out of the question's topic?

[Yuriy Toropin]

Hi Wade,
check the message box

[Juan Conejero]

Hi Yuriy,

check the message box

If you have something to say that you think isn't appropriate for this forum, please feel free to say it openly. This is a place where constructive criticism and discussion are always welcome. Don't worry, it won't hurt anybody, or anything like that.

[twade]

Juan,

How are you aligning the five images?

Yes, DynamicAlignment with only one point on the comet.

ImageIntegration should do a fine job removing all the stars.

For some odd reason it isn't.  You see the result.  I guess it was a combination of not allowing enough time between images (i.e. lack of spacing between stars) and the thin cirrus.

With five images you could try percentile rejection, or averaged sigma clipping. Basically any rejection method will work.

I tried them all with limited success.  I posted the best result.

I can post them if you like; however, it will have to wait until next week.

Thanks,

Wade

[Yuriy Toropin]

 
As you wish, Juan, copy of my original message to Wade is below. I've mentioned about FitStacker several times so it's not a big deal.
 

***
Hi Wade,

Let me share with you the best kept secret
that will not be really welcomed on public part of PixInsight forum 'cause it will bless other piece of software vs. PixInsight.

Anyway, I really encourage you to download and try FitStacker software written by Ivan Ionov, fellow Russian astroimager. It can be taken here
http://qhy.narod.ru/FITStacker9.zip
also, set of MS Redistributable libraries could be needed if they've not been installed on your Win system, can be downloaded here
http://qhy.narod.ru/vcredist_x86.zip

It performs only image integration, so you should submit set of aligned images (say from PI) to it.

Features that makes it unique and very effective in integration:
1) Dynamic spline interpolation of background for each image - this is what exactly will help you in your fight with gradient due to cirruses;
2) Iterative Sigma clipping with "soft borders" (Sigma Fade parameter in FitStacker) - lead to smoother transition around sigma clipped areas;
3) Integration is optimized to get the best S/N for the sum based on S/N of individual images - it's known that images should be combined with weight ~(S/N)^2 to get the best S/N in the sum.

Download, install and start the FitStacker, after that workflow is pretty straightforward
(see screenshot below for reference or go tho this link)

1) Press Add... button to select and open images to integrate.
Could be used several times to add images from different folders, etc -> images will be loaded in FitStacker, Merge Parameters dialog will become available (if it is dissapeared for some reason - press Adaptive button). Images should be aligned.

2) (Several) background areas should be choosen to measure Noise and Background.
Use left mouse button + dragging on any image to select the first area, to add other hold Shift and use left mouse button dragging on image again (see screenshot for example). Green boxes will appear on images.

3) (Several) object should be select to measure Signal.
Use Right mouse button to select the first object, hold down Shift + dragging of Right Mouse button to select any additional objects. For comet it makes sense to select single objects, comet's head (see screenshot).

4) You can adjust scale and gamma of image representation, see View group on toolbar. Drag image with mouse and CTRL pressed down. You can scroll through images via button on toolbar or with arrow keys. Delete any image that is REALLY too bad. List will show dialog with measured S/N for all images.

5) Set correct parameters for integration in Merge Parameters dialog (start with ones from screenshot, I underlined everything you have to take care about).
Set "Float" in drop box to the left of Save PNG... button in the toolbar -> in this case the sum will be prepared in Float 32 bit format ready to be saved and then opened in PI.

6) Start integratin with Create button in dialog. And wait! you will see which pixels will be filtered out during the process. While we set 5 iterations in sigma clipping it will take some time to finish the process.

7) As soon as integration will be ended the Save As... dialog will show up (and you will see sum in FitStacker for now). Save the sum.

That's mainly it. The only parameters that really requires some play could be Normalize Area (pix) - it could be made larger for larger images, good rule is just divide X size of your image by 4 or 5 to get the size.
Also, Sigma Clip could be played with a little bit to control the level of "filtering out", higher parameter will lead to less restrictive filtering.

Try it! Sum all frames to get "Synth L", after that you will be able to sum only R, G, and B to get color channels.

Let me know if you will have any questions. FitStacker consistently gives better results than PI in case of limited "difficult" subsets. PI is better on longer homogenious sequences.

All the best,
   Yuriy

[Niall Saunders]

Hi Yuriy

that will not be really welcomed on public part of PixInsight forum 'cause it will bless other piece of software vs. PixInsight

Not at all - that would NOT be the way I see things. If one of us knows of a method of achieving some miracle of astro-image processing, and it is NOT part of the PI 'stable' then we should embarce this method with open minds.

What is the worst that can happen?

We should WELCOME the efforts of others. We should invite them INTO the world of PI, not shun them away from us.

There are many ways to skin a cat, and the big sharp knife is not always the most effective method

Cheers,

[Yuriy Toropin]

Hi Niall,

We should WELCOME the efforts of others. We should invite them INTO the world of PI, not shun them away from us.

Come on! That was exactly what I did some time ago, proposing Ivan Ionov, author of FitStacker, to try PixInsight for himself and to apply his energy and development talent in PI environment.

[Juan Conejero]

Hi Yuriy,

Thanks for the information.

We should WELCOME the efforts of others. We should invite them INTO the world of PI, not shun them away from us.

Come on! That was exactly what I did some time ago, proposing Ivan Ionov, author of FitStacker, to try PixInsight for himself and to apply his energy and development talent in PI environment.

He definitely would be welcome, as all developers willing to work on the PixInsight platform. Modestly and honestly I think PixInsight has a lot of interesting and exciting things to offer for anybody seriously interested in development of image processing software tools, in any fields of technical imaging, not just astronomy. However this is just my opinion and I am biased, obviously. What is for sure is that I am working 24/7 to ensure that this is and will be possible. In fact, I invest most of my development time and effort in making PixInsight better as a development platform, more than as an image processing platform. This is my bet for the future, and as always happens when one bets, I can lose.

Anyway let me comment on a few points of your message:

3) Integration is optimized to get the best S/N for the sum based on S/N of individual images - it's known that images should be combined with weight ~(S/N)^2 to get the best S/N in the sum.

Our ImageIntegration tool does exactly this in PixInsight. This is our "noise evaluation" weighting method. The most important point that requires special attention in this method is the algorithm(s) used to evaluate the noise. In our case, we apply a multiscale noise estimation algorithm described by Jean-Luc Starck and Fionn Murtagh in their paper Automatic Noise Estimation from the Multiresolution Support, published in PASP in 1998. This noise evaluation algorithm is part of the standard implementation of wavelet transforms in our PixInsight Class Library (PCL), and hence is readily available to developers.

2) Iterative Sigma clipping with "soft borders" (Sigma Fade parameter in FitStacker) - lead to smoother transition around sigma clipped areas

The next version of ImageIntegration (perhaps released with PI 1.6.1, time and tests permitting) implements an iterative algorithm that largely improves the performance of sigma clipping. I don't know how your friend has implemented this, but my algorithm basically iterates sigma clipping restricting it to previously rejected regions, using a wavelet-based technique to isolate structures within the rejection mask. I've been working on this recently, but unfortunately have had to stop it because version 1.6.1 has more urgent tasks. These improvements are very necessary IMO.

1) Dynamic spline interpolation of background for each image - this is what exactly will help you in your fight with gradient due to cirruses

Hmm, I tend to disagree with this one. Image integration is actually a preprocessing task, as I see it, and hence I don't want too fancy things done at this stage. I want to ensure that the original data, including their linearity, are strictly preserved, and only apply simple operations with a sound statistical basis. The idea of 'interpolating the background' doesn't seem to fit well in this scheme, but this is of course my personal point of view, without knowing what this tool is doing exactly, so I may be wrong here.

[Yuriy Toropin]

Thanks for comments, Juan!

3) Integration is optimized to get the best S/N for the sum based on S/N of individual images - it's known that images should be combined with weight ~(S/N)^2 to get the best S/N in the sum.

This is our "noise evaluation" weighting method. The most important point that requires special attention in this method is the algorithm(s) used to evaluate the noise. In our case, we apply a multiscale noise estimation algorithm described by Jean-Luc Starck and Fionn Murtagh in their paper Automatic Noise Estimation from the Multiresolution Support, published in PASP in 1998. This noise evaluation algorithm is part of the standard implementation of wavelet transforms in our PixInsight Class Library (PCL), and hence is readily available to developers.

This sounds great and definitely Noise is evaluated better in PI, but... how the Signal is evaluated?
In FitStacker it's straightforward - direct sum of (S-B) for selected area is calculated, if needed, averaging by several selected regions. Naive "aperture photometry". That gives "fair" signal for the objects of interest and any variation of transparency, background, etc. will be correctly considered in optimization by S/N.

As I've mentioned above, PI is the champion when we're talking about integration of homogenious set of data (say, the same objects, exposures from the same night), but in case of non-homogenius data where S/N and background of images can varry widely, FS will give better results now. It's checked on several data set including ultra deep M51 one or on subsets from different nights for the same object..

Note: Evaluate Noise is turned off on the screenshot below 'cause PI noise evaluation failed on some images of the M51 set...

2) Iterative Sigma clipping with "soft borders" (Sigma Fade parameter in FitStacker) - lead to smoother transition around sigma clipped areas

The next version of ImageIntegration (perhaps released with PI 1.6.1, time and tests permitting) implements an iterative algorithm that largely improves the performance of sigma clipping. I don't know how your friend has implemented this, but my algorithm basically iterates sigma clipping restricting it to previously rejected regions, using a wavelet-based technique to isolate structures within the rejection mask. I've been working on this recently, but unfortunately have had to stop it because version 1.6.1 has more urgent tasks. These improvements are very necessary IMO.

Please, check below screenshots of "ultra-deep M51 mega sum" obtained in FitStacker and PI. From it it's obvious what's the difference. That Sigma Fade parameter in FitStacker defines how steep (sharp) is the "border" between clipped and non-clipped region is, or, in other words, how smooth transition will be. Difference is obvious. If it will be improved in the next version of PI - that will be great!
Also, effect of "background interpolation" is visible on that screenshot. I don't know the way how to correctly normalize background in PI especially in the case where some images have wide "black" borders.

1) Dynamic spline interpolation of background for each image - this is what exactly will help you in your fight with gradient due to cirruses

Hmm, I tend to disagree with this one. Image integration is actually a preprocessing task, as I see it, and hence I don't want too fancy things done at this stage. I want to ensure that the original data, including their linearity, are strictly preserved, and only apply simple operations with a sound statistical basis. The idea of 'interpolating the background' doesn't seem to fit well in this scheme, but this is of course my personal point of view, without knowing what this tool is doing exactly, so I may be wrong here.

Well, I would agree with what you've said, but in case when you have some "exclusive" data that you will not be able to re-shoot (and the comet's shots are of this kind for sure) then any trick that will give acceptable results will work. And this trick, while not be scientifically perfect,  works fine allowing to use "partially ruined data".

The question that is not addressed in FitStacker is how to consider FWHM differences in construction of the "optimum sum"? And here PI with all that "multiscale wavelets processing" can really shine if some appropriate optimization will be proposed...

If you're interested, I will FTP'ed downscaled archived aligned subsets of the "grand sum" together with integration results for them obtained in FS so that you will be able to yesy and fine tune Integration on that set also.

[twade]

To all,

I gave FITStacker9 a shot.  It did improve things a little, but still was unable to remove a lot of the stars.  I guess I'm going to have to give up on this image.  I hate not being able to process this image, especially given the fact I ended up with some pretty good data.  It's mostly my fault as to why I can't remove the stars.  While checking over my calculations (after the fact), I forgot to add a padding of two pixels.  This would have been an extra two minutes for each cycle.  However, looking at the image, I am surprised most of the stars can't be removed given the spacing between each sequence.  I do realize there is no hope for the brighter stars, but the fainter stars seem like they would have enough spacing.  Here's an integrated image with no rejection.

http://www.northwest-landscapes.com/images/comets/2009R1_06182010_lum_no_reject.jpg

The color data is fairing a little better, but it was shot at 3x3 binning -- pretty useless without the luminance data shot at 1x1.

Is anybody interested in "playing" with the data?

Wade

[Juan Conejero]

Hi Wade,

Could I try out your data? I'd like to try with the new version of ImageIntegration, which I've just finished for PI 1.6.1.

[twade]

Juan,

Could I try out your data?

Absolutely!  I look forward to seeing if you can make it work.  Or at the very least, devise a method to remove the remaining stars.

I'll upload the data this evening.  I assume the upload method and password has not changed.  Right?

Thanks,

Wade

[georg.viehoever]

Hi Wade,

attached is a quick experiment using erosion (MorphologicalTransform). It causes heavy artefacts if you look closely, but if you are interested in a large scale picture only, it should work. I have no doubt that using the right parameters with ImageIntegration can give better results, because you'll have a lot more information to play with than with this single image. Let's see what our Chief Magician produces...

Georg

[twade]

Georg,

erosion (MorphologicalTransform)

This might come in handy once most of the stars have been removed.

Wade