New Tool: DynamicPSF

[RBA]

By the way, congratulations for a long-awaited tool. So far the implementation looks impeccable to me, and I can see it as a great foundation for other more high-level tools developed on top of it.

I do have some questions but I'll hold until the documentation is available, as most of the questions will probably be answered there.

Again, congratulations, and THANKS!

[DarrenS]

Excellent work Pix team,  8)

Ran the dynamic_psf with deconV on a couple of images last night, really impressive results.

Range selection tool works well too. Made a mask for the above Decon very quick.

Darren

   
 

[Jordi Gallego]

Congratulations and thank you very much for this great tool Juan
It was really needed and indeed looks very, very promising.

Regards
Jordi

[sleshin]

Juan,

Thanks for taking the time to explain MAD. I do believe I understand. Just don't spring a quiz on me.

Steve

[astropixel]

But I will suss it out with the tried and trusted harry method , consisting of fumbling around in the dark till I find the light switch 

Happy to know that I'm not alone I got the bit about a new tool - after that

[RBA]

But I will suss it out with the tried and trusted harry method , consisting of fumbling around in the dark till I find the light switch 

Happy to know that I'm not alone I got the bit about a new tool - after that

Don't get dizzy with the equations. I would expect that the documentation being written will also explain things in a more empirical way, so in the end it's all about understanding the suitability for the different functions being offered, the "art" Juan described when selecting candidate stars, and of course, the values returned by DPSF. Most of the data DSPF returns is not that hard to understand, and for those values that might be, take your time to read Juan's descriptions.

Many of us "aesthetic astroimagers" don't rely on many of these measurements for what we do, but eventually, it's a good call to start looking at the data we're capturing in a different way, and eventually use that knowledge to our advantage for "what we do". The way I see it, and after using it for a bit, DPSF is probably the most enticing tool you can find to do just that.

[Howard]

Dear Juan:

I am a little late to the DynamicPSF party, but hope I can get in a few questions about the algorithm, in addition to exclaiming a huge "Thank You!" for another extraordinary PixInsight tool !

DynamicPSF returned two clusters of fit angles in my first trial, what looks to be a situation somewhat similar to the data in Philippe B.'s very helpful example. The clustering in my case is somewhat tighter, within about 5 degrees of zero+, and within about 5 degrees of 180-. You suggested to Philippe that he discard the stars with small angles, but could this be indicative of the degeneracy associated with an ellipse whose major axis is close to zero degrees? This seems likely to happen for example when the ellipticity is produced by random guiding errors of different amplitude in RA and Dec, assuming that the camera is similarly oriented (which happened to me with the frames I'm working on now, shot on two very windy nights). Does this seem like a reasonable expectation? Is there a way to force theta=0 in the fit?

Two followup questions: when I regenerated the fit after Deconvolution, the angles for individual stars seemed fairly randomly distributed from zero to 180 (this may also be the case in Philippe's example). Is this a sign that the PSF provided the Deconvolution with the necessary kernel to reduce the ellipticity in the initial data? On the other hand, I was surprised that the aspect ratios were not significantly improved. Perhaps I have not used one of the tools correctly, or am incorrectly interpreting the results?

Finally, regarding the regeneration of the parameters of the fit: for a fixed image, the parameters tend to wander around a bit with repeated regeneration. I gather this reflects the inevitable "soft landing" in the iterative Levenberg-Marquardt method (though very slight with this data)? Do you have any recommendations for appropriate use of regeneration in this context?

Many thanks,
Howard.

[Juan Conejero]

Hi Howard,

Thank you.

DynamicPSF returned two clusters of fit angles in my first trial, what looks to be a situation somewhat similar to the data in Philippe B.'s very helpful example. The clustering in my case is somewhat tighter, within about 5 degrees of zero+, and within about 5 degrees of 180-. You suggested to Philippe that he discard the stars with small angles, but could this be indicative of the degeneracy associated with an ellipse whose major axis is close to zero degrees? This seems likely to happen for example when the ellipticity is produced by random guiding errors of different amplitude in RA and Dec, assuming that the camera is similarly oriented (which happened to me with the frames I'm working on now, shot on two very windy nights). Does this seem like a reasonable expectation? Is there a way to force theta=0 in the fit?

It seems pretty reasonable to me. You can force circular functions by checking the Circular PSF check box on the PSF Model Functions section.

If I've understood it well, your case doesn't seem similar to Philippe's example. In his example he was getting a quite coherent set of measurements except a few ones that were clearly outliers (nonstellar objects, probably). In your case it seems that you're getting two sets as a result of uncertainty in the fitted rotation angles. Are you getting aspect ratio values very close to one? When the stars are nearly circular, fitting elliptical PSF functions becomes an unstable process and the fitted rotation angles tend to be meaningless. This situation can get much worse for low SNR data (noisy raw frames for example). You know that this is happening when the fitted rotation angles are disperse and don't follow any spatial distribution pattern. In these cases circular functions are more meaningful and tend to be more accurate.

when I regenerated the fit after Deconvolution, the angles for individual stars seemed fairly randomly distributed from zero to 180 (this may also be the case in Philippe's example). Is this a sign that the PSF provided the Deconvolution with the necessary kernel to reduce the ellipticity in the initial data?

I think so. If the angles become randomly distributed this means that the angles are not meaningful at all, and hence that the stars are closer to circular, following the discussion in the preceding paragraphs.

On the other hand, I was surprised that the aspect ratios were not significantly improved.

Well, it seems that your stars were already quite circular. The aspect ratio is a relatively uncertain parameter. Rotation angle is more sensitive.

Finally, regarding the regeneration of the parameters of the fit: for a fixed image, the parameters tend to wander around a bit with repeated regeneration. I gather this reflects the inevitable "soft landing" in the iterative Levenberg-Marquardt method (though very slight with this data)? Do you have any recommendations for appropriate use of regeneration in this context?

This happens because the initial set of parameters for a fitted PSF function is slightly dependent on the initial point chosen to detect the star, or search location. For each star, the search location defined when you manually click on the image is not the same, in general, as the location automatically selected by the regeneration routine. For slightly different initial values, the L-M algorithm may choose one among several valid solutions, due to uncertainty in the data (different paths on the solutions space). With high SNR data this is usually unnoticeable, but noisy data are more prone to make these variations noticeable. The variations should always be very small, though. What are the magnitudes of the differences that you are observing? Do the fitted parameters stabilize after the first regeneration?

(9:24 - Edited to clarify the last paragraph)

[Howard]

Hi Juan:

Thank you very much for your very detailed and informative reply.

The stars in my initial image have an average aspect ratio of about 0.88 (I typically get roughly 7% deviation from roundness, but gusty winds played havoc with auto-guiding this time), and the ellipticity in the stars is quite apparent to a visual inspection. The ellipses look very closely aligned with one axis of the frame, corresponding to the Dec axis which had a much larger amplitude of random guiding errors than in RA on those gusty nights. In this case, using the elliptical fitting functions with the constraint theta=0 might be more efficient, though now that I think about it again, imposing that constraint isn't really necessary in actual practice, since one can average over so many stars that one will anyway end up extremely close to the expected orientation. In fact, DynamicPSF returned an average theta=178 degrees using about 20 stars!

I wondered if the fitted aspect ratio might show a noticeable improvement in the deconvolved image using the exported PSF, but the average actually decreased slightly, to 0.83. But as you suggest, this doesn't appear to be as significant a number for the final image, since the stars actually look appreciably more circular than in the original .

About the variations in fit parameters with regeneration, this is surely the sensitivity to initial conditions that you pointed out. The variations are very small, as expected for a good extremum. (Out of curiosity, when regenerating, does the algorithm start from the last estimate of the parameters?)

Many thanks again,
Howard.

[Howard]

Dear Juan:

I hope I can trouble you with one more go at the problem of improving the aspect ratio in Deconvolution using DynamicPSF.

I was wrong that my stars showed an improvement. Their "football" aspect is actually quite severe (due to those very gusty winds), and shows no improvement in either visual appearance or in the fitted parameters after Decon using the exported PSF. In case I've goofed in the use of these tools, or if only a miracle could save this poor data, I hope you might take a look at the two attached screenshots of a small portion of the image, viewed at 200%, before and after the Decon.

The screenshots include a portion of the DynamicPSF dialog, along with the average parameter values. The screenshot after Decon also includes the exported PSF, and a part of the Deconvolution dialog. Please forgive these very busy screenshots, as I tried to pack in as much useful information as possible .

The football shape is readily apparent at this magnification, and one can see that there is essentially no change in the aspect ratio, or the orientation of the ellipses, although the FWHMs do happily decrease by about a factor of 2.

The ellipse angle is consistent with a small rotation of the camera axes relative to the telescope axes, as can be seen from the diffraction spikes of the bright star near the lower left, which makes sense since the gusty winds caused a greater amplitude of random guiding errors in Dec than in RA.

Any advice would be greatly appreciated !

Howard.

[Nocturnal]

Hi

I tested this new function, mainly for PSF extraction

I finally got around to trying the recipe that Phillipe posted. Unfortunately I don't get anywhere near those results. Extracting the PSF is easy enough, thanks for showing how to do that. I used the method that Juan advised we use. Perhaps my starmask is problematic again but I also noticed that the preview size has a -huge- impact on the effect of deconvolution. Same process applied to a larger preview gives a completely different result to a small sub section of that preview. This is surprising to me as deconvolution (little as I understand it) works on the local scale and does not involve extracting wavelet layers which change with the size of the image. Looks like experimenting with previews isn't really possible with deconv.

In any case, I either get a completely snowed out image or one that has larger stars with snowy edges. Neither effect is very desirable.

Still, extracting the PSF is neat. I wish I could use it for something.

[Hans Pleijsier]

Some personal observations on the Dynamic PSF tool:

- a gradient in the frame is not helpful for the dynamic-psf-process. Better loose it in advance by applying bckgrnd-extraction or (better:) flatframes;

- When I deconvolute images with the artificially generated PSF I see relatively fast results with simple starfields. However low-signal frames with non-star-objects in the field require more work; I experience snowy results now and then;

- on my macbook I have to draw a little box around each star and then click the star before dynamic-PSF picks it up; a simple mouseclick on a star is not enough.

- that said, I like this tool a lot because it gives me a <<scientific>> basis to justify my poor image-quality.

gracias muchos.
HP

[Nocturnal]

Hi Hans,

gradients and flattening aren't necessarily related. Even a well flattened image can still have gradients from LP for example. DBE subtraction can not solve flattening issues. I have not tried to use DBE in multiplicative mode, perhaps it works.

I have to say I think both multiplicative (flattening related) and additive (DBE/gradient related) should not affect star statistics. Multiplication will make a star brighter but won't change the shape. Subtraction presumably removes the same values from all pixels in the star unless the gradient is very very very steep in that spot.

I look forward to hearing from Juan about this.

I'm glad to hear you also got the snow effect. Doconv works fine on planetary images for me but for DSO not so much.

[Juan Conejero]

Hi Howard,

The key to evaluate these PSFs is the beta parameter. Note that before deconvolution you had beta values around 6, which is typical for well sampled images. After deconvolution, beta has decreased to 1.6 - 1.8. The beta parameter controls the overall shape of the Moffat fitting function; lower beta values lead to narrower function profiles (leptokurtic distributions).

The following figure shows the difference between two Moffat functions for beta values of 6 and 1.5, respectively.



To generate these plots, use the following script:

Code: [Select]

#include <pjsr/UndoFlag.jsh>

#define MOFFAT_BETA  1.5

var M = Matrix.moffatFilterBySize( 64, MOFFAT_BETA );

var w = new ImageWindow( M.cols, // width px
                         M.rows, // height px
                         1,      // numberOfChannels
                         32,     // bitsPerSample
                         true,   // floatSample
                         false,  // color
                         "Moffat" + format( "_%.1f", MOFFAT_BETA ).replace( '.', '_' ) // identifier
                        );
var v = w.mainView;
v.beginProcess( UndoFlag_NoSwapFile );
v.image.apply( M.toImage() );
v.endProcess();

w.show();
w.zoomToFit();
Change the value of MOFFAT_BETA to render different function shapes. The script generates a new image window where you can apply the 3DPlot script to generate the 3D renditions. Note that you can also define a small preview covering a star in your image and use 3DPlot with it.

So you have achieved narrower star profiles with deconvolution. This result --as long as you don't get objectionable ringing artifacts-- denotes a nice deblurring effect.

Now let's think of the r PSF parameter. The r parameter measures the aspect ratio (sy/sx), or the ellipticity of a star. It's true that aspect ratios have not decreased significantly after deconvolution, as you probably expected. However, take into account that the aspect ratio is a relative measurement. So although their ellipticity has not varied significantly, in absolute terms (in pixels) your stars are much closer to their ideal shapes. After heavily stretching the image in a nonlinear way, star halos may still show slightly elliptic shapes, but this can be fixed with star masks and morphological operations.

[Juan Conejero]

Hans, Sander,

Relatively slight gradients don't degrade DynamicPSF performance, in general. For each star, DPSF computes a mean local background (the B parameter), which is a constant signal level added as a pedestal. The A parameter (peak value) takes this pedestal into account, as do the rest of fitted parameters.

However, a very steep gradient can lead to less accurately fitted PSFs. All fitting functions assume a flat, horizontal base plane and a strong/steep gradient violates this condition. A steep gradient acts like an inclined plane, on top of which the PSF fitting routines have to find average B values and approximate function parameters as a compromise. Note that the same problem arises when fitting a star over a steep 'natural' gradient, such as a nebula. For this reason the best PSF fitting candidates are isolated stars over free sky background areas.

I have not tried to use DBE in multiplicative mode, perhaps it works.

As long as the gradients are multiplicative, It does work very well. An incoming video shows precisely an example of this type of correction. The problems begin when gradients have mixed additive and multiplicative origins, as usual.

- When I deconvolute images with the artificially generated PSF I see relatively fast results with simple starfields. However low-signal frames with non-star-objects in the field require more work; I experience snowy results now and then;

This is normal. Deconvolution only works for high SNR data. Low SNR regions should always be protected with suitable masks. When deconvolving low SNR data, the uncertainty to differentiate between noise and signal can invalidate all premises of the deconvolution process. When this happens you get noise structures enhanced, which is probably what you identify as 'snow'.

- on my macbook I have to draw a little box around each star and then click the star before dynamic-PSF picks it up; a simple mouseclick on a star is not enough.

This is very strange and should not happen. Can you upload one of the images where you're getting this behavior?

- that said, I like this tool a lot because it gives me a <<scientific>> basis to....

I'm glad to read this. With PixInsight I try to provide a scientific approach to image processing. Only through a scientific approach to the data can one grow personally and technically as an astrophotographer. If more people understood this, astrophotography wouldn't be so plagued of the pseudo-magical approaches and retouching tricks that are restraining its development.