deconvolution is very touchy and can take a lot of trial and error to get right.
luckily now the DynamicPSF (
http://pixinsight.com/forum/index.php?topic=3261.0) process exists and can be used to synthesize a PSF representative of your image from averaging several star profiles. that's very useful to have as a proper starting point for the PSF. before that it took some work to find the right PSF.
there's a thread on the forum called "deconvolution deringing wrecks images?" (
http://pixinsight.com/forum/index.php?topic=2727.0) which contains a lot of good knowledge on deconvolution. juan pretty much did a tutorial in that thread.
to get a good result, you need to turn on deringing, both global and local. the local deringing takes a star mask. i think it's okay to only cover the brightest stars in that mask. note that the local deringing image is not used as a mask per-se, but as a hint about how to apply the deringing itself. as for the global deringing, getting the bright-side and dark-side parameters for that can take a bit of work. you can try with a ridiculously large coefficient and see what it does to the image. you need to keep backing this off until you are not getting gibbs effects and also not making a mess of the image - too much deringing can really screw things up. sometimes my global dark deringing amounts are 0.05 or less. usually i dont have to mess with the global bright slider. sometimes i'll do 50-100 iterations of RRL deringing, but other times this is too much and i end up at 20 iterations or so. i have not really figured that out yet. anyway i think there are a lot of details on this in the thread mentioned above.
of course you should have a luminance mask to protect the background. what i usually do is extract the luminance from the linear image, and then stretch that pretty strongly, clipping the blacks. i then remove the first 4-5 atrous scales to soften it up a little bit. you can play with the mask brightness to try to protect more/less and see what you get.
usually toward the end of processing i'll use the same luminance mask (or another one that protects more of the fainter parts of the nebula) and do atrous sharpening on the first 2-3 dyadic scales. recently i've been using the atrous tool to apply noise reduction on the background by inverting the sharpening mask.
but i am by no means an expert on this stuff and understand very little of the theory.
