Deconvolution gives dramatically different outputs from series of frames

gvasseur

Member
Sep 14, 2014
14
2
Greetings,

I'm a bit at a loss after 2 days of experimentation. I had 4 short nights of Ha on sh2-275 (Rosette). When I process the first 2 nights, my master can be nicely deconvoluted with 60 iterations). When I add the 2 next nights, it doesn't work properly anymore, rendering "fractal-like" patterns.

If that's related, I had to mitigate a lot of high rejection issues by doing processing separately for each night. Doing it for the 4 nights at once would create a massive amount of rejects (I would see the Rosette clearly with the rejection pixels).

I posted the 2 masters here (https://drive.google.com/drive/folders/1MHd9Qms71_qEWpnZTeJkTRrM4h-IqSWa?usp=sharing): one works (the Sh2-275-Ok-Deconvolution.xsif) and the other one doesn't.

I would like to understand what I am missing to avoid these issues in the future.

Many thanks for your support!

G
 

pfile

PTeam Member
Nov 23, 2009
5,325
64
what are your exact deconvolution settings? saving a process icon, zipping it and attaching it here would be the best option.

what you are reporting is not totally surprising; every image is different and the decon settings may need to be tweaked even if it is the same object. the statistical properties of the image change when you add subs and that can necessitate tweaks to decon.

btw, 60 iterations sounds like a lot.
 

gvasseur

Member
Sep 14, 2014
14
2
Greetings "hvb" and "pfile"!

Thank you for getting back so quickly. Here are the before and after.
- I use a mask (not a range mask)
- a starmask
- a PSF model generated by the psfimage script

I guess a lot of things can go wrong.

I gave it more thought and realize that it could come from the pixel rejection during the integration. I attached a stretched screenshot too. I'm not sure how to mitigate that properly. I use the Weightbatchprocessing and I only get limited rejects if I do 1 process for each night.

Screen Shot 2020-04-04 at 3.40.30 PM.jpg




Screen Shot 2020-04-04 at 3.33.33 PM.jpg

Screen Shot 2020-04-04 at 3.34.04 PM.jpg
 

pfile

PTeam Member
Nov 23, 2009
5,325
64
that 2nd screenshot looks to me a little like the global dark deringing is too aggressive, if those rough looking patches are what you are talking about.

btw, one thing i've done instead of a mask is to increase the noise thresholds for the first 3 scales - like maybe 8, 5, 3 instead of the default 3,2,1.

as for the rejection you might want to blink thru your frames and see if there is one that is just super bright. actually, you can tell how many frames were bad by zooming in on the rejection image and looking at the pixel values. for instance if you have 10 subs, then the values of the pixels in the rejection map are going to only have the values 0.0, 0.1, 0.2, 0.3, etc. 0.1 means that pixel was rejected from only one image (1/10), 0.2 means that pixel was rejected in 2 images (2/10), etc. so depending on the value you'll know how many bad frames to look for.

in this case since you see structure in the high rejection map, it means either your rejection parameters were set too high, or one or more images are super bright and for some reason could not be normalized properly.
 

gvasseur

Member
Sep 14, 2014
14
2
Thank pfile!

That was fast! I'm grateful for your patience.

I re-run a batch process, double-checking the level of my Cosmetic Corrections just in case. There are so many hot rejects.

I have 19 subs of 10 minutes. Sounds like the readout is 0.3 - 0.4. Does this mean 2 frames were too high?

I'll rerun a more aggressive selection with blink.


Screen Shot 2020-04-04 at 4.23.30 PM.jpg

Screen Shot 2020-04-04 at 4.27.58 PM.jpg
 

gvasseur

Member
Sep 14, 2014
14
2
Ok, so I re-ran 2 tests:

1 after removing 4 frames that were a bit lighter: no success
1 with more tolerance on the rejection (bumped it up from 3.0 to 7.0 in the batch processing)

I still get quite some rejects.

Here are my batch processing parameters.

Screen Shot 2020-04-04 at 4.46.47 PM.jpg

Screen Shot 2020-04-04 at 4.46.57 PM.jpg

I don't recall running into such problems with previous versions of PI. I haven't imaged since November and back then, with the same process icons, I was doing ok.


Screen Shot 2020-04-04 at 4.42.45 PM.jpg
 

pfile

PTeam Member
Nov 23, 2009
5,325
64
i think this is just typical of light pollution - there are some pretty strong gradients in your images. are you calibrating these with flats and darks at all or just integrating the raw lights? it would definitely improve matters to calibrate your images.

probably LocalNormalization can help with this problem but first you need to come up with a clean master by integrating these subs and not worrying too much about what's getting rejected, and do a very careful DBE on the master. then use that master as an LN reference for the subframes, taking care to tune the scale parameter.

anyway i'm not sure the rejection issues are necessarily related to your deconvolution troubles, so using LocalNormalization to attack the gradients is probably not strictly necessary.

rob
 

gvasseur

Member
Sep 14, 2014
14
2
Yes, I do calibrate with bias, darks and flats (30,000 ADU). These do help a lot.

Thank you so much for the help Rob. I'll work on what you described (I need to digest it, this is a bit next level for me ;). I'll get back to you tomorrow with hopefully some good news.

Never a dull moment on this never-ending learning curve!
 

pfile

PTeam Member
Nov 23, 2009
5,325
64
ok i ask because you posted only the original files from the camera and i didn't think it made much sense for me to work on those images since the results would be different from what you are getting if you were using calibrated images.

yeah there's a lot to learn here...
 

gvasseur

Member
Sep 14, 2014
14
2
Wooooot! it worked so nicely with LN!!! That will become my Go-To integration process. A huge step for me!

Thank you so much Rob!

Now that I have a really clean image, I'll work on the deconvolution issue. I still get this, no matter how many iterations I do.

May I bother you with one last hint on where to look?

Screen Shot 2020-04-05 at 11.04.44 AM.jpg
 

pfile

PTeam Member
Nov 23, 2009
5,325
64
ok you just need to be careful with LN because it can produce its own artifacts. notably you may get halos around stars and bright features if the scale is set too high in LN. but of course the higher the scale, the more and more LN degenerates to full-frame normalization. anyway if the scale you chose (or the default) worked OK on this image then nothing to worry about for now.

keep decreasing the global dark until the weird worms go away. maybe go down to 0.01 and see what happens, then if that is OK you can go up until you see artifacts again. also note that what happens on a preview with deconvolution might not reflect what happens on the full image just due to the way the algorithm works. so even if it works out OK on the preview it may not be OK on the full image.

rob
 

pfile

PTeam Member
Nov 23, 2009
5,325
64
I uploaded the calibrated and registered frames here if you have a minute to "Blink" through them.


Not sure to know when a brighter frame is brighter.
well you can load them into SubFrameSelector and look at the median value. i looked at your uncalibrated subs and they were not terrible, no huge outliers, which is why i concluded you're just looking at strong LP gradients rather than clouds or a frame that was spoiled by the moon.

rob
 

gvasseur

Member
Sep 14, 2014
14
2
Oh wow, now I understand! All this time I was looking at the preview while the full-frame was working.

I worked on the full-frame and got nice results - see screenshot (It was still only 190 minutes of light). I guess I'm going to revisit many of last year's processing.

Thank you also for the advice on the global dark! I doubled checked if there was any artifact after the LN, but I seem to have been spared. I'll keep the point in mind.

Screen Shot 2020-04-05 at 12.40.43 PM.jpg
 
  • Like
Reactions: pfile

gvasseur

Member
Sep 14, 2014
14
2
Once again, thank you Rob for all your help. I made so much progress.

My Rosette was only 3 hours of data and thanks to you, I could get something out of it.

Sh2-275-f-small.jpg
 
  • Like
Reactions: pfile