PixInsight Forum (historical)

PixInsight => Release Information => Topic started by: vicent_peris on 2017 May 12 02:31:12

Title: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: vicent_peris on 2017 May 12 02:31:12

Hi,

People coming to my workshops already know that we were working on an important update to the ImageIntegration tool to allow a better rejection of meteors and other "track-like" outliers. Well, I'm happy to announce that we finally finished the development of this improvement to this critical tool. I post here some examples.

Below you can see a comparison between the regular integration and the integration with the new improved tool. It's a 15 frame integration using Linear Fit Clipping with the same settings in both cases. This data set has originally 75 frames, but I manually selected all the frames where there was a meteor:

(http://forum-images.pixinsight.com/20170512/M81-1.jpg)

(http://forum-images.pixinsight.com/20170512/M81-2.jpg)
Image provided by Frank Willburn.

This update is able to handle the rejection of big planes:

(http://forum-images.pixinsight.com/20170512/M51-4.jpg)

And perfectly reject RBI artifacts:

(http://forum-images.pixinsight.com/20170512/M51-3.jpg)

Below you can see the comparison between the current version of ImageIntegration and the new one coming in a few weeks:

(http://forum-images.pixinsight.com/20170512/M51-1.jpg)

(http://forum-images.pixinsight.com/20170512/M51-2.jpg)
Image provided by Frank Willburn.
This new feature also helps rejecting stars in a comet-aligned image set:

(http://forum-images.pixinsight.com/20170512/ISON-1.jpg)
(http://forum-images.pixinsight.com/20170512/ISON-2.jpg)

Of course, the rejection of the stars depend greatly on the speed of the comet and the optical system, but will be always better than in the current version of the tool.

One advantage of this new feature is that it is a completely automated solution, so it doesn't require any manual rejection of image bands. We hope you'll like this tool and the new ones that are coming in PixInsight 1.8.5. As always, our goal is to provide you a state-of-the-art solution for your imaging.


Best regards,
Vicent.

Title: Re: New in PixInsight 1.8.5: Large-scale pixel rejection
Post by: Juan Conejero on 2017 May 12 03:56:42

Here is a side-by-side comparison to show the typical performance of the new large-scale pixel rejection algorithm:

(http://forum-images.pixinsight.com/20170512/ls_rejection_comparison-tn.jpg) (http://forum-images.pixinsight.com/20170512/ls_rejection_comparison.jpg)

Click on the image for a full size version (http://forum-images.pixinsight.com/20170512/ls_rejection_comparison.jpg)

On the left, the result of a normal integration of 10 calibrated, demosaiced and registered DSLR raw frames, using Winsorized sigma clipping rejection with clipping points set to 5 and 4 sigmas, respectively for low and high clipping. On the right, the same integration with the new large-scale pixel rejection algorithm enabled. The plane trail has been completely rejected. It can be rejected equally well even if the high clipping point is raised up to 5 sigmas in this example, which I have selected because I think it's quite representative of this kind of problems in typical data sets.

With large-scale pixel rejection, now we can overcome virtually all problems caused by large and bright spurious objects with high uncertainty borders, such as plane and satellite trails and flashes, meteors, stars on sky flat frames, RBI artifacts, etc. As Vicent has shown, integrated comet images can also benefit from a much better rejection of trailed stars. The algorithm works the same way for high and low outlier structures, so it can also reject large dark artifacts as well. Since large-scale rejection works in a completely automatic way (basically, you only have to enable it), getting the most out of your data sets can now be easier than ever.

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: andreasmax on 2017 May 12 04:54:20

very good news! cannot wait to check out this feature on my comet frames.

thx for this!

andreas

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: Ignacio on 2017 May 12 09:24:31

Great news, thanks!

Is there a cost to this implementation? Say, in terms of final general noise? I ask because of the optionality of enabling this feature.

Ignacio

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: vicent_peris on 2017 May 12 09:47:24

Hi Ignacio,

You should expect a signal to noise increase since you'll always need a more aggressive rejection to reject those outliers without large-scale rejection.

Best regards,
Vicent.

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: Juan Conejero on 2017 May 12 09:47:35

No cost at all. Actually, it's just the opposite. Since large-scale pixel rejection is so efficient to reject large artifacts, one can use considerably higher clipping points, just as if the plane and satellite trails, meteors, etc. were not present. The end result is higher SNR, much better rejection of outliers, and easier use of the ImageIntegration tool.

I am considering the possibility to enable large-scale rejection by default, at least for bright structures.

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: Ignacio on 2017 May 12 10:00:47

Understood, thxs!

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: mmirot on 2017 June 11 06:50:17

Do you think the RBI removal is as good as using the flash camera feature?

There is the obvious smear of bright pixels around stars etc. That seams to correct well.

What about the smearing other parts of the image,  not so bright structures?

You think it will recover some signal to noise in these regions? 



Max

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: vicent_peris on 2017 June 16 15:48:19

Hi!

I just wanted to show one experiment we did today with data by Rick Stevenson. We are really happy with the results, so I thought it's maybe worth to share it here. It's a very difficult example because there's literally a "highway" of satellites in the image. With the new large-scale pixel rejection we were able to completely reject all those trails with minimal noise increase.

Attached you'll find an animation without any rejection and with the large-scale pixel rejection.

Hope you like it!
Best regards,
Vicent.

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: ChoJin on 2017 June 17 01:04:38

that's really impressive. Please release 1.8.5 asap, I'm currently processing some data which would greatly benefit from this :)

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: mmirot on 2017 June 17 18:07:16

Yes Juan,

Get this one out. Too cool.

" Sound of fingers tapping on desk"

Max

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: joelkuiper on 2017 August 03 04:40:10

Looking forward to this! One thing I noticed is that sometimes the clipping params were decent for normal ImageIntegration, but not when subsequently doing upsampling with DrizzleIntegration, will this new setting integrate with the Drizzle workflow?

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: OldSkyEyes on 2017 August 04 01:59:12

Congrats on the new update!

Does this automatically work in the current BatchPreprocessing Script?

Thanks

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: dougggg on 2017 August 04 13:43:47

I just downloaded 1.8.5. I'm trying to remove a satellite trail from an image.

Is there a quick start user guide for large scale rejection? I'm trying to use it in ImageIntegration but it doesn't seem to have any effect on an image stack containing 4 images with 1 image with a satellite trail. Layers/Growth are set to 3 and 4, reject high large-scale structures is clicked.

Doug

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: javajunkie2121 on 2017 August 04 16:36:33

Quote from: dougggg on 2017 August 04 13:43:47

I just downloaded 1.8.5. I'm trying to remove a satellite trail from an image.

Is there a quick start user guide for large scale rejection? I'm trying to use it in ImageIntegration but it doesn't seem to have any effect on an image stack containing 4 images with 1 image with a satellite trail. Layers/Growth are set to 3 and 4, reject high large-scale structures is clicked.

Doug

I second Doug's request..this is a VERY useful update and will be VERY popular..if someone would please give some guidance on settings that would be wonderful

I'm seeing a better identification of multiple trails in the high_rejection map and a better end integrated image using 12 images (2 with satellite trails) and  using the default 2/2 for layers growth.  Not sure if number of subs is reason it's working better.  I did not change the pixel rejection sig low (4) and high (3) default settings.  I tried it on a 30 sub integration with 4 known bad subs with trails..rejection map looks awesome..but end result looks same..maybe original algorithm was smart enough when there are 30 subs to eliminate trails.

I am unclear though on the role of the "reject low large-scale structures" setting when trying to eliminate bright satellite trails?  I've run the integration with and without it and I'm not detecting a difference?

jeff

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: chris.bailey on 2017 August 05 00:48:16

Jeff

I did a bit of playing with this yesterday. With reject large scale structures enabled you can crank up the sigma low and high (say 6 and 5) and still get the same level of rejection and get some SNR improvement as a result, in fact, with clean individual frames (Cosmetic Correction) you can dispense with the normal rejection altogether. As I tend to image the faint SNR and Sharpless objects this is a significant advantage.

Chris

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: javajunkie2121 on 2017 August 05 08:46:08

Hi Chris:  I tried this with 12 blue subs of M51, 2 of which had star trails. I left the layers/growth values for high large scale structure pixel rejection at defaults of 2

..as soon as I start increasing the pixel rejection sigma low/high values from default 4/3(windsorized sigma clip), I get star trails returning?

also I tried adding "low large scale structure" rejection to high large scale structure rejection with sigma low/high back at defaults...compared to high structure rejection alone I think I'm seeing some loss of detail in the galaxy?  Is the low large scale structure intended for other targets?

jeff

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: ngc1535 on 2017 August 06 14:27:24

JavaJunkie....

Could this be a method of rejecting dust donuts (due to flat errors) just like what I was looking for?

-adam

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: pfile on 2017 August 06 15:19:54

indeed it should adam, and i am remiss because i had trouble rejecting dust spots during the beta period and i flaked out on providing the sample data to juan. i can't tell if my trouble was due to misconfiguration of large scale rejection or not. i need to follow up with juan on that.

rob

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: chris.bailey on 2017 August 07 00:42:32

Quote from: javajunkie2121 on 2017 August 05 08:46:08

..as soon as I start increasing the pixel rejection sigma low/high values from default 4/3(windsorized sigma clip), I get star trails returning?

Jeff

Is you data dithered in acquisition?

Chris

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: javajunkie2121 on 2017 August 07 07:30:47

yes Chris...was doing a medium dither via PHD2/SGPro with subs

jeff

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: chris.bailey on 2017 August 07 08:01:38

Jeff

I'm stumped then as the half dozen data sets I have thrown at it have shown similar or better rejection with significantly higher Sigma values and shown a worthwhile improvement in end result S/N.

Chris

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: mmirot on 2017 August 07 09:08:38

I was able to do a better job with satellite trails than in the past with 9 subs. 

This still required a sigma high of around 2.5 -2.9  using a Winsorized sigma set. 


I am using the large scale high settings of 2 for scale and 4 or 5 for growth. Only small traces on trails remain now remain.

These are best setting that I could see working with my images.  I don't know if this is truly optimal or typical.

Large scale low did not seem to have any impact.
I suspect Large scale low won't be usefully for satellite trails, RBI artifacts and cosmetic rays. These are bright structures.

I had been using Windsorized sigma high of 3.5 - 4.5.
However, I was also removing bright trails from the stack to use 3.5 - 4.5 too. 

Max

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: ddahlstrom on 2017 September 15 13:54:53

I was having difficulty figuring this out.  I have an image with 7 600s DSLR subs, one having a satellite trail.  Turning on Large-Scale Pixel Rejection (reject high) at the defaults of 2 and 2 seemed to do nothing, and raising the values higher yielded no improvement.  I eventually found the secret sauce for this image to be lowering layers (high) to 1 and increasing growth (high) to 7.  This, at last (and with default sigmas), completely eliminated the satellite trail in the integrations.   

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: lucchett on 2017 October 05 08:35:55

Hi, I read the thread and from my understanding activating Large scale rejection seems always beneficial.
Am I right that I should consider this a default option for any Image?

Thank you,
Andrea

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: jkmorse on 2017 October 19 16:08:57

Juan & Vicent,

I am just starting to scratch the surface on the new 1.8.5 tools, but this one is magical in its results.  I am going back to redo old processing projects and am amazed at the improvement generated by large-scale pixel rejection when I compare the old results with the new.  It just about eliminates the distracting blotchiness in my backgrounds and otherwise makes for a far cleaner image, with no discernable damage to good signal.  And the high and low rejection maps are things of beauty in showing the problem areas that have been fixed.

Just had to say thanks,

Jim

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: Kerrywnm1 on 2019 February 01 11:14:13

All,

I am not having that great of results using ImageIntegration and large-scale pixel rejection to remove the satellite streak superhighway in my images. I have posted over on "Image Processing Challenges"  (https://pixinsight.com/forum/index.php?topic=13238.msg80235#msg80235) the info and a link to my data. Maybe some of you could please take a look and tell me what I can do?

Thanks,
Kerry

Title: Re: New in PixInsight 1.8.5: new large-scale pixel rejection
Post by: pfile on 2019 February 01 11:22:48

hi, as i said over there the best current technique is to paint over the satellite trails with white or black lines using pixelmath, and then normal pixel rejection will take care of it.

rob