Pixel replacement for saturated areas.

[Jack Harvey]

For my last image I used a process I am familiar with but not smart enough to duplicate in PixInsight.  The exposures of 900 sec on Orion left the trapezium burned out.  SO anticipating this I had taken some 120 sec exposures i all three filters.  After obtain the master frames of both the 900 and 120sec for each channel I did a "Missing Values" pixel replacement in CCDStack.  The process is as follows:

Open  a Long duration and a short duration Master of one channel
Normalize the long duration to the short duration
Set a limit to reject all pixels above a certain value (i.e. 40,000) and execute on the Long duration
Grow the rejection by 2-6 pixels
Now execute what they call Replace Missing Values
Set the image weight of the Short duration to 0.00001
Combine the two images with a Mean or Average

The result is you replace the saturated (above 40,000) pixels of the Long duration image wit the unsaturated short duration pixels for the saturated areas only?

I suspect this can be done in PixInsight and if it does not take a huge amount of math statements in Pixel Math I would be interested (?script)

Thanks   Jack

[Nocturnal]

Hey Jack,

I think Vicent wrote the HDRComposition script which does what you want. I used it for M42 last year. Haven't tried it with this years data yet.

[mmirot]

It would be nice to be able to do this. It might work with other things, blooming etc.


Max

[Jack Harvey]

That would be great Sander.  Unfortunately I have never gotten the script to complete.  It always crashes after a period of time.  But will try again with renewed interest

[Nocturnal]

Hi Jack, I will give it another try again soon and let you know how it works out.

[Simon Hicks]

It always crashes on me as well.....well not a crash but an 'out of memory' fail. It does all the image manipulation and masking in 64bits I think. This is mathematically superb and great if you have a 64bit OS with 16GB RAM....but unfortuinately a lot of us don't. I therefore wonder if it would work more universally if the script had a 'Work in 32bit' option for us mere mortals who have to use their work laptop. 

Someone will now tell me that this will degrade my data......but I just want to be able to use the script.

[Niall Saunders]

It's funny, but I tried to do something 'similar' last night - I had a three-sub data set of IC434, calibrated with slightly temperature mis-matched darks, which left the stacked frame covered in nasty 'black spots'

In AIP4WIN, I could use an implementation of 'Threshold' or 'MinMax' to replace all ADU values 'below' some threshold with a single ADU 'of' some user-specified value.

But, I wanted to be able to do this in PI, and felt that PixelMath would make it a simple task.

However, maybe my brain was just fried, but I couldn't come up with a 'simple' expression in PixelMath to achieve this, and felt that going down the PJSR route was overly complex - inevitably ending up with some algorithm that would replace the 'black' spots with some algorithmically (automagically) statistical 'best fit'.

Altogether a horribly over-engineered solution - where the actual 'best' solution was to simply delete the 3 ten-minute subs, and to wait (months  ) for another clear night, and to capture the data properly, INCLUDING a decent set of temperature-matched Darks (not just whatever I had lying around that was 'close enough'  )

It seems that a set of MaxMin, Threshold (etc.) type tools would come in handy, or a brief tutorial to remind/show us how to accomplish it in PixelMath.

Any takers?

[Juan Conejero]

Hi Niall,

replace all ADU values 'below' some threshold with a single ADU 'of' some user-specified value.

Easy. Enter this expression in the RGB/K slot:

Code: [Select]

iif( $T < low, fix_value, $T )
and this in the Symbols slot:

Code: [Select]

low=0.025, fix_value=0.05
where the numbers are just examples; replace them with your actual values.

Don't forget to disable the Rescale option if you don't want to rescale the image to the [0,1] range at the end of the process.

If you feel more comfortable working with integer sample values in the native output range of your camera (what you call ADU):

Code: [Select]

iif( $T < low/range, fix_value/range, $T )

Code: [Select]

low=1638, fix_value=3277, range=65535
If your images are RGB and you want different replacements for each channel, you'll have to disable the "use a single expression" option and enter different expressions in the R, G and B slots.

Hope this helps.

[Juan Conejero]

Hi Jack,

Before continuing, I must say that our HDRComposition script is light years above that procedure in sophistication and accuracy. Yes, HDRComposition eats memory and runs extremely well on a 64-bit version of PixInsight, not so well on 32-bit versions with less than 4 GB of RAM. When we implement it as a regular process most of these memory problems will be overcome.

Open  a Long duration and a short duration Master of one channel

File > Open

Let's rename the images as L and S for the sake of simplicity in PixelMath expressions.

Normalize the long duration to the short duration

Normalization is a multipurpose word that can refer to many different things. In this context, normalization could be done in this way:

Code: [Select]

(L - Med(L)) * AvgDev(S)/AvgDev(L) + Med(S)
applied to L with the rescale option disabled. This expression will do two things:

- Scale the long exposure to adopt the average signal levels of the short exposure.
- Replace the mean background of the long exposure with the mean background of the short exposure.

There are other ways to do this, but I personally tend to favor average deviation (the AvgDev function) as a robust estimator of dispersion. Of course, your images must be in either a floating point format or in 32-bit integer format, or you'll probably lose some data.

Set a limit to reject all pixels above a certain value (i.e. 40,000) and execute on the Long duration
Grow the rejection by 2-6 pixels
Now execute what they call Replace Missing Values

The "Grow the rejection" part involves building a mask and applying a dilation filter. This would be quite easy to do, but I don't understand why growing a rejection mask is of any use here, and I in fact can see a lot of problems and very few advantages by doing so (but I may be missing something here). Ignoring this part, your task would be simply:

Code: [Select]

iif( L > 40000/65535, S, L )
applied to L with rescale enabled. This expression simply replaces L with S where L is above 40000 in the 16-bit range. Since we have previously scaled L to have the same signal levels and background as S, both images are statistically compatible. The final rescaling is useful here to ensure maximum usage of the available numerical range.

Set the image weight of the Short duration to 0.00001

I think this makes no sense if you perform the combination with a PixelMath expression as above.

Combine the two images with a Mean or Average

Better than this would be an adaptive combination. This can be done very easily with a variant of the above PixelMath expression:

Code: [Select]

iif( L > 40000/65535, L*S + (1 - L)*L, L )
There are other variants that we can explore, for example a weighted combination. But first let's see if any of these can work out something useful

[Jack Harvey]

First steps looked promising but last step failed?
PixelMath: Processing view: Orion_Blue_Master
Writing swap files...
48.02 MB/s
*** Error: Unknown function identifier: iff(Orion_Blue_Master>40000/65535,Orion_Blue_Master*Orion_Blue_Master_120sec+(1-Orion_Blue_Master)*Orion_Blue_Master,Orion_Blue_Master)
Reading swap files...
566.85 MB/s
<* failed *>

[Juan Conejero]

iff(Orion_Blue_Master>40000/65535, ...

Just a typo: instead of iff, try with iif (inline if).

[Niall Saunders]

Thanks for that example Juan - I am sure that I managed to achieve this once before - but the other night my brain just wasn't up to the task  

Cheers,

[Jack Harvey]

I got this to run but it left a well delineated darker patch where the replacement occurred (trapezium on Orion).  SO did not work like I hope :-((

[Niall Saunders]

it left a well delineated darker patch

And this, of course, is where the 'feathered mask' approach available in PS has the advantage.

The 'simplistic' (though 'sophisticated') layering that this instance of PixelMath provides, is based on a very 'hard' transition point - namely the threshold value of 40000 (in the example given).

If the 'decision to replace' had an element of 'fuzziness' (akin to the 'Feathering' option in PS) then the hard edge would be blurred. The same applies to the Star Alignment & Mosaic process shown in Juan's recent tutorials.

We may yet have to wait for the next stage of development from Juan - although I still feel that a combination of PixelMath and PJSR might yet offer an interim solution.

Cheers,

[Simon Hicks]

I've done this before with a mask....its easy enough to do as long as you only have two images.

Start with Juan's scaling expression....i.e. reduce the illumination of the brighter image down to that of the dimmer image. Then create a mask that shows what you want from one of the images and cuts out all the rest....and have nice sharp edges on the mask where you want them and nice fluffy edges where you want them.....so the mask might be a complicated combination of star and nebula masks for instance.....but you guys know how to make masks!!!

Then remember that the mask will have values from 0 (black) to 1 (white).....so you can use the mask in the PixelMath expression. I can't for the life of me remember exactly what I used, but it was something that said 'use the dim image where mask = 0 and the bright image where mask = 1 and use the weighted combination for intermediate values'. I can work out the expression tonight....but I'm sure one of you will be able to do it quicker!