Author Topic: Managment of "synthetic luminance" and RGB questions (NOW MOSTLY DBE !!)  (Read 20065 times)

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Managment of "synthetic luminance" and RGB questions
« Reply #15 on: 2010 April 16 08:36:33 »
In fact, K stands for "gray", on grayscale images. If you want to read the luminance along with RGB, you should use RGB+L ;)
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #16 on: 2010 April 16 10:40:10 »
Niall/Carlos
I was refferring to "hovering" over my 3 stacks ie the RG and B...
So these are greyscale images before combining...I was looking at the dark/sky areas....wondered why they were all different...I guess the k value does not mean much here...
I have not actually done much "hovering" over the color (RGB combine)...
What can this hovering tell us..?
I mean is it telling us anything about the quality of the data..?
Am I correct in just "tossing" the RG and B together in color combine..?...as is..out of DSS...?
Is there EVER a time when doing DBE or HDR etc on each sub is a good idea..?

ALSO (while I am here)...there was a suggestion to apply Decon to data BEFORE stretching...but the noise gets pretty bad...even with a Lum mask...
Once I have stretched a bit and applied ACDNR...it goes better...
Is this just a sign of weak data or...?

Niall....I will be shipping my yacht back in a few weeks...You are more than welcome to "crew" your way over...
What size life preserver do you wear.. >:D
Carlos...You are booked on the Queen Mary...May 3,7AM..@ Southhampton.
My chopper can pick you up.
Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Managment of "synthetic luminance" and RGB questions
« Reply #17 on: 2010 April 16 13:37:23 »
Hi Dave

Theorically there should be no difference from applying DBE or HDRW to those grayscale images, since those algorithm work in a per channel basis (unless you use hue preservation, or luminance masks). There are many processes that work that way. The advantage on usign a multichannel, RGB image, is that all the parameters will be the same, and thus it is easier to prevent hue modifications (this happened sometimes with the old SGBNR if you applied different values for each channel). Also, a RGB image allows you to use more sophisticate algorithms that include other color spaces.
One reason to apply a process to a single channel would be to correct optical problems (chromatic aberrations, etc.) or other strange effects. Also you may try that for more efficient noise reductions, but you always have the risk of changing the colors.

Deconvolution "should" be applied always before stretching. The idea behind that process is to recover the "purest" image, before it was degraded by the atmosphere, optical problems, etc. This cannot be done after stretching. Then, deconvolution losses its original purpose and becomes something like a high pass filter (sharpening). For that task other tools may yield better results, like wavelets.
Noise is the big enemy of deconvolution. If there were no noise, recovering the "true" image would be a very simple problem. Since noise for it's nature is random, there is no way of telling if you are sharpening a true edge, or just increasing noise contrast. For that reason very complex models are build to try to determine what is signal and what noise. In PI, in presence of high amounts of noise the best tool is a Richardson-Lucy regularized deconvolution. You'll have to carefully fine tune all the parameters (noise, deringing, etc etc etc). It's quite time consuming, and needs a lot of trial and error processes, but it is worth.

Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #18 on: 2010 April 17 14:03:52 »
Carlos
Thanks
When you say that NR "may be applied to single channels"...you are not recommending it..?
What is the risk of applying DBE to single channels..?
My M101 was a struggle... >:D
When I did the RGB combine...and "neutralized the background"...(which I notice DBE does almost as as well as background neutralization..comments?)...
Anyways..the "corrected" background still has quite a color gradient...where did I go wrong...the stretch shows some blue upper left...down to some green lower right...
And this is going right THROUGH M101...

So its tough to correct in DBE...
I have to use like 400 samples...then I fret I am damaging M101... :'(

So I had the "brilliant" idea of "cleaning up" each channel on its own...
Problem there is that post DBE on each one...they are "quashed" to the left...no room to manouver...
Which I guess is OK...
This seems to work...
Here is RGB with per channel DBE;
http://www.flickr.com/photos/daveh56/4526475770/

And it seems "OK"...

Anyways...maybe I am rambling...well,no.I KNOW I am rambling...
I think what I should do is post the RGB image done the "standard way"...
I will do a pass of DBE...and post it here...to show my struggles :'(

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #19 on: 2010 April 17 14:23:10 »
Here it is...
http://www.flickr.com/photos/daveh56/4528718673/

Nice eh..? >:D

So this is where maybe (now..!!) I am asking about DBE and how best to apply it...
small samples..?
Lots of samples...(I think yes,to this..)
BUT...It looks like my samples will have to "march through" M101...no avoiding this...
I am suspiscious that I am sucking precious juices out of 101 here..... >:D

Comment anyone...
I will play with "color correction" more...
Can this fix a color gradient...?
Thanks to anyone listening...any advice/feedback gratefully considered...

PS Randy Nulman (who started this whole prior "scrum team fiasco"...(of mine) was fairly positive on this image... 8)
(well...the final version...not this !!)

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #20 on: 2010 April 17 14:51:16 »
OK...
(Now I am really talking to myself..I suspect.
I was not doing the "color calibration" routine correctly...(!!!)
Onwards and upwards..!!
Thanks Dave >:D

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #21 on: 2010 April 17 15:24:35 »
Dave
Here is the RGB after CORRECT use of;
1)background neutralization
2)color calibration
3)DBE

http://www.flickr.com/photos/daveh56/4528861817/

Dave >:D

PS is this why the "official position" around here is RGB and no LUM...?
Because I can ONLY use DBE for any gradient issue in the LUM data..correct?
Or does background neutralization work in LUM....?

Gee...I will go and find out....
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Managment of "synthetic luminance" and RGB questions
« Reply #22 on: 2010 April 17 16:17:21 »
Sorry, it was nap time :D

First of all. DBE works in a per-channel basis. There is absolutly no difference on applying it to the RGB composite or to individual channels. In fact, working on the RGB composite allows you to easily determine if there is any gradiens, as the background hue will change.
Now, DBE "flattens" the backgroung. Itself has nothing to do with background color calibration. Why it works that way if you substract the model image is because the model after all is the sky. What I do, to not change the color of it, is to add a pedestal (I use the median of each channel and the correspondent pedestals; the same is true for division, with a multiplying factor).
The reason why I said that independient (external) per channel modification may lead to hue variations is precisely because you don't see how it interacts with the other channels, generating color. It is better to evaluate them all at the same time. DBE is not applied to the luminance, although, if you have a true luminance, you should apply DBE before combining (and also to the RGB data).

Some tips about DBE, from my experience/likeness:
- Use as few samples as you can. Don't overload the image with them. Trust the interpolation algorithm :) Add samples that indicate variations on the gradient, or to fine tune the results, not just in case.
- Try to use big enough samples. I work with 10px to 20px boxes. This helps the statistical rejection algorithm, and makes more meaningful samples.
- In many cases you'll need to iterate. Don't worry if you have to apply DBE again. Just avoid increasing wild changes, or local "contrast". In other words, if you see that the background is worse, go back and place new samples or move the ones around the problematic areas. Remember that DBE is not making an exact interpolation between you samples. It has a smooth parameter that allows a more rigid surface to be fitted to the data. This prevents inaccurate samples to do much damage, but at the same time there may be some residual gradients.
- Avoid at all placing samples over objects, and faint features. Make use of virtual samples from symmetrical properties of the image, or even create samples with fixed value over them, if normal samples cannot do the job alone.


Hope this helps :)
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #23 on: 2010 April 17 16:55:55 »
Carlos
Thanks !

Call the Queen Mary purser and ask for a cabin upgrade,and "the Captain's special"...(wink,wink)... >:D
That is ALL SO helpful !!

But also this use of background neutralization and color calibration...
If I use LOTS of samples...well..it is magic !!
I have not been getting the full benefit due to undersmpling all the problem areas...
Really...brilliant !
I am all excited now...!!
And I will be printing out that DBE advice.

A few more quick questions ( >:D)
(They can wait a while...I am good and happy for a while now...go have a nice dinner,grab some sleep and then get back to me... >:D

1)If I have too high a threshold...is this when I may damage my data..??
Is it better to keep this low and ADD samples manually ?
2)Smoothing...can you expand on this a bit ?
3)Can you comment on "creating samples with a fixed value"...??

I feel there is still a TON in DBE I do not fully understand...
Thanks again for youur kind and prompt reply...
I was mostly kidding about talking to myself...
But I am a blabber...
Here is the latest LRGB using these tools better;
http://www.flickr.com/photos/daveh56/4528936599/
Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Managment of "synthetic luminance" and RGB questions
« Reply #24 on: 2010 April 17 17:49:03 »
1) I usually increase a lot the Threshold. I rather like to include some "bad" pixel than having poor statistical samples. Also, what he thinks to be the expected background value may not match my opinion :)
In 99% of cases I add samples manually. If it were so easy to put automatic samples, I would use ABE :D At least for a first iteration.
2) I don't touch that parameter, but in very rare situations:
a) If you are not sure if the samples are placed over true background sky, or it shows small variations from noise/big stars/etc, it is better to increase a bit the smoothness of the model.
b) If you have few background samples, or there are strong variations, decrease the smoothness to create a more "accurate" model. Also take a look to the weight of the samples.
3) You are able to create a sample and manually set the pixel value. This may be very tricky, but in very difficult cases may be the only way to incorporate a meaningful sample. Before digging into this, try any symmetries that may cover that area.
Trick: Let's assume that you can place samples on a very narrow space, in one quarter of the image (for example, you are following a dust line in the milky way's core). If there is an obvious radial pattern in the gradient, create radial symmetries with different amount of "sides", specially along contiguous samples. For example, use 7 for one of them, and 8 with the other. This will create a richer sample field, that will avoid strange radial patterns. Also you may increase the smoothness.

Another trick: use a VERY aggressive STF to inspect the results. Make sure you are adding the pedestals (and not rescaling) so the mean flux of the sky is preserved. Use previews with PixelMath to quickly inspect the image, "blinking" between the result and the original image.


Yes, there is a lot to learn :) You have to give it time, and experiment with the parameters. This is the only way to gain experience and get the "feeling" where to go next. That intuition is based more on trial/error than theoretical knowledge (that helps too, of course), since nothing is better than "seeing" what actually happens. The same is true with almost every process in PI :)
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Managment of "synthetic luminance" and RGB questions
« Reply #25 on: 2010 April 17 18:27:21 »
Carlos
Must be 2-3 AM in Europe...get some sleep !
Maybe you are imaging..lucky man !!

OK..I am ashamed to admit it,..but this "pedestal" thing has me beat...
I tried to ignore it...but you keep referring back to it... >:D

WHAT is it?
HOW is it "measured"..?
What is it done with it..?
Why is it used..?

Is "pixel math" used/needed?...(remember-I am allergic +++  !)
I have been just requesting a "subtraction"...is this where it is in reference to...?
Oh Oh...I am showing my ignorance...I thought we just subtracted the background and boogied along...all so quick.
I should be taking the image and the background into pixel math somehow..????
NOOOOOOOOOOOOO..... >:D

No,seriously...can you expand on it and how I use it in DBE...
I am 100% lost about it...
PS you can "practice and experiment"....but learing from others is 50% of the trick...
That's why I never stop asking (dumb) questions...
Hope to chat some more tomorrow...

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Carlos
Just went to your web site...and see you are in the America's....well,its still early !!
I assumed you were in Spain...why ?
Lots of time for you to stay up late and answer all my questions !!

Anyways....Sitting here....thinking......insomnia strikes...(and clouds)
Pixel math...pedestal....hmmmm
What CAN it all mean...??

So I should be using PM to subtract the background PLUS or MINUS an extra bit...?
Do I use a fractional value of the background maybe..?
How do I enter that value into the "black art world" that is PM...?

"Summa 3-kj*-=+"...or something frightful like that...you will have to REALLY spell it out for me...

again do I add or subtract it though?...hope all is revealed.
I find just clicking "subtraction"...I am getting MIGHTY dark results...almost clipped a bit,after stretching...maybe this has been costing me faint data...Gadzooks !!
So maybe I should be using this "Pedestal" thingy....

Again I await further details...
Interestingly...WAY back...I used PMath ...in DBE...I know...it feels like years ago...
Maybe this was in the free or trial version...?
I was so dumb, I though this was the only way to do it...
I also was stretching the extracted background (a bit,by trial and error)...and THEN subtracting it...
Maybe I was close to inventing a pedestal of sorts... >:D

Dave
PS I added the only known digital image I have of me to my profile...taken off my Yahoo profile...
a 30KB image...I upsampled it in Pixinsight !!
(and of course updated the gear...)
« Last Edit: 2010 April 17 19:13:20 by dhalliday »
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
It is not unlikelly that I go to bed at those hours, but I live in Chile, and here right now is just 10pm :)

You don't have to worry about the pedestal. Latest versions of DBE included a section, at the bottom, about model application. To simplify, enable the Normalize option (this will calculate and apply the pedestal automatically), discard the model, and apply DBE to the target image. Now, to compare results, just have to undo/redo. Not as fast as with a preview, but helps a lot. You may create duplicates of the image with the best results so far, to compare with other set of parameters/samples.
(I've been here for a while, so sometimes I forget the latest additions :P )

A quick description of what is that famous pedestal: Think a bit on the flat frame calibration scheme. You want to correct the data information from uneven illumination from optical design, dust motes, and also different pixel response. So, you create a "model", by imaging a uniform screen (or by any other method). The result is a image, with pixel values. If you perform a straight division, you modify the flux of the entire image. As a trivial experiment, think that every pixel is divided by 0.25 (no reason for that particular number). It is equivalent to multiplying the image by 4... everything changes. So, if you want to keep the image as closely as you can to the original state, in terms of flux, you must normalize the flat frame. Usually this is done by dividing it by a constant number, like the median of its pixels, or the average. Or, what is the same, dividing the light frame by the "raw" flat, and then multiplying by it's median/average/other value. The same can be said for additive corrections (sky subtraction, from light pollution gradients or other effects). To keep the original flux constant, we have to normalize the model, or add a pedestal, witch is calculated in the same way.
WARNING: the real background has noise, small scale variations and other elements that make it not as smooth as the model. A straight subtraction, without pedestals or rescalations will yield in a heavy data loss. That operation is in some way equivalent to a very aggressive black point setting, at the Histogram, clipping all the data below the maximum of the "data bell".
Back on topic, current DBE's and ABE's implementations do the work for you, so forget all that PM stuff :)


PS: Nice image! That reminds me to upload mine :P Tomorrow... maybe :D
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Carlos
I am using version 01.05.06.0525...
(upgrades give me nite sweats...)
Does normalize "fix my pedestal problem"...??
If so great...!
I am in a heated discussion with some folks who tell me the data has "lost something/looks clipped"...
(T Tuggle)
So I will redo..
And listen...if I upload the RGB masters somewhere...what is the best format to keep everyone happy ??
You are a swell guy,Carlos !
We are sweeping your QM cabin for electronic bugs as a "comp"...
(But then I just realized it will be a long helicopter flight from S America to Southampton...pack a survival suit. >:D

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Carlos
Here is a RGB reprocess with "normalize" applied to the combined RGB data...
http://www.flickr.com/photos/daveh56/4530945563/
or
http://www.flickr.com/photos/daveh56/4530945563/sizes/l/
(large)
Color/processing may be a bit different,but...

Indeed,I think it is deeper..(!!)
I note the histo after DBE no longer has that "clipped"/squashed to the left (my words) look...
Comments ??
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME