Pink stars after integration - DSLR data

[magnusl]

Hi!

I'm working with some DSLR data, Canon 70D camera, on M101. When I integrate all my subs, I get pink stars, most of all the saturated stars. The pink is there already after integration, and then remains through deconvolution and arcsinhstretch. So: what is going on here, and what to do? See the small png image below, as well as a 3-d-plot of the image. The xisf-version of the same small image is here:

https://www.dropbox.com/s/bipa2yzs0y7msny/M101_integration_DBE_DBE_Preview01.xisf?dl=0

I've tried the Repaired HSV Separation script with various settings, but cannot solve it. Same with Color Calibration and Background Neutralization.

What I could do, is to utilize a star mask + color mask and reduce the magenta saturation in the stars. But I'd like to understand why the stars become pink, what is wrong in my processing here, rather than using too much manipulation to get around it.

So: any ideas or suggestions for how this happens (debayer - issue??), and how to best manage it?

Magnus

[John_Gill]

Hi,

I see you have done a similar thing but this might help with the pink stars.  After doing "ArcsinhStretch" I noticed pink stars, I went back to the Linear state and then ran the Script ---> Utilities ---> Repaired HSV Separation ---> Check V - no repairs, Select Repair level 0.2 - 0.7 (Repair level 0.35 and Radius 32 worked for me) ---> Ok. This generates a number of images.  Now run ChannelCombination ---> HSV ---> add H, Sv and Unrepaired_V images and apply.  Finally stretch the image with ArcsinhStretch/MaskedStretch/HistogramTransformation.

If the pink stars are still there go back to the linear state and try again with different setting in the Repaired HSV Separation script.

space is not black
John

[John_Gill]

Do a search for "Pink Stars" on the Forum and there is the explaination of what is happening to your image.

[magnusl]

Hi!

Thanks for your replies!!

Well, I've tried the Reparied HSV separation, with a wide range of settings, to no avail. And: I do have the pink stars already after integration, not just after Arcsinhstretch, but at the very "beginning".

So, have you tried running the script on the xisf-image I posted? Does it work? It would be wonderful, if you could try and see how to make it work.

I've been through a lot of threads about pink stars, but so far have not found anything that really seems to fit my situation - if you know of it, I'd appreciate a link or hint. I am quite lost here.

Best,

Magnus

[magnusl]

Adding some detail: It seems not just to be the saturated stars that are pink, but many stars in the image. I understand that PI can produce pink when stars are saturated, and the recommendation is to add shorter exposures with HDR composition. However, that is not an option for me, this is data from a year ago. And interestingly, Repaired HSV Separation seems not to make much of a difference here. So I really don't get it, why would not saturated stars become pink?

THe full integrated image, after DBE, is available here, as well as a few single subs:

https://www.dropbox.com/sh/fs744ujp8oed79d/AACfnztnt-E98N8km8Tv2dUBa?dl=0

Magnus

[pfile]

i think the "core" of the problem here (no pun intended) is that DSLR data is 14 bit data in a 16-bit container, so saturated pixels are not represented as 1.0 in PI's internal format, but 0.25.

you might try the Rescale process set to "individual channels" against the linear version of the image and see if your star cores are then 1.0 in every channel.

rob

[magnusl]

Hi!

Thanks! This is beginning to make sense. I've just quickly tested, and it makes a difference. But more questions surface (which is a sign of developent in my book).

I ran a statistic on the image, then did a rescale individual channels, and redid the statistics. See attached images. So, at the start, the red channel seems to utilize the whole 16-bit range, but both green and blue are restricted to roughly 14-bits (if I count correctly). After rescale, the are all using the whole 16-bit range, and indeed, the pink cores are gone!! At least for now...

But why on earth is the red channel using a wider range than the others?

And: what more precisely does this Rescale do? I find no documentation for it. Should I even run it on the individual subs before integration, or after integration, before anything else, like DBE (which was my intuition)?

Thanks indeed for a very helpful piece of information and advice!!

Magnus

[pfile]

i am thinking maybe the large values you see in the red channel of the "before" image are not actually star cores but hot pixels. i think to really figure it out you'll have to carefully make a preview that includes a bad star but no hot pixels and see what the statistics say.

there is a thread here about rescale somewhere, but it really just describes that it expands the data to fill the entire range of the data. having said that, i wonder if it throws out outliers first because obviously those supposed hot pixels i'm talking about could cause the tool to not rescale that channel. PI generally does pretty robust outlier rejection in most of it's processes where that would be important.

it turns out the source code for Rescale is available in the PCL library, however, Rescale turns out to be one of the methods that is associated with Image objects - the Rescale process just asks the image to rescale itself - and i don't think that source code is available. so i think juan would have to comment about how ImageVariant::Rescale() actually works WRT outliers.

seems like the Rescale source code contains a version that can be run from the PI console, and so if you type "Rescale --help" into the console you'll get some more info about what it does in the different modes.

rob

[magnusl]

Hi!

Good point! So I tried a very small preview with the bad star that I already used in this post, just the star. As far as I can see, no hot pixels. And when I reset the STF, the star is clearly saturated. Doing a 3-D-plot, the star is clearly saturated, that is, with a flat top, and no other peaks in the preview, nowhere near the value of the star. This small preview gives me max values for R = 65247, G = 13933, B = 15944.

So I extract the R, G and B channels and make a 3-D-plot for each, to see if it is saturated in all channels, although at different levels (so to speak), see attached image. And sure, doesn't it look like it is flat on all three, but on very different levels?

Now, if I rescale, it seems that I get saturation at the same ADU levels on all channels. See attached image number 2. And with the interesting "depression" on top that I guess is what causes the pink stars after Arcsinhstrech and that can be handled by Repaired HSV Separation script. But on similar levels!

Does this make sense?

Now, my images were taken with an Astronomik CLS-filter, that does influence the color balance. But why should the red be transformed to full 16-bit range, but green and blue not?

Further on, when I do the rescale, I get mean values that are very different, the red being far far below, so a very strong green image. If I do a color calibration to get rid of this, suddenly the range for green and blue are lowered again, although not to the same values as before. Hrm. So if I now do the Rescale - how to best proceed from there?

Magnus

[magnusl]

Hi!

Some more details - I'm not sure if it is clarifying or confusing.

Something seems to happen with the saturated stars in the calibration process. Before calibration, the files are all 14-bits. You can see a raw file in the dropbox-folder. Deebayering it shows all three channels are about equal, and examining the saturated star as above, shows saturation in all 3 channels at the same level.

However, after calibration, the saturation is at different levels. Now the red channel seems to show the saturated star at about 65000, but the green and blue is still around 16000. Max values, however, are about similar, around 65000.

So my hypothesis: it is hot pixels in green and blue channel that are responsible for the values above circa 17000 in the calibrated image. But not in the Red channel!!

And true enough - after cosmetic correction, the max values are very different between the channels. And from the calibrated image and onwards, the saturation in red is far higher than the blue and green, as shown in the 3-D-plots above.

In my dropbox-folder are files: the raw, ending with cr2. The calibrated, ending with _c.xisf, and the cosmetic corrected, ending _c_cc.xisf.

Any idea what is going on here? Am I on to something or am I chasing ghosts?

Magnus

[pfile]

thinking about this a little more, i think what could be happening is that the application of the flat is modifying the non-saturated values to near saturation in one or more channels. how well color balanced are your flats? ordinarily this should not matter if your data fills the 16-bit space but it could be an issue here with the 14-bit to 16-bit stuff if one of the flat channels is brighter than the other. the CLS filter definitely gives you a very blue cast and it will do that to the flats as well unless you use a slightly pink light source to counteract that.

maybe this is an argument for what SGP does when it saves a DSLR image as fits - it does the rescaling to 16 bits at that time. in theory to mimic this you could preprocess your CR2 files by simply running rescale on them and saving them as xisf using ImageContainer. tbh i have not thought this all the way thru; i think if you do this to your lights you might have to do it to all of your calibration frames as well even before integrating the calibration masters and running ImageCalibration.

rob

[sharkmelley]

White balance is the usual reason for the saturated parts of stars turning pink.  In the raw file, the Red, Green and Blue channels will all have the identical saturated values.  White balancing boosts the red and green channels turning those saturated parts to pink.

Mark

[magnusl]

Hi!

Mark, when do you mean there has been a white balancing in the process? These in my mind strange differences between the channels occur after calibration...

Magnus

[pfile]

yes i think WB can do it too - really anything that tries to change the 14-bit saturated values can lead to weird colors.

if you have not done any color calibration in your flow then it seems to me the flattening is the next logical place where the pink cores come from.

rob

[magnusl]

Hi again!

I'm looking into the idea of rescaling the subs before calibrating. And also rescaling the calibration files. However, I run into an issue with my flats.

In 14-bits, one master flat has has maximum 4851 and median 3789 ADU, wich is about 25 % of the maximum ADU in 14-bits, those most often work nicely for me, I think, with the 70D. But: there is nothing saturated here. THe maximum is 4851, not the full 16383, even when it is made in PI and saved as an xisf flat master.

However, rescaling that flat gives me not very good values. without debayering (for simplicity's sake now), I get a maximum of 65532 and a mean value of 50165. That makes sense, if the Rescale scales so maximum value in original image is now at the "ceiling" - even though it was just about 25 % before.

If I take an unprocesses flat cr2-file and do the same, I have maximum of 9846 in the cr2, and when rescaling it becomes 57866, that is proportionally far higher. This time however, the mean stays at a better 30093, but it also means the gradient in the sub is far steeper now, than in the original sub.

Does this make sense....?

So, if I should rescale the flats, how should I do that, to preserve the ADU percentage they are taken with, and to preserve the gradient in them? Or is this not doable?

I'm beginning to think it might be better to calibrate in another program, and import to PI as calibrated fits-files, preferably already scaled to 16-bits by then.

Magnus