Pink stars after integration - DSLR data

[pfile]

well i was basing this idea on the fact that SGP will do the scaling to any type of file it saves as fits from a DSLR. the scaling done there is probably different as it's the scaling built in to dcraw, which is what PI used to use for CR2 files (the DSLR_RAW module). at any rate it's my understanding that SGP would do this to flats as well as bias and darks, but perhaps not. to be honest, it doesn't make sense to rescale a bias or dark. it is configurable in dcraw after all, and SGP knows what kind of file it's saving, so it could turn off the scaling for calibration frames. i would have to ask in the SGP forum.

PI now uses libRAW and i just looked at the RAW file handler - it has a similar rescaling function but it is called "No Highlights Clipping" and that is checked when the user clicks "pure raw". the tooltip says:

"when this option is enabled, highlights will be left unclipped resulting in various shades of pink. With this option disabled, highlights will be clipped automatically to solid white"

i wonder if the right thing to do is simply to only rescale the lights while calibrating. it looks like the RAW module supports the format hints 'clip-highlights' and 'no-clip-highlights' so you could pass 'clip-highlights' in the input format hints while calibrating your light CR2 files.

anyway because my fact checker is always lurking, please know that it's been a few years since i've used a DSLR with PI, i have encountered the same problem but never fully solved it, and am just throwing out ideas here. without pulling out some of my old data and doing experiments, i can't be sure that what i'm suggesting is the right course of action or whether or not it would even work. i think you are on the right track with this stuff though.

rob

[bulrichl]

Hi Magnus,

I took a look at your calibrated (and cosmetically corrected) light frame, "M101_70D_Light_300_secs_2019-04-10T01-52-10_026_c_cc.xisf". I split this frame into the CFA channels (with SplitCFA) and viewed the histograms of the CFA channels. Whereas the peak shape of CFA1 to CFA3 looked as expected, the peak shape of CFA0 looked very weird (see appended histograms). Besides I was surprised that the peaks of the CFA channels CFA1 to CFA3 were one upon the other -- this is very unlikely in a calibrated light frame, because the flatfield correction normally shifts the peaks to different locations.

There are several possible causes for the outcome. In order to decide what went wrong, one would have to get access to the Master Calibration Files as well. Furthermore, the following questions should be answered:
- which settings in Format Explorer / RAW were used?
- which Master Calibration Files were used?
- how were the Master Calibration Files prepared?
- which settings were used for ImageCalibration?
- were any additional manipulations performed?

Bernd

[magnusl]

Hi!

I'm not really sure I follow your thinking here, but yes, I can reproduce those histograms with the cc-file. However, before cc, that is, the one that is only calibrated, does not have that strange hump on the CFA0 (that is red, right?) channel.

I use a Astronomik CLS filter, that suppresses red a bit, if I am right. The subs taken directly from the camera are somewhat green when looked at as color images (any raw capable application, for instance). So I guess there is something going on in the red channel there.

I've added calibration files to a subfolder of the dropbox folder, I believe you can access that. They are produced by a standard "Inside Pixinsight"-process. That is: darks are integrated but not calibrated, flats are calibrated and integrated.

Settings for raw and image calibration are attached below.

So what does this tell you?

I'm still struck by the fact that the strong stars look saturated in 3-d-plots, but does NOT reach full maximum ADU. I wonder if that means that there is something strange going on in the transformation from 14 to 16 bits - essentially, that the weaker parts of the image are transformed correctly, but the strong stars are not. Thus the very different ADU levels of "saturated" stars in the different channels in the calibrated image, but not in the raw file.

Magnus

[magnusl]

Rob,

Thanks, sounds useful. How do you mean by "i wonder if the right thing to do is simply to only rescale the lights while calibrating". Not rescale previous to calibrating?

So I tried passing the option "clip-highlights" to the calibration of one frame. However, the "saturated" star that I am focusing on here, still is very pink. And interestingly enough: just splitting the R, G, B channels for the star, again produces radically different values.
R max: 51736
G max: 13352
B max: 15663

While in the uncalibrated, they are almost identical....R = 15311, G = 16400, B = 16331

But:

I tried rescaling just the raw, then calibrating the rescaled raw. See the attached images below. If I FIRST rescale, the values in the three channels are far better preserved. If I calibrate the cr2-file, they end up with vastly different levels, for the strong/saturated star.

Now, how can we understand this? Even if there is something strange with my flat, here the results seem to depend on when in the process the light-sub is rescaled. The master calibration files are already rescaled, as they are integrated and so on.

Magnus

[bulrichl]

Hi Magnus,

yes, I missed the calibrated, not cosmetically corrected file.

In the calibrated frame the CFA0 (red) channel again looks quite different than the rest. The histogram peak is much broader (much more noise), and there are horizontal structures that are not visible in the other channels.

I also tried to calibrate only with the MasterDark (no MasterBias, no MasterFlat). Then the histogram peak of the CFA0 channel comes out much more narrow. So I guess that (at least part of) the issue is related to the quality of your MasterFlat. There is little signal in the CFA0 channel compared to the other channels. The horizontal structures are already weakly visible in the CFA0 channel of the light frame calibrated with only the MasterDark.

How many flat frames were used for the preparation of the MasterFlat? Maybe you have to use a larger number of flat frames.

Furthermore it stands out that the flat frames are underexposed. The red channel of the MasterFlat is by far (by factor 4 - 5!) the weakest, so this channel is concerned the most:

            MF_CFA0     MF_CFA1     MF_CFA2     MF_CFA3
count (%)   99.98763    99.99954    99.99933    99.99942
count (px)  5041956     5042557     5042546     5042551
mean        824.647     4391.798    4371.928    3515.716   <-
median      833.568     4445.423    4427.996    3557.699   <-
variance    3410.628    78454.199   75361.941   54379.268
stdDev      58.401      280.097     274.521     233.194
avgDev      54.944      269.616     264.493     227.833
MAD         51.681      247.832     243.658     217.574
minimum     0.441       266.044     54.010      51.771
maximum     1794.439    4951.006    4945.752    4051.001


I suggest you to make new flat frames with exposure time x 2.5, same light source.

Presumably you took sky flats? Such strong discrepancies between the color channels can cause trouble in the preprocessing of images captured by OSC cameras. Maybe a much better result would be obtainable by using a light source with white light (plus more flat frames).

Bernd

[pfile]

this weak red channel thing is a common problem with the CLS filter. if you put white light into it, you end up with frame that is strongly dominated by blue and green and the red channel is very weak.

when i used one of these filters i used a slightly pink T-shirt over the objective to try to even out the channels in the flats.

but as bernd suggests you can also just make more flats to get the SNR of the red channel up. that's what's important. it's easier to get a high-SNR flat by making it white and exposing as far to the right while still in the linear range of the sensor, but of course more frames works too.

rob

[magnusl]

Hi!

Yes, I've been playing around with the flats today as well, I totally see what you are pointing at (I think). What dawned on me, was that very low values in a channel that you divide with, might produce very high levels in the resulting image. That needed to dawn on me - how do you say, the coin fell down or something (there is such a saying in Swedish, cannot find the English....).

I'm going to play with the flats. I cannot take new, though, because this was in April, and much has happened since then. But I think I can work with what I have. Interestingly, some of my flats are just slightly more exposed, and they produce radically different results, it seems.

Thanks!

Best,

Magnus