Determining proper exposure for DSLR Flats

[STEVE333]

I have a Canon T3 DSLR camera. I use it with one of three different filters (to help with LP): (1) IDAS LPS-D1 for Galaxies, or (2) Triad Tri-Band filter, or (3) STC Duo-Narrowband filter for Nebulae.

I use an LED Tracing Pad as the light source for creating Flats. Some T-Shirt material is stretched across a Needlepoint Hoop to keep out any wrinkles. The hoop is placed over the front of the Dew Shield and then the whole assembly is pushed flat against the LED Pad. The setup is shown in the first attached picture.

With each of the three filters the R channel tends to have a significantly lower signal than the G or B channels (because of the filters and/or the Led Pad). My goal is to have all channels with signal levels between about 50% - 90% to ensure good signal-to-noise and good linearity.

I check the signal level for each channel by taking a Raw image and then loading it into PixInsight and looking at the image with HistogramTransformation (I'm sure other programs have similar tools). This presents a histogram showing the R/G/B levels in the image.

The second picture shows three Histograms: The leftmost histogram is for the setup shown in the first picture. Because the Red channel is too weak compared to the G/B channels I put a layer of Red cellophane between the LED Pad and the Needlepoint Hoop. I used Red Cellophane to attenuate the G & B channels more than the R channel. The middle histogram is for one layer of red cellophane and the rightmost histogram is for two layers of red cellophane. Notice how the cellophane tends to bring the G/B channels closer to the R channel.

The third picture shows the histogram for the setup with two layers of red cellophane and an exposure time of 0.6 sec. The Red channel still isn't up to 50%, but, it's close and that is as far as I went (I was getting concerned about possible multiple reflections with too many layers of cellophane).

The Flats have worked very well.

I hope this is useful for someone else.

Steve

[STEVE333]

The main reason for posting this approach of using PI (or any other program that will accurately read the signal levels from Raw images) to determine the signal levels for the three channels is because of my using BYEOS. I really like BYEOS for controlling the camera, but, the  Histogram that it presents is very inaccurate and misleading. The author of BYEOS knows about the problem, but, apparently has not yet found a workaround. I originally used the BYEOS histogram to determine the proper exposure time for taking flats. Once I checked it using this approach (with PI) it became evident that my flats were vastly underexposed.

I'm hoping this post will help anyone else using BYEOS to avoid the same mistake.

Steve

[bulrichl]

Hi Steve,

I also used a EOS 600D (T3i) for astrophotography. My camera is modified (filter exchanged). I don't use BYEOS. If I understand it correctly, BYEOS saves all images in Canon's raw file format, CR2.

Obviously BYEOS does not display a linear histogram, so it is not suitable for judging whether the flat frames are properly exposed. However, the PixInsight histograms that you showed are not from the raw data as well: they are histograms of debayered flat frames. If  you want to judge the intensities of the raw data you have to take a look at the raw monochrome Color Filter (CFA) images.

I would do the following in order to properly check the exposure of the flat frames:

1) Set RAW format preferences to 'Pure Raw' (Format Explorer, double click on 'RAW').
2) Load a flat frame. You'll see the monochrome CFA image.
3) Call HistogramTransformation. Set plot resolution to 16-bit (64K) and horizontal zoom to 4 (with this setting and the slider left at the leftmost position, you are viewing the rabge of intensitiaes from 0 to 16383).
4) Call ImageStatistics. Set 16-bit (64K) and uncheck 'Unclipped'.

The range of the intensities is 0 - 16383 as the camera has a 14-bit AD converter. The T3i has a bias offset of 2048 ADUs. With my camera, saturation occurred at about 15300 ADUs. So the usable range of intensities with this camera was 2048 to 15300 ADU. The center of this range is at about 8700 ADUs.

The linear range of modern DSLR cameras is quite large. You want the exposure to be plenty, but must avoid safely that pixels will be saturated (disregarding the small number of hot pixels).

I suggest you not to attenuate the R and G channels by red cellophan. Who tells you that this material is sufficiently homogeneous and does not introduce artifacts to your MasterFlat? Set the exposure so that the mean of the flat frames is around 8700 ADUs with the settings given above (see appended screen section. This is simple and worked well for me. It is not fatal that the red channel is somewhat weaker than green and blue ones.

-----

Why were your flat frames debayered? Definitely all the calibration (and optionally CosmeticCorrection) shall be done with the raw data, i.e. with monochrome Color Filter Array (CFA) images. Debayering shall be performed just before the alignment step.

Bernd

[STEVE333]

Hi Steve,
I would do the following in order to properly check the exposure of the flat frames:

1) Set RAW format preferences to 'Pure Raw' (Format Explorer, double click on 'RAW').
2) Load a flat frame. You'll see the monochrome CFA image.
3) Call HistogramTransformation. Set plot resolution to 16-bit (64K) and horizontal zoom to 4 (with this setting and the slider left at the leftmost position, you are viewing the rabge of intensitiaes from 0 to 16383).
4) Call ImageStatistics. Set 16-bit (64K) and uncheck 'Unclipped'.

The range of the intensities is 0 - 16383 as the camera has a 14-bit AD converter. The T3i has a bias offset of 2048 ADUs. With my camera, saturation occurred at about 15300 ADUs. So the usable range of intensities with this camera was 2048 to 15300 ADU. The center of this range is at about 8700 ADUs.

The linear range of modern DSLR cameras is quite large. You want the exposure to be plenty, but must avoid safely that pixels will be saturated (disregarding the small number of hot pixels).

I suggest you not to attenuate the R and G channels by red cellophan. Who tells you that this material is sufficiently homogeneous and does not introduce artifacts to your MasterFlat? Set the exposure so that the mean of the flat frames is around 8700 ADUs with the settings given above (see appended screen section. This is simple and worked well for me. It is not fatal that the red channel is somewhat weaker than green and blue ones.

-----

Why were your flat frames debayered? Definitely all the calibration (and optionally CosmeticCorrection) shall be done with the raw data, i.e. with monochrome Color Filter Array (CFA) images. Debayering shall be performed just before the alignment step.

Bernd

Thank you for the detailed response Bernd.

I have struggled with how to set the DSLR_RAW settings. So, I followed your suggested steps (1)-(4). The first attachment shows my previous Histogram on the left, and, on the right is the new histogram created following your suggested steps. Although they are "similar" they are definitely not the same. So, clearly it does matter which way the histogram is created. I trust your way. Fortunately, the levels look pretty good using your approach. The mean value is 8035 which is just a little lower than your suggested 8700 value.

As for using the cellophane I will continue to experiment with it to see how it works. I understand your concerns, but, I don't yet know that it is not homogeneous. It should become evident pretty soon.

"Why were my flat frames debayered?" I didn't debayer the flat frames. However, the DSLR_RAW settings I was using in the past are shown in the second attachment. Maybe those setting explain the debayering.

Thanks again for your help. Much appreciated.

Steve

[bulrichl]

OK, I am curious about the results (comparison of the integrated stacks, new exposure without red cellophan vs. with red cellophan).

-----

The other point is maybe even more important: perform the calibration (and optional Cosmetic Correction) with the RAW format preference to 'Pur Raw. (the setting is memorized, so you will have to set this only once).

The correct workflow is:

     ImageCalibration -> (optional: CosmeticCorrection) -> Debayer (RGGB, VNG) -> StarAlignment -> ImageIntegration

Even better results can be obtained, if you don't debayer but use CFA-Drizzle. In this case the workflow is:

     ImageCalibration -> (optional: CosmeticCorrection) -> StarAlignment (Generate drizzle data checked) -> ImageIntegration (Generate drizzle data checked) -> DrizzleIntegration (Enable CFA-Drizzle checked)

In DrizzleIntegration, the path to the calibrated light frames have to be input under 'Format hints/Input diectory', if they reside in a directory different from that where the Drizzle data files are.

I hope this helps.

Bernd

[STEVE333]

OK, I am curious about the results (comparison of the integrated stacks, new exposure without red cellophan vs. with red cellophan).

-----

The other point is maybe even more important: perform the calibration (and optional Cosmetic Correction) with the RAW format preference to 'Pur Raw. (the setting is memorized, so you will have to set this only once).

The correct workflow is:

     ImageCalibration -> (optional: CosmeticCorrection) -> Debayer (RGGB, VNG) -> StarAlignment -> ImageIntegration

Even better results can be obtained, if you don't debayer but use CFA-Drizzle. In this case the workflow is:

     ImageCalibration -> (optional: CosmeticCorrection) -> StarAlignment (Generate drizzle data checked) -> ImageIntegration (Generate drizzle data checked) -> DrizzleIntegration (Enable CFA-Drizzle checked)

In DrizzleIntegration, the path to the calibrated light frames have to be input under 'Format hints/Input diectory', if they reside in a directory different from that where the Drizzle data files are.

I hope this helps.

Bernd

Thanks for the continued inputs Bernd -

I clicked on the Pure Raw button at the lower left corner of the RAW Format Preferences window and then clicked OK. Hopefully that sets the "Pure Raw" preference.

I've never tried the DrizzleIntegration approach, so, that will be my next experiment with the same Lights data.

Steve

[STEVE333]

Well...........

I tried to do Drizzle Integration but wasn't able to get it to work.

I'm doing it manually (no BPP) and must have one of the settings (raw cfa?, etc.) incorrect.  Sigh!!!!!!

I may just stick with the Debayer approach for now.

Thanks again for the help.

Steve

[gianpri]

"Even better results can be obtained, if you don't debayer but use CFA-Drizzle. In this case the workflow is:

     ImageCalibration -> (optional: CosmeticCorrection) -> StarAlignment (Generate drizzle data checked) -> ImageIntegration (Generate drizzle data checked) -> DrizzleIntegration (Enable CFA-Drizzle checked)"

But I know it is not possible to use StarAlignment without first using Debayer. Am I wrong?

[bulrichl]

But I know it is not possible to use StarAlignment without first using Debayer. Am I wrong?

No, you're absolutely right. I forgot the Debayer step which is essential for the second workflow as well. Correct workflow for CFA-Drizle:

     ImageCalibration -> (optional: CosmeticCorrection) -> Debayer (RGGB, VNG) -> StarAlignment (Generate drizzle data checked) -> ImageIntegration (Generate drizzle data checked) -> DrizzleIntegration (Enable CFA-Drizzle checked)"

In DrizzleIntegration, the path to the calibrated light frames has to be input under 'Format hints/Input diectory', if they reside in a directory different from that where the Drizzle data files are.


@Steve: I guess that's the reason why it didn't work for you. I am sorry, please try again.

Bernd


P.S. Let me add that CFA drizzle will improve the result only if you have a sufficient number of light frames and the light frames are well dithered.

[STEVE333]

Gentlemen (Bernd and "Gianpri") -

Thank you, thank you, thank you! I just completed my first successful DrizzleIntegration. With all of the adjustments applied the DrizzleIntegration worked perfectly.

I started this thread thinking I understood a new approach to making better Flats. With Bernd's help I now do know how to make better Flats. However, along the way, Bernd pointed out that using DrizzleIntegration would produce better results than Debayer. This in turn led to my being educated in the world of DrizzleIntegration.

The attachment shows the regular integration and Drizzle integration images side by side. The improvement in the stars is clearly evident.

I now have another PI "tool" to use in transforming barren Light frames into amazing Astrophotos.

Thanks again for taking the time to share your knowledge.

Warmest regards,

Steve

[acmalko]

Hi

All of this seems familiar for me as I had the same questions during last months

I'm using an app on my galaxy tab3 to produce a pink screen that is very useful for acquire flats.
All three channels are then equals.
Today I'm using iso 100 as I've recently learned that 1 100-iso-flat = 8 800-iso-flats

Last thing, be careful not using bayer drizzle (=cfa enabled) with scale=2 and drop shrink=0.9
Use scale=1 and drop shrink=1

[STEVE333]

I'm using an app on my galaxy tab3 to produce a pink screen that is very useful for acquire flats.
All three channels are then equals.
Today I'm using iso 100 as I've recently learned that 1 100-iso-flat = 8 800-iso-flats

Last thing, be careful not using bayer drizzle (=cfa enabled) with scale=2 and drop shrink=0.9
Use scale=1 and drop shrink=1

Clever idea with the tablet and the pink screen. Armed with your idea, I found a way using Page Color in Microsoft Word to adjust the color of the laptop screen to almost any color I want. I will give it a try next time I make Flats. This should allow me to "line up" the histograms for the R/G/B channels.

I tried using scale=1 and drop shrink=1 for DrizzleIntegration. The result is that it looked nearly identical to the normal non-Drizzle Integration. The only difference is that it took about 5 times as long! Drizzle is definitely time consuming.

Steve