Can't create a proper Master Flat from EOS R6 files

H

hajoKt

Member
Hi there,

I'm having trouble creating a master flat frame in Pixinsight based on CR3 raw files from my new EOS R6. The flats were shot with a flat field light and an exposure time of 0.5sec (ISO200, @~f4). I'd checked the histogram on the camera screen. It looked OK.

In Pixinsight, a single flat frame is only displayed very darkly. The Image Statistics in 16Bit gives a minimum/average/maximum of 2218/4478/6894.

Pixinsight_ImageStatistics.JPG


The master flat looks even worse: 226/1105/ 1720.

For comparison, I looked at a single image in APT. The histogram looks reasonable here. I double checked that there was not autoscaling applied. The same applies to Affinity Photo. Curiously, the histograms are a bit different. However, they show normal values and are not remotely as dark as in Pixinsight. The next to images show screenshots of APT (light blue background is a crop of the rendered single flat frame) and Affinity Photo:
APT_RAW_CR3_unstretched.JPG


Affinity_Hitogramm.JPG


I have no idea what went wrong here. Whether a setting is wrong or whether there is an error in Pixinsight. But, almost always the problem is in front of the screen.

I've isolated the creation of the flats from the rest of the processing. The WBPP screenshot shows the associated configuration:

WBPP-Configuration_ND.JPG


I would be very grateful for any hint!

I have stored a single Raw File and the MasterFlat on the following drive: https://c.1und1.de/@714447422012528392/eXOXkcEOS9ii01foLkXCcg

If I want to to calibrate the lights with this flat I get the following error: !!! Error: No active light frames in the group.

Best regards

Hajo

PS: I looked at all the individual flats with Blink. They all look the same! Incidentally, I have never had any problems with my OSC ASI071MCpro.
 
Last edited:
Hi Hajo,

the Analog Digital Converter (ADC) of the Canon EOS R6 has a bit depth of 14 bit (range 0 to 16383 DN). When when the proprietary raw format (CR3) is used for saving the data, no scaling is applied by LibRaw, the software that is used for raw image decoding by PixInsight's RAW Format module. This differs from the behavior of the camera driver of the ZWO ASI071MC. So the intensity values that are displayed for the flast frsmes in PixInsight's Statistics process are OK, they have to be related to a maximum possible value of 16383. Apparently the histograms of APT and Affinity Photo are not scaled liearly, or they show a histogram of a stretched image.

However, if the MasterFlat shows median values of 226/1105/1720, something went wrong in the generation of the MasterFlat. How was the MasterFlat generated? If you provide WBPP with light, dark, flat and bias frames (or flat-darks), it should work well.

Please also see my guide https://pixinsight.com/forum/index.php?threads/for-beginners-guide-to-pis-imagecalibration.11547/ for a detailed description how to generate master calibration files and what to consider in the data redution.

Bernd
 
Dear Bernd,
Thanks for your evaluation of the issue.
I made sure that APT was not stretching the image. I purposely hit the clear button of the histogram window.
WBPP was only run with Flats and the associated Dark-Files. An image of the WBPP window (in this case including lights) is contained in my initial posting. I also have added a link to a single raw and the master flat file.
Will have a look at your image calibration guide shortly!
Kind regards,
Hajo
 
APT and Affinity both are probably stretching the image "behind the scenes" - if they use libRAW or DCRAW to display the image and histogram, libRAW and DCRAW can debayer, stretch and gamma boost the image depending on the settings the developers chose. given that the screenshot from APT looks light blue, it seems almost certain to me that this has happened.

Affinity would almost certainly do this for sure as it is a terrestrial photo application. APT may be less likely to do this but it might be a case where they do it for OSC images but not for mono fits images from dedicated astro cameras. there is a way of course to get libRAW/DCRAW to debayer an OSC image but not stretch it, which is probably what APT should be doing if it isn't already.

when i used a DSLR for astro i'd just use Av + 2EV when taking flats and that was usually good enough. although it's not important for color balance, because i was using a CLS filter i sometimes used a light pink T-shirt over the telescope aperture to rebalance the flat. this was only to make sure that the red channel SNR was good. even when a flat causes a color cast in the calibrated image, it can be undone with BackgroundNeutralization/ColorCalibration or PhotometricColorCalibration. additional PI now has the ability to scale flats independently per-channel.

rob
 
hajoKt said:
I made sure that APT was not stretching the image.
Click to expand...
I don't know what APT is displaying in the histogram. PixInsight has all necessary tools to investigate in case of issues, and I use them.

Your MasterFlat has a weird name: "masterFlat_BIN-1FILTER-NoFilter_CFA_3.xisf" - why is it so? It is (as expected) a CFA file, 5496 x 3670 pixels.

I split the color channels of the single flat and got different numbers than those specified by you. These are the median values of the channels:

single flat
CFA0: 979 (R)
CFA1: 1932 (G)
CFA2: 1935 (G)
CFA3: 1773 (B)

In the calibration of the flat frames, the MasterBias (or MasterFlatDark) is subtracted. This leads to intensity values that are diminished by about 500 DN in the MasterFlat:

MasterFlat
CFA0: 469.1 (R)
CFA1: 1428.9 (G)
CFA2: 1431.2 (G)
CFA3: 1268.2 (B)

So these numbers are in good agreement.

In this case, the flat frame exposure is indeed somewhat short and can be extended by factor 5 - 6. However, this doesn't explain the error that coccurred in the light frame calibration with WBPP.

I guess that the error message cannot be obtained from WBPP settings shown in the screen section: the Calibration Flow Diagram shows for light frames: "32 active", but the error message reads "!!! Error: No active light frames in the group" - that doesn't match.

I would need the detailed WBPP logfile.

Bernd
 
Last edited:
Dear Rob,
Thanks for the further explanations. That's actually a strange behaviour of the the 2 apps. I'll give your Av + 2EV-approach a try tomorrow. Maybe I also try not to use WBPP but do calibration and integration in separate steps.
Hajo
 
Dear Bernd,
Thank you for all your efforts. I will run the process again tomorrow and generate the corresponding log files.
Hajo
 
I have redone the Flats and the associated Darks increasing the exposure by a factor of 4. Now one single raw file looks OK, but the Master Flat still appears to be underexposed. I have added the new files including the log of the Master Flat generation (WBPP) on the cloud drive. I also have done a complete integration run with WBPP. This time it succeeded but the resulting Master Light was red (RGB Channels linked). I didn't manage to create a usable image of it on the fly with Background Neutralization and DBE. I have added an image (M45 with Canon EF 200mm) of my old processing without flats on the cloud drive.
BTW: This image was shot in Andalucía (Cabo de Gata) while rehearsing for comet Leonard using my Fornax star tracker.
Hajo
 
hajoKt said:
I have redone the Flats and the associated Darks increasing the exposure by a factor of 4. Now one single raw file looks OK, but the Master Flat still appears to be underexposed.
Click to expand...
Same procedure as before results in the following median values in the new single flat:

CFA0: 2450
CFA1: 6453
CFA2: 6458
CFA3: 5838

... and the following in the resulting new MasterFlat:

CFA0: 1953.1
CFA1: 5990.2
CFA2: 5996.1
CFA3: 5372.2

Again, the values in the MasterFlat are expected because in flat frame calibration, the MasterFlatDark (with a median value of about 500 DN) is subtracted from the flat frames. I guess optimal flat frame exposure would have been accomplished with an extension of the exposure time by factor 6. However, the exposure of the avalilable MasterFlat (x4) is definitely sufficient.

hajoKt said:
I also have done a complete integration run with WBPP. This time it succeeded but the resulting Master Light was red (RGB Channels linked). I didn't manage to create a usable image of it on the fly with Background Neutralization and DBE.
Click to expand...
It is quite normal that the integration is displayed with a strong color hue when in STF the color channels are linked - you should be used to that from the ASI071MC. In the ImageCalibration process (and in the WBPP script as well) you can choose whether one master flat scaling factor is calculated (from the average of all color channels) or a master flat scaling factor is calculated for each of the color channels. While in the former case, an additional color shift is affected by light frame calibration, this does not happen in the latter case.

The uploaded WBPP logfile only contains the generaton of the master calibration files, but not the light frame calibration. Please upload the resultant Master Light from the complete run with WBPP as well as the WBPP logfile that contains the light frame calibration. The essential question seems to be whether the MasterFlat matches your light frames. This is not a question of flat frame exposure though.

Bernd
 
Thanks again Bernd,
I've read in your Tutorial about the introduction of a color shift through flat frame calibration. I'll rerun the WBPP with single color channel scaling. and provide the Master Light and the corresponding Logfile.
Hajo
 
Now I've rerun the WBPP script with single channel scaling. Looks better now. What I can spot though, is a kind of donut structure in the middle of the frame: darker center and brighter ring. The logfile and the master light are now copied to the cloud drive
Hajo
Ps: Wie gut, dass das Wetter hier so bescheiden ist. Da kann man sich mit seiner eigenen Unfähigkeit beschäftigen. Dummerweise wartet mein Weihnachtsgeschenk darauf, eingesetzt zu werden. Und keine Hoffnung, jedenfalls für Deutschlands Mitte.
 
I see the problem in the integration. The background contains a dark ring. I extracted the color channels from the integration and to the green channel applied ABE once with function degree 1 (in order to only remove any linear gradient caused by light pollution) and then with function degree 8. The resulting background model looks like this:

background _model_G.png

Artifacts like this are possibly caused by stray light. However, it is nearly impossible to remove such artifacts by background modelization, even more in a region of the sky that contains faint scattered light from interstellar medium. You should try to remove this artifact by avoiding stray light in the optical train. See e.g. the article that I cited in my guide about this topic: Alan Holmes, "Flat Fields and Stray Light in Amateur Telescopes",
https://diffractionlimited.com/flat-fields-stray-light-amateur-telescopes/ . Maybe capturing sky flats could be the solution? I don't know.

Perhaps someone experienced with such artifacts can chime in? @ngc1535 ?

Bernd
 
Dear Bernd,

Maybe the issue with the ring is due to the set-up of my Lens/Flatlight. I took an idea from Cuiv, who simply used a couple of step-down rings screwed together to stop down. I thought that's cheap and effective. Maybe this approach creates the donut problem? I don't know. I have to do some testing and read the article you referenced.


Flatfield-Set-Up_EF200mm_s.JPG

I consider my original problem as being solved. Obviously one can't trust the LCD screen of the camera, nor APT, nor Affinity. You have to multiply the exposure time that you get from looking at the LCD-screen of your camera by a factor of ~5. And don't forget to tick ""Separate CFA Fat Scaling Factors". Thanks again for all the effort.

Hajo
 
hajoKt said:
Maybe the issue with the ring is due to the set-up of my Lens/Flatlight. I took an idea from Cuiv, who simply used a couple of step-down rings screwed together to stop down. I thought that's cheap and effective. Maybe this approach creates the donut problem? I don't know. I have to do some testing and read the article you referenced.
Click to expand...
In my view, this is possible. There are a lot of adapters that reflect some light and generate artifacts in astrophotography.

Bernd
 
Yes, this does look like circularly (cylindrical) scattered light. Flocking (black) the adapters, if possible, would be an easy and excellent test to confirm or eliminate as a source is difficulty.
-adam
 
I have been redoing the flats once more. This time without the step down rings (max aperture of the EF 200). It seems as it (the donut) has gotten even worse. Once I'v got some black ink, I'll redo the flats once more.

I also tried to fix this donut shape by using Axial Symmetries in DBE. This idea stems from Adam Block. He discribes this method in his YT-Video "An example of using Pixinsight's DBE creatively. And I must say it works quite well. It mitigates the problem to a certain extent without harming the interstellar clouds to much.
Hajo
 
I take my flats with an Aurora flatfield panel balanced (without any special light seal) on top of the (vertical) telecope aperture just like your image. I use this with configurations from my 80mm Esprit ED to my 200mm EdgeHD. I use the APT CCD flat aid to determine my flat exposure (so I don't look at any histograms, I just set a target ADU value). I always take my flats at night, so the background is dark. I've never had any problems like this.
 
Hi Fred,
Yes, I also use an AURORA with a 1.2 ND filter film. The tests were now run afterwards in a darkened room. I use APT - besides NINA - also regularly. As far as I know, the Flat Aid in APT does not support DSLR/MILC. That's why I chose the histogram approach. I assume that your good experience relates to both your CCD/CMOS cameras and the 1100D?
Hajo
 
I confess I haven't used my 1100D for astrophotography since I got my Aurora panel, so I've never used them together. Although the aid does say "CCD" I use it happily with my CMOS cameras, and I don't see why it wouldn't work with the 1100D. If I get a free night I'll give it a try.
 
You must log in or register to reply here.
Back
Top