Nikon Z6, overexposed image, "Statistics" max. only at 25%??

[PieterVuylsteke]

There is no special thing at all about Nikon cameras. If you really did read my text, you have found the following text passage (regarding the exposure time of flat frames):

"5.5 Flat frames: determining the appropriate exposure time
Flat frames have to be captured through the telescope or lens applying an uniformly illuminated field as possible. Too short exposure time for flat frames can cause nonuniform illumination, e.g. when the light source flickers with low frequency (depending on the power source, the illuminant and the use of a dimmer) or when a mechanical shutter of a dedicated astro camera is used. So check with your equipment whether there is a lower limit for flat frame exposure time in terms of uniform illumination.

Illumination level and exposure time for flat frames have to be controlled in order that the peak in the histogram is in the region of linear response of the sensor. Normally this is the case at about half of the saturation intensity. At first determine the maximum intensity value (in ADU) in the histogram of an overexposed frame (= saturation intensity). Take half of the saturation intensity as the approximate target mean value of a flat frame."

The key term here is saturation intensity. This recommendation is a general one, it works with every camera, for Nikon cameras as well. So I don't have to modify or complement my guide in this respect.

Have a nice day.


Bernd

Well, maybe i did not read it good enough

Or maybe i was just too unexperienced to understand what you meant, at that time ?
And should explanatory texts not be at the level of the unexperienced ?

But look at the title of this thread, and maybe that prooves the confusion with a lot of nikon users.

You know, one of my friends in our astroclub has lots of years of experience with pi. I had to send him three times a link that prooves that a good lighted nikon flat is at around lightlevel 0.125 if you measure it in pixinsight. He just did not believe me.

The confusion is there.

 

[PieterVuylsteke]

I made another error typing previous message. Please be not offended if yo did read it before me changing it.

 

[PieterVuylsteke]

Do you know how many different camera types people use for astronomy?
I hope the PI team has better thing to do with it's time than writing user instructions for each camera type.
There is no PixInsight fault in the processing of DSLR data. It is well documented (on these discussion forums, which is where you will find most of the useful information), that raw input from DSLRs is input (by the very widely used 3rd party LibRAW library, not by the PI code) at the native ADC resolution (or whatever resolution the camera is configured to export in its raw files). It is easy to rescale it if you want to.

I'm not aware of any "bad drivers". What exectly are you refering to?

I hope nobody has ever suggested this. Your flats (as always) should be taken at ~50-70% max ADU. If you do not rescale the (14-bit raw) data this will give you a normalised value ~0.125 in PI; if you are uncomfortable with this - rescale the data.

Omg

 

[PieterVuylsteke]

First of all many thanks for the explantations! Great guys here
What I didn't say until now is: my goal is to get (in the near future) into astrophotography without messing around with a laptop but only with my DSLM and a standalone autoguider, and a mount of course. But back to topic:

second:


I have to agree here. Maybe it was my fault because I actually know from the settings menu in my Z6 that it saves images in 14 bit raw only, so if I would have thought by myself about it... anyway, it's history. I am sure that PI loads the raw file correctly and those 0.25 are okay.

third...


I've read that statement further above and now again. Why that? I saw some videos where they said that I shoud aim for the histogram to be in the middle of the backside monitor of my DSLM. I guess I should read Bernds guide, maybe this would answer me some questions. Is there enough dynamic range when flats are exposed that dark?

And now go on discussing please, it's very interesting!

Daniel

Do not let them confuse you.

If you expose your flat half way saturation level, as bernd explains nicely, pixinsight will show 0.125 when you measure the pixels.

In the pi histogram the peak will NOT be at 50%, it will be at 12.5%.

And if you look at the histogram in your z6 itself, it will be totally overexposed. Yes that is normal, because that histogram is not linear any more. I think almost all beginners aim for 50% of the histogram on incamera histograms, i did. But they say you sjoild not.

I will not proove it, but am rather sure such a flat will do perfectly its job: showing dust spots and vignetting. The camera is totally designed for that, not really for working at 2.5 stops overexposed, which gives that 50% of saturation.


Oh yes, another one, try to use the statistics process, and put ithe readings at 14bit, because your photograph is 14bit, totally logical isnt it?

It is not a bug what you see. It is perfectly normal, that 4000. But it is extremely totally confusing, if one is a beginning astrophotographer,

And do not even try to explain it is total confusion.

 

[fredvanner]

And if you look at the histogram in your z6 itself, it will be totally overexposed. Yes that is normal, because that histogram is not linear any more. I think almost all beginners aim for 50% of the histogram on incamera histograms, i did. But they say you sjoild not.

I will not proove it, but am rather sure such a flat will do perfectly its job: showing dust spots and vignetting. The camera is totally designed for that, not really for working at 2.5 stops overexposed, which gives that 50% of saturation.

I've no idea what this is trying to say. Let us be completely clear about this.
This has nothing to do with PixInsight!
Your flat frames should ideally be exposed at the highest exposure that keeps all pixels within the linear response range of the sensor. Modern sensors are linear (enough) up to ~90% max ADU. So aiming for a "back of camera" histogram at ~50% is safe. Aiming for 25% (if anyone is actually suggesting that) is really too low (though it will not be a disaster). Using my Aurora flat light on OSCs typically gives me ~25% in the red channel when adjusted for max < 75% in the green / blue channels - so I manage with 25%.
If you import your frames into PI as raw format, they will be imported as 14 bit data (max ADU 16383, so 50% =8192). Since PI assumes a 16-bit (max ADU 65535) format (because there is nothing in the raw import interface that tells it not to), this will be represented as a normalised value of 8192/65535=0.125. This is a data representation issue; it has nothing to do with the choice of camera exposure. This can be simply rectified by scaling the raw input (PixelMath $T*4).

 

[PieterVuylsteke]

I've no idea what this is trying to say. Let us be completely clear about this.
This has nothing to do with PixInsight!
Your flat frames should ideally be exposed at the highest exposure that keeps all pixels within the linear response range of the sensor. Modern sensors are linear (enough) up to ~90% max ADU. So aiming for a "back of camera" histogram at ~50% is safe. Aiming for 25% (if anyone is actually suggesting that) is really too low (though it will not be a disaster). Using my Aurora flat light on OSCs typically gives me ~25% in the red channel when adjusted for max < 75% in the green / blue channels - so I manage with 25%.
If you import your frames into PI as raw format, they will be imported as 14 bit data (max ADU 16383, so 50% =8192). Since PI assumes a 16-bit (max ADU 65535) format (because there is nothing in the raw import interface that tells it not to), this will be represented as a normalised value of 8192/65535=0.125. This is a data representation issue; it has nothing to do with the choice of camera exposure. This can be simply rectified by scaling the raw input (PixelMath $T*4).

Omg.

 

[fredvanner]

More than once on this thread there is a suggestion that PixInsight should document particular camera-specific issues. e.g.
"It is already long known that a nikon photo is not loaded correctly ...", "Where did you warn nikon users...".
It should be noted that PI is an image processing program. PI itself does not know anything about cameras. Many users capture data in standard format files (e.g. FITS); PI understands this image format and can import it (as well as other image formats, such as TIFF, PNG, JPEG etc.). There are dozens of proprietary camera formats (generically described as "raw" formats - there is no single raw format), often several different formats from the same manufacturer. These contain image data in a wide variety of resolutions (at least 8, 10, 12 , 14 and 16 bit per pixel). Several years ago a guy called Dave Coffin wrote a Linux command line open source program called DCRAW that converts the raw input formats for several different cameras into a standard transportable image format (PPM). Over the years this utility was upgraded to support more and more cameras. Some years ago a community developed to convert this command line program into a reuseable library, called LibRaw. DCRAW is no longer supported, so LibRaw had become the principal resource for raw camera input. LibRaw is aware of the different raw camera formats but PI only sees the converted output. This is typically in the form of a 16 bit per pixel image, plus whatever metadata LibRaw chooses / is able to extract from the raw file. Since this does not usually include the raw ADC precision PI doesn't know if the data is 8, 10, 12, 14 or 16 bit range.
If a user is unhappy with the limitations of the LibRaw import interface, they can seek an alternative 3rd party raw file converter, or they can use a data capture program that exports images in the FITS (or better still .xisf) format.
But let me repeat the bottom line: PixInsight does not know about Nikon (or Canon or Fuji or Pentax or...) cameras. Users on the PI forums may be able to share their experience with different camera types, but don't expect to find this in the PI documentation.

 

[PieterVuylsteke]

pffft....

 

[bulrichl]

Your flat frames should ideally be exposed at the highest exposure that keeps all pixels within the linear response range of the sensor. Modern sensors are linear (enough) up to ~90% max ADU. So aiming for a "back of camera" histogram at ~50% is safe. Aiming for 25% (if anyone is actually suggesting that) is really too low (though it will not be a disaster). Using my Aurora flat light on OSCs typically gives me ~25% in the red channel when adjusted for max < 75% in the green / blue channels - so I manage with 25%.

Apparently you (Peter and Fred) talk at cross-purposes regarding the term "back of camera" histogram.

Most DSLR cameras have a display and can show a histogram of the saved images. I guess this is what Peter meant by "back of camera" histogram.

This histogram is a support for daylight photography only, usually totally useless for astrophotography. It is not a histogram of linear data, but shows a histogram of the data after an initial stretch.

I know that the "back of camera" histogram of my Canon EOS 600D (= Rebel T3i) is not linear at all. Here is an example that I captured just now:

I captured a white wall with Av automatic operation (user chooses aperture, camera computes appropriate exposure time), and an exposure correction set to +2 2/3. This is the "back of camera histogram":

The peak is displayed near the right edge, at 240 ADU in the range of [0,255], this is about 15/16 of the available range.

and these are histogram and statistics when the raw (CR2) file is opened in PixInsight with RAW Preferences set to 'Pure Raw':

You see that this frame would be exposed perfectly for a flat frame. However, the "back of camera" histogram reports that it is hopelessly overexposed.

I guess with a Nikon (or any other brand) camera this is similar. Bottomline: forget "back of camera" histograms. If you exposed flat frames according to the "back of camera" histogram, these flat frames would be severely underexposed.

All the rest of what Fred wrote is correct though.

Bernd

 

[fredvanner]

It never occurred to me that the histogram on my camera was not linear! Because I always use the histogram in APT when I'm capturing flats, I've never compared it with the camera histogram - I just assumed they would be the same. That does make it much more difficult to judge the correct exposure for flats using just the camera - it is surely unmanageably cumbersome to have to download each test image to see what the average brightness is.

 

[PieterVuylsteke]

Thank you Bernd for taking your time to explain what I meant to say, highly appreciated.


It is a very nice example of the mountain of surprices a beginning DSLR astrophotographer has to go through. Once you know it, it is "utterly evident", but the first time you remark it, is is "utterly surprising".

That is not the fault of PixInsight, of course, and I of course did not want to complain about that, but it is a fact that it is surprising.


Another confusion point I had when starting to learn PixInsight was this. I do mention it in another posting, but some printscreens make it more understandable.

If you open a NEF flat file, and choose the 16bit format, you get this image (same as in your posting) :

But, then, totally stupid me, thought, tiens, my camera is 14bit, should I not choose 14bit ?
And then you get this statistics :

Only 2200 ? And the nonlinear histogram on the back of the camera is showing 2.5stops overexposed ??? Can it be more confusing ?

Nowadays it is "utterly understandable", but at that time it was "utterly confusing".


Or you look at the histogram in Histogram Transformation, and see this (for a correct Nikon Z-series flat) :

So here indeed, one has to aim at 0.125 in the histogram.

Do not forget that looking at the histogram (on the back of a camera, or even inside of the viewfinder on new camera's as the nikon Z-series) is the single most important thing in making landscape photographs. So that reflex is "utterly understandable", and yes it is not usable in astrophotography. Indeed.

This is the last post I will make in this thread, and yes, thanks again, Bernd, for explaining what I meant to explain.
It does make me look less stupid. Much apreciated.

 

[fredvanner]

The "8, 10, 12, 14, 16-bit" display modes in "Statistics" were a bit of a puzzle to me originally.
The key thing to understand is that inside PI, the image is stored as normalised floating point values in the range 0 to 1.
For any given internal pixel value, say x, the n-bit display mode will convert x to x*2^n ADU.
This gives the correct value if the input data has been scaled correctly, so that max ADU +1 scales to 1.
Since raw input is always scaled as 16 bits, you must select 16 bit mode to read out the actual ADU values.

It follows that if you have n-bit raw input files, they must be multiplied by a factor of 2^(16-n) to be correctly scaled.

 

[bulrichl]

That does make it much more difficult to judge the correct exposure for flats using just the camera - it is surely unmanageably cumbersome to have to download each test image to see what the average brightness is.

The following procedure is feasible with manageable effort:

With a DSLR camera in manual mode, it is advisable to once perform an exposure bracketing and determine the correct exposure time for flat frames with given optics and illumination level. If you use a flat field generator, this exposure time value (in manual mode) usually can be used forever for given optics (assuming that the brightness of the flatfield generator is reproducible).

For daylight or twilight sky flat frames (changing illumination level), the use of Av automatic operation is convenient. At first, again determine the correct exposure time (at constant illumination!). Then you have to determine the appropriate exposure correction: switch to Av automatic operation (user chooses aperture, camera computes appropriate exposure time) and determine which exposure correction will result in this correct exposure time. Now you can use Av automatic operation with this exposure correction value for capturing daylight or twilight flat frames. With my camera (Canon EOS 600D = Rebel T3i), the appropriate exposure correction value is +2 1/3 to +2 2/3.

Bernd

 

[Juan Conejero]

Hi Fred,

The key thing to understand is that inside PI, the image is stored as normalised floating point values in the range 0 to 1.

Not actually. Images are always kept in memory with their current pixel sample format unaltered. Supported formats are 8, 16 and 32-bit unsigned integers, 32 and 64-bit floating point real, and 32 and 64-bit floating point complex (although complex-valued images cannot be managed directly in the graphical user interface). All pixel value normalizations are generated dynamically on-the-fly by different interface components and tools. The normalization to [0,1] is just a useful and powerful way to express pixel values in a universal virtual format, able to represent any supported pixel data format uniformly and transparently.

 

[fredvanner]

Images are always kept in memory with their current pixel sample format unaltered

I stand corrected . I guess that I am rarely looking at the raw subs, and as soon as almost any processing happens the processed image is in FP.
However, I think my description of the statistics display modes still works correctly (with the normalised image as a useful "universal reference" format).

 

[dswinnard1]

The following procedure is feasible with manageable effort:

With a DSLR camera in manual mode, it is advisable to once perform an exposure bracketing and determine the correct exposure time for flat frames with given optics and illumination level. If you use a flat field generator, this exposure time value (in manual mode) usually can be used forever for given optics (assuming that the brightness of the flatfield generator is reproducible).

For daylight or twilight sky flat frames (changing illumination level), the use of Av automatic operation is convenient. At first, again determine the correct exposure time (at constant illumination!). Then you have to determine the appropriate exposure correction: switch to Av automatic operation (user chooses aperture, camera computes appropriate exposure time) and determine which exposure correction will result in this correct exposure time. Now you can use Av automatic operation with this exposure correction value for capturing daylight or twilight flat frames. With my camera (Canon EOS 600D = Rebel T3i), the appropriate exposure correction value is +2 1/3 to +2 2/3.

Bernd

Just today I was looking at grossly over-exposed (from my photographer's perspective) frames from my Fuji X-T3 wondering why the statistics readout didn't show max. values for the saturated pixels. Finding this thread was a great help.

Based on Bernd's notes above I've determined my flats are quite underexposed. (though to this beginner's eye they seem deal with the dust and vignetting satisfactorily). I will have another go at the flats for my recent captures tomorrow and see what the differences are when I expose according to the above guidelines. (I too have been using "histogram at half-way" on the back of the camera approach).

Thanks for the thorough discussion,