Strange results and questions related to ImageCalibration and DrizzleIntegration

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
Hi Stéphane,

OK, I am glad that it worked fine for you. Definitely the artifact is NOT present in the light frames (and I don't know why). Probably you are right that the difference that builds up between AF / non-AF pixels in the dark frames is dependent on exposure time.

Since you wrote that there was no difference between the conditions for light frame acquisition and dark/bias frame acquisition, perhaps Yannis asked the right questions (in post #20):
"How do you make the darks/biases? With or without a lens/telescope attached?"

I guess that you will need a MasterFlat in order to judge whether the application of dark frame optimization is advisable with this camera or not. The criterion is the complete removal of "amplifier glow", and that can be judged better when the vignetting is removed by flat field correction.

Bernd

P.S.:
Yet some more questions:
In which format (RAF or FITS) did you capture the frames?
What is the reason that the FITS files that you sent don't contain the mosaic pattern?
How do you correctly debayer the calibrated frames? Currently it is not possible to explicitely set the mosaic pattern to 'X-Trans' in PixInsight.
 
Last edited:

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
Hi Yannis,

I have seen dark frames and a MasterDark of a Sony ILCE-6300. These did not show a similar artifact caused by AF pixels. The Sony ILCE-6300 has a bayer pattern instead of a X-Trans mosaic pattern. I don't know whether the CFA is the only difference in the hardware of the two cameras. Definitely the in-camera processing of the digital data must be very different.

The effort to modify the MasterDark and MasterBias may seem high at first glance, but actually it is not. These modifications would be the only ones that add to the normal preprocessing workflow. On the other hand, one would of course need MDs for different temperatures with a not cooled camera, and all MDs would have to be modified.

I don't think that marking all these pixels as bad is a good idea: there are 506,016 pixels out of 24,321,024 pixels overall (= 2.1 % after all), and in the rectangle (x=1510, y=505, w=3010, h=3017 corresponding to 9,081,170 pixels) it makes even 5.6 %.

The AF pixels are not easily identifiable in a dark frame, but in a 3-min MasterDark it is no problem. So if you can upload a MasterDark of X-H1 and X-Pro2, I can identify the AF pixels and make up a corresponding PixelMath equation. However, I am not able to program a JavaScript script.

Bernd
 

Yannis Doukakis

Active member
May 19, 2020
29
0
Thessaloniki, Greece
Thanks Bernd for the detailed PixelMath. In any case, MasterBias and MasterDark can be re-used in subsequent imaging sessions with the same temperature, so it is not a process you HAVE to do on each imaging session.
One thing I did not like with the RAF files, is that their EXIF does not contain (or I could not find it with exiftool) any info on temperature, unlike the CR2 files of Canon. So you have to keep notes on the temperature of each session, if you plan to re-use.
 

Stéphiou

Member
May 16, 2020
15
2
Hi Bernd, Hi Yannis,

How do you make the darks/biases? With or without a lens/telescope attached?
I always make darks and biases with my lens/telescope covered and attached.

The criterion is the complete removal of "amplifier glow"
I agree, and it seems that amp-glow is totally removed from the light frames when the PixelMath procedure defined by Bernd is applied on MasterDark and MasterBias prior to the calibration of light frames.

In which format (RAF or FITS) did you capture the frames?
What is the reason that the FITS files that you sent don't contain the mosaic pattern?
How do you correctly debayer the calibrated frames? Currently it is not possible to explicitely set the mosaic pattern to 'X-Trans' in PixInsight.
I capture the frames only in RAF format (JPG possible, FITS not possible).

I have no idea why the FITS files do not contain the mosaic pattern, to be honnest I do not remember if I have saved the FITS files from RAF format or from CFA frames (in XISF format) or from debayered frames (XISF format).

I use the following settings:

  • to create raw CFA images:
1590161097703.png

  • to debayer CFA images:
1590160996583.png

One thing I did not like with the RAF files, is that their EXIF does not contain (or I could not find it with exiftool) any info on temperature, unlike the CR2 files of Canon. So you have to keep notes on the temperature of each session, if you plan to re-use.
Indeed! Temperature probe is not integrated for sure.
 

Stéphiou

Member
May 16, 2020
15
2
I wonder if it is possible, that the extra optics these pixels have, to do phase detection, get confused as they are supposed to do, on an out of focus image, especially with a camera without lens attached (only cap on body)... Just wondering...
Sorry I have no idea at this moment, but I shoot only under full Manual mode, that means AF photosites should be not used, maybe that can help you:
  • Focus mode: M
  • AF+MF: OFF
  • Focus check: OFF
  • Shutter AF: OFF
  • Shoot without lens: ON

Find an interesting paper regarding Fuji CFA design:
 

Stéphiou

Member
May 16, 2020
15
2
I hope this will fix the issue.
I have checked the corrected MasterDark using FFT magnitude component.

First, I have compared different 256*256 regions on the same horizontal axis:
  • region #1 defined inside the square region
  • regions #2 and #3 defined 32pixels away respectively from the left and the right of the square region (32pixels in order to minimize amp-glow pollution in region #2)
  • region #4 defined inside the lateral high intensity amp-glow region
1590243470103.png

Only region #1 shows some power spectrum peaks.
No interested power spectrum peaks revealed in regions #4, #2 and #3. Just note the significative "fingerprint" related to amp-glow signal.
After some quicks maths according to the FFT magnitude component related to the region #1, we find the same periods or I would say more precisely, the same PDAF pixel pitchs (p): 3 pixels (highest 2048/p), 4 pixels (medium 2048/p), and 12 pixels (lowest 2048/p).


Second, I have compared the FFT magnitude component related to the 2048*2048 centre region (inside the square patern) of the not corrected MasterDark vs the corrected MasterDark to check the quality in terms of correction efficiency:

1590239188613.png 1590236591001.png
Left: FFT magnitude component related to the not corrected MasterDark
Right: FFT magnitude component related to the corrected MasterDark

The FFT magnitude component related to the corrected MasterDark shows that pixel pitches of 4 pixels and 12 pixels (related to vertical pixel pitches) are completely eliminated (y)
Just some very faint symetric dots (a little brighter than the background signal), which are related to the pixel pitch about 3 (related to horizontal pixel pitch), remain.
Because the "periodic" nature is still present in ($T-a) term, I would suppose that these faint dots could be subsequently completely eliminated:
  • if $T-a value is replaced by the mean or the median of the nearest neighbour pixels
  • another solution could be to replace the ($T-a) value by a random pixel intensity value of let's say between ($T-a)-stdev and ($T-a)+stdev, but don't know if this is possible with PixelMath
FFT magnitude component related to the 2048*2048 centre region of the corrected MasterDark:

1590246830739.png

Finally, related visual impact on the corrected MasterDark frame of these faint dots in power spectrum is not visible, that "mathematicaly" confirms:
  • the ugly square pattern is expunged
  • no more artefacts are added
  • the correction is very clean!
  • thank you again Bernd! (y)😁
 

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
Just some very faint symetric dots (a little brighter than the background signal), which are related to the pixel pitch about 3 (related to horizontal pixel pitch), remain.
Because the "periodic" nature is still present in ($T-a) term, I would suppose that these faint dots could be subsequently completely eliminated:
  • if $T-a value is replaced by the mean or the median of the nearest neighbour pixels
  • another solution could be to replace the ($T-a) value by a random pixel intensity value of let's say between ($T-a)-stdev and ($T-a)+stdev, but don't know if this is possible with PixelMath
Stéphane, I really appreciate your feedback, thank you.

It is true that there is a small residual irregularity in the corrected MasterDark, and it is related only to the AF pixels. This is visible as well in the histogram: the peak in the histogram is slightly distorted. In fact the stdDev of the corrected pixels is smaller than for the rest of the pixels in the rectangular AF-pixel region.

I'll take a closer look tomorrow and report back to you.

Bernd
 
  • Like
Reactions: Stéphiou

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
Hi Stéphane,

I created a new grayscale image (Image02), w=6032 and h=4032, value=0.01 and added (applying the NoiseGenerator process) a normal distribution noise with amplitude 0.00005. This amplitude was a guess.

Then I performed the correction of the "MasterDarks_LFC66-0.018%_NoBiasCal.xisf" with the following PixelMath equation:

Code:
iif(x()>1509 && x()<4520 && (x()-1)%3==0 && y()>504 && y()<3522 && ((y()-1)%12==0 || ((y()-5)%12==0)),$T-a+Image02-mean(Image02),$T)
This correction seemed to suit better, please try it.

Bernd
 

Stéphiou

Member
May 16, 2020
15
2
Hi Bernd,

I have created the new image as you have proposed, then have applied the new correction equation to the "MasterDarks_LFC66-0.018%_NoBiasCal.xisf" using this new image: I get the same result than using the 1st correction.
The symetric dots remain after FFT calculation, and each very faint dot show similar very faint intensity than using the 1st correction equation.
I suppose Image02-mean(Image02)~0 using NoiseGenerator process.

In my humble opinion, the focus should be to find a correction equation without the value $T which would introduce the periodicity.
Maybe we have to consider these pixels as hot pixels, in the spirit of the following? (ref.: David Ault - Advanced PixInsight PixelMath Operations)

1590438173207.png
 

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
Stéphane, I forgot to mention that you again have to complement:
Symbols: a=0.00038291931
when you perform the correction. Obviously you did not use the symbol a.

Verification with FourierTransform according your proposal in post #26 produced the following results after correction as described in my post #28: see "FT_results.JPG": there is no residual pattern in the previews 1 to 3.

Bernd
FT_results.JPG
 

Stéphiou

Member
May 16, 2020
15
2
Bernd, please run a Fourier Transform on the 2048*2048 centre region which is entirely included in the rectangular region defined by PDAF pixels.
Then check the magnitude component.
I got the following result showing 8 very faint dots related to a pixel pitch about 3 in X and Y axis:

1590513245643.png

Could I have screwed up somewhere?
 

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
You are right, in post #30 I only showed FFTs of the 256 x 256 previews. However, my FFT looks differently than your's. The centre FFT of the 2048 x 2048 preview is appended, see "DFT_magnitude_2048.JPG".

Then I did the correction by applying the mean of the 8 neighbors with the following PixelMath equation:
Code:
iif(x()>1509 && x()<4520 && (x()-1)%3==0 && y()>504 && y()<3522 && ((y()-1)%12==0 || ((y()-5)%12==0)),(Pixel($T,x()-1,y()-1)+Pixel($T,x(),y()-1)+Pixel($T,x()+1,y()-1)+Pixel($T,x()-1,y())+Pixel($T,x()+1,y())+Pixel($T,x()-1,y()+1)+Pixel($T,x(),y()+1)+Pixel($T,x()+1,y()+1))/8,$T)
The FFT of the centre 2048 x 2048 preview looks even better, see "DFT_magnitude_2048_neighbors.JPG". (I appended the FFTs in zoom setting 1:3. The appearance of the FFTs change considerably depending on the zoom factor though. Actually the FFTs should be viewed in zoom 1:1 or grater.)

Are there residual repetitive artifacts in the calibrated images visible when you use the former correction method?

The stdev of the 2048 x 2048 previews differ considerably:
Code:
            Preview_2048    Preview_2048_neighbors
count (%)   100.00000       100.00000
count (px)  4194304         4194304
mean        1017.487        1017.481
median      1016.474        1016.513
variance    80.442          37.624
stdDev      8.969           6.134
avgDev      4.267           4.115
MAD         3.355           3.219
minimum     981.263         981.263
maximum     6050.551        4426.895
This might be caused by some warm pixels.

Bernd
 

Attachments

Stéphiou

Member
May 16, 2020
15
2
Dear Bernd,

Then I did the correction by applying the mean of the 8 neighbors with the following PixelMath equation:
I have done the same correction (by applying the mean of the 8 neighbors) on "MasterDarks_LFC66-0.018%_NoBiasCal.xisf", but I do not get the same result than you because the 8 very faint dots remain:

1590621646149.png

Don't know how to explain that.


Are there residual repetitive artifacts in the calibrated images visible when you use the former correction method?
So, yes, residual repetitive artifacts are visible in the 2048*2048 centre FFT magnitude component related to a calibrated image when the former correction method is used...but no PDAF pattern is present in the 2048*2048 CFA calibrated image centre!

1590621006770.png
Left: 2048*2048 centre of a calibrated image
Right: related FFT magnitude component

And I get the same result with a not calibrated image.
 
Last edited:

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
I do not understand the differences between FFTs that you and I find.

I also don't understand what you want to express with the statement that there is a just very small residual pattern in the FFT. It is pretty normal that a section of a MasterDark shows a pattern in the FFT.

E.g. there is a slight difference in the stdDev of columns and rows of the corrected MasterDark (preview 2048 x 2048):
Preview_2048_neighbors 6.13

iif(x()%3==0,$T,0) -> 7.06
iif(x()%3==1,$T,0) -> 5.70
iif(x()%3==2,$T,0) -> 5.53

iif(y()%3==0,$T,0) -> 6.06
iif(y()%3==1,$T,0) -> 5.54
iif(y()%3==2,$T,0) -> 6.74


Probably this is caused by the setup of the sensor. However, in my view your corrected MasterDark is exceptionally clean in this respect, so I would not complain about the camera.

For me the actually interesting question is whether the calibration result is improved by such a correction of the MasterDark or not. In this respect, it seems doubtful to me to perform a removal of warm pixels from a MasterDark (by applying neighbor averaging for the AF pixels): we don't want to produce a clean MasterDark, but we want to improve the calibration result. So an improvement in the calibrated light frames has to be proved in order to justify a correction method. The right comparison would be: visually compare the calibration results using the uncorrected MasterDark with calibration results using the corrected MasterDarks. Other aspects don't seem to be useful in my humble opinion. The bottom line is: if there is no difference visible in the calibrated light frames regardless whether a corrected or uncorrected MasterDark was used, then this correction is unnecessary and should not be performed.

Honestly, I don't want to spend more time on devising and testing more correction methods for your MasterDark. I guess by now you know how the PixelMath process can be used and do it by yourself. If you have questions specific to PixelMath, it is probably better to create a new thread.

Bernd
 

Stéphiou

Member
May 16, 2020
15
2
Hi Bernd,

I agree with you to do not spend more time on finding new corrections.

As mentioned in post #17 and confirmed in post #26, I have concluded that your 1st correction works great, the ugly square pattern is expunged, no more artefacts are added, the correction is very clean, and so no further investigations have, by that time, been requested ;)

You can also find in post#17 the final integration of light frames calibrated with corrected MBias and corrected MDark by applying your 1st correction.
Concerning the differrences we discussed just above, I did a mistake somewhere for sure.

So, thank you again for taking time to investigate deeper with NoiseGenerator process or the “mean 8 neighbors” correction, giving intriguing results. According to your results, I suppose the "mean 8 neighbors" correction should be the correction giving the best calibration quality.

Finally we have defined and validated a method to correct a critical artefact generated by the X-TransIII sensor used under astrophotography conditions.

Additionnal very good point: this correction method should be so efficient and easily adjustable to other X-Trans sensor generations using PixelMath process in accordance with the PDAF related area properties (size, pixel pitchs, pattern etc): this point has to be demonstrated.

From my experience after testing some other dedicated astro processing softwares, PIXINSIGHT is the only one, at this time, to enable "state-of-the-art" full calibration of subframes acquired with X-TransIII sensor (calibration using MFlat not tested).
Thanks to the powerful PixelMath process!

So I'm pretty sure that X-Trans sensor astrophotographers are extremely graceful to you, Juan, and Pix Dev Team!

Kind regards,
Stéphane.
 

bulrichl

Well-known member
Nov 2, 2016
746
45
La Palma, Canary Islands
Stéphane, as mentioned in post #21,
I guess that you will need a MasterFlat in order to judge whether the application of dark frame optimization is advisable with this camera or not. The criterion is the complete removal of "amplifier glow", and that can be judged better when the vignetting is removed by flat field correction.
I encourage you to capture flat frames, prepare a MasterFlat and compare the results of a fully calibrated series with differently corrected MasterDark and MasterBias, and with / without the application of dark frame optimization.

The flat field correction will result in a much more aggressive STF Auto Stretch, and deviations between the different versions can be detected easily.

Bernd
 
  • Like
Reactions: Stéphiou