Strange results and questions related to ImageCalibration and DrizzleIntegration

Stéphiou

Member
May 16, 2020
15
2
Hello,

I'm a beginner with Pix and this is my 1st thread in this forum :)
Firstly, thank you, Juan and all PixTeam for this powerfull tool!
Secondly, I'm facing some issues, don't know if I'm wrong with my process flow or if I'm reporting some bugs.

I'm shooting with the Fuji X-T20 X-TransIII sensor (it's start pretty bad) and I'm playing with Pix v1.8.8-5 Ripley (x64) under W10.
Shooting conditions under Manual mode (in my m81/m82 field example) are:
  • Bias: 100frames - 1/4000s @ 1600 ISO (mechanical shutter)
  • Darks: 20frames - 180s @ 1600 ISO (mechanical shutter)
  • Lights: 42frames - 180s @ 1600 ISO (mechanical shutter)

1. ImageCalibration process icon:
Ugly "square" pattern (related to green autofocus tracking photosites of X-TransIII sensor?) appears after ImageIntegration under most of ImageCalibration conditions with CFA files (see below, left picture). It seems there is only one set of conditions which eliminate this "daube" (see below, right picture):
  • without calibration of darks/MasterDark and lights with MasterBias
  • with optimized MasterDark + detect CFA
1589650932078.png 1589647706741.png
Left picture is obtained after calibration of darks and lights with MasterBias, showing amp-glow is correctly removed when ugly pattern remains.​
Right picture is obtained after only calibration of lights with optimized MasterDark (+detect CFA enabled), showing the ugly pattern is totally removed when amp-glow remains.​
So my 1st question is: have you an idea why this ugly pattern is not eliminated when ImageCalibration is done with MasterBias calibration (and with optimized MasterDark + Detect CFA)?
Please note that this "square" pattern can also be eliminated after ImageCalibration with debayered files (lights...).​


2. WBPP (v1.4.5):
If I set exactly the same successfull ImageCalibration conditions (without MasterBias but with optimized MasterDark) and use exactly the same files when I run WBPP, I do not get the same result than after running ImageCalibration process icon:

1589651137224.png
So my 2nde question: have you an idea why WBPP script do not give the same successful results than "manual" process flow (process icons)?
Because we set exactly the same Debayering parameters.

3. Drizzle Integration after cropping single debayered lights:
I would like to extract the background from single debayered lights in order to eliminate a huge part of the gradients before ImageIntegration.
Before background extraction, I run a Crop step in order to suppress distorted field and vigneting occuring all around calibrated light frames.
So ABE process applied on cropped calibrated light frames does the job with function degree set to 2.
But later in my process flow, after DrizzleIntegration, I get the following result:

1589656041935.png

This is a part of my initial field with distorted stars and vigneting.
After checking logs in Process Control tab, I realize that DrizzleIntegration use latest CFA frames (calibrated light frame after cosmetic correction in my case), which are not cropped. OK, so I check in Pix forum and I found that it is perfectly normal: Thank you rjbokleman for the thread and Juan for your clear answer!
So I have moved my Crop step just after the CosmeticCorrection, as the latest process step before Debayer process.
Then I get the following result (and same result with DynamicCrop) after Debayer process (at zoom 1:1 on m82) of the sequence:

1589656391753.png

It seems that Debayer process doesn't like cropped CFA frames.
Is it normal?
What strategy is recommend for extracting background from single frames before DrizzleIntegration?


Thanks for your reading.
Stéphane.
 

Attachments

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
Hi Stéphane,

short answers:

1. + 2. Neither dark frames nor the integrated dark frames should be pre-calibrated with the MasterBias. Calibration of the MasterDark is only needed when dark frame scaling (in PixInsight: dark frame optimization) is applied. It can be performed in the calibration of the light frames.

What do you understand by "MasterBias calibration?"
What do you understand by "optimized MasterDark"?

For a better understanding of the ImageCalibration process, please read my guide: https://pixinsight.com/forum/index.php?threads/for-beginners-guide-to-pis-imagecalibration.11547/ .

3. You must not rotate or mirror or crop CFA data! These operations of course will change the bayer/mosaic pattern.

Bernd
 
Last edited:

Stéphiou

Member
May 16, 2020
15
2
Thank you Bernd for your quick feedback.


1+2:
Sorry I was not clear:
- I have also tested the two following different ways giving the same result:
- MasterDark obtained by integration of calibrated dark frames (with MasterBias)
- MasterDark obtained by integration of dark frames, then MasterDark calibration with MasterBias enabled during calibration of lights frames (with MasterBias and calibrated MasterDark)
- "with MasterBias calibration" meant with calibration of light frames and dark frames with MasterBias
- "optimized MasterDark" meant dark frame optimization is enabled

I'm just trying to find the best way to work with frames acquired with X-TransIII sensor.
I have also calibrated lights frames with different MasterDarks showing different pixel rejection values, and MasterDark and lights frames with different MasterBias showing different pixel rejection values, without any success to remove this ugly pattern.
Please find below a comparison of different ImageCalibration conditions with MasterDark and MasterBias showing pixel rejection value respectively of 0.018% and 0.022%:

1589664130670.png
- MasterDark not calibrated with MasterBias
- Integration of light frames calibrated with MasterDark only (dark frame optimization disabled)
=> ugly pattern

1589665100395.png
- MasterDark calibrated with MasterBias
- Integration of light frames calibrated with MasterDark and MasterBias (dark frame optimization disabled)
=> ugly pattern

1589664654802.png
- MasterDark calibrated with MasterBias
- Integration of light frames calibrated with MasterDark and MasterBias (dark frame optimization enabled)
=> ugly pattern

1589670937221.png
- No reduction of light frames (MasterDark and MasterBias not used)
- Integration of light frames
=> ugly pattern totally removed, but amp-glow signal remains

1589673661687.png
- Integration of light frames calibrated with MasterDark only (dark frame optimization enabled)
=> ugly pattern totally removed, but amp-glow signal remains

Using WBPP, I have not found any good calibration conditions of light frames in order to removed this ugly pattern as well.

So, it seems good data reduction of frames (with MasterDark only or with MasterDark and MasterBias) captured with X-TransIII sensor is not possible, that means we can work only with light frames, correct?

But I do not understand why the ugly pattern is also removed after calibration of lights only with MasterDark and dark frame optimization enabled.



3. I didn't know that, thank you for these information!
So, if we want to use DrizzleIntegration with background extraction from each calibrated lights, we have to extract the background from each calibrated lights between Debayer process and registration, without cropping calibrated lights previously: correct?

Thank you in advance.
 

ngc1535

PTeam Member
Feb 1, 2014
526
65
AdamBlockStudios.com
There is so much stuff- things are too complicated here and need simplification.
If I might suggest- I do not think your data qualifies for drizzling. It isn't undersampled- so you might consider removing this bit of the operation for the moment.
-adam
 

Stéphiou

Member
May 16, 2020
15
2
Dear Adam, thanks for your reply.
Yes you are right.
The few users of X-TransII&III&IV sensors merit a dedicated pre-processing guide definitively.
 

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
I agree to Adam. It is difficult to follow what you did.

If possible, please upload a light, dark and bias frame each (in FITS format), the not pre-calibrated MasterDark and the MasterBias (both in XISF format) to a filehoster and post the link. I will take a look at it.

Bernd
 

ngc1535

PTeam Member
Feb 1, 2014
526
65
AdamBlockStudios.com
First of all..that camera does not have a normal bayer matrix. It is a completely different pattern.
I am not aware that PixInsight (or any software) can handle this?
I can only hope this explains the square bit.
Some of your frames were completely black.
I suspect it would be necessary to start form the raw frames and not masters.

The First image is correct. The amp glow is gone.
Does a single raw light frame have the square? You did not demonstrate this. However...based on your last image the square is not there.
The square needs to be in the LIghts and the Darks.
If it is not in both... nothing is going to work correctly.

-adam
 

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
Stepháne, thank you for uploading the files.

The bias frame, the MasterBias and particularly the dark frame and the MasterDark contain 506,016 pixels in a rectangular central region (x=1510, y=505, w=3010, h=3017) that have increased intensity values. Probably these pixels are the AF pixels. Strangely this intensity increase is NOT detectable in the light frame. This is the reason for the artifact that you showed in calibrated light frames and the final integration result.

I am curious whether you used a different camera driver or another acquisition software or file format for light frames than for bias / dark frames, or whether different camera settings were used. I have no other explanation for these deviations. If this is not the case, this artifact has to be corrected (in the MasterBias and the MasterDark). This possibly will be achievable with PixelMath. Tomorrow I will try to develop a correction method for the affected pixels in the MasterBias and the MasterDark.

Another issue is that PixInsight cannot detect the mosaic pattern from the FITS files, so the 'Auto' setting in the Debayer process will not work with your frames. However, the basic problem in this case is: whereas a bayer pattern can be set explicitly in the Debayer process, this is not possible with your mosaiced FITS files since a 'X-Trans' setting is currently not available in the Debayer process. The only workaround for this that I am aware of would be to use RAF file format instead of FITS file format. If you want to try this approach, please note that then ALL frames of a project (light, dark, bias and flat frames) must be in RAF format.

Bernd
 
  • Like
Reactions: ngc1535

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
Hi Adam,
Bernd... so I was on the right track?
you were right that this camera (sensor) doesn't have a bayer pattern, it is the more complex X-Trans mosaic pattern (see https://en.wikipedia.org/wiki/Fujifilm_X-Trans_sensor ).

However, the increase of intensities in some pixels in a rectangular region of the sensor is not related to the mosaic pattern but to special Auto Focus photosites that are arranged in the rectangular region in a specific geometric layout. I don't understand why this artifact did appear in dark and bias frames but not in light frames. I hope that Stéphane will answer my question concerning different camera driver / acquisition software / file format. If this is the cause of the issue, a correction of MasterBias and MasterDark would not be reasonable, but freshly captured calibration frames would be the correct solution.

Bernd
 

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
Hi Stéphane,

I am still awaiting a reply to my questions:

Did you use a different camera driver or another acquisition software or a different file format for light frames than for bias / dark frames, or did you use different camera settings?

Bernd
 

Stéphiou

Member
May 16, 2020
15
2
Dear Bernd, Adam,
Many thanks for your reply!

Sorry for my late feedback, I was working on this issue.
For your information, it is possible to (visually) see this large-scale pattern at the surface of the sensor under particular low-angled light.
This square central region related to Fuji X-T20 sensor is ~3000*3000 size and image artefact is known since X-TransII (X-T1, X-T10...) sensor was lunched (in 2014?)...this is why I have mentioned "it's start pretty bad" in the introduction of my first message.
I note also that this square area seems to increase with X-Trans generation (lower size for X-T10 X-TransI sensor).

Concerning the source of this square pattern, just apply ChannelExtraction on a debayered dark frame, you will get:
1589895978550.png 1589896024277.png 1589896183906.png
Blue channel (bloody) Green channel Red channel

So this demonstrates this ugly square pattern is only related to (AF) green photosites in the 3000*3000px central region.
As a matter of fact, green photosites are the most sensitive for movement tracking vs red photosites and blue photosites.


Does a single raw light frame have the square?
Using Pix or any photography softwares, raw frames caputered with X-TransII senbsor (lights/bias/darks in .RAF format) have not this square pattern...fortunately for Fuji! 😁


I am curious whether you used a different camera driver or another acquisition software or file format for light frames than for bias / dark frames, or whether different camera settings were used.
I just use my X-T20 as is, do not think some acquisition softwares are compatible with Fuji system...
Fuji raw format is .RAF and the same following settings are applied to capture lights/bias/darks:
Firmware: Fuji 2.01 (latest)
RAW recording: Uncompressed
Long exposure NR: OFF
Pixel mapping: I do it one time just before starting my session (before shooting lights/darks/bias)


The square needs to be in the LIghts and the Darks.
The square pattern must be present in light frames, dark frames, and bias frames, from my point of view.
Please find some comparisons between individual CFA frames and Masters below:

Comparison of one CFA bias frame vs one CFA dark frame vs one CFA light frame
1589886846213.png 1589886985484.png 1589887101824.png
Individual CFA bias frame Individual CFA dark frame Individual CFA light frame

Comparison of MasterBias vs MasterDark vs Integration of not calibrated light frames (no substraction of MasterDark neither MasterBias):
1589887858215.png 1589887946153.png 1589888224600.png
MasterBias MasterDark After integration of not calibrated light frames


Shutter speed related to bias frames and dark/lights frames is respectively 1/4000s and 180s.
Noise intensity related to AF green photosites seems to be much lower than read noise and this square pattern seems to be acquisition time dependent

At first sight, this square pattern is only "observed" (by STF auto-strech, boosted or not) on individual dark frames, not on bias frames and light frames, while stacking bias frames (MasterBias) reveal this square pattern.

But I do not understand why this square pattern is not revealed after integration of not calibrated light frames.
Could be interested to compare individual darks (and related MasterDark) captured at relative higher acquisition times (let's say 30mins or 1hour) at the same temperature (not easy), or rather at the same acquisition settings but at relative higher sensor temperatures (easy 😁), just in order to increase thermal noise signals.
Maybe Fuji engineers have integrated a specific correction operation in signal processing chain to suppress this square pattern when captured signal is above a certain signal threshold, who knows...

Let us know how to suppress this ugly square pattern by ourselves.
Zooming on a region at the frontier of this square pattern from the MasterDark:
1589892323352.png

We observe periodic bright green pixels in X and Y axis with different periods in the 3000*3000 area:
  • in X axis: one "period" about 3 bright green pixels of one vertical line of pixels
  • in Y axis: one "period" about 12 bright green pixels of two horizontal lines of pixels separated by 4 pixels
So I'm pretty confident that it is possible to suppress this noisy square pattern by FFT processing or Pixelmath operations, with ROI defined according to the size and position of this square area. But I have never played with FFT or Pixelmath for now, and the size of the square could be a problem with FFT.

Maybe also Juan has some ideas.

Stéphane.
 
Last edited:

Stéphiou

Member
May 16, 2020
15
2
Please find one raw frame for each light/bias/dark sequence here:


At this moment, I can say that after integration of not calibrated lights or calibrated lights, we obtain an integration result:
  • without the ugly square pattern:
    • if CFA light frames are calibrated only with MasterDark + dark frame optimization enabled + detect CFA
    • if CFA light frames are not calibrated
    • if RGB light frames are calibrated with/without RGB MasterDark and with/without RGB MasterBias, but this is not the right way to do!
  • with the ugly square pattern:
    • if CFA light frames are calibrated only with MasterDark + dark frame optimization enabled + force CFA (I have just checked after calibration of light frames under these conditions: ugly square pattern remains in debayered individual calibrated light frames)
    • if CFA light frames are calibrated only with MasterDark + detect CFA (dark frame optimization disabled)
    • if CFA light frames are calibrated with MasterBias + MasterDark calibrated with MasterBias + dark frame optimization enabled + detect CFA
    • if CFA light frames are calibrated with MasterBias + MasterDark calibrated with MasterBias (dark frame optimization disabled)

Screenshot of integration results (with boosted STF AutoStretch):

1589900116966.png 1589899538081.png 1589899500660.png
Left: CFA light frames are not calibrated
Center: CFA light frames are calibrated only with MasterDark (dark frame optimization disabled)
Right: CFA light frames are calibrated only with MasterDark + dark frame optimization enabled + detect CFA


1589899870124.png 1589899992159.png 1589900760032.png
Left: CFA light frames are calibrated with MasterBias + MasterDark calibrated with MasterBias (dark frame optimization disabled)
Center: CFA light frames are calibrated with MasterBias + MasterDark calibrated with MasterBias + dark frame optimization enabled + detect CFA
Right: RGB light frames are calibrated with MasterBias + MasterDark calibrated with MasterBias + dark frame optimization enabled + detect CFA



Too bad that we can not obtain a good result, i.e. without the square pattern, if CFA light frames are calibrated with MasterBias and calibrated MasterDark!!!

For information:
  • X-TransI sensor (X-M1...) does not integrate movement tracking functionnality (so no ugly square)
  • X-TransII-III-IV sensors (X-T1, X-T10, X-T2, X-T20, X-T3, X-T30, X-T4...) integrate movement tracking functionnality...
  • ...but don't know if X-T4 gives this artefact (I'm pretty sure X-T3 and X-T30 do)
Hope we will find a solution for the community of Fuji X-Trans users!
 
Last edited:

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
As I wrote in my previous post (#9) the rectangular pattern is detectable both in the MasterBias and (stronger) in the MasterDark.

Please do the following:

1) Load the MasterDark ("MasterDarks_LFC66_0_018__NoBiasCal.xisf")
2) Make this preview: (x=1510, y=505, w=3010, h=3017)
3) Save the MasterDark with option 'Preview rectangles' in section 'Embedded Data' enabled
4) Make the MasterDark (not the preview) to the current view
5) Open PixelMath, copy and paste the following PixelMath operation:
Code:
iif(x()>1509 && x()<4520 && (x()-1)%3==0 && y()>504 && y()<3522 && ((y()-1)%12==0 || ((y()-5)%12==0)),0,$T)
and execute
6) Open Statistics with option 'Unclipped' disabled (this is the default) and jot down the mean value of the preview (= 1017.746)
7) Make the MasterDark (not the preview) to the current view
8) Undo the PixelMath operation
9) Edit the PixelMath equation by swapping '0' and '$T' at the very right
and execute
10) Open Statistics with option 'Unclipped' disabled (this is the default) and jot down the mean value of the preview (= 1042.841)

This shows:
From 6): the mean of the pixels in the preview except the AF pixels is 1017.746
From 10): the mean of the AF pixels is 1042.841
The difference (= 25.095) is the value by that the AF pixels are greater than normal pixels in the preview for the MasterDark. Normalized to the range [0,1], this is the value 25.095/65536 = 0.00038291931

Now applying the correction:
11) Make the MasterDark (not the preview) to the current view
12) Undo the PixelMath operation
13) Edit the PixelMath equation: replace ",$T,0)" by ",$T-a,$T)" at the end of the equation,
in the Symbols field, input a=0.00038291931
and execute

Save the modified MasterDark as "MasterDarks_LFC66_0_018__NoBiasCal_new.xisf".

Accordingly the MasterBias shall be corrected (the correction value has to be determined analogously.

These new MasterDark and MasterBias should be used for the ImageCalibration. I hope this will fix the issue.

Bernd
 

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
As long as you don't have flat frames (that would have to be calibrated with the modified MasterBias), you don't need the modified MasterBias at all.

Calibrate the light frames with the modified, not pre-calibrated MasterDark only.

Bernd
 
  • Like
Reactions: Stéphiou

Stéphiou

Member
May 16, 2020
15
2
Dear Bernd, your method works great using MasterBias and MasterDark!

1589958572589.png1589961093063.png1589960911542.png
Left: new MBias
Centre: new MDark
Right: integration of CFA light frames calibrated with the new MBias + new MDark calibrated with the new MBias + dark frame optimization enabled + detect CFA

We got a plan! Brilliant! (y)(y)(y)
And thank you so much for the class!
 
  • Like
Reactions: Yannis Doukakis

Yannis Doukakis

Active member
May 19, 2020
31
0
Thessaloniki, Greece
The exact area that seems to give problems is where the Phase Detection pixels are located. That's what we see in the above pictures. These are the "periodic" pixels we see in the above picture.
Probably, as these are on the sensor "left" and "right" pixels, to aid focusing, that's why on a well focused light frame you do not see them. I saw them also on my cameras on Bias and Darks. The actual pixels may be smaller? different bias?
I used BatchPreProcess and did not have any visual artefacts on the resulting image.
Does anyone have darks/flats from Sony A6XXX which share the same chip as Fujifilm (different CFA)? Do they have the same issue or Sony uses different ways to "hide" the AF pixels?
 

Yannis Doukakis

Active member
May 19, 2020
31
0
Thessaloniki, Greece
Maybe we have to mark all of these pixels as bad and forget about them. PixInsight may offer a script to do this automatically. I do not own one, but in X-T3 / X-T4 the area would probably be bigger, covering 96% of the sensor.
We can test various machines, if pixels are located in the same exact position in all sensors. If needed I can send you dark frame for X-H1 / X-Pro2.
 

Yannis Doukakis

Active member
May 19, 2020
31
0
Thessaloniki, Greece
Another thing that came to my mind. How do you make the darks/biases? With or without a lens/telescope attached? I wonder if it is possible, that the extra optics these pixels have, to do phase detection, get confused as they are supposed to do, on an out of focus image, especially with a camera without lens attached (only cap on body)... Just wondering...