Author Topic: Winsorized Clipping for Dark Frames  (Read 707 times)

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Winsorized Clipping for Dark Frames
« on: 2018 December 21 09:34:02 »
Hello,

While integrating dark subframes I am noticing that I cannot use Winsorized Clipping as it causes the process to take a very long time. When I use Sigma Clipping with dark subframes the process is as expected.

Is Winsorized Clipping not appropriate for dark subframes?


Thanks

Farzad

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1416
  • We have cookies? Where ?
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #1 on: 2018 December 21 11:27:08 »
Hi Farzad,

Well, I'm going to stick my head well above the parapet here, as it has been a long time since I read any of my Statistical Analysis textbooks (and probably equally as long since I experimented with the setting options in PixInsight!!).

However, both of the forms of Sigma Clipping that you referred to require a data set (in our case, 'the ADU values of a single pixel in a group of images') that contains data with some form of (more or less) 'normal distribution' of data values. From the data set, it is then possible to determine the 'mean' of the data values, and the 'standard distribution' (the 'sigma' values) of the ADU data about that mean.

When it now comes to 'clipping', what is being referred to is the treatment of any individual ADU value in the data set, should that value lie outside the standard distribution level (or some scaled value thereof).

In the case of 'basic' Sigma Clipping, all ADU values outside of the distribution limits are replaced with zero-value data. In contrast, Winsorised Sigma Clipping replces the out-of-limits ADU values with the actual ADU value at the limit.

In both cases, the original, and now moodified, images are re-evaluated - repeating the calculation of mean and SD, and the production of a further set of data. The process iterates until no further improvement is obtained for the mean value, thus giving the final image.

However, crucial to the success of these clipping methods is that the original data set (hence images) contains data with a distribution curve that allows the analysis to take place (i.e. there has to be 'some' SNR to work with).

If the data has very poor SNR, the clipping processes have little or nothing to work with. This is the case with Biases and Darks - in these cases, a simple arithmetic mean is all that is required when creating Master frames.

FlatLights and ImageLights do benefit from statistical clipping however, especially if the image dataset have been fully calibrated to maximise the SNR in the first place.

I hope this very short explanation is
a.) correct, as best as I can remember things
b.) helps you understand what is going on
c.) encourages you to at least look up Sigma Clipping on the internet
d.) prompts others with a better grasp of the process to add their comments
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #2 on: 2018 December 21 12:32:36 »
Thanks for taking the time.

Your "short" explanation does not say "if" what I am experiencing as what causes integration crippling (Winsorized Clipping) is or is not appropriate to use to integrate dark frames.

I have done more than just research Sigma Clipping online, and frankly, as impressed as I am with the internal sophistication of PI (which is why I am sticking with it), I don't feel like I need to know so much of the under-the-hood activity at the image processing levels that I am currently involved with.

And yes, I am also hoping that someone else will eventually respond to my question and tell me if Winsorised Clipping is not the right method for integrating dark frames, and if it is, then why should PI get its winds knocked out of it when I get it to integrate 20 dark frames using Winsorized Clipping.


Thanks again.


Farzad

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4018
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #3 on: 2018 December 21 12:43:04 »
what kind of files are the input files? i use windsorized sigma-clipping all the time on 8300M darks/bias/lights all the time with no performance impact.

if you are integrating directly from CR2 or NEF files, this might have something to do with the problem.

rob

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #4 on: 2018 December 21 12:46:08 »
These are files that come out of my monochrome camera. I think they are the dot fit ones. I used to have great experience with these as well until a few days ago when we underwent two major updates to PI.

Farzad

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4018
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #5 on: 2018 December 21 13:17:09 »
OK - so its not the CR2 thing. i have not done a large integration since upgrading PI so i will have to see what happens next time i do that. but in short i dont think it should be normal for this to happen.

rob

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #6 on: 2018 December 21 13:26:34 »
Thanks, Rob. Yes, it is the general consensus in various communities that Winsorized Clipping, however an overkill for integrating Dark frames, should not be behaving in the way that it does.

I am hoping that my reporting this could be of some use to the developers.

Farzad

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #7 on: 2018 December 21 19:35:04 »
Here is a snapshot of my integration settings. I am integrating 38 subs in this case and it is taking a while.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6415
    • View Profile
    • http://pixinsight.com/
Re: Winsorized Clipping for Dark Frames
« Reply #8 on: 2018 December 22 04:06:00 »
Hi Farzad,

Winsorized sigma clipping is a perfectly adequate pixel rejection algorithm to integrate 38 dark frames. The problem you are reporting is very strange and I can't figure out any reason for it.

As always, to solve a problem I have to understand it first. To understand it, unless the issue is trivial—which is not the case here—, I need to reproduce it. Can you please upload a set of these raw frames where the problem can be reproduced? That would be of great help to diagnose it.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #9 on: 2018 December 22 08:25:10 »
Hi Juan, thanks for the reply. Yes, I have been studying Winsorized and it is a fantastic algorithm and it doesn't look like it shouldn't be used. I am uploading the files to DropBox. Once they are uploaded in a few minutes they can be reached using this link:

https://www.dropbox.com/s/z3vlm0g89xtj204/DARKS%20-20%2C%20300S_300sec_1x1__None_frame40.zip?dl=0

Thanks again.

Farzad


Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6415
    • View Profile
    • http://pixinsight.com/
Re: Winsorized Clipping for Dark Frames
« Reply #10 on: 2018 December 26 05:47:30 »
Hi Farzad,

Thank you for uploading the data. The problem with these dark frames is that they use a very limited range of pixel sample values. On average they have an effective bit depth of about 6.5 bits.

To verify this, you can use the following script:

Code: [Select]
/*
 * Effective dynamic range by histogram quantization.
 *
 * This script can evaluate ranges not larger than MAX_RANGE.
 * To evaluate ranges smaller than the 16-bit range, set MAX_RANGE to 65536 below.
 * This will speed up calculation considerably, and will require much less
 * memory than the default value.
 */
#define MAX_RANGE 10000000

function CountValues( image )
{
   for ( var c = 0; c < image.numberOfChannels; ++c )
   {
      // Generate a 16-bit histogram for the current channel
      var H = new Histogram( MAX_RANGE );
      image.selectedChannel = c;
      H.generate( image );

      // Count the number of nonzero histogram levels
      var N = 0;
      for ( var i = 0; i < MAX_RANGE; ++i )
         if ( H.count( i ) != 0 )
            ++N;
      console.writeln( format( "   %2d   | %8u | %5.2f |", c, N, Math.log2( N ) ) );
      console.flush();
   }

   image.resetSelections();
}

function main()
{
   var window = ImageWindow.activeWindow;

   if ( window.isNull )
      throw Error( "There is no active image window!" );

   console.show();
   console.writeln( "Calculating effective dynamic range..." );
   console.writeln( "<b>", window.currentView.fullId, "</b>" );
   console.writeln( "Channel |   Values |  Bits |" );
   console.writeln( "--------|----------|-------|" );
   console.flush();

   CountValues( window.currentView.image );
}

main();

Just open Script Editor, create a new JavaScript source code document, copy the code above, and save it as a new file with the .js suffix. Load one of your dark frames and run the script. As you'll see, each of these dark frames use 90-95 discrete pixel values, which means that they are approximately 6.5-bit images. Our ImageIntegration tool has not been designed to work with extremely reduced bit depths like these.

Another problem is that these images have a median absolute deviation (MAD) of zero. This happens because more than one half of the pixels are equal to the median. This property, along with the reduced bit depth noted above, causes these frames to provide very little information about the dark current of your camera. To visualize this problem, you can use the following PixelMath expression:

$T == median($T)

Since MAD is zero, you can't use any statistical estimator of variability based on MAD, such as MAD itself, BWMV, Sn, or Qn. To integrate these frames, you have to use the average absolute deviation from the median, or the percentage bend midvariance, which don't use and are not related to MAD.

A different, although probably related problem is the fact that Winsorized sigma clipping is indeed very inefficient for these images. I have to investigate why exactly this happens, but I am sure the algorithm is performing a huge amount of iterations because the limited range of values makes it very difficult to find convergence.

I hope this analysis will help you understand and diagnose the problem.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #11 on: 2018 December 26 06:35:55 »
Thanks for taking the time and evaluating these images. I will give the script a try.

I have a couple of questions for you:

[1] Could the quality of these dark frames indicate something is wrong with the camera?

[2] When you say "...Winsorized sigma clipping is indeed very inefficient for these images" are you saying it is inefficient because the flaws or anomalies associated with these images in particular or dark frames in general?

Farzad

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6415
    • View Profile
    • http://pixinsight.com/
Re: Winsorized Clipping for Dark Frames
« Reply #12 on: 2018 December 26 08:18:03 »
Hi Farzad,

You're welcome!

Quote
[1] Could the quality of these dark frames indicate something is wrong with the camera?

I don't think so. This looks more like wrong configuration settings in the software you have used to acquire these dark frames. This is just what I suspect, but I may be wrong of course. You'll have to investigate why this happens, but at any rate, 6.5-bit data is clearly wrong.

Quote
[2] When you say "...Winsorized sigma clipping is indeed very inefficient for these images" are you saying it is inefficient because the flaws or anomalies associated with these images in particular or dark frames in general?

I refer to this particular data set. Winsorized sigma clipping works well for any type of images, including dark frames.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #13 on: 2018 December 26 08:32:02 »
Thanks, Juan.


Offline Farzad_k

  • PixInsight Addict
  • ***
  • Posts: 139
    • View Profile
Re: Winsorized Clipping for Dark Frames
« Reply #14 on: 2019 February 06 16:08:01 »
Hello, Juan.

I am back, looking at this issue a little closer, trying to see if I can solve the issue.

I hear you saying that the condition of these dark frames does not necessarily mean something is wrong with the camera and it could be in the image capture settings. Also that the 6.5 bit is just wrong.

I have acquired new dark frames, watching the camera settings, and I am just using HDR setting of my ZWO ASI 1600 with zero gain and 100 for offset, 16 bit. Besides these there are no other features that I can set. What should be the right bit for the dark frames and should there be an exception that the bits are different for light frames than dark frames. I am sorry that I don't understand this bit bit - I have compared a comparable light frame using your script and it shows bit of more than 10.

You recommended a certain method of integrating these low bit dark frames. I wonder if the resulting integrated dark master would be useful to calibrate my light frames at all.

Thanks a lot.

Farzad