Author Topic: ImageIntegration Time Difference Between Linear Fit & Winsorized Sigma  (Read 2780 times)

Offline GeneralT001

  • Member
  • *
  • Posts: 71
Hi,

On my computer I can run 256 Bias frames through ImageIntegration to produce a Master Bias in about 3 minutes. If I leave all the settings the same but change the Pixel Rejection to Winsorized Sigma it takes like an hour to process the same Bias files. Why?

Also, running the Winsorized Sigma Pixel Rejection seems to get the "Pixinsight Not Responding" message fairly often if I try doing anything else while processing with Winsorized Sigma clipping.

I am on Windows 10, i7 6700K, 32Gig mem, SSD and using a RAM Disk of 16Gig.

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Hi,

Have you tried monitoring your PC performance whilst running the process? (Go to the START menu, type in Perfmon, and select the .EXE file that should be offered. Then you might want to consider monitoring your total commited RAM memory, as well as the %age of time used by each processor. There are a few useful YouTube videos out there that can help you with setting up PerMon - I have it running on one of my auxiliary monitors so that I can see 'what's happening' as PixInsight starts to munch its way through available resources!!).

The next thing that can (sometimes) make a difference is to set two parameters in the ImageIntegration process - the BufferSize and the StackSize. These normally default to BufferSize=16MB and StackSize=1024MB. However, the default options can (will ??) usually result in fairly intensive use of swap files. If you examine the ToolTips on these parameters, you will glean the following:

Set BufferSize = W x H x C x 4, and
Set StackSize = BufferSize x N, where

W = image width, in pixels;
H = image height, in pixels;
C = number of colour channels (e.g. 1 for Mono, 3 for RGB);
N = number of images being stacked.
(the multiplying constant of '4' should be changed to '8' for Linear Fit integration)

Doing this, for an image stack that I was recently working on (30 x RGB images, each 3800 x 2500 pixels) pushed my memory usage up to 80% ov my available RAM, and that was without doing linear fits.

Interestingly, the CPU usage for each of my 4 cores, was pushed hard against the 100% end-stop - demonstrating that PixInsight will happily grab all the CPU resources it can get at. And, because of that, you either accept that PixInsight 'has priority' (and so all other PC activity will be compromised), or you - as a mere human - try and 'get in its way' and make the CPU try to do stuff that PixInsight would rather you left alone.

Finally, as a matter of interest, for Biases, I would have thought that simple 'additive averaging' is all that is required - no statistical pixel rejection is required (nor is any noise estimate required). All pixel values, in all the images, have equal wright, and should be processed as such. And, it would be very questionable whether you would see any improvement in your Biases by processing as many as 250 subs (irrespective of how 'easy' it might be to acquire that many Biases in the first place).

What kind of camera are you using? Does it have TEC cooling? Is the TEC set-point reliable? If so, then you may have absolutely no need for Biases whatsoever, you would only need Darks, Flats and Lights.

Anyhow - I hope some of this helps.

Cheers,
Niall
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline GeneralT001

  • Member
  • *
  • Posts: 71
Wow, thanks for the reply. A lot of info. The camera is an ASI1600 with TEC cooling. You are probably right in that I don't need any Bias frames.

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Well, looking at the spec for the camera, you would need quite a lot of PC resources to be able to handle that many Biases, when the CCD sensor is 16M !!

Also, think carefully about your TEC setting. Here in the UK, I run my camera at -15C, a temperature that is easily achieved year in and year out. It is relatively pointless driving the temperature down to the lowest you can get. Apart from always then running the risk of icing up the CCD window, you may not be able to hit that temperature on a warmer night, meaning that you might then need to have to take a new set of darks just for that session.

I have one set of darks for each of the typical exposure times that I use (2.5s for my Flats and then either 300 or 600 seconds for my Lights), and I create a MasterDark for each of the three sets of 30 raw subs. Those MasterDarks have, in my experience, not changed with any real significance for the lasy six years. I tend to grab a new set of Darks 'every so often', but only if the mood suits me, and only if I have nothing better to do with my time. I have compared MasterDarks over the years (by using a PixelMath expression to subtract one image from another, and then looking at the result in the Image Statistics window - I usually see an Average ADU of zero, or very close to zero, suggesting that there is no significant difference between the images).

Spend more time on processing the Lights, and make your creation of the Calibration Frames as efficient as possible.

Hope this helps.

Cheers,
Niall
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline GeneralT001

  • Member
  • *
  • Posts: 71
Hi Neil,

I'm onboard with all that you have said. My only caveat with the ASI1600 is that there is no need to go longer than 1 - 2 min for LRGB (as low as 5 sec for L) and a max of 300 sec for NB. The camera has a low read noise so doesn't take much time to get a proper exposure - plus amp glow becomes more of an issue going past 5 min.


Thanks