Buffer / Stack option - image integration

[Peter S]

Hello,
I am new to this forum and a beginner with PI, mostly taking landscapes and now trying to combine landscape + milky way.
When stacking images I got "out of memory" with various configurations.

I read the documentation and applied the formula for stack.

I started with 10 lights (no darks, na flats, no bias), each 11608 x 8708 pixels,  RGB, TIFF 16 bit, each image appr. 580 MB size
on a Windows 10 64 bit Intel i7 3930k @ 3.2 GHz with 32 GB RAM, Pixinsight 1.8. 64 bit.
setup:
Combination: Average
Normalization: Additive with scaling
Weights: dont care or Noise evalution
Scale estimator: iterative k-sigma

The maximum I achieved until now was stacking 3 images without "out of memory"


Any recommendations concerning settings buffer and Stack ?

Thx, Peter

[Niall Saunders]

Hi Peter,

Let's move back a stage - what camera are you using, how are you capturing the image, and how are you storing ut (prior to trying to import it into PixInsight)?

An image of the size that you are quoting is more than a little bit impressive! Is this image the result of having already created a panorama outside of PI?

If so, does this mean you are trying to stack 10 panoramas on top of each other? How did you build each panorama? What process did you use to create the alignment and overlap of the images?

If you have a large number of image frames, all taken whilst pointing at different 'image centres', then it would be more 'normal' to use the Mosaic-building capabilities of PixInsight to aling, stack, and 'normalise' all of the images (to eliminate the 'seams' of your mosaic). This way, you would only ever have 'one' (very) large image.

In any case, even if you left the ImageIntegration values for the Buffer and Stack sizes at their default settings, you should not run  out of memory. You 'might' incur a time penalty, but that should be trivial with your i7 CPU. You should read up on Windows <PerfMon> utility - this allows you to visualise key parameters of your hardware on a 'live' graph. You can monitor CPU and Thread usage, RAM usage, SwapFile usage, etc.

Come back to us with more information, if you can.

[Peter S]

Hi Niall,
thanks for your reply!
Firstable an update / further informations:

It´s a PhaseOne XF with IQ3 which has natively 11608 x 8708 pixels (100 MPX....) - so there is no mosaic or panorama.
The big files resulted from developping the RAWs with CaptureOne Pro (TIFF, 16 bit).

I did 10 lights, 10 darks, ...bias....flats. The lights were taken with tracking (Polarie, Vixen), the stack of lights shows very good alignement.

(the only issue that I have with those images is a not 100% perfect focus)

Firstable I tried PI "preprocessing tool" and took all images (darks, bias, lights.....).
When I encountered the memory issue I reduced and took only the lights (TIFF).

I also tried  PI Image integration process

Now I tried with the PhaseOne RAW files (each 142 MB, RGB = 3 channels a 16 bit) : same problem.
Monitoring the resources (PerfMon utility) showed several GB "free" / unused during the whole process until suddenly (shortly before finishing?) there comes the "out of memory".
Only one time I had a Windows warning (out of memory)  - while the PerfMon showed 8GB unused.
I suppose there are peaks that are not monitored.

Another problem is that I have to decide now wether to buy PI or not -  because I am running trial period........

May there be a problem because the images show stars / milky way AND landscape?
- but the stack of three shows blurry landscape (as expected) and nice stars

I´d rather prefer to get PI running than to use command-file-utilities.....

[Niall Saunders]

Hi Peter,

OK - every day is a learning day - and today I learned that I will never have enough money to buy all those pixels 

But, we still need to think through your problem(s).

Firstly, I would be happy to state that the issue of 'landscape' being present in your image data should not make any difference to the stacking of your ten 'mega-images'. PixInsight is, basically, just stacking a lot of individual pixel-columns - each with 3 x 10 16-bit values (3: one for each of R, g and B; 10: one for each image). It doesn;t care 'what' the values are - each will simply be between 0 and 65535.

I also do not see why memory restriction should be an issue either. If you simply leave the Buffer and Stack parameters in the ImageIntegration Process at their default, then PI will just take (much) lomger to stack the images as it will only be able to work on smaller sections of data at a time (in comparison to the overall image).

The Buffer Size that you would require to handle you images is based on the formula "W x H x C x 4" where your W = 11608, your H = 8708 and your C = 3. This gives you a requirement for 3.389 GB of RAM for this.

The stack size is based on "W x H x C x 4 x N" where everything is as above, with N = 10. So, you need a further 33.9 GB of RAM for this.

This means a total of around 38,175 MB of RAM to avoid having to swap data to and from your working drive(s).

But, you only have 32GB of RAM fitted to your machine - so there might be issues - especially considering that there will be a finite amount of RAM taken up running the OS and any other concurrent packages at the same time as PixInsight (PerfMon might be bale to give you a good 'base-line' of RAM usage prior to loading PixInsight, and then repeated agian just before starting the ImageIntegration Process).

What happens if you take the RGB TIFF images, but only try stacking 9 instead of 10? (Or 8, or 7, ...)?

Perhaps Juan, or other PTeam members will jump in and comment?

[Andres.Pozo]

Have you tried to convert the TIFFs to XISF before preprocessing?

[Niall Saunders]

Hi Andres,

Would the XISF concersion actually make any worthwhile difference? Yes, the images could be 'condensed' into a smaller file size whilst still 'on disc'. However, once they are read-in to PixInsight for subsequent processing (of any type), wouldn;t they simply be expanded back to 'full-size' and therefore require just as much RAM as they would had they been read in as TIFF files?

[Andres.Pozo]

Niall, ImageIntegration doesn't need to load all the images at the same time when the image format allows to read them by rows. In this case PI can process really big datasets with "normal" amounts of memory.

FITS and XISF allow this. The RAW formats don't. But I don't know TIFFs are read. This is why I suggested to change the format of the files.

Andrés.

Hi Andres,

Would the XISF concersion actually make any worthwhile difference? Yes, the images could be 'condensed' into a smaller file size whilst still 'on disc'. However, once they are read-in to PixInsight for subsequent processing (of any type), wouldn;t they simply be expanded back to 'full-size' and therefore require just as much RAM as they would had they been read in as TIFF files?

[Peter S]

Hi Niall and Andres,
....yes I know it sounds crazy ($$$) but it´s an upgrade, an interest-free loan and bought by company....

I did no XISF conversion

I´ll try with those settings (thanks for the formulas - different than in the documentation!) and`ll work through it from 4  (RAW) to .....? images.

.....but what if in addition stacking of darks, bias, flats is wanted ... ?   and actuallay I want to stack 20 images of each...
 

 

[Peter S]

I´ll save as XISF 32 bit floating point
or 16 bit
?

[Niall Saunders]

Hi Peter,

Well - I have thought about what Andres has said about the way that PixInsight might handle RAW-format images (and how PI might not be able to handle these images on a 'line-by-line, or row-by-row basis) and that might make sense. It would be great to hear from the likes of Juan, who would be best placed to advise whether non-FIT and non-XISF images do present stacking issues.

Most (all ) of the other pre-processing steps that you may choose to use are not anywhere nearly as memory-intensive as ImageIntegration - they nearly all just perform their actions on a single file at a time. So, your only enemy would be 'time'.

There is no doubt about it, II is a very memory-hungry process. If you are really serious about working with these huge images, then perhaps you could look at a memory upgrade for your PC - hopefully you have a motherboard with 8 x DDR4 memory slots, and hopefully you are only using either 2 x 16GB or 4 x 8GB and so you have spare slots available. (I very much doubt that you would be using 8 x 4GB memoru modules, but it isn't impossible). If so, then either seek assistance from the supplier of the machine (so that you can match the new modules to the originals) or try to do this yourself. If you really had to, then you could even sell the modules that you already have, and uograde from 32GB to 64GB of RAM.

I have just purchased a top-of-the-range PC myself, but finally decided to settle on 32GB of RAM (based on 4 x 8GB DDR4 3300MHZ modules) which would allow me to easily upgrade to 64GB when funds allow this. Given that the maximum RAM is 128GB, going to that level would have virtually doubled the purchase cost of my machine!!

[Niall Saunders]

I´ll save as XISF 32 bit floating point
or 16 bit
?

I would imagine that the RAW images that you would be converting would be 16-bit integer, so it might be necessary to save in 32-bit float

[Peter S]

...Buffer Size: This gives you a requirement for 3.389 GB of RAM for this.

...maximum buffer size is 1024 MiB....

settings just used are: Buffer 1024 MiB, Stack 1o GB  - it worked in 2:30 minutes:
higher stack leads to memory issue

Integration of 10 images:
Pixel combination ......... average
Output normalization ...... additive + scaling
Weighting mode ............ don't care
Scale estimator ........... average absolute deviation
Range rejection ........... range_low=0.000000
Pixel rejection ........... sigma clipping
Rejection normalization ... scale + zero offset
Rejection clippings ....... low=yes high=yes
Rejection parameters ...... sigma_low=4.000 sigma_high=2.000
* Using 5640 concurrent pixel stack(s) = 10239.54 MiB

RAWs too!

As soon as I choose Weights: Noise evaluation or the option evaluate noise or Scale estimator: iterative k-sigma I get the memory issue.

this is a big step for me.....
the noise evaluating option is something I´d like -I ordered 32 GB RAM but I´m afraid it´s not the solution




 

[Niall Saunders]

Hi Peter,

Well you're getting somewhere !!

...maximum buffer size is 1024 MiB....

Sorry - I wasn't quoting from the tooltips or the system documentation, all I had to hand was my 'notebook of really useful information' (into which I perpetually scribble all sorts of notes about all sorts of subject material, from how to program with multi-threading in VisualBasic to how to analyse the ray paths through the likes of a Ritchey Chrétien OTA). The pencilled scribble told me 'how' to calculate the sizes, but not what the PI limitations were !

higher stack leads to memory issue

Obviously, your PC has basic memory requirements to do its 'own stuff' (including providing PixInsight with enough RAM in which to work).

I ordered 32 GB RAM but I´m afraid it´s not the solution

Perhaps it will go a long way to helping - given that you are 'running out' of RAM when you hit that 10GiB buffer. Add another 32Gib to your machine and perhaps a lot (if not all) of that would be available to PixInsight. [I hope that you took the time to purchase 'matched' memory - often the advice is to try and source memory from the same 'batch', but that is not always possible)]

Let us know how the extra memory works out - and post us a link to a (lower resolution !!!) final image so that we can all appreciate your efforts.

[Juan Conejero]

Hi Peter,

The latest versions of ImageIntegration compute image statistics using high-level parallelization. See this release information post.

With high-level parallelization, your machine has to load 12 images simultaneously (because your i7-3930K CPU has 6 physical cores, 12 logical processors). Now in order to compute the required statistics for noise evaluation and image normalization, each running thread has to make several duplicates of the entire image. This is a lot of memory with large image files.

You can disable high-level parallelization as follows:

- On the ImageIntegration tool, click the Edit Instance Source Code button on the tool's bottom control bar (void square icon).

- On the code editor window, look for the following line:

P.useFileThreads = true;

- Change it to:

P.useFileThreads = false;

- Click the Commit button (green check mark, top-right of the window).

- Now you should be able to integrate your files.

Without high-level parallelization and 96 Mpx files, it will be quite slow, so you'll have to be patient. Image statistics are computed and stored in an internal cache, so this is normally a one-time process. Once stored in the cache, these properties will be retrieved automatically each time you integrate the same files.

The advice given by Andrés is also very important. Use the BatchFormatConversion script to convert your TIFF files to the XISF format. This will allow you to integrate your images with incremental disk file read operations.

Let me know if this helps.

[Niall Saunders]

Thanks Juan,

It's always nice to know what is really happening behind the scenes. What had initially appeared to be a 'simple' scenario has developed into one that has hopefully provided useful new information for not just me, but any other visitors to this thread.