How PixInsight manages the RAM

[Dominique]

Hi everybody,

First of all, excuse me for my poor english, it's not my natural language.

secondly thank you to PixInsight for offering a free trial period of 45 days

My question is about ram managment in Pixinsight. I search the web and i don't find an answer.  The only answer I get is : you need a lot of RAM.

My computer is a Core 2 DUO E8400 with 4Gb RAM running vista 64bits.

Yesterday, I use the process to create a master bias. I have 54 CR2 files ( 8.27Mb ). It take 3568 sec ( 1 hour )  to achieve the process.
I run the task manager to see what Pixinsight is doing. My CPU usage stay behind 10% but my RAM usage is 100% after a few minutes. My swap file increase to 9Gb.

So, I don't understand how Pixinsight manages the RAM ?
54 files of 8Mb , that's just 432 Mb. How is it possible that my computer need about 13Gb Memory to process these 54 files ?

Today I send a mail to a friend to ask him what kind of computer he uses to process 200 bias. He says he has a core I7 with 8Gb under windows 7 64bits. The process takes about 10 minutes.

I do not understand why with only 4Gb more memory, Pixinsight seems to run well. I do not take into account the processor because the mine is not solicited during this phase.

So, how Pixinsight manages the RAM ?

Thank you in advance

[Phil Leigh]

I had the same problem; the solution is to convert CR2 files to fits using the Batch Format conversion script. RAW files cannot be opened incrementally like fits files and require huge amounts of RAM to manipulate - this isn't a problem just for PixInsight... but Pi keeps all the images in cache to work its magic.

If you convert the files your problem will go away.

[Dominique]

Thank you Phil

I have converted my files and now it only takes a few minutes. My RAM usage is always under 100%

The only software I used before was IRIS and the first operation is to convert the files. I'll have to think about.

I see many tutorials in French and they never convert files. But they still have a small amount of files and it works very well.

Now, as I saw in another topic about the conversion, it is advisable to choose (raw bayer CFA) in the options file.
With this option, there is only one channel and the rejection of pixels is not the same.

If I convert in raw bayer rgb as before I have the same rejection for the three channels.

Now, I don't know what is the best for DSLR ? Convert in raw bayer cfa with 1 channel or convert in raw bayer rgb with 3 channels.

Thank you

[Phil Leigh]

I believe I am correct in saying that both options are equal in the latest version of PixInsight. I would not expect to see any difference in the end result. The data is the same... It is just a different way if structuring the file content.

The key thing is to ensure that the files are not being debayered during the conversion. Debayering must only take place after calibration and before alignment,
Regards,
Phil

[Dominique]

Thank you again but I don't think it's the same.

As I said , the amount of pixel rejection is not the same.

In my example I have 0.155% Red 0,288% Green and 0.134% Blue

When I convert in raw bayer cfa and create the master bias I have a total of about 0.750% rejection

So I don't think this is the same. But I don't know if it give in final a better result or not.

Regards.

[pfile]

when you load the CR2s directly into ImageIntegration, have you used the same format hints ("raw cfa") as when you use BatchFormatConversion ?

[Dominique]

Hello pfile,

After Phil's good advises, I search on the pixinsight forum about conversion.

I see two topics with differents advises :

http://pixinsight.com/forum/index.php?topic=1911.0

http://pixinsight.com/forum/index.php?topic=1868

I just follow the two and I have not the same results for the two. This seems logical.

But those answers are from 2010. I'm just wondering if one method is better than the other.

thanks

[Juan Conejero]

As for image calibration, the only differences between RAW Bayer RGB and CFA formats are:

- In RGB Bayer format, each image occupies three times more space in RAM and on disk.

- In RGB Bayer format, the dark frame optimization factors can be calculated more accurately. This is because in RGB format we can compute per-channel optimization factors, while the CFA format forces us to compute a single dark optimization factor for the whole image. Per-channel factors should be more accurate in theory, but the difference is usually negligible in practice.

So the recommended format is RAW Bayer CFA (monochrome). This is how the BatchPreprocessing script works, and this mode can be selected on ImageCalibration and BatchFormatConversion with the "raw cfa" input format hints.

As for pixel rejection, I don't see why there should be any differences. Both formats load exactly the same data; they just organize them as different data structures in memory. Once the images are debayered, the results should be identical.

[Eddy Timmermans]

Juan,

Is it better(faster) to use the BatchFormatConversion first and then load the FITS files in de BatchPreProcessing script ?

Or can I just as well load the CR2 files in the BPP script as I do now ?

I have 16 GB of RAM and last time I loaded 178 lights, 60 darks, 20 flats and 100 bias in one go.
It took quite some time but it all went smoothly.

Would it have gone faster if I first converted everything to FITS ?

Eddy

[Dominique]

Thank you all for your help.

[georg.viehoever]

...
Would it have gone faster if I first converted everything to FITS ?
...

Converting to FITS before BPP saves a lot of time and RAM because ImageIntegration (used internally by BPP) can handle FITS files much for efficiently than the other formats.
Georg

[Dominique]

I can confirm that I have no more memory problem since I converted my files.

Converting saves my RAM