Buffer / Stack option - image integration

[Juan Conejero]

Another option is decreasing the maximum number of processor cores allowed for the entire PixInsight platform:

- Select EDIT > Global Preferences on the main menu.

- On the Preferences window, select the Parallel Processing and Threads section on the left panel.

- On the right panel, uncheck the Allow using all available processors option.

- On Maximum number of processors used, you should see 12 on your machine. Lower it to 6 for example.

- Press F6 or click the Apply Global button.

This will limit the number of processors used by all processes, not just ImageIntegration. However, since less threads will execute concurrently, perhaps this will allow you to integrate more images with high-level parallelization enabled.

A faster way to do this is entering the following command on the Process Console:

parallel -p=6

To reset parallel processing settings:

parallel --reset

To see all options available for this command:

help parallel

[Juan Conejero]

Hi Niall. As always, thank you for your great support, within and beyond this forum. I highly appreciate it, as I am sure all users do.

[Juan Conejero]

Just an additional note about this. Besides the recommendations given in this thread, I'll try to make ImageIntegration more flexible in successive versions, especially easier to use regarding memory consumption. Sorry for the limitations of the current implementation in this regard. I'll try to do my best to improve it.

[Niall Saunders]

No thanks needed Juan - although it is appreciated!

I just give freely of my time, helping others as others have helped me in the past, always ready to stand corrected or to learn something new.

We, as the global PixInsight community, don't thank you guys in the PTeam often enough. For example (warning: scary thought ahead) would Peter have even been able to put his question to 'anybody', if he wasn't trying to use PixInsight to integrate his 10 mega-images? (And that doesn't take into account that you, Juan, have already added - yet another - item onto your ToDo list, to improve PI even further). And all of this within 48 hours 

And, finally, at least you now have a(nother) PixInsight user who will be trying to make PI work with huge datasets - so you have another means of ensuring that PixInsight serves not just the current needs of the 10Mp imagers, but also of the future needs of 100Mp (or even 1Gp) imagers.

[Peter S]

...firstable: thanks a lot for your help! This makes PI even more worth it.   

Meanwhile I succeeded with calibrating 10 (100 MP IQ3 RAW or much larger XISF) DARKS / FLATS / BIAS.
slowing down the machine helped (I applied "Juan´s twin brake": no parallelization, only 6 proc´s) - today I´ll get 32 GB more (matched) RAM.....

Calibrating the Lights with the Master BIAS and Master DARK and Master FLAT resulted in several errors like
 "Ch 0 / 1 / 2 has no correlation with Master Dark" (something like this.....late tonight).

The masters (BIAS, DARK, FLAT) look "good" when stretched with STF.
The resulting lights have a weird colour cast. StarAlignment is not possible because "no stars found".

I´ll be back when the RAM is installed....

[Niall Saunders]

Hi Peter,

"Ch 0 / 1 / 2 has no correlation with Master Dark"

I get that as well - it doesn't seem to affect the process, and I don;t know what causes it (if someone has already explained this, I apologise as I have already forgotten 'who', 'when' and 'where' !!)

The resulting lights have a weird colour cast

Again, don't worry about this right now - colour casts always seem to be an issue when you are dealing with Flat-corrected-Lights from DSLR and OSC imagers. I have long postulated that, due to the inherent division that takes place in the Flat correction, the the resultant image will have a 'cast' equivalent to the ratio of the intensities of each channel in your MasterFlat.

In other words, if your MasterFlat was a 'NeutralGrey', then there would be little or no colour cast after the Flat-correction. However, if your MasterFlat had a predominantly Blue cast (for example), then the 'higher' ADU figures for the Blue channel would result in 'lower' figures in the final image.

I have played around with my MasterFlats to try and overcome this issue - splitting the MasterFlat into its four individual channels (one each per filter in the original CFA). then using PixelMath to try and align the four histogram peaks before finally recombining the CFA channels to build a new,'Normalised'(), MasterFlat. (PixInsight now has built-in tools to perform these steps)

I would love to hear from anyone using an OSC/DSLR who has either tried my approach, or who thinks there is any advantage in doing things this way. Similarly, I would also like to hear from anyone who can explain why there is no real need for these extra (time-consuming) steps in the pre-processing phase.

[msmythers]

Niall

My quick thought on the color cast and flats is as long as the resulting final integration is linear and none of the data has been clipped or lost in anyway then good color calibration should be able to bring all channels into a proper neutral image. I've experimented the same way you have I think and have found just for my own sanity, because I don't like looking at a colored flat I try to make the flats as neutral in color as possible. Then I let PI do to it's thing during color calibration.


Mike

[Peter S]

...okay, thanks! ....that is too much work for me..... and too complicated for me....

I´ll compare 2 ways:
- the whole process as executed (BIAS, DARKS, FLATS, LIGHTS), trying to avoid that colour cast, Alignment, stacking, Robertson-Lucy-deconvol., stretching...
vs
- the "modified Roger Clarke proposal": just  RAW development (LIGHTS, eventually with Camera-Auto-Darks) with lens correction (instead of FLATS) etc., Alignment, stacking, Robertson-Lucy-deconvol., stretching

[Niall Saunders]

Hi Peter,

...okay, thanks! ....that is too much work for me..... and too complicated for me....

 

Don't worry, and don't over-complicate things (too soon!!).

Stick with your first method, as Mike said in his reply, PixInsight can make a very good job of sorting out colour-cast issues anyway (and the soon-to-be-released 'new' tools wield even greater power).

I'm not that enthusiastic about your second proposed method. I have little, or no, faith in noise-elimination algorithms that take place 'in camera' (quite an apt phrase really because, in camera, rooted in Latin, translates quite nicely to 'in the box, hidden from view' - exactly what you face when you select de-noising options in a DSLR - those algorithms are never published or made available in any detail whatsoever). The issue is that, to be really effective, all of our pre-processing (or calibration) routines have to be be heavily reliant on 'strong' statistical processes - which the very ethos of PixInsight. If they are not, then it is more or less impossible to differentiate between a 'signal' value and a 'noise' value.

So, any 'in camera' routine would really need multiple Biases, multiple Darks and multiple Flats in order to be able to effectively reduce noise prior to generating a 'clean' RAW image. I don't believe that I have encountered any DSLR body with internal software that performs image-calibration in any way that even comes close to what astroimagers expect - but I am willing to stand corrected.

As you say though - you could at least 'try' the two different methods in your setup. After all, what you are using is not really a 'high-street' DSLR camera 

[Peter S]

update:
Image Integration with 64 GB RAM, buffer set to 1024 (max.), stack set to 35 GB: processing 10 images (120 MB each) went fluently; using all processors, parallelization enabled.
1.)
...even if it works now, do I miss something concerning the formula?:
calculating the amount of stack: W×H×(12×N + 4)  makes:  11608 x 8708 x (12 x 10 + 4) = 101.082.464 x (120 + 4) = 12.53 GB - with monochrom images / 1 channel?
so with RGB : x3 = 37.6 GB ?
2.)
converting IIQ RAW with BatchFormatConversion (commercial version meanwhile) there is by default: input format hints: "raw" "cfa"
This wasn`t the case with the trial version
I got monochrome images.
I read the documentation and understood format hint "raw" = read RAW as "raw as can be"

What does "cfa" mean or: what is the difference between the options "cfa" and " no - cfa"?
- load a monochrome image but with all the sensor informations not converted into RGB?

...what do I have to do after converting with those settings?

[Niall Saunders]

Looking good Peter 

Picking up on your queries:

My notes state that Buffer Size is W x H x C x 4, where :
W - image Width, in pixels
H - image Height, in pixels
C - number of Channels in the image (MONO and RAW = 1, COLOR = 3)
4 - becuase every pixel is expressed, internally, as a 32-bit (4-byte) Float variable
(4 must change to 8 if you are working in 64-bit resolution)

So, for your 'mono' (or 'raw') images, at 11608 x 8708, this gives approx. 386MB

Once the images have been DeBayered, the Buffer Size would need to increase by a factor of 3, to approx. 1.13GB

Stack Size is simply 'N times the Buffer Size', where N is the Number of images being processed.

Remember, Buffer Size is independent of the number of images being processed. However, as you point out the maximum size is 1GB - whuch is just slightly less than you 'require' if you were processing 3-channel 'color' images. I would not worry about this too much though, as the next (imminent) release of PixInsight should be able to perform the ImageIntegration process on 'raw' images - i.e. without the requirement to first DeBayer all the images prior to integration/stacking.

Also, just keep in mind that a 'RAW' image is not the same as a 'MONO' image - even though they might both appear as 'greyscale' images in PixInsight. A 'RAW' image implies that the data does actually carry colour data, encoded within the image as a function of 'pixel position in relation to the CFA'. the 'CFA' is the Color Filter Array that has been placed immediately in front of the imager sensor (at the time of manufacture, and which cannot be removed or otherwise overcome).

The actual imaging sensor may even be identical between two different imagers - one with the CFA (allowing RAW One-Shot-Color - OSC - or RAW DSLR images to be acquired); and the other with no CFA (allowing only MONO images to be acquired, even though secondary WB or NB filters may be placed in the optical path).

One of the key differences netween an OSC imager and a DSLR camera is the fact that a DSLR is (usually) capable of pre-processing the RAW image into a COLOR image within the DSLR software itself. Most (all ??) OSC cameras make no attempt to do this whatsoever, relying totally on PC-side software to DeBayer the image (i.e. to extract the color information with knowledge of the CFA filter layut). A DSLR camera allows a RAW (un-DeBayered) image to be extracted from the camera, to be proceesed by various types of 'DarkRoom' software - one of which is PixInsight. Performing the DeBayer process outside the DSLR software, especially with the likes of PixInsight, allows a far more detailed, statistically-based, suite of algorithms to be used to increase the fundamental SNR of the image. You just cannot do that inside the camera (yet, because I see no reason why PixInsight could not simply be used as the internal OS of a modern camera (of 'any' type  )

So - again a bit of a lengthy reply, but maybe you can now answer some of your questions, and generate others!

[By the way, did you purchase a full license for PixInsight ?]

[Peter S]

Niall - again: thanks!
Yes I purchased a full license - is there another option? 

The preferences in this / my actual  version are different to the trial version where one opens debayered interpolated images (at least in my case). I was not sure if the images were "mono" or RAW un-DeBayered or if  I just forgot to activate colour display anywhere...

processing images,  I learned now that DeBayer is best done after calibrating and just before registrating / integration
do you agree?
DeBayer options: RGGB, VNG

It seems I must pull out all the stops with PI because the images just out of the camera (before registr. +  stacking) are really not bad. But as mentioned earlier - yet to compare. And until now I do "just milky way" and no deep sky images. I may sound like a tourist to you astro pros...

[Niall Saunders]

Hi Peter,

I may sound like a tourist to you astro pros

Not at all - and, anyway, your imaging equipment means you are travelling First Class, on Concorde (whereas I am operating the pedals, truning the engines to keep us airborne !!)

At the moment, my personal preference is to fully 'calibrate' every raw frame (i.e. with Biases, Darks, Flats and FlatDarks). Only then do I perform a DeBayer process, to every calibrated Light. Then I perform StarAlignment, and finally I can consider ImageIntegration.

Why?

Well, if I DeBayer before ImageCalibration, then the calibration amster frames also have to be DeBayered - and every time you invoke the DeBayer process, you introduce noise and therefore you must reduce the SNR. Exactly the opposite of our goals. My process order means each Light frame is only affected 'once' by the DeBayer process (and, hopefully, any loss of SNT can be pertially recovered by subsequent steps).

What if I miss out on the DeBayer step, and simply perform SatrAlignment followed by ImageIntegration - leaving only the very last MasterLight to be DeBayered - does that yield better SNR results?

Well, extreme care is required here. Remember the DeBayer process is intimately connected to image  pixel positions with respect to the CFA layout. So, if you mess with that relationship - for example by invoking StarAlignment before the DeBayer stage, then you run the serious risk of reducing SNR (by increasing chrominance noise) because you are no longer working with the 'true' CFA relationship.

I dabbled with the alternative sequence, for a brief period, before deciding that - for me - I felt that I could not accept the 'hope' that the CFA mapping was affecting the final MasterLight. I don't (any  longer) have the evidential frames that I was experimenting with at the time, so anyone who wishes to try both methods should do just that. After all, empirical results are often all that are needed to select a 'result' from a 'choice'.

If anyone reading this has statistical data or explanations that justify one method over another, the I would be delighted to hear from them.

My thoughts, for your particular under-sampled wide-field images, is that my first method may be the most appropriate for you (especially given the power of your PC and its available RAM).

As far as your concerns about the differences between the Trial and Licensed versions (there are only these two options) is that you may just not have invoked any upgrades on yout Trial install (or, indeed, the Trial version is simply a 'fixed' release version that is only ever updated, by Juan, when he chooses to update it). Maybe an Uninstall/Reinstall cycle might have moved things forward slightly.

In any case - don't worry - I am sure that you will be delighted with the 'fully licensed' version of PixInsight - especially when you finally get to see what is marching inexorably down the Beta-test path towards you!!

Juan has performed his usual magic - I keep on telling him just to turn PixInsight into an Operating System - capable of running Windows, Linux, MacOS in a 'virtual' environment for those PI users who have unsatisfiable urges to do any work outside of the PixInsight GUI 

[Peter S]

Thanks Niall!
Meanwhile I did proceed like you recommended and got a nice Master Light Frame -
now I am struggeling with Deconvolution
I try to do it like R Bloch in his Youtube-tutorial (PSF, StarMask, "Deconvolution mask"). 

My thoughts, for your particular under-sampled wide-field images, is that my first method may be the most appropriate for you 

- YES

With 11608 Px, pixel size 4.7 µ x 4.7 µ and a 45 mm wide angle lens I get a (Subframe) scale 21,5 px - do you agree?

For stars I get a FWMH around 1,0 px with a 60 MP back and 28 mm lens
(my actual test series with the 100 MP Back is bad focussed and I get FWMH around 5 px....  )

By the way - analysing the sensor with the tool resulted in a

dark current: -1. # IO e/sec

[Peter S]

sorry, has to be:
scale: 21,5 arcseconds per px