Calibration: What i m doing wrong?

[Niall Saunders]

Sander,

With pipeline I mean that a single image is processed at a time all the way up to the final image combination step, without leaving memory

You presumably meant to clarify that "the user is not directly involved with moving intermediate images to and from HDD storage". Surely, as image sizes grow, even with large amounts of memory, there will always be a situation where information has to be 'swapped out' to HDD storage during a process, even a fully automated process.

I agree that the current PixInsight 'phased' approach requires the user to remain intimately involved in 'organising' the data created during all the intermediate steps - which is time-consuming, and which does NOT suit the sometimes haphazard, random, approach I have seem some users applying when it comes to 'organising' their data. I also therefore fully acknowledge that the PI method really only suits the 'anal' amongst us - those who, like me, are willing to sacrifice a LOT of time and effort to collating and renaming data into understandable storage structures, and who are also quite happy to take the time to organise, rename, and save all the intermediate PI stages, as ProcessIcons, to help them understand what the h3ll it was that they actually did all those weeks ago - when they created that un-processable 'masterpiece-in-the-making' image

Everybody has their own approach (thankfully), and will use methods that THEY are most comfortable with. A few of them (actually, all of 'us', I suppose) are also then 'curious enough' to seek out new ways of doing things, no matter how 'strange or awkward' they might initially seem, at first, and then some of us will just use those new (PI) methods to the exclusion of all others, just because those methods suit our approach.

Until, of course, another - even better - method comes along, irrespective of how 'strange or awkward' THAT new method might initially be to implement

[Nocturnal]

Hi Niall,

no, I can't see a future where PC memory can not keep up with imager sizes. Calibrating an image 100% in memory is a trivial matter from a resource perspective and I don't see that ever changing when users have 'reasonable' PCs and cameras. Clearly you might get into trouble with 20 Mp subs and PI running on a netbook but I don't consider that reasonable. Since the image gets changed as it's being processed it doesn't even require more and more memory as it goes through the pipeline. Only when the light has been calibrated and registered do you need to save it to disk so that the final image combination is not stack size limited. Both PI and DSS use disk based final images for combination. DSS uses a hidden mechanism that may not even be based on individual files (no idea, don't care), PI requires that you have the individual registered lights on disk.

I think any reasonable stacker should use a comparable memory footprint regardless of the number of lights being used. DSS works this way and PI does as well. The current phased approach is memory friendly as it uses secondary storage extensively. A pipeline would require similar amounts of memory as each image would be processed one at a time and then saved to disk before integrating. Actions such as dark subtraction or flattening don't increase the memory requirements for each sub, they modify the image itself. Debayering naturally requires more memory. But if someone's PC can't debayer a single sub in memory they should take up wood carving or knitting

[ammcdavid]

Hello all,
I don't want to take over Theirry's thread but I also do not want to duplicate the topic so I will dovetail in here and see how it goes.  Anyway, downloaded my trial version of PI 1.6 today and have started working with it.  I also managed to get some of my own data with my DSLR - 6 dark frames and 36 lights @ 10 seconds each on a tripod.  I don't expect much but when I checked the lights with STF there was data there so I am encouraged to at least see what I can get out of it.  Here is what I have done so far...

- Opened the RAW darks in PI and saved them as fts files (I think there is a batch conversion process in my Canon software to make this faster in the future).

- Performed ImageIntegration per the PI tutorial and this yielded 3 additional images labelled "rejection low", "rejection high", & "integration".  At this point I am pretty sure the Integration is the item of interest and is now my averaged dark frame to use on my lights.....is this correct?

The next step in the tutorial talks about Master Flat creation with the IC tool but I don't have any bias data and only 6 darks so I am not sure if this step applies.  Plus, when I step through the selections, I uncheck Master Bias and leave Master Dark checked but it wants a file listed ......is that my new "integration" file created above?  If not, where and/or how do I get this file?  Is 6 darks enough to create a Master Dark that means anything with only 10 second exposures?

Sorry for any dumb questions....if I can get thru this, I should be able to apply at least this dark frame to all of my lights and then start combining them.  I feel kinda dumb.....stuck on the pre-processing and haven't even sniffed the actual processing yet...

Anyway, any guidance is really appreciated.

Andy

[pfile]

Hello all,
I don't want to take over Theirry's thread but I also do not want to duplicate the topic so I will dovetail in here and see how it goes.  Anyway, downloaded my trial version of PI 1.6 today and have started working with it.  I also managed to get some of my own data with my DSLR - 6 dark frames and 36 lights @ 10 seconds each on a tripod.  I don't expect much but when I checked the lights with STF there was data there so I am encouraged to at least see what I can get out of it.  Here is what I have done so far...

- Opened the RAW darks in PI and saved them as fts files (I think there is a batch conversion process in my Canon software to make this faster in the future).

stick to PI for all RAW file handling. the canon software can't produce FITS to my knowledge and anyway the canon software is going to do nonlinear transforms to the pixel data when doing the raw conversion. PI uses the package called "dcraw" to debayer the raw file and will treat the data linearly.

- Performed ImageIntegration per the PI tutorial and this yielded 3 additional images labelled "rejection low", "rejection high", & "integration".  At this point I am pretty sure the Integration is the item of interest and is now my averaged dark frame to use on my lights.....is this correct?

yes, "integration" is the image stack.

The next step in the tutorial talks about Master Flat creation with the IC tool but I don't have any bias data and only 6 darks so I am not sure if this step applies.  Plus, when I step through the selections, I uncheck Master Bias and leave Master Dark checked but it wants a file listed ......is that my new "integration" file created above?  If not, where and/or how do I get this file?  Is 6 darks enough to create a Master Dark that means anything with only 10 second exposures?

right... save the master dark to disk, and then point the calibration tool at the file.

you should make some bias frames. it's very easy, just set your camera to the fastest shutter speed possible, set the ISO the same as your lights, and fire off 50 or so shots. then make a master bias frame per the tutorial.

you need the master bias frame because in order to make a "scalable" master dark, you need to subtract the bias noise out of the master dark. if you want to proceed without bias frames, you can really only do dark subtraction of the lights, and you should then turn off dark optimization, otherwise the bias noise that's in the master dark will get scaled as well, and that's bad.

[Niall Saunders]

- Performed ImageIntegration per the PI tutorial and this yielded 3 additional images labelled "rejection low", "rejection high", & "integration".  At this point I am pretty sure the Integration is the item of interest and is now my averaged dark frame to use on my lights.....is this correct?

and

yes, "integration" is the image stack

Correct - just don't totally ignore the two 'rejection' images. Have a close look at these, with an AutoSTF applied to each. Whatever you can 'see' in these images is actually image data that has been 'removed' from the final 'integration' image. If you can 'see' any kind of 'structure' or 'shape' in the rejection images, and if that structure or shape has anything to do with some actual structure or shape in the final image, then you may not have the clipping values set quite right. You might want to think about tweaking the sliders (one at a time) and seeing if you can 'eliminate' the 'structure' from the 'rejection' images - by doing this you are putting the otherwise 'rejected' data back into the final image - where it will most likely IMPROVE the overall SNR, wherever it can.

Basically, the rejection images should, ideally, never contain 'valid data' - the only 'structures or shapes' that you want to see should be associated with blooms, dead pixels, hot pixels, column defects and cosmic ray strikes. Other than that they should really only be showing random noise.

You should also look at the Console Output after the ImageIntegration process has completed - at the end of the listing there will be figures associated with 'how many' pixels have actually been rejected, both as an actual number count, and as a percentage of the total number of pixels in the image. My personal 'rule of thumb' is to try and keep the TOTAL (i.e. for both 'high clipping' and 'low clipping') to less than 1% of the overall number of pixels in the image. Why 1%? No reason, it just 'feels good' to have retained 99% of my original data.

And, even with an el cheapo imager like the DSI II, it is actually quite easy to stay within the 1% overall data loss, whilst still eliminating a lot of the nastiness that used to get through. It just takes a wee while to get used to what the rejection sliders are doing for you - but, because PI can/will re-use the statistical data each time you re-ren the ImageIntegration process, the re-run times are actually pretty short. Well worth the effort, in my opinion.

[ammcdavid]

You should also look at the Console Output after the ImageIntegration process has completed - at the end of the listing there will be figures associated with 'how many' pixels have actually been rejected, both as an actual number count, and as a percentage of the total number of pixels in the image. My personal 'rule of thumb' is to try and keep the TOTAL (i.e. for both 'high clipping' and 'low clipping') to less than 1% of the overall number of pixels in the image. Why 1%? No reason, it just 'feels good' to have retained 99% of my original data.

Niall,
I did notice the clipping values as a percentage and each were in the .3-.5 range for a total of, yes, less than 1%.  The low clipping and high clipping both showed (after some STF massaging) a striking cross hatch pattern in the horizontal and vertical directions and from examples I have seen online are similar and representative of the sensor defects.  Other than that, I saw no discernable structure but until I gain some experience, I may not know what to look for.  I am trying to process some bias frames now to process with the darks.

Andy

[Niall Saunders]

OK Andy,

What to do is to try moving the sliders 'the other way' - so that your rejection image DO start to show 'real structure' (play with the Reject High slider first, for example), and note that the %age of rejected pixels also starts to increase.

Don't worry - you are TRYING to 'ruin' your IntegrationImage at this stage

Keep moving the slider until you DO see the structure start to appear, and then start working your way back in the other direction, until the structure has disappeared far enough that you feel confident that the 'useful' information that you would otherwise have been clipping has now been 'put back' into the IntegratedImage.

Now repeat the process for the RejectLow side of things.

I have found that, so long as my imaging capture process then does not deviate too much from what I had been using, I can then re-use THESE slider positions as a very good 'first approximation', with the result that I can be pretty confident that I will get a - more or less - consistent 0.5% rejection at each end of my outliers.

It sounds complicated, but it actually becomes VERY intuitive after a while.

[ammcdavid]

OK Andy,

What to do is to try moving the sliders 'the other way' - so that your rejection image DO start to show 'real structure' (play with the Reject High slider first, for example), and note that the %age of rejected pixels also starts to increase.

Don't worry - you are TRYING to 'ruin' your IntegrationImage at this stage

Keep moving the slider until you DO see the structure start to appear, and then start working your way back in the other direction, until the structure has disappeared far enough that you feel confident that the 'useful' information that you would otherwise have been clipping has now been 'put back' into the IntegratedImage.

Now repeat the process for the RejectLow side of things.

I have found that, so long as my imaging capture process then does not deviate too much from what I had been using, I can then re-use THESE slider positions as a very good 'first approximation', with the result that I can be pretty confident that I will get a - more or less - consistent 0.5% rejection at each end of my outliers.

It sounds complicated, but it actually becomes VERY intuitive after a while.

Okay, and thanks!  I understand what you are saying......maximizing the signal by finding the structure threshold as signal is hard enough to acquire as it is without throwing it away needlessly.

Andy

[RobF2]

I wish there was some way to aggregate your images, such as a container object that could point to the images, then the cal'd images, then the registered images - perhaps smart enough to look for the original names with suffixes denoting calibration and integration etc.  You could then perhaps drag this object onto a series of process icons for cal, registration, integration.

Its so frustrating ATM that you constantly have to navigate back to the directory with your images, then re-specify where the cal'd or registered images go etc.  I guess this fits in with the pipeline described earlier - you want to put your images in, apply certain defaults or saved settings from process icons, and perhaps at the end go back in a tweak some settings or processes in the very rare instance that you're not happy with what PI supplies at the end....

[mmirot]

I am being tough on the PI calibration UI but I do think it is user unfriendly
I expect only the best from the PI team.  I have confindence they will improve this.


If the core processes are the best in the world that is great.
However, it does not help when novices and experinced users struggle with these modules.
Many will end up going back to DSS or MaxIM DL to do calibration and preprocessing.

BTW most professionals would use a pipeline to process their data.
IMO, It is a feature that is a nagging hole in the PI menu.

Max

[georg.viehoever]

...
BTW most professionals would use a pipeline to process their data.
IMO, It is a feature that is a nagging hole in the PI menu.
...

I entirely agree with this. While the set of tools provided by PI for this purpose is certainly complete (and probably was before ImageIntegration was provided --- after all you can do it all with PixelMath  ), a tedious and repetitive task such as calibration, registration and integration needs to be automated as much as possible. For the moment, DSS is my tools for calibration, writing out the calibrated files. Registration+Integration I mostly  do with DSS, and occasionally (when I need to control) with PI.

I would love to switch entirely to PI if it would have an interface for the lazy ones like me that allows for pipelined processing such as DSS, hitting some buttons and seeing the result the next morning. I just need my sleep 

Georg