Calibration: What i m doing wrong?

[thierry31380]

Hello,
Finally i have bought Pixinsight yesterday and i do not regret it.

I'm trying to calibrate frames using the tutorial by Vicent PERIS.

All seems work fine but after calibration, the light frames still have hot pixels

sorry for my poor english

[Niall Saunders]

Hi Thierry - and congratulations on making the move to PixInsight.

Without a more detailed explanation of all of your steps, I am going to try and guess what you are doing, and seeing, at each stage of what I 'hope' is your procedure.

You seem to be starting off with, at least, Lights and (presumably) Darks (perhaps / maybe Biases as well - but, no matter)

You use ImageIntegration to create a MasterDark, and ImageCalibration toapply the MasterDark to all the Lights - giving CalbratedLights

Is it at THIS stage that you are still seeing 'hot pixels' in the CalibratedLights?

If so, it could just be that your Darks (and hence MasterDark) need to be examined more closely, along with the procedure that you are using to create the MasterDark. Whenever I have 'suspicions' about my MasterCalibration frames, I like to open an 'original' one of the source files, alongside the MasterCalibration file of the same type. And I then like to use the HistogramTransfer tool to 'flick between' the two images - effectively letting me 'blink compare' the two Histograms. They really should remain more-or-less 'the same' - I certainly know if there is an impending problem if I see 'significant' differences - like histogram peaks jumping around from side-to-side. That really just should NOT be happening.

You can also seriously consider doing the same kind of 'blink-compare' using the Statistics tool as well - again looking for hugely different values in the Statistics table.

And, even using Nikolay's excellent <Animator> script, you can 'blink-compare' your new MasterCalibration image with the source images that were used to create it.

At the end of the day, your MasterCalibration image - no matter which 'type' of source data it came from, should really always just be a 'statistically smoothed' version of any individual image from the same series.

OK - so let me assume that there were no significant differences, I personally would not be too concerned - at this stage - if my CalibratedLights still had either Hot or Cold pixels present. When I subsequently StarAlign and ImageIntegrate the series, I would be hoping that 'Pixel Rejection' would help to eliminate these 'outliers' - and the process is usually VERY successful at doing so.

However, to be so successful, you DO NEED to have 'some dither' between images. If your setup is as problematic as mine, this should NOT be difficult to achieve Just having the 'true image' move with respect to the CCD frame from image to image will help ensure that hot and cold pixels can be eliminated (more or less) by ImageIntegration.

And (as is always the case ) one 'however leads to another : However, you also need quite a few source images, each with some 'dither', for the Pixel Clipping algorithms in ImageIntegration 'to get their teeth into' - the algorithms need to have enough 'good data' to be able to detect the 'bad date', and hence eliminate the hot and cold pixels as outliers.

Finally, there is work-in-progress that will take a user-defined DefectMap image, and apply a correction algorithm for each pixel defined as being 'defective'. I haven't used this myself yet, so I can't really comment much further.

I appreciate that English may not be an easy language for you to ask question in (or to understand if answers, like this, are received in English). If you feel better, ask your question in whichever language you are most comfortable with - I am sure that everybody will do their best to try and help, wherever possible.

Use two languages if you feel that helps.

[mmirot]

The process in PI is needlessly complicated.
Better to just try out DSS.

Max

[Juan Conejero]

Hi Thierry,

Welcome aboard.

All seems work fine but after calibration, the light frames still have hot pixels

We'd need to take a look at your master calibration frames and to one of the problematic light frames in order to know what's happening. Can you provide more details about your calibration workflow?
 

[Juan Conejero]

The process in PI is needlessly complicated.
Better to just try out DSS.

In my opinion, PixInsight's calibration, registration and integration tools aren't needlessly complicated. Preprocessing isn't a completely automatic task in PixInsight, but that doesn't mean it's needless. We provide full featured, accurate and flexible tools in all image preprocessing areas.

I have nothing against DSS. Just at the contrary, I think it is an excellent application, but yours isn't a very encouraging comment. Anyway, always feel free to make these comments here, as PI Forum is open to every opinion and criticism.

[thierry31380]

Thank you very much for your replies

for now I will continue to process my pictures with DeepSkyStacker, time to get used to PixInsight. In some time i will do another try and i will ask your help if i have problems.


Thank you one more time, i go back playing win PI  

Thierry

[Harry page]

The process in PI is needlessly complicated.
Better to just try out DSS.

Max

Hi
 I am Mr PI you know that but the calibration tool is to contrived for easy use    , I did see juan being asked for help on scripting this but I know he is busy , so I will keep chipping away till he gives in 

Harry

[Enzo De Bernardini]

We provide full featured, accurate and flexible tools in all image preprocessing areas.

And I love that! With PI we have control over every little detail, and each image has different needs. I think it is a wonderful advantage.

Enzo.

[Nocturnal]

Hi Juan,

Max said 'needlessly complicated', not 'needless'. Big difference even if you feel it's "needfully complicated"

[Nigel Ball]

The process in PI is needlessly complicated.
Better to just try out DSS.

Max

I'm on my trial period with PI, about day 10 I think. Having used many other packages incl DSS I've found PI very logical and staightforward in the approach to Image Combine.

The problem, IMHO, with DSS is it is a bit like a black hole you don't really know what is happening under the hood. With PI we have total control and therefore flexibility over the details of the Combine Method. I prefer this flexibility but I do appreciate that for a newbie to image it could be a bit off putting ..... you could always just leave the default settings maybe - that's what I did with my first combine and it wasn't that far off

Like all programs good raw data gives good images 

Nigel

[Nocturnal]

Hi Nigel,

with DSS you also have complete control over the combine methods so that argument doesn't really stick. Yes DSS is a black box but one that's well documented. Because it is an image processing pipeline it's much more efficient. It opens a raw light, dark subtracts and flat divides in memory before debayering and registering. It also does automatic master file maintenance.

PI writes all intermediate files to disk and has to read them back in every time because it's a phased approach (create master files if needed, do all dark subtraction for all, then flatten all, then debayer all, then register all, then stack all).

For PI to be a competitive stacking solution the phased approach needs to be replaced with a pipeline. The raw materials are there to do this but it's going to take considerable effort. Someone who's hurting for a good stacking solution for Linux/FreeBSD or MacOS will have to step up to the plate. To me it's not worth the trouble because I'm perfectly happy with DSS. Plenty of other interesting things I can do with scarce free time. Besides, if I want to experiment with PI image combination I can still have DSS calibrate and/or register my images first.

Of course there are different work preferences and the phased approach is actually very common in other software as well. Some people swear by this approach and therefore don't need a PI integrated stacker.

Anyway, as I've explained before I don't think current PCL and PSJR can effectively combine so many processes to create a true integrated stacking solution. You'd spend weeks just replicating all the GUI elements and making sure the value from each maps correctly. That's monk's work and I'm no monk Perhaps once PCL and PSJR can bring up process GUIs then this becomes an option for a mere mortal like me. Juan can do it sure but not me.

[Niall Saunders]

Some people swear by this approach and therefore don't need a PI integrated stacker

Hi Nigel, Sander,

I think that Sander's quote is 'key' here, The pipeline approach is, after all, a phased approach as well, only the user has all the phases 'concealed' by the pipeline itself.

What it really boils down to is whether the user needs to, or wants to, become involved in each, or any, of the individual phases required in the overall pipeline.

Personally, I do - or at least I 'did' - want to be involved. After all, it is only by becoming intimately involved with the individual phases, and how they have to all be brought together, that you ever really get a feel for what is needed for the overall process to work. It is also a fact that, by analysing the results at all of the intermediate stages, a user can get a feel for where their own process may be failing.

Obviously, like Sander, I now believe that I actually have a fairly solid understanding of all the individual 'phases', and so a 'pipelined' approach will simply save me time.

What I also know is that it will not actually save me that much time, especially because I am just anal enough to want to keep all the intermediate data anyway. OK, so my preferred 'manual phased' approach is nothing like as fast as DSS, but it is pretty fast once I have pre-sorted all my source data into properly named files and sub-directories. And I get to 'see' what is happening, and can identify where my capture process has failed long before I ever get to see a final image.

It is also worth noting that I 'grew up with' the very, very, basic "Command Line Interface" provided in the excellent (and free) "IRIS" software - a package that forced you to slow down and actually understand all the steps needed for proper image calibration.

So, it really does come all the way back to user choice - DSS will perform very good Image Calibration but, in my opinion, the level of sophistication of the calibration being performed is nothing like as good as that available in PixInsight. But, in PI you (currently) have to still 'work hard' to attain the better results. And the same applies to the various ImageIntegration steps used by both packages - where, again, the results available from PixInsight are 'better' than those available from DSS, but at the expense of having to 'work harder' to attain them in PI.

Does that mean that everyone should really be using PixInsight for their pre-processing? Not at all - in exactly the same way as some folk will happily not bother with PI for any part of their image processing steps. As any individual's experience grows, they will find specialist tools that will offer a processed result that was 'better' than the way they used to do things.

And, in my case, PI is actually doing that for ALL the stages that I have needed in the past - even though I fully agree that a calibration, alignment and integration 'pipeline' for PI will be very useful in the long run. My biggest concern is that, given such a 'super tool', will novices ever take the time to fully understand any of the individual 'phases' actually involved?

Which also then begs the question, "Do novices actually need to fully understand what a tool, or process, is actually 'doing' when they press the 'go' button?"

[Nocturnal]

Hi Niall,

I'll just note here that PI has a lot of voodoo-magic processes where 99% of us don't have a clue how things work internally. Should that stop us from using them? Of course not. Should we force ourselves to do manual wavelet layer extraction and then use pixelmath to perform the HDRWT magic before recombining? No. 'Interesting' and 'educational' as it is, ultimately software should make life easier and not force us to do manual steps that can be automated. Flattening is well understood and given a set of images there's really only 1 way to do that. Why allow people to do that incorrectly? You gain nothing. All the knobs and dials that the calibration and stacking processes in PI provides are still relevant for an automated pipeline. The thing that's automated is the sequence in which you do things.

Anyway.

I'd love some someone to show quantatively what I am missing by using DSS to stack instead of PI. As indicated I can see how some of the combine methods will work better for imagers with dark current but my camera doesn't have any so I don't see appreciable differences.

[Nocturnal]

Hi,

one more clarification. With pipeline I mean that a single image is processed at a time all the way up to the final image combination step, without leaving memory. In a phased approach all images get the same step performed on them before moving on to the next step. That means intermediate images have to be written to disk and then read back when the next phase comes. A PI integrated stacker can still implement a phased approach but perform the steps automatically. I think that would be half a solution as it's still phased. The preferred way would be to implement a pipeline.

[Niall Saunders]

Hi Sander,

For those of us who do have the luxury of using cameras with 'no dark current', then there is an argument that can be made for not needing Darks (either FlatDarks or LightDarks). There is also then the argument that Biases are all that are needed to 'DarkSubtract' the Lights and Flats.

With only Biases thus being needed, it could also perhaps be argued that there is no need for sophisticated pixel rejection algorithms in the ImageIntegration process (an area where, I feel, PixInsight is 'better' than DeepSkyStacker).

The same argument could therefore also be made for your Flats - if you take the time to make enough of them, and also take the time to make them with care, then very simple 'Averaging' may be all that is needed to generate a 'statistically clean' MasterFlat.

And DSS will do that perfectly adequately, as well as correctly applying the MasterFlat to the Bias-corrected series of Lights.

Thus far, there is perhaps little difference between the two packages - given that the user is capturing data with a 'top-quality' CCD imager (remember, I am NOT, so all of the arguments above DO NOT APPLY - in other words PixInsight definitely makes my interim 'masters' MUCH better, in terms of eliminating 'statistically evaluatable noise')

So, now, you have a series of CalibratedLights - and these need to be Aligned (or Registered), and then ImageIntegrated (or Stacked).

Does DSS perform a better job than PI when it comes to image alignment? I don't know. I suspect not, but I also believe that the actual difference between the two may be very small, given enough suitable referece stars in the image series. Certainly I have never been able to 'fault' DSS in this area (or, indeed, in any of its capabilities - other than working with Meade DSI I and II OSC images).

So, no real reason, so far, to 'choose between' the two methods (phased or pipelined) - other than the amount of 'manual effort' needed to (currently) achieve results in PI.

But, I still maintain that PixInsight achieves 'better results' when it comes to the final stage of ImageIntegration - simply because of the (controllable) power of its pixel rejection algorithms, and the way PI can 'evaluate noise' for each image in a stacking series, allowing each image to 'only add what it should' to the final output.

That said, of course DSS can (easily) be used to calibrate and align the source data, leaving the final stage of integration (or stacking) to be done in PI.

So, no, I am not 'disagreeing' with you - there IS a perfectly good case for saying that it is 'easier' to get images pre-calibrated in DSS - perhaps even right through to stacking them as well. And, by doing this, a lot more time is left for the user to get into the 'meat' of the processing phase - which is where they will then have ALL of the Power of PI at their disposal (and, of course, a huge learning curve as well ).

I hear what you are saying about the 'voodoo-magic' of PI - and, yes, I agree that this applies to image calibration as well, but, let's be honest here - there is NOT really a lot of voodoo involved in getting your images calibrated in PixInsight. It really is more of a case of having to 'feed the pipeline' with the correct data, at the correct time.

And that is where DSS excels, above PixInsight - the process CAN be 'automated' -  you load the apporpriate image series into the appropriate containers, configure the process to do what you want, press the 'GO' button and watch the gears a-whirring in your PC whilst DSS churns out your master image. Does that 'help' the novice? Yes, sure, just the same way as calculators nowadays help our children divide 98 by 7.

For someone who HAS a good understanding of the process, they CAN then make an educated choice about which method they will use to get their data cakibrated. For someone who does NOT have a clear grasp of the stages involved in calibration, then actually EITHER package will help 'teach' them what needs to be done - and I would happily support either for that purpose. But, for those who are willing to put the effort into calibrating with PixInsight, they will just be using 'more powerful' tools to learn the process.

And, I am NOT condoning 'allowing people to calibrate their data incorrectly' - in fact I am a proponent for exactly the opposite. I am all for empowering users to make the effort to fully understand the process, so that they DON'T do it incorrectly, even though they have FULL CONTROL of the powerful tools provided by PI.

Don't get me wrong, I fully support the notion of an automated 'pipeline' process, tied in directly with the native GUIs of the sub-processes (and thus 'evolving' as the sub-processes evolve) - and I fully expect this to become available as PI matures further. And, when it does, I believe that there will be no 'need' to turn to DSS to 'simplify' the process.

However, do we need it 'today' (or even 'soon')?

No, not at all. For those who care to 'work' (or 'struggle', for those who really feel that the existing PI method is such a 'chore) the method in PI, all the 'phases' are currently available, bringing with them all the power of ImageContainers, ProcessContainers, etc., and the ability to save workflows to disk for later re-use, or re-play.

For those who may be intimidated by this extra level of complexity, or for those who just don't need the full Power of PI at this stage of their processing workflow, then there is no reason why they should not use DSS, or CCDStack, or Iris, or Nebulosity, or the myriad other programs that are available out there.

So long as they have ended up with as good as possible a 'master image' for all the PixInsight processing stages that will be ahead of them