PixInsight 1.6.1 - New RANSAC fitting rejection algorithm in ImageIntegration

[RobF2]

Nikolay,

You really want to get the monster?

My advice is to avoid asking that kind of questions to these guys. They tend to say yes when they see the words "want" and "monster" together  

Beer? someone said beer?  

Yes! Yes! Yes!   (to monster AND beer : )

I actually had in mind the bare essentials to start with - a script that called saved process icons preconfigured with your library bias/darks/flats and usual integration settings.  You would run it for each filter then manually do RGB or LRGB combines.  Could even be a button on your fantastic animation script that called process icons for Image cal, alignment, then stacking - that alone would be a huge benefit saving have to specify related images over and over.  I know some people don't like the idea of library calibration images, but a lot of us do work this way.  

The "monster" with tabs and endless configuration for each filter/channel would be the ultimate - I get frightened of asking the IT guys at work for the ultimate though 'cause they get frightened off and I get nothing.....

Perhaps this should go to another thread though if you do run with it Nikolay - to get us out of Juan's excellent integration explanation?

[Jack Harvey]

Howdy, Guess i am a little late to the party.  Only one small point.  I note you have included a tab for DarkFlat n your proposed Frankenstein of processing tools.  I have yet to hear a good argument as to why Flats need to be dark calibrated.  Bias Yes! Dark, not so much?

[mmirot]

I wonder if we nonchalantly threw this in as a request to Nikolay if he'd do it with default parameters as part of his animation script.....

Friends, if you like to see simple calibration (by master files w/o dark optimization) in animation script, I will do it. It's easy to implement. You really want to see result w/o dark optimization?

When are we going to get a script or something to help link the work flow process together?
ie calibration, registration, and integration.

For me it's: integration ( masters ) > calibration > registration > integration.
How you like to care all settings for this long process? You want to see 7 tabs (bias/dark/flat/darkflat/calibrate/register/integration) and many-many buttons/checkers/slider/list on every tab?
You really want to get the monster?

Sounds like a good plan. You might want to look at MaxIMs calibrate and Stack commands for ideas.
They did a nice job on the worth flow but PI has better processes now.
You did a great job on blink. I think you can handle it!

Max

[mmirot]

The more images, the better linear fit is possible, and hence the new algorithm will also perform better. It's a matter of testing it and see if it can yield better results than the other rejection algorithms available, for each particular case.

Great explaination.

We will have see how it works out in practice.
I commonly get 15-25 images, so it is worth a try.

The hardest problem to deal with is slight changes in transparency this can occur even on very good nights. These end up being complex gradients. It is hard to identify the inliers.

(  Of course this does not fix the gradients , we still do DBE after our integration)

Max

[Niall Saunders]

Hi Jack,

Howdy, Guess i am a little late to the party.  Only one small point.  I note you have included a tab for DarkFlat n your proposed Frankenstein of processing tools.  I have yet to hear a good argument as to why Flats need to be dark calibrated.  Bias Yes! Dark, not so much?

The reason I have included this is that this is how I have ALWAYS calibrated my imaging data. My Flats tend to be acquired as 2s exposures (or more, for NB data). And, as far as I am concerned, there is enough 'dark' signal at 2s to allow it to be statistically evaluated and therefore correctly eleiminated by using a MasterFlatDark.

I am 'not comfortable' with Biases being used to calibrate Flats, unless the Flats can be demonstrated to have no 'significant' signal attributable to 'thermal noise'. Certainly, because of the ultra-short Bias exposures, no statistical 'thermal' signal can be obtained. So, if the same cannot be said for the Flats, then using only a MasterBias to calibrate the Flats will still leave the MasterFlat with a thermal-noise content.

Then, is there not also the issue with the likes of full-frame CCDs? There is a finite minimum shutter-speed that these will operate with, and that shutter-speed is SIGNIFICANTLY longer than typical exposure times for Bias frames. So, I would want to be using multi-second FlatDarks once again, and just not bothering with Bias frames at all (because, they really just will NOT be 'true' Bias frames, because their exposure times will be too long). Have I just failed to understand something here?

In any case - I think it would be wrong to 'force' users to work with Biases as FlatDarks. I think that the option should remain a 'user option'.

Cheers,

[mmirot]

Keeping the option is fine.

Unless your camera has very high dark current or your flats are several minute long you are most likely wasting your efforts. 

My camera generates a only a few electrons under one minute.

I could be helpful with a uncooled camera such as DLSR in warm weather.

Max

[NKV]

Beer? someone said beer? 

Juan, take beer, beer from Harry and all who really want to see the 'monster'. And just for fun, I'll be happy to send you a beer, if you collect all the tools in the 'one-click monster'.

I really not understand why so many people want to see DSS inside PI ? Maybe because everyone like use familiar tools... or not want to get best result from given sources. Some times I like to see fast result, but it's just for nothing or just for web publication. Really I see only one situation where the 'monster' might be useful: generation of images for a garbage.

I used DSS and other software like 'one click' preprocessing, but I newer happy with final result. Yes, we can to set all the settings at once, then press the button "make nice" and wait for a masterpiece. But do not come Masterpieces ... maybe better to focus on each step separately. Really there is not enough in PI is a tool for analysis. I prefer control and analyse result of everyone step, so IMHO the module structure of PI processing is best of the best solution for best result.

Friends, you really think the 'monster' can generate APOD ? Or any reason why you not want to do preprocessing step-by-step.

Also, I don't want to see PI in one line with other 'one click software'.

[Jack Harvey]

Niall  My understanding is with modern cameras, the build-up of dark current is minimal if the flat exposure time is kept reasonably short. Typically, dark current may be ignored at exposure durations less than 30 - 45 sec. Only a master bias subtraction is needed to calibrate flat frames

Nikolay  Darn I was hoping there would be a checkbox that would send the appropriately sized image to Jerry Bonnell et al at APOD for automatic publication:-(   Seriously I agree that for quality imaging every step of the processing needs to be carefully monitored.  It takes hours, days or even weeks to gather your data, why does it have to be processed in a few minutes?  Just my two cents worth.

[mmirot]

Beer? someone said beer? 

Juan, take beer, beer from Harry and all who really want to see the 'monster'. And just for fun, I'll be happy to send you a beer, if you collect all the tools in the 'one-click monster'.

I really not understand why so many people want to see DSS inside PI ? Maybe because everyone like use familiar tools... or not want to get best result from given sources. Some times I like to see fast result, but it's just for nothing or just for web publication. Really I see only one situation where the 'monster' might be useful: generation of images for a garbage.

I used DSS and other software like 'one click' preprocessing, but I newer happy with final result. Yes, we can to set all the settings at once, then press the button "make nice" and wait for a masterpiece. But do not come Masterpieces ... maybe better to focus on each step separately. Really there is not enough in PI is a tool for analysis. I prefer control and analyse result of everyone step, so IMHO the module structure of PI processing is best of the best solution for best result.

Friends, you really think the 'monster' can generate APOD ? Or any reason why you not want to do preprocessing step-by-step.

Also, I don't want to see PI in one line with other 'one click software'.

After I remove the bad images automation usually works just as well as step by step.

If you have the option to save images at each step you can troubleshoot your problems fairly easily.

Likewise if you want to integrate again with different parameters you have aligned and calibrated images.



It is just another tool .

You can spend your time on where it counts best.
You must validate manually that your calibration and aligments settings generally work without problems.
I find that once I have this working for a image set or camera the rest goes smoothly.

"you really think the 'monster' can generate APOD"

It is not about fast quick and dirty methods just to generate a pretty picture.
It is about an effective process.
I just don't like to waste my time starring at a busy hour glass cursor doing the same thing over and over again.

If you don't have the desire to do this someone will eventually take it on.

Max

[Harry page]

Hi Juan

I use many dozens of images in my stacks and anything that helps is great , and I have offered you wine before , now you want beer

Niall

I never use darks full stop . and I must say if your camera gains statistically important dark current in a few seconds , its time to get a new camera 
But seriously you can add more noise by doing this and this is one reason why most people use just a bias frame

Nikolay

I am up for the monster and can do draught beer if its required 

I do not see making the process more user friendly a disadvantage and I do not always want fast results , but why make things harder than they should be.
And as I said before I do not use darks , so skip this if you like ( Calm down everybody I was only joking )

I think you have made a rod for your own back by doing the brilliant blink script 

All hail the Nikolay  , I am trying hard here , you don't want to see a grown man beg 

Regards harry

[mmirot]

Pehaps Juan can make calibrate, align and integrate work with process container ?

[NKV]

calibrate, align and integrate work with process container ?

Process Container is a 'monster' at 99%.
Juan, please add 1% ("save to file" in ImageIntegration module)  

And about External Objects:
Possible to open dialog via script:

Code: [Select]

var p = new ImageIntegration();
p.launchInterface();We can Run via p.executeGlobal()
But how to enable user to use control elements of External Objects dialogue and wait closing for collect settings from ?

[RobF2]

Sounds like a good plan. You might want to look at MaxIMs calibrate and Stack commands for ideas.
They did a nice job on the worth flow but PI has better processes now.
You did a great job on blink. I think you can handle it!

Max

+1 re Maxim's logical tabbed workflow model (although what you're supposed to do on each tab can be a bit archane)

I understand and agree with what people are saying about unreasonably expecting a masterpiece from full automation, but also suspect many of us are very pleased with the routine settings we use for image cal and alignment - PI is just THAT good at producing a quality end result.  Please remember though I'm talking and thinking as a relative CCD newbie who shoots between 30mins to 4 hrs on most objects.  I'm extremely happy with my QHY9 and PI processing - but would prefer to skip a few of the very repetitive steps early on at all possible.....   

Fascinating discussion....

[Niall Saunders]

Hi all - and really sorry about your original thread being hijacked, Juan

However - the Monster - well, it isn't really a Monster at all - it should just be a way of coordinating all you raw data files, and the processes that you will fine-tune to calibrate these.

In my mind, you would not even 'feed' the Monster in one go. There is nothing against each tab being capable of being 'switched off' for a time. Spread the Monster's meals over several session, making sure that each is fine-tuned (to avoid 'indigestion' - Monster-Barf is truly horrible stuff )

As each interim stage is fine-tuned, it gets left alone. The data each stage provides may obviously be used in subsequent stages. Eventually all sub-sections have been fine-tuned, and the top-level images can be created using the "GoForIt" button. But, perhaps as important, is to then be able to save the entire calibration configuration for later re-use.

That is really what we are missing I suppose - a simple (PCL-based) interface that combines II and IC (perhaps along with deBayer and Star Alignment)

What we need to therefore start with is a 'Baby Monster' that is based on ImageCalibration - but where this new SuperCalibration process allows us to ALSO specify the files that will generate the MasterBias and MasterDark frames - along with how these subs should be integrated in order to create the said MasterFrame.

Don't get me wrong - I am PERFECTLY happy with the 'manual method' that I already use - especially now that I can save individually selected ProcessIcons. A key part of my astroimaging pleasure comes from being able to recreate EVERY single step in the processing process, so any interface that makes that easy for me is an advantage.

Cheers,

[mmirot]

My work flow currently.
Inspect images remove bad ones including bad flats.

Setup MaxIm DLs calibration by adding new flats for each filter.
MaxIM will allows calbration all image with different filters simultaneously.
This is a huge time saver.
MaxIm automatically makes Masters from the flats. 
Another huge time saver.
I generally have Master Dark and Bias in setup already.
MaxIm will make a new Master Automatically any time you give it a new set  offlats, Bias, darks.
Another huge time saver.

I often process/Calibrate a single image from each filter just a first check.

However after this, There is no reason not process this as a batch.
There are few real options for calbration once it is set up.
Or aleast,  I tend to use the same setting such dark as optimization etc.

Calbration results should not really vary once you find the best settings.

If you want to bang your head on the wall by manual making every master by hand and doing each set at a time go a head.

I can tell my results will be the same as yours unless you make a mistake with pixel math etc.

I can produce well calibrated images in MaxIM in this manner.
 I often do , then use them in PI.

I would use PI Calbration more often if it was more user friendly.
I wound like to try using the overscan feature in PI.
I doubt it will prevent me from a APOD since 30-40 % were processed in MaxIM this way.

Next consider registration. Once you choose a reference frame and elminate frames that won't align you are fine.
It is always the same. No real no settings to mess with.
The software tells you if you can't register a image.
It is fairly binary, it works or not.
PI tells you this and how well your registration did.

What do you want to do each image set one at a time? 
Or perhaps hand star register with 5 stars for 46 images?

There is no reason to not batch these processes. There is nothing to be gained.

Integration is the only step that may require some tweeking.
If your automated work flow saves the aligned images then you change your settings and reintegrate.

My 2 cents.

End of message

Max