PixInsight 1.6.1 - New RANSAC fitting rejection algorithm in ImageIntegration

[mmirot]

Keeping the option is fine.

Unless your camera has very high dark current or your flats are several minute long you are most likely wasting your efforts. 

My camera generates a only a few electrons under one minute.

I could be helpful with a uncooled camera such as DLSR in warm weather.

Max

 

[NKV]

Beer? someone said beer?  :-*

Juan, take beer, beer from Harry and all who really want to see the 'monster'. And just for fun, I'll be happy to send you a beer, if you collect all the tools in the 'one-click monster'.

I really not understand why so many people want to see DSS inside PI ? Maybe because everyone like use familiar tools... or not want to get best result from given sources. Some times I like to see fast result, but it's just for nothing or just for web publication. Really I see only one situation where the 'monster' might be useful: generation of images for a garbage.

I used DSS and other software like 'one click' preprocessing, but I newer happy with final result. Yes, we can to set all the settings at once, then press the button "make nice" and wait for a masterpiece. But do not come Masterpieces ... maybe better to focus on each step separately. Really there is not enough in PI is a tool for analysis. I prefer control and analyse result of everyone step, so IMHO the module structure of PI processing is best of the best solution for best result.

Friends, you really think the 'monster' can generate APOD ? Or any reason why you not want to do preprocessing step-by-step.

Also, I don't want to see PI in one line with other 'one click software'.

 

[Jharvey]

Niall  My understanding is with modern cameras, the build-up of dark current is minimal if the flat exposure time is kept reasonably short. Typically, dark current may be ignored at exposure durations less than 30 - 45 sec. Only a master bias subtraction is needed to calibrate flat frames

Nikolay  Darn I was hoping there would be a checkbox that would send the appropriately sized image to Jerry Bonnell et al at APOD for automatic publication:-(  Seriously I agree that for quality imaging every step of the processing needs to be carefully monitored.  It takes hours, days or even weeks to gather your data, why does it have to be processed in a few minutes?  Just my two cents worth.

 

[mmirot]

Beer? someone said beer?  :-*

Juan, take beer, beer from Harry and all who really want to see the 'monster'. And just for fun, I'll be happy to send you a beer, if you collect all the tools in the 'one-click monster'.

I really not understand why so many people want to see DSS inside PI ? Maybe because everyone like use familiar tools... or not want to get best result from given sources. Some times I like to see fast result, but it's just for nothing or just for web publication. Really I see only one situation where the 'monster' might be useful: generation of images for a garbage.

I used DSS and other software like 'one click' preprocessing, but I newer happy with final result. Yes, we can to set all the settings at once, then press the button "make nice" and wait for a masterpiece. But do not come Masterpieces ... maybe better to focus on each step separately. Really there is not enough in PI is a tool for analysis. I prefer control and analyse result of everyone step, so IMHO the module structure of PI processing is best of the best solution for best result.

Friends, you really think the 'monster' can generate APOD ? Or any reason why you not want to do preprocessing step-by-step.

Also, I don't want to see PI in one line with other 'one click software'.

After I remove the bad images automation usually works just as well as step by step.

If you have the option to save images at each step you can troubleshoot your problems fairly easily.

Likewise if you want to integrate again with different parameters you have aligned and calibrated images.



It is just another tool .

You can spend your time on where it counts best.
You must validate manually that your calibration and aligments settings generally work without problems.
I find that once I have this working for a image set or camera the rest goes smoothly.

"you really think the 'monster' can generate APOD"

It is not about fast quick and dirty methods just to generate a pretty picture.
It is about an effective process.
I just don't like to waste my time starring at a busy hour glass cursor doing the same thing over and over again.

If you don't have the desire to do this someone will eventually take it on.

Max

 

[Harry page]

Hi Juan

I use many dozens of images in my stacks and anything that helps is great , and I have offered you wine before , now you want beer

Niall

I never use darks full stop . and I must say if your camera gains statistically important dark current in a few seconds , its time to get a new camera  >
But seriously you can add more noise by doing this and this is one reason why most people use just a bias frame

Nikolay

I am up for the monster and can do draught beer if its required  ;D

I do not see making the process more user friendly a disadvantage and I do not always want fast results , but why make things harder than they should be.
And as I said before I do not use darks , so skip this if you like ( Calm down everybody I was only joking )

I think you have made a rod for your own back by doing the brilliant blink script 

All hail the Nikolay  :angel: , I am trying hard here , you don't want to see a grown man beg 

Regards harry

 

[mmirot]

Pehaps Juan can make calibrate, align and integrate work with process container ?

 

[NKV]

calibrate, align and integrate work with process container ?

Process Container is a 'monster' at 99%.
Juan, please add 1% ("save to file" in ImageIntegration module)  

And about External Objects:
Possible to open dialog via script:

var p = new ImageIntegration();
p.launchInterface();

We can Run via p.executeGlobal()
But how to enable user to use control elements of External Objects dialogue and wait closing for collect settings from ?

 

[RobF2]

Sounds like a good plan. You might want to look at MaxIMs calibrate and Stack commands for ideas.
They did a nice job on the worth flow but PI has better processes now.
You did a great job on blink. I think you can handle it!

Max

+1 re Maxim's logical tabbed workflow model (although what you're supposed to do on each tab can be a bit archane)

I understand and agree with what people are saying about unreasonably expecting a masterpiece from full automation, but also suspect many of us are very pleased with the routine settings we use for image cal and alignment - PI is just THAT good at producing a quality end result.  Please remember though I'm talking and thinking as a relative CCD newbie who shoots between 30mins to 4 hrs on most objects.  I'm extremely happy with my QHY9 and PI processing - but would prefer to skip a few of the very repetitive steps early on at all possible.....  >

Fascinating discussion....

 

[Niall Saunders]

Hi all - and really sorry about your original thread being hijacked, Juan :'(

However - the Monster - well, it isn't really a Monster at all - it should just be a way of coordinating all you raw data files, and the processes that you will fine-tune to calibrate these.

In my mind, you would not even 'feed' the Monster in one go. There is nothing against each tab being capable of being 'switched off' for a time. Spread the Monster's meals over several session, making sure that each is fine-tuned (to avoid 'indigestion' - Monster-Barf is truly horrible stuff :)

As each interim stage is fine-tuned, it gets left alone. The data each stage provides may obviously be used in subsequent stages. Eventually all sub-sections have been fine-tuned, and the top-level images can be created using the "GoForIt" button. But, perhaps as important, is to then be able to save the entire calibration configuration for later re-use.

That is really what we are missing I suppose - a simple (PCL-based) interface that combines II and IC (perhaps along with deBayer and Star Alignment)

What we need to therefore start with is a 'Baby Monster' that is based on ImageCalibration - but where this new SuperCalibration process allows us to ALSO specify the files that will generate the MasterBias and MasterDark frames - along with how these subs should be integrated in order to create the said MasterFrame.

Don't get me wrong - I am PERFECTLY happy with the 'manual method' that I already use - especially now that I can save individually selected ProcessIcons. A key part of my astroimaging pleasure comes from being able to recreate EVERY single step in the processing process, so any interface that makes that easy for me is an advantage.

Cheers,

 

[mmirot]

My work flow currently.
Inspect images remove bad ones including bad flats.

Setup MaxIm DLs calibration by adding new flats for each filter.
MaxIM will allows calbration all image with different filters simultaneously.
This is a huge time saver.
MaxIm automatically makes Masters from the flats. 
Another huge time saver.
I generally have Master Dark and Bias in setup already.
MaxIm will make a new Master Automatically any time you give it a new set  offlats, Bias, darks.
Another huge time saver.

I often process/Calibrate a single image from each filter just a first check.

However after this, There is no reason not process this as a batch.
There are few real options for calbration once it is set up.
Or aleast,  I tend to use the same setting such dark as optimization etc.

Calbration results should not really vary once you find the best settings.

If you want to bang your head on the wall by manual making every master by hand and doing each set at a time go a head.

I can tell my results will be the same as yours unless you make a mistake with pixel math etc.

I can produce well calibrated images in MaxIM in this manner.
I often do , then use them in PI.

I would use PI Calbration more often if it was more user friendly.
I wound like to try using the overscan feature in PI.
I doubt it will prevent me from a APOD since 30-40 % were processed in MaxIM this way.

Next consider registration. Once you choose a reference frame and elminate frames that won't align you are fine.
It is always the same. No real no settings to mess with.
The software tells you if you can't register a image.
It is fairly binary, it works or not.
PI tells you this and how well your registration did.

What do you want to do each image set one at a time? 
Or perhaps hand star register with 5 stars for 46 images?

There is no reason to not batch these processes. There is nothing to be gained.

Integration is the only step that may require some tweeking.
If your automated work flow saves the aligned images then you change your settings and reintegrate.

My 2 cents.

End of message

Max

 

[varmint]

especially now that I can save individually selected ProcessIcons.

How do you do this?  Copy/paste the code in the History explorer?

Can you save Process Containers?

I tried looking for answers to this last night and couldn't...  I think I ran into a thread from a year ago that said this wasn't available, but I am not sure.

Thanks,

Jim

 

[georg.viehoever]

Hi Juan,

Enter linear fit clipping:

...
The remaining question is: is linear fit clipping the best rejection algorithm? I wouldn't say that. Linear fit rejection requires a relatively large set of images to work optimally. With less than 15 images or so its performance may be worse or similar to Winsorized sigma clipping. To reject pixels around a fitted line with low uncertainty, many points are required in order to optimize it in the least average deviation sense. The more images, the better linear fit is possible, and hence the new algorithm will also perform better. It's a matter of testing it and see if it can yield better results than the other rejection algorithms available, for each particular case.
...

I must admit that this method of identifying outliers and the "true value" of a pixel appeared extremely suspicious to me. After all, who says that a distribution without outliers would be well fitted by a linear function after sorting the samples? So I did an experiment with Excel, generating 50 samples with a poisson distribution (as a CCD would do), and adding some polluting signal (a sine) to this that was intended to simulate changing sky conditions such as haze. As can be seen in the attached screenshot or Excel 2007 file, your approach would come close the "true" value of 8000 in both the unpolluted and the polluted case. Quite a surprise for me.

Is there any literature that describes the method? What are known cases where this method produces bad estimates?

Kind regards,
Georg

 

[georg.viehoever]

Hi,

I did some additional research.

In essence, this image is a Q-Q plot http://en.wikipedia.org/wiki/Q-Q_plot for a uniform distribution http://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29, i.e. when all values generated by the random process (here: CCD values) are equally likely within a certain range, the plot will be similar to a straight line. For distributions where values at the extremes become less likely (e.g. Poisson or normal distribution), the curve will always look more or less like an inverted S.

Basically, finding "outliers" always means that you have some idea how the data should be distributed, and filtering out those that do niot fit under this assumption. I think for CCD data, assuming a Poisson distribution http://en.wikipedia.org/wiki/Poisson_distribution is a valid approach. So to further refine this approach, it may be a good idea to use a Q-Q plot for a poisson distribution instead of one for a uniform distribution.

Just some ideas, not practically tested....

Georg

 

[Niall Saunders]

Quote from: Niall Saunders on June 24, 2010, 15:08:21
especially now that I can save individually selected ProcessIcons.

How do you do this?  Copy/paste the code in the History explorer?

Can you save Process Containers?

I tried looking for answers to this last night and couldn't...  I think I ran into a thread from a year ago that said this wasn't available, but I am not sure.

Thanks,

Jim

Jim - I am moving the reply to your question into the <Tips and Tricks> section

Have a look at http://pixinsight.com/forum/index.php?topic=2025.0

Cheers,

 

[Emanuele]

How come I dont have RANSAC Fitting Rejection under ImageIntegration ?

I just have Linear Fit Clipping.....

E.

 

[Juan Conejero]

Hi Emanuele,

I decided to drop RANSAC fitting in favor of the new linear fit clipping rejection algorithm. Sorry for the confusion I've generated with this post; I'll add a note to inform about this at the top of the thread.

 

[Emanuele]

Oh, no worries juan, thank you fir clarifying that.