PixInsight Forum (historical)
Software Development => PCL and PJSR Development => Topic started by: bitli on 2010 May 14 11:19:05
-
Hello,
Here is a first version of a script that aligns and integrates already calibrated FITS images, using the naming conventions of lightbucket.com (works fine with images produced with LB0002, as they are already calibrated).
You can add a bunch of files and the script will find the files to align and integrate (using file name pattern). It creates the alignment files in a working directory, then it integrates each channel in a view, and finally combine the channels if they are LRGB or RGB (if only L, it is left alone). The registration will adapt the size of the image to the reference image (selectable, the first L by default), so binned channels can be integrated too.
The convention to follow is to have the input calibrated files ending in one of -Red, -Blue, -Green, -Luminance (the script currently does not do any calibration).
It is recommended to have an empty workspace, as I have limited control on the generated image ids.
The script is somewhat complicated, but it is structured so that I can add other naming conventions, and extend the operations. My goal is to support PI naming (unfortunately the R ending is used for Red and for Registered), add an automatic calibration based on master frames and take more advantage of the fits keywords (some files have an FWMH already calculated, this could help select the best reference image). The GUI needs some polishing too and maybe some key properties could be made settable by the user. The name of the script may change when it is more general.
Comments welcome.
-- bitli
-
Hi
Many thanks for your hard work ;D
Have had a go and would say
1) Why the restricted file names , would be hard work to rename hundreds of files
2) How can we alter the sigma settings as the default is always used ???
Please keep at it as It will be very valuable addition :D
Harry
-
1: Why the restricted file names , would be hard work to rename hundreds of files
2> How can we alter the sigma settings as the default is always used ???
Thanks for the comments
1: Self fishness. I wanted to avoid setting the file type/channel for my usage, and it works fine with my files :). But I may add a capability to add by file type/channel like in DSS, or to select the type/channel. Not sure how to make easy at UI level.
2: I assume you refer to sigma high/low in image integration ? I could add the parameters. I just want to keep the UI simple with possibly a few key parameters, so it is possible to create an image relatively quicky. If needed, a second manual pass is always possible.
I do not know if it is possible show the user interface of a process (instead of executing it), so that the user can tweak a few parameters and then execute. This could be an alternative. I could also stop at the intermediate step so the user could have a look and stop or continue. However I have limited free time ;)
-- bitli
-
Hi
I did try altering the settings in the intergration tool , but had no effect ???
Harry
-
Sounds good! I will give it a try shortly.
If your interesting in doing more we could use a script to make masters calibration frames, bias, dark and flats
Max
-
Hi,
HArry, for the sigma, can you please check the history of the resulting image? If I change the valeu in the script, the resulting image has the new values. MAybe setting the flags to generate the rejected values images could help finding the problem.
Otherwise yes, I am interested in doing more. If there is a script that generates masters, then it would be easy to use them for calibration in my script. I am not sure how many steps we can automate in practice, as at some point the running time can be quite long.
-- bitli
-
Hi
I did check the consol and whatever settings I changed in the process( Intergration ) it just used the default ???
Harry
-
Hi Harry
Processes are objects... once you create a new instance, in this case on the script, it's parameters are initializated with the default values. So, the script should import the settings from the opened process instance by the main application. Don't know how to do this :P I'm not involved on the other side ;)
-
Why the restricted file names , would be hard work to rename hundreds of files
Hi Harry,
You want to get a hold of "Sequentialize.exe" from a company called "Polybytes" - it is a free stand-alone (it does NOT get 'installed', it just runs from where you 'store it'). Polybytes no longer seem to have it on their own website, but a Google search does find 3rd party distributors who still have it for distribution.
It is a great little utility - I use it for EVERY session, to get my filename structure they way 'I' want it !!
Cheers,
-
Hi bitli! I can't see any link in your post! (with Firefox, nor IE, nor Opera). What is happen? Tanks :surprised:
-
Forget it! :surprised: :surprised: Thanks!!
-
Looks Great but I can't use it with this naming convention.
How about using/ matching the fits key word filter.
Max
-
How about using/ matching the fits key word filter.
Max
That's my suggestion too and the way I'm planning to do it in the AutoCalibration script that I am working on. There is information about accessing the fits header info in this post,
http://pixinsight.com/forum/index.php?topic=1904.0
Mike
-
Yes, it was my original idea, but I was not able to read FITS keywords before the correction of Juan. I may adapt it later.
Anyhow Light bucket seems to require special processing for calibration due to their non standard FITS files.
-- bitli
-
As far as I know you can't download individual calibration files from LB. As you can't have access to the individual bias, dark and flat frames, you can't generate your master calibration frames. In this way you lose control over an extremely important part of the whole process —with high probability the master calibration frames are wrong, IMHO (see Vicent's tutorial (http://pixinsight.com/tutorials/master-frames/en.html)). Besides incompatibility problems caused by the use of floating point FITS files, it is too bad that you can't generate your masters with PixInsight's calibration tools.
-
LB isn't the only option out there. If they don't fix their problems, take your money somewhere else :)
-
I have to agree with Sander - and I am totally un-biased here as I have never used any remote telescope service.
However, you have to keep in mind that those who provide such a (commercial) service are 'vendors', and if you are not getting what you are paying for, stop paying :moneyinmouth:
That said, perhaps they (i.e. the 'astute' vendors) will realise that many users will want to purchase JUST the raw data, not the data as massaged by some process over which they have no control.
Certainly, given the cruddy weather we seem to be having here all the time nowadays, I might yet have to resort to purchasing 'remote photons' - either that, or I am moving down to Juan's country cottage, or over to Carlos' remote mountain hideaway ::)
Cheers,
-
A few years ago I attended a talk at NEAIC presented by a technical director of one of the scope services out there. I mouth fell open when he said that all you get is a jpg, raw FITS data was planned for later. I think their goal was to attract highschool science classes but still.
-
If only I had a car and money... I would be picking chicks somewhere ;) :D :D
-
As far as I know you can't download individual calibration files from LB. As you can't have access to the individual bias, dark and flat frames, you can't generate your master calibration frames. In this way you lose control over an extremely important part of the whole process —with high probability the master calibration frames are wrong, IMHO (see Vicent's tutorial (http://pixinsight.com/tutorials/master-frames/en.html)). Besides incompatibility problems caused by the use of floating point FITS files, it is too bad that you can't generate your masters with PixInsight's calibration tools.
It would be nice to continue working on a script thank automates this work flow. I still use MaxIM DL since it is easier than going through all the steps manually.
Max
-
Hi Max
If we keep winging , someone will listen >:D
Harry
-
It would be nice to continue working on a script thank automates this work flow. I still use MaxIM DL since it is easier than going through all the steps manually.
Max
And to Harry as well,
This is one of the scripts I was thinking of giving a go, though I don't really know exactly what I'm in for yet, it seems that someone will likely beat me to it... ;D
Jim
-
H Jim
You write and I wll supply the beer :D
Harry
-
I'm attempting to draw the conversation from the 1.6.1 release (http://pixinsight.com/forum/index.php?topic=2013.0) on this subject back to a thread that is on-topic.
Nikolay
I am up for the monster and can do draught beer if its required ;D
Regards harry
I started drafting a text file of the "process" so I could get my hands around what might be needed in a script when I realized this could really be a "monster"...and now my confidence is rattled.
I think I'm going to create a block-diagram to streamline what I was thinking. It seems like it's roughly in alignment with comments from the other thread, but I have to admit this looks a bit daunting to get started on. I was planning on mainly calling functions from the other already existing scripts/processes (wanting to keep them separate so they could be updated without impact to the "easy button" script).
Getting time is going to be my challenge (family vacation's starting this week, and it sounds like NKV may already be on it?). I'd use this as a learning exercise anyway...
Jim
-
One thing that's very nice in Maxim is the way it remembers where all your calibration masters are, unless you decide to override. Likewise does a great job of recognising all the filters, producing individual masters should you want to save them.
I guess the reality is that it is likely to be a monster to cater for all the different ways people like to work - bit like Nikolay's comment about not being so sure why we want to "build DSS inside PI".....
-
Just note: PixInside on 99% ready to align, integrate and calibrate via one click. ;)
to get 100% we need push Juan to add options "save to file" in ImageIntegration module.
Also if we not like to use ProcessContainer and want to write own dialog, we must ask Juan to add possibility open external module like subprocess dialog.
-
Is anyone working on this?
Max
-
Hi
As said , a small amount of work from Mr Juan and this would be sorted , So how about it master ;D
Harry
-
I wish I knew JScript.
Max
-
I wish I knew JScript
Hi Max,
I said that as well - about a year ago. Then I realised that I 'did' know JScript - because it is - more or less - 'English'
OK, so I had to learn a whole load of fancy steps, and Juan (and others) were willing to help there. But, by looking through other people's efforts (thanks to the fact that we ARE so willing to 'share' our code) I was able to put together my first PJSR script quite quickly.
And I tried to make sure that it was as well commented as I could make it - so that 'I' would be able to understand it the next time I had to work on it, or any other PJSR script. I am sure that, given a 'requirement' and a little bit of knowledge and skill, you too would probably be able to create 'something' in PJSR - helped (as usual) by the excellent folks here on the Forum.
Go on - give it a go. You know you want to ::)
Cheers,
-
I'm not sure PJSR is the right tool for the job. The modal nature of the dialogs really turns me off from writing complex dialogs. Once the super-duper stacker script runs you can not use PI for anything else. If you want to examine the result you need to close the dialog so you need to have some kind of filelist save feature so you don't have to start from scratch each time. And then you have to basically copy all the parameters from the processes you use into the script. It'll be huge.
I'm sorry to be pessimistic about this, someone please explain how this realistically can be done.
-
Sander, I would rather see it all as standard feature in the modules.
Juan has not seem to be to interested in this.
For some reason Juan suggest it could done in a script. Perhaps he was just speaking from fatigue
Max
-
Sander, Max,
Yes - I also agree that this feature would be better served as a PCL module, not as a PJSR script. Unfortunately that would also restrict the number of people able to code the process in the first place. I am sure that there are folks like me who can work with PJSR, but who have no idea where to start in PCL.
This may well have to be something that the likes of Juan works on - with guidance from users like us. This allows the interface to 'mature' naturally. It is certainly NOT a process that needs to be rushed, after all we have been able to calibrate in PI for quite some time now (even before the advent of the ImageCalibration process) - it just takes a whole lot of 'manual effort'.
I am willing to wait several more PI 'cycles' if needs be.
Cheers,
-
'It just takes a whole lot of 'manual effort'.
That's and understatement.
Any improvement in the modules would be helpful. There is a lot of feature that I thought would be basic.
For example, just about every program I have ever used allows you to choose a set of images to automatically produce a master as part of basic cal.
Also, if the modules where able to work serially in the process container we might not be having this conversation !
Max
-
Hi again Max,
Just bear in mind how PixInsight is developing - the very concept of image calibration is very much an afterthought. The fact that you CAN calibrate and integrate image data in PI is testament to Juan's herculean efforts. Even with the hard manual effort currently required to achieve full calibration and integration in PI as things stand just now, I still believe that the final outcome (due to the technology 'behind the scenes') still outperforms other packages in the marketplace.
Like I said, I would happily wait another year for a full-blown calibration and integration process. At least I would know how it had developed, and would have had a say in the way I wanted to see it work.
Don't get me wrong, if I had the tiome and ability to put this together myself, I would - gladly. But I don't get to retire for a few years yet, so - unless my Lotto numbers are due for an appearance, I have to keep hammering the keyboard for 'someone else' in the meantime ::)
Cheers,
-
Hi,
if it would be possible to save settings for the calibration, alignment and stacking modules then the script wouldn't need to deal with those. It would simply instantiate the processes which in turn would load their default settings and receive file lists from PI. This would be doable in PJSR as this is a short lived window. It would still need the ability to save file lists and parameter settings though. Ouch.
-
It needs to hand off the processed image set from cal to, alignment then intergration. It should use current settings, this would include choice of reference image. It would nice to calibrated version of the reference frame.
The abilty to calibrate to flats and make masters from within the cal module would be very helpful.
Hopeful Juan is reading this.
Max
-
Yes - as much as I love PI, when I've got limited data that's unlikely to benefit from the extra Smarts, I give in to temptation and just throw it all into Maxim's nice Stack routine.
-
That's the problem most people give up on it and use something else insted.
Not as user friendly as many packages.
Max
-
If you're on Linux or FreeBSD your choices are limited and you'll use what you can. On Mac there's Neb2 which is a little easier to use but not that much. On windows there are plenty of options.
-
I am willing to write this in PCL if I am not duplicating someone else's efforts.
I like the ease of DSS, but I DO get better results when I go through Vicent's calibration steps. And I really enjoy working with PixInsight in general. It appeals to my inner software developer :P
So - here is my question: Will I be duplicating functionality that is already going to be built in PI in the near future? And is anyone else already doing this?
Let me know - we can agree on what the basic requirements will be, and I will start coding...
Cheers,
Dave
-
Hi Dave,
I'd wait with tackling this until it's possible to bring up individual process GUIs through either PJSR or PCL. Without that you'd end up most of your time replicating GUI items from all the processes you need to integrate. That or it should be possible to save process options and restore them later. Perhaps this is possible by having the user create a process container that contains all the processes with their options. I assume it's possible to read Process container contents from PCL and PJSR. It's a bit of a cludge but could save significant development time.
-
Hi Dave
I don't think that this will be developed by Juan (or me) in the shot/mid term. AFAIK, we both are more of "divide and conquer" guys than "do it all".
Having said that, almost everything is currently working as separate processes. So, a script to automate everything should not be too hard to create. Of course, there is the problem of working with large arrays of data, with "unknown" format, and one wants to have everything inside the same code. So, it should be well organized. :)
On a parallel note, I'll be writing a process module that performs some of this tasks, in a much more simpler way. I have to calibrate and align microscopic image for a project I'm working on the university. I may release the code of the "preprocessing" steps, in a month or two. The workflow shares some steps, but then it diverges and needs other steps not applicable to astronomy. Anyway, if anybody wants to write something from the scratch, this may help a bit.
-
So maybe PCL isn't quite ready to do this in a re-usable sort of way? I am not really familiar with PCL yet, but I guess what you are saying is, for example, I can't load the ImageIntegration module's UI portions from another module?
-
No, I don't think that's possible. I recently started a new thread about that.
http://pixinsight.com/forum/index.php?topic=2202.0
-
No. Processes are self contained objects. Don't communicate. The only exception right now is Process Container.
I always wanted more interaction between processes... for example, dropping a HT icon into the STF window to load the parameters, or viceversa (isn't this implemented yet, right?) :D