PJSR Script to 'Calibrate' images

[Niall Saunders]

Hi all,

Now that Juan has managed to implement Image Registration and Star Alignment into the 'core' of PI, and as a worthwhile (I believe) exercise prior to building full calibration facilities into PI as well, I wonder whether the 'open code' nature of PJSR is perhaps an ideal place to create the 'calibration processes' that can then be moved into PI once they have been 'thrashed out' by us, the users?

I already have a script that creates appropriate 'Tree Lists' of selected files, and I am working on adding 'Tabs' to the dialogue window, so I see my 'core' code as being 'mostly written'.

My thoughts are to provide a Tab for each image data type, i.e. one for Lights, one for Darks, etc (in a similar fashion to DSS, I suppose). For each image type, having specified the files that you wish to 'collate', you would then have the option of specifying 'how' you want the (background, i.e. 'core') Image Integration process to be applied.

For example, for the Lights, you may choose to apply either a MasterDark or a MasterFlat, or both. But you would not specify an integration process, as your images would still need to be registered (Star Aligned) before you would want to combine them. Obviously, this stage could be catered for as well - but I plan to walk before I run !!!

For the Darks tab, you would have a choice of 'how' to combine the images that you have selected - my initial choice would always be to 'Median Combine', and probably without any form of ReScale(0.0, 1.0) function. But the options would be presented to the user - giving full flexibility (including the possibility of just selecting a single image, already created with the intention of being used as a 'MasterDark')

Flats and FlatDarks (the next two tabs) follow their counterparts of Lights and Darks, except that - in this special case - you CAN combine the 'integrated' and (perhaps) dark-corrected F;ats, as they will NOT require to be Star Aligned first. However, the MasterFlat would perhaps need to be 'normalised' (under user control) such that either the 'maximum' or the 'median' (or the 'mean') of the MasterFlat was adjusted to be '1.000000' prior to being 'divided' into each of the (dark-calibrated) Lights.

My scripts already allow for destination folders to be defined, and they also allow for file-overwriting, if needed.

I am also convinced that the 'experts' amongst you can share your knowledge regarding hot, cold, warm, stuck and dead pixels. And those with NABG imagers can make suggestions based on your experiences as well.

Hopefully, as the fundamentals of the required processes are established, Juan (or our new 'PCL' expert, Sander !!) can then start converting the PJSR code into 'core' code for a future release.

Any comments?

Cheers (with fingers poised over keyboard, ready to start coding !!!!!!!!),

[Juan Conejero]

Niall, and all,

I wholeheartedly approve and support this initiative. I am sure there are many members of this forum that have experience and knowledge enough (much more than I, of course) to help in carrying out this task. I'd be delighted to see a set of calibration tools written around the PJSR and/or PCL frameworks.

For image calibration tasks in particular, I think the scripting interface can be the best option. As I understand it, there are no performance penalties in using JavaScript to implement image calibration, since all performance-critical sections would be implemented as calls to framework routines (which are native multithreaded routines). The main advantages of JavaScript in PixInsight are that (1) it is cross-platform by nature, and (2) scripts are easy to test and require no additional development tools.

If this is being proposed as a collaborative effort, I think the scripts (or modules) should be published under a recognized free-software license, preferably GPL. But of course this is just my opinion; as you know my policy is always to respect and not to interfere with the decisions of developers, who are free to release under the terms and licenses of their choice.

Of course, you know I'm always ready to help in anything you may need with PJSR and PCL programming.

[vicent_peris]

Hi Niall,

for the moment, two comments:


It's imperative to make dark combination through a pixel rejection algorithm. Darks are long, and will have the same cosmic rays as light images. Trust me, there are CCDs with big pixels that can capture *a lot of* cosmic rays in 20 minuts. Simple median combine will not an ideal solution.

You don't need to normalize the flats. I make the division multiplying light by the average signal of the flat: light*Avg(flat)/flat.

This is the entire expression I make to calibrate my images:

((light-bias-dark)*Avg(flat))/flat

Prior to applying this expression, you must substract the bias image to the light, dark and flat images. You can also make it in one step:

((light-dark)*(Avg(flat)-Avg(bias)))/(flat-bias)

In case of not having daks, you must substitute the dark by the bias frame.

This equation gives you the same flux intensity than prior to calibration. For visualization through STF of linear images, can be useful to add the pedestal of the bias frame after calibration. This would be the equation:

(((light-dark)*(Avg(flat)-Avg(bias)))/(flat-bias))+Med(bias)


Hope this helps you!
Vicent.

[Niall Saunders]

Thanks Vincent,

These are the sorts of comment that I am going to need prior to starting the code.

What folks have to bear in mind is that I image in mediocre conditions with mediocre equipment (LX90 / ED80 and DSI-II imagers), so I don't come across the problems that folks see when they are imaging with 'top-end' equipment, under skies that easily tolerate a 20-minute exposure.

At the moment, all I have done is 'glanced' over your formulae - I will need to write it all down, step by step, so that my  brain can assimilate the steps. What I also want to do is to see if 'my' simple median combine actually 'fits in' with your stages, but as a 'simplistic' method.

Obviously, what I would hope to be able to do is to allow the user to decide what level of complexity they wish to emply in their calibration process.

So, your feedback si a good start. Are there other users out there who can contribute (or condemn   )

Cheers,

[georg.viehoever]

Niall,

the documentation for Deepskystacker http://deepskystacker.free.fr/english/index.html (User Guide. FAQ, Technical Details, How to create better images) gives a good overview of what is involved in calibration. As far as I can see, the DSS method is fairly complete and covers quite a lot of use cases.

Since DSS is such a good product, and free: What would we gain from doing it in PixInsight? Are the methods available in PI more powerful/more exact than the DSS implemetations. Would it be faster?

So before we start such a fairly complex project: What is the benefit?

Georg

[peregil]

Hi Georg,
For exemple, DSS only runs under windows. If you're a linux user you can't use DSS.

Pere

[georg.viehoever]

Linux, of course, is a good reason. Still not convinced - use Wine to run DSS on Linux (according to some messages on the internet that actually works, I did not test myself).

Keep the reasons coming    ,
Georg

[peregil]

Keep the reasons coming    ,
Georg

I'll think about it,   

Pere.

[Niall Saunders]

Hi Georg,

I hear what you are saying - about DSS already being available.

Here are my immediate thoughts - in no particular order.

a.) If DSS is capable of doing everything that we need, then why are some folk willing to pay 'silly money' (IMHO) for the likes of CCDStack ? Now, I have nothing aginst CCDStack, per se, (other than it being yet another software package that I would have to buy).

b) Which leads me to another reason - achieving the 'power' (and flexibility) of other programs within the realms of PI, i.e. the ability to achieve all of your post processing within 'one' software package

c) The next reason is that, simply by utilising the growing power of PI, we (the users) get to understand the processes better. We already see (and a few of us can even understand) how some of the new advanced processing tasks Juan has implemented for us can make our post-processing easier, or at least more effective. If we bring our combined knowledge together to tackle the steps required for 'pre-processing' then, surely, those stages become easier as well? (After all, just with one reply from Vincent, I have already 'learned something new' - something that would not have happened had I not mentioned my thoughts here on the forum)

d) Perhaps the least obvious reason - I (and others, like yourself ??) might actually 'enjoy' creating 'something useful' - a tool that will help us work with our images.

e) And this last reason may have no real merit - but, when Jack posted his recent image 'entirely processed in PI v1.5 - OTHER than for the calibration process', I just felt that a far greater achievement would have been to have created that stunning picture ENTIRELY with PixInsight.

Juan certainly made the 'best' case for creating this PJSR - and that is the 'power' required to make the PJSR already exists, as of PI v1.5.x. The PJSR code will simply act as a 'front end', to allow a user to collate their images into groups, and to define how the likes of Image Integration processes will be applied to each image data type.

In some ways I don't really see any 'complexity' in the script - the 'cleverness' will be in the 'core routines', and the 'skill' will be up to the user to select 'how' to implement the 'core routines' for their various image data sets. In which case we are NOT really improving on the likes of DSS - however, we WILL have the benefit of having 'direct access' (via Juan) to the 'power' of the core routines, and we will have direct access to the skills of 'you guys', who already know what they 'need' an imaging process to provide.

Surely a wheel is worth reinventing when the end result is a better wheel, or a wheel that you have better control of ? 

Cheers,

[David Serrano]

I already have a script that creates appropriate 'Tree Lists' of selected files, and I am working on adding 'Tabs' to the dialogue window, so I see my 'core' code as being 'mostly written'.

My thoughts are to provide a Tab for each image data type, i.e. one for Lights, one for Darks, etc (in a similar fashion to DSS, I suppose). For each image type, having specified the files that you wish to 'collate', you would then have the option of specifying 'how' you want the (background, i.e. 'core') Image Integration process to be applied.

Yeah, my thoughts were in the same direction too. An additional idea I had is give the user the possibility to use an already defined instance of PixelMath for combining images, so they can use advanced integration formulae should they wish (this is not yet possible but I would bet that the required PJSR classes appear in some 1.5.x release). For example, a set of radio buttons along the lines of "[ ] Use median combine [X] Use His Majesty vperis' formula [ ] Use my own PM instance". This could apply to the StarAlignment step too.

[georg.viehoever]

Hi Niall,

please dont get me wrong: I don't want to discourage you or anyone else from writing tools for PI. In fact, I am quite excited about the level of activity I see in this area recently. I asked my question because I clearly want to see the benefits of such a tool, because I want direct my energies into the most promissing areas.

I'll add David's idea of "user defined calibration formulas" to the reasons why we should do it. 

Georg

[Niall Saunders]

Hi Georg,

No, I am just as happy to hear any 'negative' comments because, after all, you are correct to question whether the effort is worthwhile.

That said, I am hoping that the 'framework' will help folk use the 'ImgInt' and 'StelAlign' tools more productively.

Over the next week, as time allows, I will adapt my deBayer script to let folks see how 'I' am thinking - and that is when I will be looking for criticism (constructive, or otherwise !! )

Cheers,

[mmirot]

Niall ( et al),

I am not a programmer but have used a lot of software. I am certain there are improvements in calibration that can be gained in PI over existing packages.
The excellent data rejection algorithms in PI already makes a cal module a good idea.

Here is my take of the competition so to speak.
Mostly food for thought/ perspective. I am talking about calibration and preprocessing features in general. I am not trying so make specific suggestions on what you do next.

MaxIM DL


1) MaxIm does fairly good job of calibration. However, the process has few options and is a total black box.  I like the rejection maps that PI and CCDstack produce. It tells me a lot.

MaxIM's Set calibrate:
The original idea was that you just dump all the calibrations frames into one directory and it will automatically sort out darks, flats, bin, exposure time, temp etc from the fits header. Then it makes masters and your set for calibrating images. It will even match the cal images to proper bin and closest temp and time to your working images.

The main problem is many us of don't have all our calibration images into one directory to start.

I don't feel like doing 30 mins of drag and drop into one folder just so MaxIm DL can do sorting  automatically.

The feature is a great idea but needs to be taken to next level.
That is a full relational data base function that searches a disk, or a folder,  a subfolder  for darks, bias, flats and then presents them to you logical manner for selection. 
Obviously it needs search the fits header for temp, times, filter color, bin, flat, dark etc.  It should should sort out the acquisition dates and even position angles for completeness.
If you ever acquired 4 color flats on multiple nights you would find this a major help.   Imagine shooting three objects over several weeks each a LRGB and each having separate rotation angle.  Figuring out where each flat set is maddening process.

It would a be a kick-ass, great feature if some developed calibration data base.
 I am sure large observatories build this into their processing pipelines.

(I guess this is related to calibration but is really a data/file management issue that could be addressed latter)



I generally end setting up MaxIM's calibration utility manually selecting files for bias, dark etc.  This is similar to DSS.

2) DSS. I have used it few times. I did not get good results. However, I am very unfamiliar with the settings so this means nothing.
 I can see it has some interesting features: Color/ bayer cameras interpolations, drizzle, and generation of star mask.
The UI set up for calibration is quite logical and straightforward.

3) CCDstack. It does a nice job. Work flow is kind of slow and limited. You generate each cal master frame one at time.
Then set up for calibration of one color filter  at a time.
Images have to be open taking gobs of memory. I could not use this program effectively on 32 bit machine.
Vista 64 and 4-6 gigs of ram,  it runs great. 

( Aside-- I think what most people like about this program is it's normalization and data rejection routines. You get to see exactly was is been rejected from each image prior to integration. It give a red overlay on the displayed image in the integration stack showing rejected data. If it is not selecting bad data,  cosmic ray hits ,etc, you clear the settings, and redo them with new settings.  This does give a nice control in the rejection step.  Also, stack is able to weight images of unequal exposure length. MaxIM does not do this. I am not sure about DSS.  It appears PI  now does a lot of weighting strategies   

May first impression was PI integration with sigma generated the best result I have seen to date. This is without any fuss using the default settings.  It gives a summary high/ low rejection maps that are very informative. 
However, my images were under nearly a perfect sky so perhaps normalization and data rejection have not been full tested

BTW , CCDstack this has a poor registration function. You have to pay an extra 89.00 US to get a an pluggin that actually on works)


Final thought, start simple.  Add more features as time permits.

Max
 





 


 

[Cheyenne]

Linux, of course, is a good reason. Still not convinced - use Wine to run DSS on Linux (according to some messages on the internet that actually works, I did not test myself).

Keep the reasons coming    ,
Georg

I have yet to be able to run DSS under Wine successfully in Linux.  Currently Wine is missing a required facility  that DeepSky Stacker requires (specifically GDI+ see http://wiki.winehq.org/GdiPlus ).  Trying to run DeepSky Stacker under wine gets the following error:

fixme:gdiplus:GdipCreateBitmapFromHBITMAP can only get image data from DIB sections
fixme:gdiplus:GdipCreateBitmapFromHBITMAP can only get image data from DIB sections
fixme:gdiplus:GdipCloneBitmapAreaI (0,0,48,48,2498570,0x17a730,0x32566c): stub

So so far PixInsight has easiest usable image intergration I've found in the Linux environment.