PixInsight Forum

PixInsight => Tutorials and Processing Examples => Topic started by: vicent_peris on 2010 April 07 23:40:13

Title: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 07 23:40:13
Hello,

just released a new tutorial about creating master frames for calibration. It explains several things I've learnt during the last 4 years about calibration. In fact, ImageCalibration does the image calibration exactly as I do it. In this article, I also explain how I obtain the master calibration frames.

I encourage to read it because:

- It will show you how to get the best of the new ImageCalibration tool. Good calibration is impossible without good masters.
- It will show you some practical uses of ImageCalibration and ImageIntegration tools. You will understand better some of the parameters in these tools.
- I think you will learn some new aspects about image calibration.

I plan to release in the near future an article about superflatting. For the moment, hope you will enjoy this one:

http://pixinsight.com/tutorials/master-frames/en.html (http://pixinsight.com/tutorials/master-frames/en.html)


Regards,
Vicent.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mmirot on 2010 April 08 06:34:03
Nice Article Vincent.

Can you explain what clipping the upper range at .98 does?

Should it allway be this value?

Max
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 08 07:42:01
Yes - good work Vincent.

My question (not yet having had a chance to 'experiment' with IC and the new II) concerns the difference between the 'Sigma' and the 'Range' sliders in the new ImageIntegration module.

Is there a simple answer, or should I get home and read the ToolTips, and then start 'tweaking' to see if I can figure out what they do :cheesy:

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 08 08:11:47
Nice Article Vincent.

Can you explain what clipping the upper range at .98 does?

Should it allway be this value?

Max

Yes - good work Vincent.

My question (not yet having had a chance to 'experiment' with IC and the new II) concerns the difference between the 'Sigma' and the 'Range' sliders in the new ImageIntegration module.

Is there a simple answer, or should I get home and read the ToolTips, and then start 'tweaking' to see if I can figure out what they do :cheesy:

Cheers,

Hi!

The range values will simply ignore saturated and zero pixel values, as they can't represent any physical data.


Vicent.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mmirot on 2010 April 08 09:03:33
That's what I thought. I imagine 0.97-0.99 should be optimal on the high end.

Do you save the outputs as 16 bit unsigned interger? 

Max
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 08 10:22:26
Do you save the outputs as 16 bit unsigned interger?

No, 32 bit fp, specially if you take a bunch of bias and darks. HDDs are cheap.  ;)


Vicent.


Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 08 10:37:15
Hi

I can not get your mouseover images to work !

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 08 10:40:23
Hi

I can not get your mouseover images to work !

Harry

Wait several seconds until the image changes.


V
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 08 10:48:46
Hi

I have tried that and still no joy the old tutorials work with mouse over  ???

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 08 10:51:30
Hi Again

The side bar options do not work either  ???  says error on page


Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 08 10:52:07
Hi

I have tried that and still no joy the old tutorials work with mouse over  ???

Harry

See the code of the webpage. The file names of the images are in the code, so you can download them and see in your computer. It's an alternative solution...


V.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 08 10:52:31
Hi Again

The side bar options do not work either  ???  says error on page


Harry

What browser are you using?

V.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 08 10:56:25
Hi

I am at work so on IE 8 I know go get firefox  :o  will try when I get home


Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 08 23:52:53
Harry,

Try enabling the 'compatibility' mode of IE8 (i.e. the mode that makes IE8 'compatible' with every other browser on the planet).

It is useful when, like me, you are tied to some crazy IT department's notion of 'reliable software' ::)

If possible, like me, change to Firefox, and never look back ;D

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: bitli on 2010 April 09 02:42:35
I do not want to enter the browser war, but I have the same problem on Chrome.

Looking at the development console I have tons of:

Uncaught TypeError: Cannot set property 'src' of undefined

at en.html:164, 166, 168, ...

I seems not recognizing document['MO04'] in the mouseover event (or other images), although they are defined with the proper id.

--
bitli



Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 09 04:49:15
That's what I thought. I imagine 0.97-0.99 should be optimal on the high end.

Do you save the outputs as 16 bit unsigned interger? 

Max

Hi Max - no, the most important thing is to 'acquire' your raw subs in 16-bit FITS integer (unsigned) mode. I haven't had a chance to play with the new ImgCal module yet, but I would very much doubt if it will work AT ALL if you try and feed it 32-bit Float images (for all the reasons that I have tried to explain elsewhere).

Once you have started to create the various intermediate (master) calibration frames, all of these can be saved in 32-bit Float mode - as PI 'understands' this environment, and will have internally adjusted the 'raw' data to 'fit' a [0.0, 1.0] floating-point range. So, you can work in 32-bit Float mode thereafter, and enjoy the fact that you won't have to worry about 'round-off' errors in your data.

Obviously, once you are ready to save in JPEG (for the web, etc.) you can export your final image to this 8-bit Unsigned Integer environment - but you will have saved a 'master' image in 32-bit Float anyway (wont you ???)

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mmirot on 2010 April 09 07:55:00
I think a problem might be if you created the Masters as 32 bit then tries to apply them in the calibration module to acquired 16 bit unsigned images. 

I have not checked this yet.

Perahps PI just handle the 32 bit to 16 bit scaling internally ?

Max
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 09 08:41:25
I think a problem might be if you created the Masters as 32 bit then tries to apply them in the calibration module to acquired 16 bit unsigned images. 

I have not checked this yet.

Perahps PI just handle the 32 bit to 16 bit scaling internally ?

Max


No problem with this if you generated your 32 bit files in PI. For PI, black is 0 and white 1 in 32 bit fp, or 0 and 65535 in 16 bit int. I always save my masters in 32 bit fp.

V.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mmirot on 2010 April 09 09:34:37
Ok,

That's what I hoped for. I have been saving them as 16 bits just in case.

Max
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 10 02:23:08
That's what I hoped for. I have been saving them as 16 bits just in case.

What it boils down to is that, for as long as we are all using "16-bit" cameras, the ONLY time that we need to worry about saving in 16-bit Unsigned Integer mode is when we are actually 'capturing' our raw data. Thereafter, once PI starts working with that 16-bit data, it is quite acceptable to save in 32-bit Float mode. OK, there is a very slight penalty in terms of increased HDD storage space, but the benefits are worthwhile.

And, remember, the reason PixInsight 'insists' on 16-bit UI 'raw' data is that this is the only data format that does not have to be 'translated' in order to be properly used by an image processing program. And, by that, I mean that a program would need to know a whole load of 'extra unique information' about a FITS image in order to be able to correctly make use of it. Many other programs do just exactly this - but that requires constant code revision  in order to be able to keep up with new hardware (which is what happens).

Stick to 16-bit UI at time of capture, or use a very simple ImageContainer/ScreenFormatConversion process to quickly change all your 32-bit F data into 16-bit UI and all will be well (that's how I work now, and it really takes just a few minutes after each capture session - after which I delete the 32-bit F 'raw' images forever).

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Jack Harvey on 2010 April 10 06:08:31
Want to cliffy my interpretation of this tool tutorial.  Once you are ready to calibrate a stack of light frames and have you Bias, Dark and appropriate Flat Masters plugged into the slots on the ImageCalibration tool you check the Calibrate boxes under Bias and Darks, but NOT flats??????
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 10 06:39:53
As I understand things Jack (and I am just about to sit down to my 'first run through'),

You start off with raw Lights, Darks, Flats and Bias frames

The first two steps are both ImageIntegration (II) steps, and these allow you to create a MasterDark and a MasterBias frames from your raw subs - you cannot, at this stage, create the MasterFlat because you will need BOTH the MasterDark and the MasterBias frame to pre-calibrate your Flats.

The next stage is to then use the ImageCalibration (IC) process, along with your new MasterDark and MasterBias frames, to calibrate the INDIVIDUAL Flat frames

Then you need to use II again, this time to integrate (stack) your newly calibrated Flat frames into a MasterFlat.

Only now do you have the three (master) calibration frames, which can now be applied to your Lights - again using the new IC process. as far as the tutorial then goes, you do NOT tick the 'calibrate' check box in the 'Master Flat' sub-section. The reason being that you have already just 'calibrated' the Flats.

The bit that I don't yet understand is why - at the time we are about to 'calibrate' the Lights - do we still then have the 'calibrate' box ticked for the 'Master Bias' and the 'Master Dark' sections ???

I need to go and 'have a play' - and see if I can work out the logic.

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Jack Harvey on 2010 April 10 07:47:03
You see precisely my point.  Why have the Bias and Dark boxes checked and not the Flats?
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 10 13:56:35
Hi all,

There is definetly some confusion there. Judging by fig.2 of the tutorial where 'Calibrate' is ticked for both MasterDark and MasteBias when calibrating Flat frames I also kept them ticked for the light frames (I created MasterDark, MasterBias, and MasterFlat frames as described in the tutorial, they all look nice). However the calibrated light frames are unusable (most details are gone only a few stars shown and even these can not be detected by StarAlignment which will not run). I have tried the same set (darks, flats, bias, lights) in ImagePlus and they work OK. Apparently I am doing something wrong due to the OSC nature of my files (Starlight Xpress SXVR-M25C camera). Would I need to debayer OSC files (darks, flats, bias, lights) before using them in the IC tool?

Thanks for any suggestions,

Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Nocturnal on 2010 April 10 14:01:01
Tasos,

you want to debayer before registering and stacking. So calibration (offset subtraction and flattening) should be done on raw frames. To be honest if you have a windows box you're better of using DeepSkyStacker. If you want to stack in PI you could use DSS to create calibrated subs for you.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 10 14:17:29
Thanks Sander, that was quick! So far I have been calibrating and debayering in ImagePlus (which calibrates raw frames and then debayers as you say), then registering and stacking and experimenting ... in PI. I thought IC could handle OSC files.

Thanks again,
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Jack Harvey on 2010 April 10 14:29:33
WOrking on some data I had I ran the light frame calibration with all three checkboxes checked (BIAS,DARK and FlAT) and then with FLAT unchecked.  To my eye the FLAT unchecked as stated in the tutorial is the way to go.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Nocturnal on 2010 April 10 15:43:46
Tasos, PI has a debayer module (I wrote it :) ) but you need to apply it at the right moment.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 11 09:42:13
Sander, I did apply debayer to the calibrated frames (not before), but as I said the calibrated frames were problematic before debayering(most of the info was gone). Also applying debayer to them reduced their size! I tried debayer on the uncalibrated frames and it worked OK (thus the calibration tool affects the OSC channels somehow).

Thanks,
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Nocturnal on 2010 April 11 09:46:24
Tasos,

sounds like your calibration isn't working properly. Most likely you're subtracting too much data. Verify that you're subtracting bias only 1 time. In other words, if you subtract darks, don't also subtract bias from your lights.

You must have used superpixel debayer. Try bilinear if you want to keep the same resolution.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Hans Pleijsier on 2010 April 11 09:59:10
Vicent,

Thank you for the calibration tutorial.
Unfortunately, sofar I was unable to integrate my canon cr2 raw darks into a masterdark because it is impossible to select cr2-files in the image integration process window. The monster seems to eat fits files only.

Any guidance on this?

Hans Pleijsier
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: NKV on 2010 April 11 10:04:15
Unfortunately, sofar I was unable to integrate my canon cr2 raw darks into a masterdark because it is impossible to select cr2-files in the image integration process window. The monster seems to eat fits files only.

Any guidance on this?
BatchFormatConversion  ;)
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 11 11:09:09
Sander, I was indeed subtracting bias twice since MasterDark is not bias subtracted. So I tried various combinations and the bottom line is that when I calibrate ONLY with darks then no info is lost. If I use MasterFlat and MasterDark, or MasterBias and MasterFlat or any of these two alone, then info is gone. There is something definetly wrong with the MasterBias (I will check the process again, maybe I did something stupid). Debayering is OK (I switched to bilinear).

Thanks,
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Hans Pleijsier on 2010 April 11 11:28:18
NKV,
batchformatconversion ....
thanks,  this might be a nice workaround but it's an extra step altogether.
HP
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 11 11:55:11
Quote
Unfortunately, sofar I was unable to integrate my canon cr2 raw darks into a masterdark because it is impossible to select cr2-files in the image integration process window.

Yes, this is a limitation of the current ImageIntegration process. ImageIntegration has been designed to work with incremental file reading operations, which are not possible with DSLR raw images.

This is the right procedure to work with DSLR raw images:

1. Open the Format Explorer window. Double click on the DSLR_RAW item (left column). On the RAW Format Preferences dialog, select the following set of options:

- Disable both white balance options.
- Enable the Create raw Bayer image option.
- Enable the No black point correction option.
- Click the OK button.

2. Use the BatchFormat conversion script to convert all your CR2 files into FITS files. Note that this is strictly necessary just for your calibration frames (dark, bias, flats), since ImageCalibration can load your CR2 files directly. However, I personally prefer to convert all raw DSLR images into FITS.

Now you can use the full set of PixInsight tools for calibration and integration. Note that step 1 is a one-time action, as the settings will be remembered across core application executions.

Quote
batchformatconversion ....
thanks,  this might be a nice workaround but it's an extra step altogether.

Indeed, but PixInsight's ImageCalibration and ImageIntegration tools are superior to other implementations. The BatchFormatConversion step is a small price to pay for ImageIntegration's performance.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 11 11:59:58
Quote
To be honest if you have a windows box you're better of using DeepSkyStacker. If you want to stack in PI you could use DSS to create calibrated subs for you.

With the due respect to your opinion, I disagree. DSS is an excellent application, and I can only recommend using it. In my opinion, however, PixInsight's ImageCalibration, StarAlignment and ImageIntegration tools are superior on all platforms, including Windows.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 11 12:09:36
Hi Juan

I will agree that alignment and integration are beyond excellent , but I am going to have to put calibration in the " work in progress bracket "  :'(

 Please listen to peoples concern on this matter as what should be a relatively simple application seems to be confusing  ??? well to me anyway

I wait for you to show me the error of my ways  ;D

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Hans Pleijsier on 2010 April 11 12:35:30
Juan,
thank you and ... I will be most loyal to PI  ;)
HP
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 11 12:40:38
Hi Harry,

Quote
I am going to have to put calibration in the " work in progress bracket "

Of course, I completely agree: ImageCalibration is a work in progress. Please see also my answer to other related post (http://pixinsight.com/forum/index.php?topic=1828.msg10964#msg10964).

But, man, don't cry for that :) The whole PixInsight is a work in progress, always and (I hope) forever! Do you think ImageIntegration and StarAlignment are finished tools? Not at all!  ;D
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 11 12:50:19
Hi Juan

Yes , but some things are more finished than others  :sealed:

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mmirot on 2010 April 11 13:24:33
Well Harry you a me will just have to bang on the drum until our programs friends in this community write a few scripts  :

I am sure it available in PCL and Java Script.

I would be on it myself but my programming skill are non existent. 

Max


Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 11 13:41:59
Hi Max

I also have no programing skills to offer but can manage a beat or two on the bongo's  :D   so a drumming we will go  >:D


Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 11 13:43:25
Quote
StarAlignment and ImageIntegration tools are superior on all platforms, including Windows.

I quite agree with that, however I pinpointed my problem to the very peculiar fact that my raw 16 bit bias frames (0.001 exp. time), are interpreted by PI as having values in the range of 0.5 (32000 in the 16 bit scale), thus they eliminate all info from my light frames (except bright stars). Light frames and darks (60 sec exposures), are shown OK. I opened the bias frames in ImagePlus and Astroart and their pixel values are correct there. So PI is playing some tricks here that have nothing to do with the IC or Integration tool. If this is solved then calibration, debayering, alignment and stacking can all be performed by PI for OSC data.

Any suggestions?

Thanks
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 11 13:50:05
Hi Tasos

Just to make you feel better I also have this exact problem when I try and import stuff from AA4 I have not done the full monty in PI so do not know if this will sort it or not  ???
I have tried rescaling everthing and this does not work, I have told mr juan about this and await his comments

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 11 14:19:18
Hi Tasos and Harry,

At this point I'd need to take a look at your images before trying to say something useful.

Could you upload one of your "problematic" images somewhere, so I can download it? Thanks!
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 11 14:21:34
Hi Juan

Can I upload to your new ftp or not I will put them on mine

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 11 14:38:19
Hi Harry, Juan,

The peculiar thing is that the light, flat, and dark frames are OK (all taken with AA4). It is the bias frames that PI misinterprets their value. I will try to provide an ftp link tomorrow.

BR,
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 11 14:43:23
Quote
Can I upload to your new ftp or not I will put them on mine

For security reasons we have disabled anonymous ftp on our new server. While I figure out how to overcome this limitation, better upload your images to your website.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 11 14:45:35
Quote
The peculiar thing is that the light, flat, and dark frames are OK (all taken with AA4). It is the bias frames that PI misinterprets their value.

Hmm, this is really weird. Are you sure these are 16-bit FITS files? Anyway, as soon as you upload the image I'll know what's happening.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 11 17:07:12
Hi Hans,

Agreed, BatchConversion is an 'extra step', but it is an essential step as PI was written to work in the 'FITS' environment - and your Canon images are very definitely NOT 'astronomical' images as such.

Fortunately it is a step that you only need to execute once, right at the start of your post-processing cycle. I have to do something very similar myself, becuase Meade created the Envisage program with a 'bug' that has remained uncorrected for several years now (and which will now be likely to remain uncorrected forever), which requires me to batch-convert all of my 'raw data' from 32-bit Float to 16-bit Unsigned Integer after every image capture session.

Fortunatley these 'batch conversions' are quite 'robotic' after a while - they don't really take up enough effort or time to be worth complaining about in the first place ::)

HTH

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Hans Pleijsier on 2010 April 12 09:36:21
Niall,

you are quite right ... and yes, I should buy a camera more fit for the job ... donations are welcome ...  8)

Hans
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 12 10:14:15
Hans,

I am even further down the 'camera ladder' than you are - I am still imaging with only 775 x 577 pixels, with no TEC cooler (at least, not one that is reliable to better than 5degC), and with everything else in my arsenal (other than my LX90 and my Moonfish ED80) being 'home brew'  ::))

So, no, I do not believe that you need a new camera - you (like me) may just have to accept that there have to be some 'extra stages' in the process of trying to obtain decent astronomy images :'(

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 12 14:10:52
Quote
The peculiar thing is that the light, flat, and dark frames are OK (all taken with AA4). It is the bias frames that PI misinterprets their value. I will try to provide an ftp link tomorrow.

Hi Tasos,

Thank you for uploading the image. The file you have uploaded is a 16-bit signed FITS image. However, its minimum sample value is 32768. This is indeed 'peculiar' for a bias frame. It is quite evident that the people who wrote the software that has created this image forgot to include the necessary BZERO and BSCALE header keywords in the main HDU of this FITS file. Another example of PixInsight being blamed for the mistakes made by other poorly coded applications :)

Fortunately, converting these images to the correct [0,65535] 16-bit unsigned range is very easy in PixInsight:

1. Define a PixelMath instance with the following parameters:

RGB/K expression: $T - 0.5
Rescale result disabled

Drag the blue triangle to create a PixelMath icon on your workspace.

2. Make sure that your FITS module writes unsigned integer images by default. Open the Format Explorer window and double-click the FITS item. On the FITS Preferences dialog, be sure the Default integer signedness option is set to Write unsigned integers.

3. Define an ImageCointainer with all your incorrect bias frames. The output template should be something like:

&filename;_u.fit

to ensure that the output images will be written in the FITS format. The _u postfix is of course optional, but I would include it to better identify the corrected files (_u stands for 'unsigned' here).

4. Drag the blue triangle from ImageContainer to the icon you created in step 1. This will convert all your bad bias frames into valid unsigned 16-bit FITS files that you can use with ImageCalibration.

As a side note, I personally would complain very seriously to the authors of the software that creates these images. I'll better stop talking at this point...
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 12 14:16:37
Hi Juan

Thanks for looking at that and the answer :D I will have a word with FAbio at astroart , I personaly have always found him very helpful and has alltered some software before at my behest before

Regards Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 12 14:24:22
Quote
The peculiar thing is that the light, flat, and dark frames are OK (all taken with AA4). It is the bias frames that PI misinterprets their value. I will try to provide an ftp link tomorrow.

Hi Tasos,

Thank you for uploading the image. The file you have uploaded is a 16-bit signed FITS image. However, its minimum sample value is 32768. This is indeed 'peculiar' for a bias frame. It is quite evident that the people who wrote the software that has created this image forgot to include the necessary BZERO and BSCALE header keywords in the main HDU of this FITS file.

Hi,

this happens also in Iris exactly as you have described. Iris works in 15 bit, so original pixel values are automatically remaped from 32768 to 65535.

¿Perhaps Astroart works also in 15 bit precision?


Vicent.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Nocturnal on 2010 April 12 14:41:39
Could it be AA has an Iris Compatibility mode? Nebulosity does.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 12 15:05:47
Hi Juan,
Thanks for the thorough response.

AA4 automatically decides on wether to save images as 16 or 32 bits depending on pixel values (an annoying feature which in many instances has prevented me from transfering AA4 debayered frames to PI).

I examined the flats, darks and lights all coming from the same session (including the bias frames) and all being 16 bits (AA4 supports up to 96 bits), and for some peculiar reason (probably related to this automatic fits formatting) the bias frames do not have the BZERO keyword while all the others do!

I will follow your procedure since I would like to keep all my processing under PI, and apparently IC is fine for OSC CCDs. AA4 is excellent (and cheaper than MaximDL) for integrating all the motorised gear (Mount, CCD, AO, Filter Wheel, Focuser) but not so much to my taste for processing.

Thanks,
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Nocturnal on 2010 April 12 15:18:08
Would it be possible to manually (or with a script) add the required keywords to your FITS files? I would probably go that route.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 12 15:29:20
Quote
Would it be possible to manually (or with a script) add the required keywords to your FITS files? I would probably go that route.
Well, I did what Juan suggested and took no time really!

BR
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Cheyenne on 2010 April 12 16:51:10
So.. I've been following this thread.. and have a question.. dealing with DSLR (but probably applicable to other formats).

Any suggestions on the process for creating a library of darks to create a master dark frame?  And can I assume that bias frames in this case are not really applicable?
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: NKV on 2010 April 12 18:55:36
AA4 supports up to 96 bits
RGB 3*32bit=96bit  ;)
So PixInsight support 192bit  ;D
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 14 14:02:30
Hi Tasos,

I must make an important correction here. Your bias frame is correct and it is PixInsight that is not interpreting it correctly.

Your bias image has been stored as a signed 16-bit FITS file. Hence, it stores pixel values in the range [-32768, ..., 0, ..., +32767]. Although this is conceptually incorrect in my opinion, it is indeed correct as per the FITS standard, and indeed no BZERO and BSCALE keywords are necessary in this case (that is, they would be 0 and 1, respectively) because the file stores physical pixel values.

The conceptual error, in my opinion, is in the fact that a CCD does not produce negative values in the range [-32768,0[, so storing physical pixel values as signed integers does not make any sense. However, as I've said the FITS standard allows it, and that's the only fact that counts here.

PixInsight is making the wrong assumption that the file stores valid pixel values in the whole 16-bit signed range, while the file actually stores physical values, which can only be positive, and hence the negative half of the numeric range is unused. So the problem is all ours and the file is definitely valid. This is actually a design error that I'll try to fix as soon as possible.

My sincere apologies to the software and to the persons who have written it. I hope nobody is too offended.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 14 14:29:01
Hi Juan,

Just trying to 'keep on top' of this wonderful world of 'FITS Standardisation' !!

So, given a 'standard' FITS 16-bit Signed Integer file, how does PI interpret this? Do you currently set the PI 'internal' [0.0, 1.0]int range to equate to the 16-bit SI range of [-32768, 32767]ext - or is this where you now perceive an 'error'?

In which case, is your 'error' such that you 'expect' that a FITS 16-bit SI image would only actually provide PI with the 'positive half' of the range (because a CCD, itself, cannot produce a 'negative' electron count)? In which case, are you making the 15-bit translation such that [0, 32767]ext is mapped to [0.0, 1.0]int?

Or do you just consider the image to be 16-bit, and therefore impose a translation of [0, 65535]ext to [0.0, 1.0]int?

I hope you don't mind us asking you to 'air your dirty laundry in public' - but, if we at least know what went wrong, and how it will be corrected, then we can help you deal with these issues as they arise.

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 14 15:02:09
Hi Juan

Fabio is not offended  :P  , particularly when He is right  ;D

Let there be peace in imaging land

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 14 15:14:40
Hi Niall,

Quote
So, given a 'standard' FITS 16-bit Signed Integer file, how does PI interpret this? Do you currently set the PI 'internal' [0.0, 1.0]int range to equate to the 16-bit SI range of [-32768, 32767]ext

Yes. And here is the problem. A signed integer image can be interpreted in two ways (assuming a 16-bit signed image):

(1) The [-32768,+32767] range represents the whole dynamic range to which the image is referred. Then we simply remap that range to [0,1] assuming that -32768=black and +32767=white. This is what PixInsight does right now.

(2) The [0,+32767] range stores physical pixel values, and the negative range [-32768,0[ is not used. Some applications are doing this, and it is absolutely correct because the FITS standard allows it. Unfortunately, PixInsight does not support this interpretation, but it will do in the next version. The reason why we have not faced this problem before is because we didn't have calibration in PixInsight.

The solution is, as in the case of floating point data, to let the user define how signed integer images must be interpreted. What I am not sure is which option should be the default one. (2) seems more convenient because many applications are using signed integer data in that way. (1) would guarantee backward compatibility out-of-the-box for signed integer images written with previous versions of PixInsight, but I think that almost nobody has been writing signed FITS images with PI, so a possible compatibility problem would be of marginal importance, if any. Suggestions welcome!

Quote
I hope you don't mind us asking you to 'air your dirty laundry in public'

No dirty laundry in PixInsight --just good things, bad things and some mistakes; just as they are present anywhere. When I am wrong or when I make mistakes, I have absolutely no problem to recognize it. Sometimes I am too vehement and that causes (and has always caused) me problems. But since I'm older than 19 there's little hope I'll ever change ;D
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 14 15:30:11
Hi Juan,

Quote
The reason why we have not faced this problem before is because we didn't have calibration in PixInsight

Yes - I have to agree there! After all, you are only too familiar with the long struggles that I have been having with Meade's Envisage, struggles that I thought were over forever when I started using PI in earnest, but struggles that returned to trouble me when I started using ImageIntegration and PixMath to 'manually' calibrate my images in PI 1.5.9

I think that you could choose (2) as your 'default' method - and then just wait and see what the feedback is. Some folks will never 'see' the problem, some will 'adapt' and reconfigure PI to suit their needs, and others will need to be helped. Same old same old

And, as far as I am concerned, I think I stopped changing when I was about 14 - I have been an obstinate 'pain in the behind' ever since >:D

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: bitli on 2010 April 14 22:41:28
Hi,
I think (2) would be more convenient too, it is better to have least surprised when one transfer data from another software. In case of problem it is not clear where to look if two programs are involved.

If we have a problem when reading back PI data in PI, it is always possible to refer to the documentation :-).

This could help:
-- bitli
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 15 00:44:57
(2) The [0,+32767] range stores physical pixel values, and the negative range [-32768,0[ is not used. Some applications are doing this, and it is absolutely correct because the FITS standard allows it. Unfortunately, PixInsight does not support this interpretation, but it will do in the next version. The reason why we have not faced this problem before is because we didn't have calibration in PixInsight.

So Astroart does calculations in 15 bit, as does Iris??

Vicent.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: georg.viehoever on 2010 April 15 06:23:51
(1) The [-32768,+32767] range represents the whole dynamic range to which the image is referred. Then we simply remap that range to [0,1] assuming that -32768=black and +32767=white. This is what PixInsight does right now.

(2) The [0,+32767] range stores physical pixel values, and the negative range [-32768,0[ is not used. Some applications are doing this, and it is absolutely correct because the FITS standard allows it. Unfortunately, PixInsight does not support this interpretation, but it will do in the next version. The reason why we have not faced this problem before is because we didn't have calibration in PixInsight.
How about deciding on the interpretation by looking at the values in the FITS file. If there are values in the negative range, we have case (1), otherwise case (2).

Georg
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 15 10:33:33
Hi Georg,

I am not that 'comfortable' with your suggestion - because now we are talking about PI starting to have 'dedicated code' to allow it to interpret the myriad of 'non-conforming' data methods that people are using.

I still feel that the PI approach to date is the most 'robust' - and that even means that I have to 'deal' with my own incorrectly constructed FITS files (thanks to Meade). But at least the solution is relatively simple, and I have no problems implementing it, and I have taken the time to 'understand it' - all of which helps ME remain in control.

Yes, shielding a total novice from the problem is one possible solution - but they don't 'learn' from that approach. After all, if someone actually realises that there is a problem, and can perhaps eliminate that problem 'at source', then wouldn't life would be easier in the long-term? (Not in my particular case - Envisage does NOT have the flexibility to change, and Meade have steadfastly ignored the software bugs for YEARS now)

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 15 10:57:37
Quote
The conceptual error, in my opinion, is in the fact that a CCD does not produce negative values in the range [-32768,0[, so storing physical pixel values as signed integers does not make any sense.
Indeed, and AA4 reports values in the range of 0 to 891 for the bias frame. It is beyond me why it decided to save flats, darks, and lights as 16bit unsigned and bias as signed 16bit.

Quote
So Astroart does calculations in 15 bit, as does Iris??
No, it just has a peculiar way of saving data.

BR
Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: georg.viehoever on 2010 April 15 11:10:31
Niall,

...
I am not that 'comfortable' with your suggestion - because now we are talking about PI starting to have 'dedicated code' to allow it to interpret the myriad of 'non-conforming' data methods that people are using.
...

I think one aspect of user friendly software design means do as much as possible "behind the scenes". Frankly, I would not mind if PI had some fewer dials and knobs. Even as a long time PI user I am often unsure on what setting is usually best.

How about letting the user decide? PI could offer 3 options: Automatic (what I think should be the default), and your two manual options.

Georg
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Cheyenne on 2010 April 15 13:44:40
The "number of dials and switches" isn't  a bad thing.

There is a scientific / mathematical reason for each (or most).  For those who are in the "know" about such things, it would be reasonable (and desirable) to have that level of control.

What would make sense would be to have reasonable defaults and suggestions on which knobs one should tune, for those of us that don't really know (or care) what the connection phi, psi, and filter banks and Ingrid Daubechies could ever be.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: cfranks on 2010 April 15 16:49:19
All these problems will go away with PI 1.7.  That's the release with ImageAcquisition right??  >:D

Charles
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 15 23:47:09
I have been assured that release 2.1.4 (due out at the beginning of April next year) will allow you to download a really nice picture, straight off the internet O:)
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Yuriy Toropin on 2010 April 16 06:10:06
;) Back to business -

Juan or Vicent,
or anybody else involved in the magic,

can you please comment on what does this coefficient means
   Writing output file: X:/Photo/Processing/2010_04_15_M13/0 - Flat Calib/fl-d_Luminance_12_c.fit
   Dark scaling factors:
   k0 = -0.389

and can this information be used for obtaining better calibration frames, i.e. to reject some of individual Flat Field samples before integrating them in the Master Flat?

Value of this coefficient varies widely [-0.389,0.234] during calibration of the series from 32 individual flat field shots...

Thanks a lot,
    Yuriy
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 16 07:06:50
Hi Yuriy,

Quote
k0 = -0.389

This is the scaling factor by which the master dark frame has been multiplied before subtracting it from your target frame (a flat frame in your example).

The dark optimization routine has found that by multiplying the master dark by -0.389, the noise is minimized in this particular target frame after dark subtraction. Note that since the scaling factor is negative, the master dark frame is being added in this case, instead of subtracted. This denotes lack of correlation between thermal noise in the target frame and the master dark frame. We have forced negative dark scaling factors in artificial tests conducted to test ImageCalibration. Negative factors can be due to:

- Wrong calibration frames. For example, calibration frames and target frames acquired with different sensors, misplaced frames due to geometrical transformations prior to calibration, or lack of linearity.

- Presence of other significant sources of noise besides thermal noise. For example, if we add a significant amount of uniform random noise to the master dark frame (to simulate read noise), the optimization factor tends to zero (on average), because the random noise cannot be correlated with the target frames.

- Malfunction of the dark optimization routine. This is very unlikely because the routine implements extremely robust minimization and noise evaluation algorithms (robustness has been enforced even at the cost of significant performance loss). But we can never rule out this possibility completely.

Could you upload some of these frames and the master dark you are using? This looks like a very interesting test set.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 16 07:41:08
Thanks for that Juan - that may wll be useful information in the future.

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Yuriy Toropin on 2010 April 16 09:07:29
Hi Juan,
I've prepared 82Mb ZIP archive for upload.
Please, check you personal email at pixinisht com and give me instructions on where to put it.

BTW, would you prefer ZIP of RAR will be also Ok, it's smaller, ~71Mb.

Regards,
   Yuriy
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Yuriy Toropin on 2010 April 16 09:53:43
ZIP archive uploaded.
I used Master Bias and Master Dark (for 3 sec exposures) for calibration of Luminance flat field files, obtained at exposures ~0.2 sec,
everything is in archive together with the screenshot.
To be honest, Dark calibration could be, probably, skipped in this case, dark signal is low on KAF8300 chip at -25oC, but there are still some hot pixels visible.

Anyway, will be interesting to understand what the issue was with calibration of this set for the master flat field.

Also, probably, ImageCalibration should receive some setting to reject such outliers (with negative coefficients), on decision of user?

Thanks a lot,
   Yiriy
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Harry page on 2010 April 16 10:53:54
Hi Tasos

Quote from Fabio At AstroArt

Hello Harry,
all 16 bit FITS files are 16 bit signed. (-32k .. 32k)
Unsigned files do not exist.

It's only possible to REPRESENT unsigned numbers,
in the range 0..64k using the BZERO trick.

http://fits.gsfc.nasa.gov/standard30/htmlfiles/fits_standard30index.html (http://fits.gsfc.nasa.gov/standard30/htmlfiles/fits_standard30index.html)

See 5.2 and 5.3.

Hope it means something to you

Harry
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Alexander on 2010 April 16 12:20:42
Thankyou Vincent and Juan

Went through tutorial with my Lights Darks Bias Flat frames that i gathered over the last week, had same problem with bias subs, so then applied Juan,s RGB/K expression: $T - 0.5...and everything worked through ok, I only have a Meade Dsi colour camera but PI has a way to improve any image,with tutorials like these,so looking forward to more. :)

 Alexander
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 16 12:33:41
Hi Yuriy,

Thanks for uploading your calibration frames.

I have tested ImageCalibration with your files, and my results are similar to yours. With the default 1024x1024 optimization window I have obtained the following dark scaling factors:

-0.287,
+0.135,
-0.068,
+0.029.

This shows that there is no correlation between your master dark frame and the flat frames. The +0.135 factor, which is the only one that might be questionable, is IMO well within the uncertainty limits of function minimization.

With full frame optimization (optimization window = 0), the scaling factors are respectively:

+0.013,
+0.023,
-0.019,
+0.028,

showing that there is definitely no correlation at all between your flat frames and the master dark frame. We can conclude that you should not apply this master dark frame to your flats.

The dark optimization routines implemented in ImageCalibration work on the basis that the dark frame is correlated to the target frames with respect to thermal noise. Assuming negligible read noise and neglecting other sources of noise in the images, consider the following expressions:

F = S + NT
D = k*NT

where F is a target frame (being calibrated), S represents the signal in the target frame, NT is the thermal noise in the target frame, and D is the dark frame. Dark frame optimization tries to find the scaling factor k. It works by computing the dark-subtracted target frame F':

F' = F - 1/k * D

such that the standard deviation of the noise in F' is minimal as a function of k.

When there is no shared NT between F and D, the function minimization algorithm tends to generate a zero scaling factor on average, that is, as the average of a large set of computations on an uncorrelated data set. Computed scaling factors are not always exactly zero because there are other sources of noise in the images, which have been neglected in the expressions above. Note that this neglected noise is precisely what the algorithm evaluates in order to find k. This uncertainty can yield odd results eventually, such as negative scaling factors.

The next version of ImageCalibration will issue a warning when this happens. The advantage of our implementation is that you can always know what happens with your data. This is just what we want to achieve with PixInsight.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 16 12:48:24
Quote
The conceptual error, in my opinion, is in the fact that a CCD does not produce negative values in the range [-32768,0[, so storing physical pixel values as signed integers does not make any sense.
Indeed, and AA4 reports values in the range of 0 to 891 for the bias frame. It is beyond me why it decided to save flats, darks, and lights as 16bit unsigned and bias as signed 16bit.

Quote
So Astroart does calculations in 15 bit, as does Iris??
No, it just has a peculiar way of saving data.

BR
Tasos

No. In Iris, you can manipulate the image between 0 and 32768. :) It does 15 bit calculation... at least some time ago...

V.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: harist on 2010 April 16 12:59:54
Hi Harry,

When BZERO is inserted by AA4 then the files are OK in PI (flats, lights, darks). What I do not understand is why this is not the case with bias frames (no BZERO inserted there by AA4).

On another matter when I calibrated using FLATS (with very high values I have to admit), PI could not align the calibrated files. Following your video tutorial I applied the "shift" trick to destroy the color info and everything was OK. I suppose a future version of IC could have this option for an OSC MasterFlat frame (Discard color info).

Tasos
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Yuriy Toropin on 2010 April 17 02:25:01
...
showing that there is definitely no correlation at all between your flat frames and the master dark frame. We can conclude that you should not apply this master dark frame to your flats.
...
The next version of ImageCalibration will issue a warning when this happens. The advantage of our implementation is that you can always know what happens with your data. This is just what we want to achieve with PixInsight.
Thanks a lot for checking this and for clarifications,
just another example that "classical" approach to calibration will fail in some cases!

Warning will be helpfull, but even now log in Processing Console gives enough insights on what's going on.

Thanks a lot again, Juan, I (and, IMHO, the whole PI community) really appreciate all those pieces of really valuable info on different aspects of processing you're providing on the forum!
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mattssporre on 2010 April 17 13:59:53

The dark optimization routines implemented in ImageCalibration work on the basis that the dark frame is correlated to the target frames with respect to thermal noise. Assuming negligible read noise and neglecting other sources of noise in the images, consider the following expressions:

F = S + NT
D = k*NT

where F is a target frame (being calibrated), S represents the signal in the target frame, NT is the thermal noise in the target frame, and D is the dark frame. Dark frame optimization tries to find the scaling factor k. It works by computing the dark-subtracted target frame F':

F' = F - 1/k * D

such that the standard deviation of the noise in F' is minimal as a function of k.

When there is no shared NT between F and D, the function minimization algorithm tends to generate a zero scaling factor on average, that is, as the average of a large set of computations on an uncorrelated data set. Computed scaling factors are not always exactly zero because there are other sources of noise in the images, which have been neglected in the expressions above. Note that this neglected noise is precisely what the algorithm evaluates in order to find k. This uncertainty can yield odd results eventually, such as negative scaling factors.

The next version of ImageCalibration will issue a warning when this happens. The advantage of our implementation is that you can always know what happens with your data. This is just what we want to achieve with PixInsight.


Juan,

I am a little confused here ???. My notation

D = Dark signal
Nd = Noise in the dark frame (or uncertaintaty in the obtained dark frame which we use to measure the dark signal).

Dark Frame = D + Nd

S = the light signal, the one that we try to extract.
Ns = uncertainty in the obtained light frame (also referred to as noise)

Light Frame = S + D + Ns

I should also include the offset signal (bias) and vignette factors but they are not important for what I want to ask.

If we have the exact time and temp for the Dark Frame we can subtract the Dark Frame from the Light Frame and get rid of the Dark signal, D, but the dark noise will not go away.

Light Frame – Dark Frame = S + Sqrt[Nd*Nd + Ns*Ns]

Now if the Dark Frame and the Light Frame does not match in time and/or temperature you have to scale the Dark Frame in some way. Assume the light frame is taken at temperature T0 and with exposure t0. Also assume that the corresponding time and temp for the dark frame is t1 and T1

Light Frame – Dark Frame = S(t0,T0) + D(t0,T0) – k*D(t1,T1) + Sqrt[Nd*Nd + Ns*Ns]

We need to find k, but what I hear you say is that you find this scaling by trying to minimize the noise in the resulting Light Frame – Dark Frame, i.e. by minimising Sqrt[Nd*Nd+Ns*Ns].

But how does this secure that we actually find a k such that D(t0,T0) – k*D(t1,T1) = 0?

BR
Matts
PS sorry for this long and technical premiere post at this forum, but I may have misunderstood something and then there is a chance to learn something  :)
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Carlos Milovic on 2010 April 17 16:39:17
Hi Matt.

If you take lots of dark frames, the final master dark will have a negligible amount of noise, so, so the sake of simplification, one may discard it. Once you have only noise from the light frame, at the end it is unidentifiable from the signal, so, again for simplicity, you may assume that everything is signal. Also, there is no correlation between dark current and noise or signal, so at the end it is the same thing.

In the real world, of course, things are not that pretty, but even so if you minimize the noise (or standard deviation of small samples, if you like), it is a better result than a theoretical linear escalation.
Minimizing noise guarantees to find a value for k if there is some correlation between light and dark frames (the dark current itself should be the correlation). This happens because the subtraction should follow a function such at +-infinity the noise goes to +infinity. Since it is a continuous function, there should be at least one local minimum that is a solution to the problem.

[Juan, feel free to overdrive anything I said :D]
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mattssporre on 2010 April 18 07:34:35
Carlos,

I understand that the noise (or rather uncertainty in the signal) gets smaller and smaller when adding more and more frames (dark frames in this case) :P.

The question I have is related to Juans statement that PI try to scale/optimize a master dark frame (to fit the Light Frame we would like to remove the Dark Signal from) by MINIMIZING the NOISE in the resulting calibrated Light Frame.

I just do not understand  ??? how minimizing the noise in the resulting calibrated Light Frame secures that we have removed the correct Dark Signal.

BR
Matts
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Juan Conejero on 2010 April 19 08:06:18
Hi Matts,

Quote
how minimizing the noise in the resulting calibrated Light Frame secures that we have removed the correct Dark Signal.

Because:

(i) Uncertainties in both the light frame and the master dark frame, as well as the signal in the light frame, remain constant during the whole dark scaling process. Since they remain constant, we can simply ignore all of them them during the whole process.

(ii) Dark current, as recorded in the master dark frame, consists exclusively of small-scale structures. By small-scale structures, we refer to pixel-to-pixel variations. We use a wavelet transform to isolate these variations in the first wavelet layer, as image structures at the 1-pixel scale.

(iii) A necessary precondition is that the uncertainty in the measurement of the dark current is negligible in the master dark frame.

(iv) Another necessary precondition is that the light frame and the master dark frame are correlated. This means that both frames share the same dark signal, although scaled by a factor k > 0. Our algorithm tries to find the value of k.

Note that our algorithm is purely numeric. It is not based on any physical properties of the sensor. In fact, our algorithm ignores temperatures and exposure times completely, because it does not need to know them.

Note also that we treat dark signal as both signal and noise, at different stages of the algorithm. When you integrate a large number of dark frames into a master dark frame, dark current is signal. By averaging n dark frames, you reduce the uncertainty in its measurement by Sqrt(n). However, when our dark scaling algorithm evaluates noise in the light frame after dark subtraction, the same dark current is noise. This is the difficult part because of the duality in the interpretation of the same part of the data. I'll try to explain this in more detail.

Following your notation, we have:

Master Dark Frame = DF = k*D + Nd
Light Frame = LF = S + D + Ns

where D is the dark signal, S is the image signal, Nd is the noise in the master dark frame and Ns is the noise in the light frame. Both Nd and Ns are supposed to be randomly valued and uniformly distributed. Our task is to find a good approximation to the true dark scaling factor k > 0.

Precondition (iii) says that the uncertainty in the measurement of D is negligible, that is:

|D| >> |Nd|

so we can assume:

Nd = 0

without changing anything important. This means that the quality of the master dark frame (in SNR terms) is very important, a fact that Vicent has stressed sufficiently in his tutorial. Note that this is also extremely important in our numerical procedure because, as Nd is uncertain by nature, we have no way to remove it, so each time we try out a scaling factor k we'll be multiplying also k*Nd. If Nd and D are comparable in magnitude, they may easily become undistinguished numerically, and our process will have no way to converge to a good (that is, certain) value of k.

Precondition (ii) tells us that in the morphological and statistical senses, D is very similar to Ns. Both features are composed of small-scale structures. Ns is random and can be assumed to follow a Gaussian distribution, as usually happens with large data sets having a strong central tendency. D also has a strong central tendency, since most dark current variations are quite similar. We know that D is not a random variable, but observed over a large portion of the image, its distribution is basically uniform. Uniform and random here, while not the same thing, can be treated in the same way because its properties are essentially the same for our strictly numerical purposes.

Now let's try out a value of k, call it ki:

LFi = LF - 1/ki * DF

We have said that we can neglect Nd, so we have:

LFi = S + D - k*D/ki + Ns

The above expression is evaluated iteratively to feed a minimization algorithm. At each iteration, the algorithm computes the standard deviation of the noise in LFi. When the algorithm brackets a minimum within a very small interval (< 0.005 in the current implementation), the current value of ki is returned as the true value of k.

As S and Ns remain unchanged during the whole process, what we are minimizing is actually the standard deviation of:

D - k*D/ki + Ns

Note that our multiscale noise evaluation algorithm is able to isolate S from the rest of terms, and it performs that task in a very robust way.

As I've said above, here we are treating D as if it were pure noise. As we approximate ki = k, the standard deviation of the noise in LFi reaches a minimum, which is approximately equal to the standard deviation of Ns. As Carlos has pointed out, the function to be minimized is strictly continuous and has a single minimum. An important fact that must also be pointed out, is that our algorithm makes no assumption as for the linearity of the dark signal as a function of temperature, exposure time, or any other physical acquisition condition. In fact, the dark signal could exhibit a wildly nonlinear behavior, and our algorithm would still find the optimum scaling factor k.

Hope this helps you and all users to understand how our algorithm works. It is not perfect, and we indeed have some ideas to improve it, but we think it works remarkably well.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: georg.viehoever on 2010 April 19 09:16:41
Juan,

as you know, Canon EOS cameras are sometimes plagued by the banding noise, as shown in http://pixinsight.com/forum/index.php?topic=1159.msg5638#msg5638 . This also affects the dark, flat and bias frames (primarily the darks). I think it is best described as some kind of read noise that is loosly correlated between different shots (but not exactly).

Since PI calibration mainly depends on analyzing the thermal noise: Is it necessary to remove the Canon read noise, e.g. by using the CanonBandingReduction script before calibrating?

Georg
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: vicent_peris on 2010 April 19 09:30:44
Juan,

as you know, Canon EOS cameras are sometimes plagued by the banding noise, as shown in http://pixinsight.com/forum/index.php?topic=1159.msg5638#msg5638 . This also affects the dark, flat and bias frames (primarily the darks). I think it is best described as some kind of read noise that is loosly correlated between different shots (but not exactly).

Since PI calibration mainly depends on analyzing the thermal noise: Is it necessary to remove the Canon read noise, e.g. by using the CanonBandingReduction script before calibrating?

Georg



I think it would be helpful to remove banding to each calibration frame. Once you have masters without banding, you can subtract thermal noise to light frames.


V.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Niall Saunders on 2010 April 20 05:21:25
Folks,

I hope nobody minds, but I have started a new topic, based on Juan's detailed reply, to allow the 'intricacies' of the new ImageCalibration process to be discussed away from the 'generalities' (or the 'how do I use the new IC process' type of questiopns).

Cheers,
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: mattssporre on 2010 April 20 08:15:24
Juan,

thanks for the explanation. I might need to contemplate over some of the statistical issues here but one statement in your made it click into place (at least in my head  ;))


As S and Ns remain unchanged during the whole process, what we are minimizing is actually the standard deviation of:

D - k*D/ki + Ns


I understand that your procedure do just this and that it would approach an approximation of D (given that the problem is stable etc). Your are not realy finding the optimized dark by minimising Sqrt(Nd*Nd + Ns*Ns) as I missunderstood you. It was the word "noise" that confused me since I do not see the Dark Signal as noise, just as unwanted signal.

BR
Matts
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: NKV on 2010 May 01 00:43:21
Juan, what about "Pedestal" during calibration? See pic.

My camera has a thermostabilisation CCD, but it has a temperature dependence of Bias. Average Bias increases with temperature. For example freeze CCD on-40oC ambient temperature of 20 ° C average 1924ADU, ambient 5 ° C-average 2000ADU. The mean deviation stable 5.5ADU
Therefore, if the ambient temperature drops during the session, the calibration process may fall below zero. To avoid zero clipping I deducted from the master of Bias 100-200AUD, with dark calibrated separately.
But I think it will be good if will be able to specify the value of the pedestal in future versions.
Title: Re: New tutorial: Master Calibration Frames: Acquisition and Processing
Post by: Mike Reid on 2010 May 12 07:52:57
Thanks Vicent for the tutorial!

I have a typical LRGB set of calibration frames where I'm shooting L at 1x1 binning and RGB at 2x2 binning.  I tried to follow Vicent's tutorial and these are the steps that I did to generate my 8 master calibration frames,


  01. integrate 1x1 bias frames                 => MasterBias_1x1.fit
  02. integrate 2x2 bias frames                 => MasterBias_2x2.fit
  03. bias calibrate 1x1 dark frames
  04. bias calibrate 2x2 dark frames
  05. integrate 1x1 calibrated dark frames      => MasterDark_1x1.fit
  06. integrate 2x2 calibrated dark frames      => MasterDark_2x2.fit
  07. bias and dark calibrate L flats frames
  08. bias and dark calibrate R flats frames
  09. bias and dark calibrate G flats frames
  10. bias and dark calibrate B flats frames
  11. integrate calibrated L flats frames       => MasterFlat_L.fit
  12. integrate calibrated R flats frames       => MasterFlat_R.fit
  13. integrate calibrated G flats frames       => MasterFlat_G.fit
  14. integrate calibrated B flats frames       => MasterFlat_B.fit


Does this seem correct?

Each of these steps required filling out a form and running the module.  It took quite a lot of time and there were several opportunities to select the wrong files. Since my settings for these steps (more if I acquire Ha also) rarely change this would seem to be an excellent candidate for a script.  Something more like ImagesPlus's Auto Imageset Processing would serve as an excellent model. 

Mike