PixInsight Forum

Software Development => New Scripts and Modules => Topic started by: Sean Houghton on 2010 November 26 22:48:47

Title: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 26 22:48:47
Here's my first go at a PixInsight script.  It collects the information from a test image and gives you a suggested sub-exposure length at which point the read noise is low enough relative to sky noise that it makes sense to stop and take another sub-exposure.  The background and calculations are detailed in this paper (http://www.hiddenloft.com/notes/SubExposures.pdf).

(http://www.cerebiggum.com/wp-content/uploads/2010/11/SkyLimitedExposure.png)

Currently it doesn't handle DSLR images well I use data from the FITS header that isn't present in converted DSLR raw files.  The script uses the FITS information for most of the calculations with the exception of read noise which comes from the camera presets list.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 November 28 02:54:04
Sean,

thank you for this very useful tool!
But it doesn't seem to work for me: see screenshot.
A couple of things:
1. I have an FLI ML8300, which is not present on the list
2. The FITS were captured in Maxim DL v5.09

Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 28 10:47:26
Thanks for the feedback,

It looks like some fields are missing in the FITS header for your image.  I captured most of my images in MaxIm DL 5.x and it worked; I'm using an ST-8300M. It's possible MaxIm is not populating the FITS header the same way for each camera.  Are you using a processed test image?

The fields that need to be present in the FITS header are:



I'm going to re-work the script so that you can enter in every value manually if the FITS header or preset list is incomplete.  In the meantime you can manually add them to your test image using the File -> Fits Header dialog.  From what I've read online your camera should use:

CWHITE: 25667 (although it seems like it should be 65535)
PEDESTAL: 0
EGAIN: 0.4

The read noise is 7.59

(source http://www.narrowbandimaging.com/images/u8300_ml8300_ptc_dtc_comparison.pdf (http://www.narrowbandimaging.com/images/u8300_ml8300_ptc_dtc_comparison.pdf))
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 28 10:55:11
Hi Sean,

Well done on the script, it certainly looks interesting. If you are going to rework the script so you can manually enter certain bits of data...will this mean it can be used by DSLR users? (I'm one of them  ;) ).

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Nigel Ball on 2010 November 28 14:49:46
Hi Sean

The script doesn't work for me

I'm using a QSI583 with Maxim 5.09

Running PI on Windows7 64-bit

Nigel
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 28 18:53:40
Ok, here's version 0.2.  I rely more on the presets now and less on the FITS header so it should work better for everyone who doesn't have the exact same camera as me  ;)

New features:


Also, please double check my preset numbers for your camera.  I'm using values from various places on the interweb and errors are always possible.

(http://www.cerebiggum.com/wp-content/uploads/2010/11/SkyLimitedExposure-0.2.png)
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: vicent_peris on 2010 November 29 01:30:12
Hi Sean,

good work. This weekend I was thinking on another possibility for this script... It would be a direct measure of bias and light frames.

In the utility script section you have a script for evaluating the noise of one image (NoiseEvaluation). The idea would be:

- Load a bias frame and analyse the noise.
- Load a preview of the light frame containing a background sky area. Then analyse the noise.

From the ratio between noise intensity in the two frames you can calculate the appropriate exposure time.

The main advantage I see here is that maths are very simple: just copy and paste the NoiseEvaluation script. :)


Regards,
Vicent.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 29 03:34:27
Hi Sean,

Quick question: Will this script give sensible results if I am giving it a RAW image from a Canon camera taken with an Halpha filter, i.e. so that the green and blue channels are effectively just noise?

Or should I extract just the red channel and use that with the script?

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 29 03:40:17
Update: I've just tried the script with an Halpha RAW and then again with only the red channel extracted.....both gave exactly the same results. So that seems cool!  ;)
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 29 05:08:17
Sorry...its me again!

I have a Canon 400D. I have found that the gain is 2.74 e/ADU at ISO400. ( http://astrosurf.com/buil/eos40d/test.htm (http://astrosurf.com/buil/eos40d/test.htm) ) How do I estimate the gain at ISO1600? Is it 2.74 divided by 4?

I have also found out that the readout noise is 7e at ISO400. Does the readout noise stay roughly constant with ISO? If not, how do I estimate it at ISO1600?

Any help would be really appreciated.

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 29 10:50:52
Hi Simon, I think your math is correct for the gain.  You are correct about the gain at different ISO values, to quote another page on the astrosurf site...

http://astrosurf.com/buil/50d/test.htm (http://astrosurf.com/buil/50d/test.htm)


For compute the inverse electronic gain at XXX ISO "sensitivity" use the relation: gain_XXX ISO = (400 ISO/XXX ISO) x gain_400 ISO. For example, the inverse electronic gain of 50D at 100 ISO is 400/100 x 0.57 = 2.28 electrons/ADU.


So basically twice the ISO means half the [inverse electronic] gain.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 November 29 11:55:26
Hi Sean,
I have just downloaded version 0.2 and this is what is happening to the window:
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 29 12:17:36
Just drag the sizer in the lower right corner a bit and it will pop to the correct size.  I'm not sure why this is happening, it happens to me when I use the script on my laptop, but not on a 30" monitor.  It has something to do with the vertical size of the screen.  I left the dialog as "sizable" instead of the convention of preventing resizing to work around this problem.

If any other script developers know what's going on let me know  :-\
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 November 29 12:32:10
Thanks Sean. Works fine now  :)

I am not understanding something though: I image from 6800ft, at a location where we have a 21.5SQM skies.
And the ExposureCalculator is telling me that the suggested exposure should be 53 seconds?

How is that possible?
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 November 29 12:34:26
I think it must be because I don't use darks with my ML8300. ?
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: mmirot on 2010 November 29 13:08:02
Maybe it is time you started?  :)

Max
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 November 29 14:12:57
Hmmm. Not really. ;)

The ML8300 has so low noise (7.59e) that darks are not necessary. I actually don't know of anyone with an ML8300, who takes darks.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 29 14:48:31
Using a dark image will reduce the median background value which will give a more correct value for "sky noise", but this is assuming that the dark noise is big enough to be significant.

You could manually run your numbers through the formulas given in the linked paper and see if you come up with the same results.  It's possible there is an error in the script, but I've never written code with a bug in it before*.



* this statement is completely untrue.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 29 15:31:36
Hi Sean,

I'm confused (not for the first time!). If I set the Gain and the Readout Noise at the values I've determined from the literature (see below), then if I decrease the MaxADUCount down to 4096 (I have a 12bit DSLR  :sad: ) then the recommended exposure time goes up. This is what surprises me. If you have a given number of ADU per electron, and you increase the number of ADU, then surely you need longer to fill the available ADU? I couldn't find anything on the maximum ADU count in the paper. Have I missed something or got the wrong of a stick?

Values for Canon 400D ISO 1600: Gain = 0.635, ReadOutNoise = 4.3

And I worked out that for the Canon 400D, ReadOutNoise = (1550/ISO) + 3.3 ....which might be of help or might not.

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 29 15:42:12
Hi Simon,

The pixel values that PixInsight gives me are normalized (i.e. a percentage of the max count) so in order to convert them back to ADU values I have to multiply them by the maximum ADU value for your sensor.  If you use an incorrect value it breaks the conversion from ADU to electrons.  You are close, but the max ADU value is (2^bits)-1, or 4095 for a 12 bit device, remember that "0" counts as a number too :).

I'll add some presets for more Canon cameras this evening.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 29 16:09:51
Sean,

Thanks, that explains it.

Regarding the zero, so you're telling me I've forgotten nothing  ;)

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 29 20:38:45
Ok, here's version 1.0.


Also, I've created a GitHub project for the script.  If anyone wants to contribute please feel free.  It's really easy to download the git client, pull a copy of the repository and push changes.  Hopefully the new updates system will support user contributed scripts so we don't have to publish updates on the forum.

https://github.com/seanhoughton/CalculateSkyLimitedExposure
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 November 30 00:58:34
THanks for the feedback Sean.
I ran the Starizona Subexposure calculator (on Starizona Website) and for a QHY9 (Same chip as mine) with a median background value of around 2700ADU (see screenshot attached for the statistics of the image) and a 180seconds exposure, I get an ideal subexposure time of 130 minutes.

?

I am lost.

Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Juan Conejero on 2010 November 30 06:18:15
Hi Sean,

Welcome to PixInsight development!

Nice work. If you don't mind, I'd like to include this script in the standard set released with the next version.

I have attached a modified version of your script that fixes all screen alignment and sizing problems. The most important modifications include the following:

- When you create a NumericControl control, it already includes a Label subcontrol (its label property) that you must use. There's no need to define an independent Label and a HorizontalSizer to group it with the NumericControl.

- To visually align all controls on a dialog, use the font property of Dialog to compute a minimum width in pixels for the longest label. Then use that value to set the minWidth property of all labels.

- You can (should) define the width of all Edit controls in the same way. in this case you must force a fixed width for the edit subcontrols of all NumericControl items, as Edit controls shouldn't be resizable in general. However, you must call edit.setFixedWidth() after calling the setPrecision() and setRange() methods of NumericControl, as these methods reset the edit's width automatically.

- There is no need to set the label.textAlignment property, as it is already set to TextAlign_Right|TextAlign_VertCenter by NumericControl's constructor.

- Try to avoid leaving potentially unrooted child controls in a dialog's constructor. For example, if you write:

   function MyDialog()
   {
      ...
      var foo = new Label( this );
      ...
   }


this code should normally work as expected, i.e. foo will be a child control of MyDialog, and in most cases it won't give problems. However, by doing that you take one risk: under certain conditions, the JavaScript garbage collector may decide that foo has become out-of-scope and is unreferenced, and if that happens, foo will eventually be destroyed behind the scenes, with catastrophic results. Although I have been very careful to prevent that to happen with all GUI objects, JavaScript's internals are too complex as to ensure that no unexpected garbage collection may happen sometimes.

To prevent these side effects, always define all subcontrols as properties of their parent controls. In the case above:

   function MyDialog()
   {
      ...
      this.foo = new Label( this );
      ...
   }


is safer because as long as an instance of MyDialog is referenced, its foo child control will also be referenced.

- When a dialog must be user-resizable, always try to force at least one fixed dimension —either the width or the height, as appropriate— whenever possible. In your script, I have set a fixed width of 420 pixels:

  this.setFixedWidth( 420 );
   this.adjustToContents();


which allows only changing the dialog's height. This ensures that a valid geometry will be set on all supported platforms (Mac OS X is particularly problematic in this regard).

Hope this helps.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: harist on 2010 November 30 12:52:29
Hi Sean,

Just tested your script and it works fine! I think it would be handy if the previously set ccd selection could be retained
each time the script is run.

Best Regards
Tasos
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 November 30 14:29:41
Hi Sean,

I've tested it on a number of different exposure length subs and it always seems to come out with sensible answers.

I agree with Tasos, if it could hold the last entered values, then that would be really great. Not sure if this is possible with a script though?

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 30 15:26:05
Thanks for the fixes and best practices Juan, that's helpful for the future.

I wasn't sure about the GC ramifications of using local variables for UI elements but I had assumed that adding them as children to the sizers would create a reference chain that would keep them alive.

Most of your improvements involve deleting code - my favorite kind.

Including it as part of the distro is fine.  Some people are reporting strange values so I'm going to do another pass on the math to make sure everything is correct.

Persistent settings is a bit tricky in a script.  I can write out a text file with the saved values but file permissions might cause issues.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 November 30 20:20:56
Emanuele,

I think you plugged in seconds when the page is asking for minutes  :surprised:

When I use the values you provided (including "3" for the exposure) I get 2.25m as the limited exposure, which ends up being 1.125m suggested exposure.  So far all my test images line up with the results from the web calculator you linked to.

Remember that low value can actually be a benefit because it means you can take advantage of subexposure integration and get better pixel rejection and a reduced penalty for an image with defects.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Emanuele on 2010 December 01 01:06:06
Hi Sean,

yes, my bad on that webpage.
I still don't understand how is it possible that I get a suggested exposure time of 1.25m from a 21.5SQM sky!  ???
E.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: harist on 2010 December 01 07:06:16
Quote
Remember that low value can actually be a benefit because it means you can take advantage of subexposure integration and get better pixel rejection and a reduced penalty for an image with defects.

Very true! However, faint targets are a different story and what matters then (in terms of sub duration) is to register target data!

A very interesting approach for sub duration is also presented here:
http://www.cloudynights.com/item.php?item_id=1622

Tasos
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 December 01 10:19:21
Hi Tasos,

I agree wholeheartedly with you re faint targets, especially if you are doing an HDRComposition so that the bright stars are not massively oversaturated. I really liked that paper!

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 01 12:30:17
I enjoyed a quick read of the paper and I'll do another careful one tonight.

It sounds like providing another value, "Total Exposure Time", to the script would let me go further and calculate a second subexposure value. It might be possible to provide a target image and have the script suggest an appropriate exposure based on its flux, the star saturation level, and the tides.

This is turning out to be more complicated than I expected when I first decided to 'write a simple script to learn the API'  ???

One of my goals in dusting off all of my equipment and having another serious go at astrophotography has been to audit and improve every aspect of my imaging pipeline.  This means working through hardware issues, such as PE in the mount, all the way through final image processing.  This is turning out to be a very interesting part of that process.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: harist on 2010 December 01 13:43:01
Quote
This is turning out to be more complicated than I expected when I first decided to 'write a simple script to learn the API'

I confess that I posted the article link hoping it would provide a challenge for enriching the script.  ;)

Tasos
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 03 22:44:06
I'm working on a new version of the script but after studying that paper I can't seem to make sense of a few details.  The formatting is not doing it any favors for readability.

I have two questions:


Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 December 04 01:55:11
Sean,

I think the author has used Etotal/min to denote Etotal in units of minutes......more usually wriiten as Etotal(min). But that's just a hunch, I may be wrong!

I'm not sure about the other equations. Could you contact the author for clarification?

Cheers
         Simon

***** Ignore the above....I've just re-read it......and I'm talking rubbish!!! ******
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 December 04 09:19:34
Sean,

Wow, that paper is a difficult read! (or is it just me???)

Yes, lambda is ?, but its not too clear what lambda actually is. From the paragraphs just above eqn 14 it does look like it is something to do with the target signal. And if you click on the link to the info on the Poisson distribution then lambda is defined as "a positive real number, equal to the expected number of occurrences that occur during the given interval. For instance, if the events occur on average 4 times per minute, and one is interested in the probability of an event occurring k times in a 10 minute interval, one would use a Poisson distribution as the model with ? = 10×4 = 40." And that sounds like the faint target signal level to me.

But if you go down to the bottom of the paper to the final paragraph before the Conclusions, then it talks about lambda in a different way. It looks like its a factor that you can enter depending upon how close you want to be to a normal Poisson distribution.....with a value of 15 being a "good" value. IF this is correct, then lambda is effectively a constant....one that you choose at the begining.

I've got this feeling that what the author is saying is something like this......the maths analysis assumes a Poisson distribution. A Poisson dist becomes normal when the faint signal becomes > 5, with 15 being a 'good' number. So if you put 15 into the equations, then they will be valid and will give you the optimum subexposure for the faint parts of the image that have a signal level of 15. However, if you want to get the optimum subexposure time for parts of the image that have a signal level of 10 (or even 5) then it will still hold true.

This value of 15 is the signal level of the faint part that we are concerned about during the exposure, i.e. it is not the value per minute, but the value per minute multiplied by the subexposure time.

IF any of the above is correct then it still doesn't make much sense. The equation seems to be dimensionally incorrect....S seems to be in units of sqrt(T)?

I have no idea if any of this rambling helps at all????

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 04 11:24:04
That definition of lambda is what I initially thought and how I initially implemented the code.  Somehow I missed the conclusion where he clearly states that and went off reworkng everything to use target signal.  Now I'm glad I started using git to track revisions :)

I think the remaining confusion boils down to the unclear definition of ETotal and it's components.

Quote
No matter how many sub-frames you want to stack, there is going to be a limit in total exposure time, call it TTotal. The final combined stacked photo’s total noise term,  will be

(http://www.cloudynights.com/images/optimal_files/image108.gif)

Where: ETotal is the total noise for a shot of time T.
S is the sub-frame length in units of time T
TTotal is the total time in units of time T

I'm fairly certain that ETotal is the error for the sum of all subexposure time, not for each subexposure time. However, in his example he uses the value for ETotal/minute: 16.5 instead of the value his equations would produce for an ETotal for a 45 minute T (16.5 * 45 = 684 with no camera noise).

Adding to the confusion is the equations for light pollution noise that are not consistent with each other:

(http://www.cloudynights.com/images/optimal_files/image083.gif)
(http://www.cloudynights.com/images/optimal_files/image074.gif)

???  I'm going to email the author and see if he'll take a look at this thread.

Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Nigel Ball on 2010 December 05 08:22:09
Sean

Only just got round to testing the latest version of the script

Seems to work okay but when i close the image I have run th escript on I get this error

After this I cannot run the script again without restarting PI

Windows 7 64 bit

Nigel
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Nigel Ball on 2010 December 05 08:28:19
Thanks Sean. Works fine now  :)

I am not understanding something though: I image from 6800ft, at a location where we have a 21.5SQM skies.
And the ExposureCalculator is telling me that the suggested exposure should be 53 seconds?

How is that possible?


I'm also getting some 'unusual' figures for my Luminance subs - suggested exposures of 16s on a 300s Lum sub???

That said my Ha subs appear to be almost spot on? I don't profess to understand the maths here so perhaps someone could enlighten me :-[

Nigel

Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 05 15:25:54
Ok, version 2.0 is ready for testing out. It includes my best effort at the Anstey equations for limited exposures.

Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 05 15:31:38
Quote
I'm also getting some 'unusual' figures for my Luminance subs - suggested exposures of 16s on a 300s Lum sub???

The readout noise limited exposures are very sensitive to the background ADU value.  If your skies have a high flux (e-/s) relative to readout noise you'll get low exposure numbers.  This can easily happen with an unfiltered luminance setup unless you're at very dark site.  Alternatively, if you use narrowband filters your sky flux can be so low that it can take hours to overcome the readout noise.

What sky flux value are you getting for your luminance test image?  Also, you can try running the script on a preview view that only contains background to ensure no target photons are creeping in - the script uses the image's median to approximate the background value.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 05 21:31:40
Charles Anstey, the author of the paper, has been very helpful (and patient) with my pestering.  With his help I have updated the script to use equation (16).  This version takes Ereadout into account and will provide more accurate values when ELP is very low.  However, this is fairly unlikely in most cases because with new cameras the Ereadout is so low and total integration time is so long.

Now, if only these clouds would clear up so I can get back to.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Nigel Ball on 2010 December 06 04:00:17
Sean

Downloaded and ran the latest version

I'm still getting an error when I close the image window

Nigel

PS Could you add a feature to remember the last selected camera?

Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 06 08:07:01
I'm running on OSX and don't get a crash but I'll take a look on my windows machine at work.  I think it might be related to one of the things Juan mentioned in his post.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 December 06 08:48:25
Just to confirm, I've been getting the same error message behaviour as Nigel. I'm running Win Vista 32 bit. Hope that helps.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: harist on 2010 December 07 11:57:16
Quote
With his help I have updated the script to use equation (16)

I'm sorry I couldn't follow up the thread in the last couple of days but I am glad that Charles Anstey was so helpful. I also had unclear points in my mind although I tried to work out his formulas with Excell. Thanks to your excellent (and impressively fast)work I will be able to do a lot more experimentation now. :)

One question though: is there a practical purpose for the 4 hrs (14400 sec) total exposure time (I never get anything good below 6 hrs with my level of LP, usually go for 13 hrs).

Best Regards
Tasos
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 07 20:31:52
Can those of you with crash problems please try the most recent version on GitHub https://github.com/seanhoughton/CalculateSkyLimitedExposure (https://github.com/seanhoughton/CalculateSkyLimitedExposure)?  If it fixes the crash I'll post another release here.  This might be something that Juan has to look at with a debug build of PixInsight.


Quote
is there a practical purpose for the 4 hrs (14400 sec) total exposure time

This is one of the reasons the Ereadout models are not always great.  If you have a nice dark background then you end up with a ridiculously long exposure in order to get the sky noise up to 95% of total noise.  Charles mentions this in his paper as one of the reasons not to use that model.

I think it's informative, but maybe not practical now that the Anstey limit I provide includes readout noise in the equation.

I've been considering adding some charting to the script to provide a more intuitive look at what's going on as the sub-exposure length changes.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Nigel Ball on 2010 December 08 02:47:36
Can those of you with crash problems please try the most recent version on GitHub https://github.com/seanhoughton/CalculateSkyLimitedExposure (https://github.com/seanhoughton/CalculateSkyLimitedExposure)?  If it fixes the crash I'll post another release here.  This might be something that Juan has to look at with a debug build of PixInsight.

Sean

I'm still getting the error on close of the image window with this build

Nigel
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: harist on 2010 December 08 06:42:00
Hi Sean,

I was getting the error with version 2.0 but it stopped with the latest one (windows 7, 64 bit).

Quote
I've been considering adding some charting to the script to provide a more intuitive look at what's going on as the sub-exposure length changes

with respect to the total integration time would be of great value.

BR
Tasos
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: canstey on 2010 December 10 13:08:01
This is Charles Anstey, the author of the paper in question.  I have been trying to find a simpler way to explain what is going on and why sub-exposure length matters.

Let's consider each term separately because they follow superposition.

Target signal : Goes up linearly with time so it depends purely on total time Tgt(t).  So 1 100-minute shot or stacking 100 1-minute shots are equal.
Light pollution signal : Goes up linearly with time so it depends purely on total time LP(t).  So 1 100-minute shot or stacking 100 1-minute shots are equal.  However we can simply subtract this out from the image as a bias.
Light pollution noise : Goes up with the SQRT(time) so it depends purely on total time Elp(sqrt(t)).  So 1 100-minute shot or stacking 100 1-minute shots are equal.

At this point if we had a noiseless camera, there is no advantage to shooting longer sub-frames and the implications are pretty astounding.  When advanced amateurs start using noiseless cameras, astro-imaging will take another huge leap forward in quality.  At that point, DSO imagers will be using 10-30 frames per second just like planetary imagers, collecting as many frames as they want, and then stacking only the best of the best until all the desired detail has been attained.  Look up "Lucky camera" to see the results of such a camera.

However, the current real-world "affordable" cameras all have read noise.  The total read noise in an image depends purely on the number of frames stacked and goes up by the SQRT(#frames)
Read Noise : RN*SQRT(# sub-exposures).  So 100 1-minute shots has a lot more read noise than 1 100-minute shot.

The goal of the formula is to balance the number of sub-exposures with respect to read noise and light pollution.  Reducing the sub-exposure length and having more total frames results in more total noise from the increase in read noise with no gain in target signal, fixed by total time.  Increasing sub-exposure length and stacking fewer frames gains basically nothing because the faintest signal discernible above the total noise has already been limited by the total exposure time and dominated by LP noise so lowering total read noise will not help.

From a practical point of view when you don't know how much total time you wish to spend on an object, shooting longer sub-exposures allows you to keep adding more frames if you decide to keep at it with the only penalty being that if the frame is lost because of bad guiding or other error, you lose a larger piece of time.  Use the formula to determine the practical upper limit of the total time for a given sub-exposure and know that if you are thinking of possibly putting in even more total time, you need to use a longer sub-exposure to be safe.

The problem with John Smith's formula is that it is only valid if you are stacking exactly two frames.  Eventually if you stack enough frames, read noise will become significant enough that for the same total time you would get better results for the faintest signals shooting fewer frames for longer time.  Also the implication is that shooting in LP or during the full moon somehow limits how deep you can image compared to dark skies.  Completely incorrect and an imager can create an identical result under the full moon compared to darkest skies if they image with enough total time.  The increase in total time may be a factor of 10+ but what else are you going to do for that night, nothing?  Seeing and transparency determine the absolute limits of image quality and depth.  You can't stack a bunch of 3.5" FWHM images and wind up with a 2.0" FWHM image no matter how much total time.  Don't skip great nights of seeing just because the moon is out.  The only caveat is the camera has to have enough well depth to cover the LP signal and have plenty of bits left over for the stretch to an 8-bit display.  Cameras with only 8 or 12 bits are likely to be limited by excessive or even moderate LP.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: harist on 2010 December 13 14:44:11
Quote
Use the formula to determine the practical upper limit of the total time for a given sub-exposure and know that if you are thinking of possibly putting in even more total time, you need to use a longer sub-exposure to be safe.

That would require an additional frame in the script UI where subexposure length would be set by a slider and total integration time would be a result (from eq. 17). Sean has already done an excellent job, I don't know if we can ask for more  :-[

Tasos
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2010 December 16 21:50:19
I'll take another peek at the script and see if I can put a couple more tools in there.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Simon Hicks on 2010 December 17 00:48:49
Hi Charles,

Many thanks for coming on the forum and contributing to it. I liked your explanation and it has got me thinking.

I think I have got a couple of points from it. Firstly, the faintest DSO signal is there in the sky superimposed on whatever light polution signal there is, and in fact its also superimposed on the light polution noise. Its not actually swamped by it in some sence that it is washed out and cannot be recovered....its still there, its just a case of subtracting the light polution signal, and reducing the light polution noise (more total exposure time) to a level where it is smaller than the DSO signal. This is where the longer exposures and more of them comes in.

So actually we could all image DSOs during the day as long as we took a huge number of extremely long exposures with a camera that had extremely huge well depth. An extreme case, but it seems to illustrate the point and seems to follow from what you were saying.

Just thinking out loud.  8)

Cheers
         Simon
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Nigel Ball on 2011 January 01 04:12:19
Excellent work Sean   8)

I find the script very useful, I'm using version 2.1

Am I correcting in assuming my ideal exposure should be between E limit (I) and Elimit (II) ?

BTW There is a spelling mistake in the options section Tolerance has only one 'l'  ;)
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: DarrenS on 2011 January 31 11:31:21
Excellent work Sean

any chance of adding a few ATIK cameras, 383L 314 ?

I've tried working out the figures and tbh its beyond me.

Darren
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2011 February 01 20:26:03
Hi Darren,

I've added both cameras.  If you add my PixInsight update site to your update repositories you'll get updates automatically - including this one.

Let me know if you have any problems with the update.

http://pixinsight.cerebiggum.com/
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: avastro on 2011 February 02 00:27:11
Hi Sean,
Excellent work, I added your site in my repositories update and that worked flawlessly.
Your script works fine but when I close it I get this message.
"At address 0000000077AA76FD with exception code C0000005 :
Access violation: invalid memory read operation at address FFFFFFFFFFFFFFFF"
Vista 64 bit.
By the way thanks also to Charles for the very useful explanation about his paper.
Cheers

Antoine
http://astrosurf.com/avastro/
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Sean Houghton on 2011 February 02 11:02:39
Hmm, that's not good.  I'll double check my references in the script, but I think someone with a debug build of PixInsight might have to take a look at that.  I haven't been able to get my license working on my work machine (a 64bit Windows7 machine) to test it, but I'll give it another try tonight.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: skoop on 2011 November 26 12:20:57
Excellent script.
I like to request a small update :)
Any change you can add a "save camera as default" option and remember last exposure? 

Thank you.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: blinky on 2012 September 30 13:18:31
So..... Do I just use a preview from a single un-debayered sub and unstretched sub?  If I do that I get a max exposure of around 2.5mins which is not good :-[  Also how/what is the pedestal value?
I have been taking 600s images in the hope that I can bring out some faint detail but it now seems that 2mins is it and any more is pointless :-\
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: NKV on 2012 September 30 23:31:35
So..... Do I just use a preview from a single un-debayered sub and unstretched sub?  If I do that I get a max exposure of around 2.5mins which is not good :-[  Also how/what is the pedestal value?
I have been taking 600s images in the hope that I can bring out some faint detail but it now seems that 2mins is it and any more is pointless :-\
Deduct a 600s Dark frame from the Light and try. ;)

PS some pop-up notes or separate selector for DarkFrame will very useful.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: blinky on 2012 October 01 03:19:34
Ah.... I don't bother with darks as the qhy8 does not need them, or so I was told ???
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: NKV on 2012 October 01 03:27:36
Ah.... I don't bother with darks as the qhy8 does not need them, or so I was told ???
Anyway you must subtract Dark for correct calculation. But, if your camera have zero dark current, you can subtract bias frame.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: jeffweiss9 on 2012 November 04 17:49:48
It seems most of the time I try this, the process console shows the following error:
" line 26: TypeError: img is undefined"
I'm just reading in a standard CCDSoft v5 FIT file so why does it think the image is undefined?
In playing around with it, I once got an actual result, although I don't believe the 2 minute subs that it was telling me I needed for an extremely dark site and 8 e- readout noise (Austel model, the other models gave 30sec sky-limited exposure).

Also, I gather it needs a dark subtracted image, which this wasn't.  Can the script just ask for a master dark and do the subtraction?  Dark subtracted subs are just not something laying around in the PI calibration process (calibrated frames are but they have flats divided out) and it would save some Pixelmath. In any case, I get the same result as described above if I do a Pixelmath dark subtraction first. 

I'm guessing its not finding something in the FITS header, but if PI is happy, why isn't the script happy?  The script asks you to enter the exposure time of the image.

Thanks for your help.
-Jeff
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: jeffweiss9 on 2012 November 20 21:11:51
Note, giving it a preview, rather than the image itself, seems to avoid this problem.
-Jeff
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: Robert Q. Kimball on 2014 July 31 14:18:09
I love this script.  Can I assume that with my Atik 383L (8300 chip) I can use the SBIG 8300 selection?

Thank
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: ecorm on 2019 March 04 19:16:29
What's the difference between Ereadout limit(I) and Ereadout limit(II)?

I see that Dark Noise affects II, but not I.
Title: Re: New Script: Calculate Sky Limited Exposure
Post by: ecorm on 2019 March 04 19:49:30
By inspecting the script's Javascript code, I figured out that Ereadout limit (I) comes from equation (3) in John C. Smith's paper (which I had to retrieve using the Internet Wayback Machine).

Ereadout limit (II) comes from Chuck Anstey's article, equation (9).