law of diminishing returns

[sstuder]

In Warren Keller's new book on Pixinsight there is a remark (p. 47) that really bothers me. It says that due to the "law of diminishing returns" the number of images to be stacked is limited to 30. Otherwise there is no improvment to be gained or the quality of the integrated image could even deteriorate.
I always thought that the SNR increases with the squareroot of the number of frames. In other words: the more frames the better. The "law of diminishing returns" questions this. It is derived from agriculture and seems to apply in other regions of economy as well. But does it really also apply to astrophotography? Can anyone confirm this or contribute in any other way?

[NGC7789]

I'm guessing that what is meant is that a two fold increase in SNR requires a four fold increase in integration time. In the case of 30 images you would need another 90 images (all other things being equal, which of course they are not!). This is the diminishing return Warren is speaking of. As the number of images increases, gaining an increment of SNR is increasingly "expensive" in acquisition effort. More images is always better if you are talking purely about SNR but there are other factors that weigh into the "return" side of the equation.

[pfile]

it really depends on how faint of a signal you are going for, or if you are imaging under light polluted skies. i regularly go beyond 30 subs and there is a real improvement.

for dark skies this rule of thumb is probably "true" but there are certainly situations where you need to push far beyond that number.

rob

[Niall Saunders]

That statement - "law of diminishing returns" - taken its own, is a pretty meaningless statement - and is one that I am surprised someone with Warren's background feels compelled to make.

Firstly - what if you maximise the effort to take as many good images as your viewing session permits, and then chose "only the 'best' 30 images from that session. Would the final image, created from those thirty images be 'better' than had you only used the 'best 20' images, or the 'best 40' (or even - though unlikely - the 'best 120')?

It is impossible to answer that - unless you actually have the data available to select from.

Secondly, what rules - other than 'noise analysis' can be used to determine the 'diminished return' from having worked hard to acquire extra data.

This '30-image' "law" (it isn't actually a LAW - it is a statistical suggestion, and authors should not be using it to glibly fill copy-inches in publications - irrespective of whether they earn money from doing so or not) is one that circulates around the astro-imaging community. It over-simplifies things for the novice, confuses the experienced, and is ignored by the professional.

There is only really one "LAW" (and, of course, it isn't a law - it's just guidance and advice - no matter who thinks they are in charge, no-one is going to send round the image police on some trumped-up charge of gathering more than your alloted quota of photons) and that 'law' would encourage you to take as many images are your viewing session allows, and as many images as you have the time to spend collecting them.

Only then, with however much raw data you have been able to acquire, will you be able to decide how many of those images you want to include in your post-processing attempts.

And, remember this "law" doesn't just apply to your light frames - it also applies to your darks, flats and - if you feel you have to acquire and use them - your Biases too.

I used to finish each imagng session by closing up the observatory and then just leaving the camera to generate Darks until I turned up some time the following day to look at things. From those hundreds of Darks, I would select 'a reasonable quantity' to create a Master Dark. I never bothered with Biases - totally unnecessary  for any TEC-cooled camera, so why bother? Then I started to look at my Darks, and I realised that I couldn't see any differences in the final images - irrespective of whether I used Darks of the same exposure, Darks of other exposure times, or no Darks at all - my camera produced a reasonably 'flat' image in terms of its dark noise characteristics.

As far as Flats were concerned - because I almost never mess around with the camera setup on the OTA, why would I feel that the Flats might change from one session to the next? I looked at them closely - and they didn't - so I only take new Flats if something major has changed in the optical train. (Much the same goes for focus - it doesn't change from one session to the next, so I leave it locked off and just shove on a Bahtinov mask every so often to remind me how glad I am that I didn't waste money on a motorised focuser!!)

But, like Darks, Flats are super-easy to acquire (needing nothing more than a pice of white card on the observatory wall, and a light bulb - somewhere - to illuminate it). They can be collected unattended, and as many can be taken as I can be bothered storing. So, collect lots, use lots, make a Master Flat, then throw the rest away and use the same Master Flat for months and months and months.

Of course - I have the luxury of a fixed setup, in an observatory - but, at the same time, face the perpetual challenge of perhaps only ten nights of 'good' imaging weather per year (that's also why I abandoned imaging with a monochoome CCD and filter wheel, in favour of a far more productive OSC camera).

Of course - your mileage may vary. Don't blame me if, at around three a.m., just as your thirty-first image dribbles onto your hard-drive, some night-warrior all dressed in black slithers stealthily up behind you and sets off a small thermo-nuclear device, allowing all those otherwise illegally captured photons that you have imprisoned to escape back into the ether of dark infinity.

[jrista]

Niall, I appreciate your post! I was pretty surprised by the 30-frame law mentioned by Warren as well. I regularly stack hundreds of subs, and so long as proper care is taken in managing both the subs, their calibration, and proper dithering, useful gains in SNR can continue to be made well beyond 30 frames. The situation is definitely not simple.

One must at the very least take into account the exposure length. With new ultra low noise CMOS cameras, the gain and sensitivity is often so high that getting long exposures is impossible. It is not uncommon (actually more common) to be "stuck" taking 60, 30, 10, 5 second LRGB subs with a high sensitivity, high gain CMOS camera, in which case 30 subs could be a rather meaningless amount of signal. In such situations stacking hundreds, if not even thousands, of frames is common and often necessary. Warren's comment about 30 frames and diminishing returns caused a bit of a stir on the CN BII forum, as many of us there are using CMOS cameras now, and many of us have found clear gains to stacking at least 100 subs, and often 300-400 subs.

FPN can definitely become an issue when stacking so many subs. That is where getting critical about calibration...both in the construction of clean, quality, effective masters, as well as getting effective dithering...becomes a fundamentally important aspect of pre-processing. With good calibration, dithering and pre-processing, the limitations of FPN can be pushed well beyond what might have been considered the "brick wall" with CCD imaging in the past.

[Niall Saunders]

There is another, perhaps crucially more important point nowadays, and that is more about 'how' the images are going to be post-processed - a point that Warren's Law fails to take into consideration.

Once you start processing using the power of tools available in PI, it certainly pays dividends to feed 'large' quantities of RAW data into the processing engine. The statistical analytics used at the heart of PI ensure that 'outliers' (which is, after all, exactly what 'noise' really is in this context) are easily detected, and effectively eliminated, during the very early stages of pre-processing (and image calibration).

Further, because PI performs statistical analysis at the 'pixel' (if not sub-pixel) level and, if required, using 64-bit levels, it can readily make use of all the data that it is given. Images are no longer analysed 'as a whole, and analysis is no longer restricted to 8-, or even 16-, bit levels. Instead PI considers 'columns of pixels' (and their neighbours' to effectiveley reduce 'noise' (or outliers - including 'hot' or 'cold' pixels).

PI has such a specialised approach to image analysis and processing that people who have come from the 'old school' can have quite some difficulty in applying their prior knowledge and experience to the new PI environment.

In PI there are 'no laws' - there is a lot of guidance, and you get to pick and choose what you want (or, sometimes, 'understand'  ) and then decide whether the result was what you had hoped to achieve. And, best of all, if it wasn't, you can still save that process (to one side) and use it again - this time with, perhaps, minoor alteration - and see if that was better, or worse.

And, if you mess up, you simply go back one stage, grab the saved process that 'worked best', and set off in a totally new direction. All guided by statistical analysis.

And statistics only really start to become meaningful when the data set is LARGE (not limited to "30").

_{This is a known fact, as shown by a recent survey where 73% of those who expressed an opinion in the 89 people surveyed agreed with the question put before them.}

[rdryfoos]

There is another, perhaps crucially more important point nowadays, and that is more about 'how' the images are going to be post-processed - a point that Warren's Law fails to take into consideration.

Once you start processing using the power of tools available in PI, it certainly pays dividends to feed 'large' quantities of RAW data into the processing engine. The statistical analytics used at the heart of PI ensure that 'outliers' (which is, after all, exactly what 'noise' really is in this context) are easily detected, and effectively eliminated, during the very early stages of pre-processing (and image calibration).

Further, because PI performs statistical analysis at the 'pixel' (if not sub-pixel) level and, if required, using 64-bit levels, it can readily make use of all the data that it is given. Images are no longer analysed 'as a whole, and analysis is no longer restricted to 8-, or even 16-, bit levels. Instead PI considers 'columns of pixels' (and their neighbours' to effectiveley reduce 'noise' (or outliers - including 'hot' or 'cold' pixels).

PI has such a specialised approach to image analysis and processing that people who have come from the 'old school' can have quite some difficulty in applying their prior knowledge and experience to the new PI environment.

In PI there are 'no laws' - there is a lot of guidance, and you get to pick and choose what you want (or, sometimes, 'understand'  ) and then decide whether the result was what you had hoped to achieve. And, best of all, if it wasn't, you can still save that process (to one side) and use it again - this time with, perhaps, minoor alteration - and see if that was better, or worse.

And, if you mess up, you simply go back one stage, grab the saved process that 'worked best', and set off in a totally new direction. All guided by statistical analysis.

And statistics only really start to become meaningful when the data set is LARGE (not limited to "30").

_{This is a known fact, as shown by a recent survey where 73% of those who expressed an opinion in the 89 people surveyed agreed with the question put before them.}

  Following with rapt attention.  May I ask a question?  It is that I take it you collect upward of 30 subs--more at times (often).  However, when I shoot NB, I take 30 minute subs.  20 is allot--10 hours.  30 is allot more.  The amount of noise in my 10 hour Ha stack is so small, I almost need no noise reduction.  Can sub duration substitute for time in your suggestions? 

Thanks,

Rodd

[Niall Saunders]

Can sub duration substitute for time in your suggestions?

Naturally - the answer is, and always will be, "Weeeeeeell, it depends . . . . ."

You can only take as many frames as your imaging session permits (weather limitations, site limitations, night-length limits, school-days, etc.) - and, when you then factor in exposure time, that number could end up being, statistically speaking - quite small.

But until, and unless, you acquire more frames - what you have got is what you have got.

You then have to analyse your data - and, fortunately PI gives you plenty of tools to achieve that. And you then need to use those tools to eliminate, by statistical selection, those pixels within that data set, that serve no useful purpose remaining in your final image.

If you don't have a load of raw data in the first place, that statistical elimination process becomes more and more difficult - eventually degenerating into having to 'paint out' or 'mask out' individual pixels by hand. Give PI enough raw data, and sit back and be amazed as it automagically eliminates all of that noise without any real effort whatsoever - some operations even allow you to decide 'how many pixels' or 'what percentage of pixel elimination' you actually want to achieve.

Acquire what you can - more is always better - and then let PI see what it can find hiding inside your data!

[rdryfoos]

Can sub duration substitute for time in your suggestions?

Naturally - the answer is, and always will be, "Weeeeeeell, it depends . . . . ."

You can only take as many frames as your imaging session permits (weather limitations, site limitations, night-length limits, school-days, etc.) - and, when you then factor in exposure time, that number could end up being, statistically speaking - quite small.

But until, and unless, you acquire more frames - what you have got is what you have got.

You then have to analyse your data - and, fortunately PI gives you plenty of tools to achieve that. And you then need to use those tools to eliminate, by statistical selection, those pixels within that data set, that serve no useful purpose remaining in your final image.

If you don't have a load of raw data in the first place, that statistical elimination process becomes more and more difficult - eventually degenerating into having to 'paint out' or 'mask out' individual pixels by hand. Give PI enough raw data, and sit back and be amazed as it automagically eliminates all of that noise without any real effort whatsoever - some operations even allow you to decide 'how many pixels' or 'what percentage of pixel elimination' you actually want to achieve.

Acquire what you can - more is always better - and then let PI see what it can find hiding inside your data!

  Well, my imaging sessions are defined by how much data I want to get--not how much time I have, because I can always wait until the next clear night to continue imaging.  I never complete an image in 1 night anyway.  I guess my question is really that diminishing returns issue.  If I am taking 5 minute subs--80 subs is very doable in 1 good night.  Even 160 would only take 2.  But with 30 min subs--15 is the most i get in 1 night usually (sometimes 18).  So if time cannot be used in your suggestions instead of # of subs--i am looking at 5 nights to get 75 subs.  Even I will not do this (I am impatient and use a mono camera).  I know that 30 minute subs have less noise and a higher SNR than 5 minute subs, so somehow I think "duration of subs" and "# of subs"  can be equated.  Lets start at "total exposure time" and work from there.  (its not that simple though).

[RickS]

Lets start at "total exposure time" and work from there.

That is a basic point that hasn't been emphasised so far.  The only noise that we can't mitigate is shot noise, and that depends completely on how many photons we measure, i.e. total integration time.  We don't want to do very short subs because read noise will affect SNR.  We don't want to do very long subs because it's a PITA when we lose one to clouds or guiding and because rejection works better with a good number of subs to chew on.  Anywhere in between these extremes we'll do OK and it is total integration time that will determine the quality of the result, at least in terms of SNR.

The argument about how many subs is too many is specious, IMHO

Cheers,
Rick.

[Warhen]

Hi guys, First off, thanks so much for your support! Second, never hesitate to contact me privately either here, or via IP4AP.com with questions or comments. I too appreciate this thread. I regret that for some, my statement seemed 'pat', but the book is meant to be a comprehensive, albeit, practical guide to PI. NGC7789's comments are in keeping with what I had in mind. I've certainly seen marginal improvement with the addition of more subs than 30 (premium ones), as well as reduction in noise by extremely deep cooling (<-30C), but though I completely agree with several of Niall's statements, I'm standing by mine as a rule of thumb. There is a law of diminishing returns (not my law! :>), and looking at the graphs, thirty really is a magic number of sorts. Niall, I like your term 'statistical suggestion.' I'm less fond of what I felt was a bit of an unnecessarily harsh indictment of my intentions.

SStuder, you say that I stated "...otherwise there is no improvement to be gained or the quality of the integrated image could even deteriorate." And, this is not at all what was stated. Let's look at the quote-

"There is, however, a law of diminishing returns. Statistically,
the limit from which you can obtain significant benefit is thirty images. You must
therefore strive to collect the majority of signal in the length of each individual
exposure, rather than taking excessive numbers of exposures."

I see nothing incorrect, misleading, or confusing here. I feel, that in my experience, it's a very reasonable recommendation. Thanks for allowing me this time, and again for your kind support!

[Geoff]

Juan provides a good explanation (graph included) of the law of diminishing returns after 30 images.  Read the documentation on ImageIntegration.

Remember that the 30 frame rule applies to a given subframe exposure time.  If you are taking 1min subs you won't be getting much improvement by going from 30 frames to 35.  Same story if you are taking 15min subs--not much improvement going from 30 to 35 frames.  However, the 15 min subs will give you a vastly better result that the 1min subs.
Geoff

[Warhen]

Thanks for that Geoff. I'd first been introduced to the concept prior to Juan, by another very bright friend- Kevin Nelson, one of the partners at Quantum Scientific Imaging. As Juan states "...The improvement remains significant up to about 30 images. Above 30 images, a considerable imaging effort is required to achieve a noticeable SNR increment." So, again, are there many variables to consider? Certainly. That being said, "There is, however, a law of diminishing returns. Statistically, the limit from which you can obtain significant benefit is thirty images. You must therefore strive to collect the majority of signal in the length of each individual exposure, rather than taking excessive numbers of exposures."   

[Geoff]

You must therefore strive to collect the majority of signal in the length of each individual exposure, rather than taking excessive numbers of exposures."   

Well put Warren. Sums it up nicely.

[RickS]

Here's a little thought experiment...

Say we have two cameras.  Camera #1 has low read noise and a 300 sec sub is sky limited.  Camera #2 has higher read noise and needs a 600 sec sub to get sky limited.

Taking 30 x 600 sec subs with camera #2 gives us about the same SNR as taking 60 x 300 sec subs with camera #1.  Should we have stopped at 30 subs with camera #1?  If 60 subs isn't much better than 30 with camera #1 then maybe we should have stopped at 15 subs with camera #2?

... end thought experiment

I suspect that the 30 subs rule comes about because we're used to cameras that have read noise requiring relatively long subs.  Once you get to 30 longish subs then the total integration time is such that it's a big investment in additional time to make a significant improvement.

There are folks using low read noise cameras now who get good results from many more than 30 subs and the maths for this stacks up (agree that the processing may become a PITA, though!)  IMHO the "30" rule will become irrelevant in the not too distant future.

Feel free to correct me if I've missed something in my argument.  I haven't thought about this deeply...

Cheers,
Rick.