BPP or do it manually ?

[stevek]

I'd be interested in what people think of this please?

I know the official line is to use the manual tools for pre-processing.  I have little doubt that it offers the greatest amount of control and I normally use it with calibrate only option so that I do the registration and integration manually.  However, BPP is very useful for calibration and cosmetic correction from where I can then take over.  That said, does it make /that/ much difference to the final masters?  I find it hard to tell form looking at them. Unless, of course, any differences manifest themselves further into the post processing stages?

For folks that do use the BPP script, do you feed into it the BIAS and darks that you manually made via Image integration?  Or do you run of a copy of the master BIAS and master darkthat BPP made?

Would be most interested ion other's opinions on this.
Steve

[dld]

I pre-process and integrate manually only for one reason: To assess human errors during acquisition of calibration and image data, to learn from them and try to correct them next time. A tedious process, but the skills and image quality improvements I have experienced, fully compensate for all of the burden.

Automation is useful when most factors are in control. Tracking, focusing, proper acquisition of flats, proper acquisition of darks/flat darks/bias, and correct usage of them during data reduction. In short, it is useful when you know your equipment and when you know how to reduce your data.

Unfortunately what I suspect is most people settle on automatic tools and let human errors or other problems haunt their integrated images for ever, without noticing. For me PI is more of a debugger which allows me to correct my mistakes during acquisition.

[Juan Conejero]

I know the official line is to use the manual tools for pre-processing.

If I can represent something official here, then let me say that we have no official line in this regard—we have very few official lines, actually, although important ones

The WBPP script is perfectly fine for calibration and registration (and the applicable intermediate steps such as demosaicing, etc). It is also great for weighting if you want a simplified approach to image grading, which works well in many practical cases. SubframeSelector obviously provides more control as a complete data analysis tool.

What WBPP is not for is integration of light frames. The image integration functionality of WBPP is only intended to provide a preview of the achievable result, but not for production purposes. Unfortunately, it seems we have no efficient way to convince most people that neglecting this fact can affect adversely the quality of their images. If I were to release the first version of BPP now, there would be no lights integration functionality (and I have been seriously tempted to remove it in some updates).

[Juan Conejero]

Automation is useful when most factors are in control. Tracking, focusing, proper acquisition of flats, proper acquisition of darks/flat darks/bias, and correct usage of them during data reduction. In short, it is useful when you know your equipment and when you know how to reduce your data.

Unfortunately what I suspect is most people settle on automatic tools and let human errors or other problems haunt their integrated images for ever, without noticing. For me PI is more of a debugger which allows me to correct my mistakes during acquisition.

I wholeheartedly agree with all of this.

[pscammp]

I would also like to get the best out of my data, as a 'semi beginner' in manual pre-processing I respect Juan's point about final integration of light frames but have no real
clue how to use that information in the real world.

I have always used the following tutorial for manual pre-processing:

https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html

All looks very easy but how do I know if the settings suggested at each step gives me the best possible result for my particular data condition.

Examples:

Master Bias vs Superbias - How can I tell if the Superbias is an improvement over using the standard master bias frame - Will I see it visually, in the statistics module, info
displayed in the process console ? ? ? ?

Image integration - Pixel Rejection settings ? - I could use the default settings suggested in the tutorial but how can I tell if these particular settings end up actually introducing
more noise into the final integration, OR, loosing valuable data and I've not even realized it.

Manual Pre-Processing is, for me, an exciting journey but sometimes I feel like I'm great with the steering wheel but have no clue what those pedals and that gear stick do.

It would be so nice to see video tutorials of these specific parts of the process (in depth) which helps someone like me begin to fully understand when to recognize when I have
just got those individual settings smack on and to know moving to the next stage in the process will be in the knowledge that I have just got the best possible result from the
previous process. Most tutorials I have seen on Youtube for example show loads of people going through manual pre-processing but they never go in depth about how they came
to particular set of settings and why the settings worked perfectly for their data set.

Regards
Paul 
 

[pfile]

for both of your questions - generally speaking we want to maximize SNR in the integrated result with minimal artifacts. so a first test would be whether or not the things you are trying (superbias, rejection) are giving you better SNR than without, with fewer artifacts.

on pixel rejection, you should be examining the rejection maps for signs of structure - if you are integrating an image of a galaxy for instance, and you can see traces of the galaxy itself in your rejection maps, either you have some bad frames which are for whatever reason statistically incompatible (meaning the pixel levels are complete outliers compared with the other frames, even after normalization), or you have too aggressively rejected pixels with the rejection controls.

on superbias i guess i'd try to measure the integrated image with and without superbias to see if you have a higher SNR result (and there are no weird artifacts in the subs or in the integrated result of the superbias integration.)

rob

[dld]

Image integration - Pixel Rejection settings ? - I could use the default settings suggested in the tutorial but how can I tell if these particular settings end up actually introducing
more noise into the final integration, OR, loosing valuable data and I've not even realized it.

Hello Paul,

while I was typing this, Rob gave some good directions. For one longer answer take a look at the corresponding presentation from Jordi Gallego.

While I don't consider myself a great astrophotographer, the best general advice I can give is: assess your sources (how credible is the material you are referring to), study, and try to understand the math. Take it step-by-step and experiment. Pick a single light frame and see if calibration has removed most of the hot pixels, or if it has corrected for dust and vignetting. Avoid complicated workflows involving deconvolution or local normalization. And this is half of the journey. Astrophotography comprises of a highly technical part and an aesthetic part which I believe is even more difficult to conquer.

[ngc1535]

I would also like to get the best out of my data, as a 'semi beginner' in manual pre-processing I respect Juan's point about final integration of light frames but have no real
clue how to use that information in the real world.

Image integration - Pixel Rejection settings ? - I could use the default settings suggested in the tutorial but how can I tell if these particular settings end up actually introducing
more noise into the final integration, OR, loosing valuable data and I've not even realized it.

Manual Pre-Processing is, for me, an exciting journey but sometimes I feel like I'm great with the steering wheel but have no clue what those pedals and that gear stick do.

It would be so nice to see video tutorials of these specific parts of the process (in depth) which helps someone like me begin to fully understand when to recognize when I have
just got those individual settings smack on and to know moving to the next stage in the process will be in the knowledge that I have just got the best possible result from the
previous process. Most tutorials I have seen on Youtube for example show loads of people going through manual pre-processing but they never go in depth about how they came
to particular set of settings and why the settings worked perfectly for their data set.

Regards
Paul

Videos that you want do exist. However, they are hard to explain briefly due to some of the background information that goes with them. For example you mention Pixel Rejection settings. In order to get the best out of Pixel Rejection it is not only important to understand the correct algorithm to use and what parameter thresholds to set... but also important is an understanding of what goes into the set up for Rejection. Image Normalization is a prerequisite for getting a good rejection. Pixinsight (Juan) cleverly separates normalization handling for pixel rejection and output data- to maximize the effectiveness of each. Thus an explanation of location and scale, especially the latter, goes a long way to understanding how the data is being treated internally. The pull down menu for "Scale Estimator" isn't there for show... it has real world meaning and implications!

To my knowledge no YouTube video attempts to explain what a "Iterative k-sigma/biweight Midvariance" is... but I do (or at least do my best to try).  By first explaining the concept of normalization, then scale and finally the algorithms that are used in PI in that drop down- the "pedals and gears" are elucidated. (By the way, thanks goes to Rob, pfile, and Rick Stevenson for helping me get on that right path for this!)

Then you can move (with normalized data) to see how different Pixel Rejection strategies/algorithms work. For example, statistics does give you expectations. If something is 4 sigma from a median/mean- this is equated to unlikely, bad, or invalid. Furthermore the choice of algorithm based on number of measurements is relatively well-defined- but your question goes to the next step and asks "How do I know I have the best possible result?" or "How much rejection should I get? How much is too much? etc.." That is another discussion as well and the basic idea is to maximize the S/N in the final integrated result- but to give you a sense of another expectation...

For a reasonable set of images (say 15-20)  that do not vary by much other than the typical kinds of transient and random fluctuations ( cosmic rays, instrumental stuff..etc) how many pixels would you expect or desire to reject on a per frame basis? There *IS* an answer to this question! There exist rules of thumb for expected rejection given particular kinds of input. The answer is approximately 1-2% and you will typically find this is true when using Sigma based thresholds at around 2-3 sigma when most of the noise is from Poisson statistics and cosmic rays. So you can look in the console at the rejection results for each frame to see if things make sense. Poor frames will show more rejection. The rejection maps give insight in terms of a set rejection, showing a per pixel percentage of rejection and correlations with particular frames (since you can see the "image" that results and how it relates to the integrated result in terms of structure).

-adam