Opinion and Suggestion: What PI Needs . . .

[Terry Danks]

In the announcement thread, I posted my disagreement with the approach that would seem to be taking PI in the direction of becoming an all-inclusive vehicle for hardware control and data acquisition, as well as for processing. If I sounded churlish in doing so, my apologies.

I appreciate there are tutorials on how to use PI . . . and even a small industry that produces same. I have watched many, and subscribe to one. I am grateful for all of them!

That said, I have not found what I feel really should be available. The tutorials presently available are very "tool-oriented" as opposed to "project-oriented" as I see it. What I feel is needed is an advanced series of tutorials that take a data set for a specific type of DSO and process it through from beginning to end. As I see it, about four are required, one each for galaxies, emission nebulae, globulars and galactic open clusters. Each of these objects, while similar in demands in many aspects, have some specifics more important to one than the others.

The ideal would be that all who desire to do so, be able to attend a PI workshop. I am sure I speak for many when I say "ain't gonn'a happen" for me.

Why could not these workshops be put on-line for those of us unlikely to ever see a workshop even in our own country, let alone our local area.
Money is not the issue. Charge for them by all means! I can't be the only individual more than willing to pay to access such.

I've been using PI for years, but still feel I have not scratched the surface of what could be done with my data by a true master of the software.

[miska]

I feel that PI is a very nice collection of tools, but:

- There is little done to help making a workflow. Although BPP is a step in that direction. It is a bit like Photoshop. A LOT of tools that do similar things, but differently. I prefer lightroom (LR), which guides you more, with perhaps fewer tools but better integrated into a workflow, getting you from the raw data to the final product.

- The Documentation, when it is present at all, is more like a technical manual. It doesn't help much in really using a tool for a particular case. A LOT of experimentation is necessary to get a tool to give a sensible result.

- A lot of the tools are unnecessarily complex and non interactive. It is like folows: set  a parameter - try - set parameter - try again - repeat 10 times. I would much prefer more live previews and sliders. Compare the noise reduction algorithm in LR with a few sliders where you see immediately what happens to the various noise reduction algorithms in PI. Although the PI ones are probably much "better" than the LR ones, it takes forever to get it right. More sliders and interactive previews please !

- I feel that PI, right now, is a tool for those who want to spend hours and hours optimizing each picture. I would prefer (but perhaps I'm the only one ?) to spend  (much ?) less time and get the image 80% or 90% right. Instead of spending a LOT of time, and getting it 100% right. So I'd like to have more tools that work "almost" with the default settings, and need much less tweaking. I am sure this is possible, but it needs a big investment from the software team, in testing various cases on different images, to come up with good default values, or automatize some parameter optimizations.

- I would LOVE presets and "assistants". Something that analyzes my images and figures out the right parameters (or almost the right ones) for my image.

- Performance is not very good in terms of computation time. PI takes a LOT of ressources, and I don't have the impression they are optimally used. This is perhaps not true, it's just a feeling. Certainly BPP is MUCH slower than DSS for example, for the same set of images.

So in summary: I would like the PI team to direct their attention to usability and automation, and possibly performance improvements.
Not really interested in completely new algorithms to replace existing ones, or telescope / observatory control software.

Just my 0.02 Euros worth...

[vicent_peris]

I feel that PI is a very nice collection of tools, but:

- There is little done to help making a workflow. Although BPP is a step in that direction. It is a bit like Photoshop. A LOT of tools that do similar things, but differently. I prefer lightroom (LR), which guides you more, with perhaps fewer tools but better integrated into a workflow, getting you from the raw data to the final product.

- The Documentation, when it is present at all, is more like a technical manual. It doesn't help much in really using a tool for a particular case. A LOT of experimentation is necessary to get a tool to give a sensible result.

- A lot of the tools are unnecessarily complex and non interactive. It is like folows: set  a parameter - try - set parameter - try again - repeat 10 times. I would much prefer more live previews and sliders. Compare the noise reduction algorithm in LR with a few sliders where you see immediately what happens to the various noise reduction algorithms in PI. Although the PI ones are probably much "better" than the LR ones, it takes forever to get it right. More sliders and interactive previews please !

- I feel that PI, right now, is a tool for those who want to spend hours and hours optimizing each picture. I would prefer (but perhaps I'm the only one ?) to spend  (much ?) less time and get the image 80% or 90% right. Instead of spending a LOT of time, and getting it 100% right. So I'd like to have more tools that work "almost" with the default settings, and need much less tweaking. I am sure this is possible, but it needs a big investment from the software team, in testing various cases on different images, to come up with good default values, or automatize some parameter optimizations.

- I would LOVE presets and "assistants". Something that analyzes my images and figures out the right parameters (or almost the right ones) for my image.

- Performance is not very good in terms of computation time. PI takes a LOT of ressources, and I don't have the impression they are optimally used. This is perhaps not true, it's just a feeling. Certainly BPP is MUCH slower than DSS for example, for the same set of images.

So in summary: I would like the PI team to direct their attention to usability and automation, and possibly performance improvements.
Not really interested in completely new algorithms to replace existing ones, or telescope / observatory control software.

Just my 0.02 Euros worth...

Hi,

I think there are basically three points you don't understand:

- PixInsight is designed to let people to understand what they are doing. It's not (and never will) based on black boxes.

- PixInsight is actually very well optimized. It's not slow at all. The extra time to apply some of the processes is what let's you have better images. For instance, you'll get *always* better FWHM values and noise levels by preprocessing it with BatchPreprocessing, compared to DSS. If it takes more time it's not because it's slow, but because it's more sophisticated.

- You can get a pretty decent image with very simple workflows in PixInsight. Some of them are actually much easier to apply in PixInsight than in the competition software packages.


Best regards,
Vicent.

[troypiggo]

...
- There is little done to help making a workflow. Although BPP is a step in that direction. It is a bit like Photoshop. A LOT of tools that do similar things, but differently. I prefer lightroom (LR), which guides you more, with perhaps fewer tools but better integrated into a workflow, getting you from the raw data to the final product.
...
- I would LOVE presets and "assistants". Something that analyzes my images and figures out the right parameters (or almost the right ones) for my image.
...

I have a set of process icons that I've saved - when loaded they're down the right hand side of the workspace, and from top to bottom they're in the order of processing required.  99% of the time, if I just work my way down that list, the end result is a decent image that I'm usually quite happy with.  If not, then I can go back and tweak some things in between if necessary.  But they do follow what I think you're referring to as a "workflow".

Have you tried something like that?  Kind of presets and workflow in one.

[mschuster]

miska, to save time, you might try focusing on a few processes that make a big difference, and don't bother with others. This will free up time to work with your occasional, "special" image that you really want to tweak. For example, for my Ha blog images, I found that Deblur, STF, CT, LHE, and nothing else suffices for me.

Regards,
Mike

[msmythers]

I myself am very happy with PI. For most images that amateurs take I don't believe there is a set workflow that can work every time. There are to many variables that change that we can't control.

That being said I have a very simple process icon for DSLR images that I use as a quick image check. It uses 6 tools in this order.
Crop   Aggressive crop of possible bad edges due to stacking and other errors.
ABE    Not using default settings. Background is neutralized for sudo color balance.
ColorCalibration  Using defaults. Not a real color balance but gives a good enough result to look for issues.
MaskedStretch    Not using defaults. Only 50 iterations. Higher target background while applying a little clipping.
Curves   Change Contrast and color saturation.
ACDNR  Some noise reduction of both lightness and chroma.

That the whole post processing. This quick and can be used to identify areas in the image that need work and plan for the proper tool to work with.

 
 Mike

[miska]

Oh, don't get me wrong. I do like some of the tools in PI, I wouldn't have paid a LOT of money for PI without that.

But when I compare for example how it is difficult to test some parameters in noise reduction (even if I can limit those tests to previews) to how easy it is to apply a noise reduction with a mask in LR. Well, it's no contest. So yes, for sure the LR noise reduction is for cave-men (although...), there must be a way to make it more user friendly.

I have now found "my" workflow, and I can work:
- BPP
- DBE, and background neutralization
- Histogram transform
- HDR-wavelet thingy
- Some noise reduction (wavelets, or another form).
- TIF, to get me out of here for the fine retouching :-)

But thanks guys for the suggestions !

> I think there are basically three points you don't understand:

What I don't understand, is the frankly condescending attitudes of some high up PI staff, and this is not limited to this comment in this thread.
If I didn't "understand" that PI is not a hallmark of software usability, I apologize. It must be that my abilities to understand this kind of thing is limited.

This thread is about opinions on how to make PI better. I gave my suggestions. If they are not shared by anybody, fine. But dismissing them as lack of understanding is just showing a lack of listening skills...

Cheers,

Miska

[Juan Conejero]

I always appreciate all attempts to help us. However, many of the statements and requests here seem to miss the point of the PixInsight project to a large extent. Perhaps the best way to understand why PixInsight is as it is, and why it has been designed and is being developed that way, is asking: How can I enjoy it? You can enjoy PixInsight if you basically:

- Understand astrophotography as a means to grow your knowledge and skills, both technically and aesthetically.

- Know that astrophotography is a very demanding discipline requiring knowledge, patience and perseverance.

- Pursue the excellence.

- Like complexity and diversity.

- Like analyzing and solving problems.

- Are open to different ways of solving problems.

- Are more interested in the process than in the end result.

- See image processing as an enjoyable task.

- Want to learn something new each time you process an image.

- Want to respect the original data to the maximum possible extent.

- Want to solve image processing problems algorithmically.

- Want to have full control on every aspect of your image processing.

- Want to make your own decisions, including the possibility to make mistakes.

- Are not scared about following long paths to find solutions to complex problems, including intensive trial/error work.

- Like science, technology and development.

- Value professional, multi-platform, innovative and comprehensive software development projects.

Clearly, this excludes simplified "one-button", "we think for you", painting and retouching solutions. If you want to paint your images or make them up easily with arbitrary hand-painted masks, layer tricks, magical recipes and other "master lessons" on brushes and pencils, then definitely PixInsight is not for you.

If you think that PixInsight is expensive, then you should rethink what you want to achieve with astrophotography. Freeware is very popular because it is free (gratis) and hence seen as "a work of love" by most, but professional software development of a large and complex project like PixInsight cannot exist without remuneration. If you really think that 230 EUR (about 260 USD as of writing this) is too much for PixInsight, then IMHO, there is something basically wrong—irrespective of whether you like PixInsight or not, and irrespective of whether you can afford it or not—in the way you understand this discipline.

[Cosmick]

- Understand astrophotography as a means to grow your knowledge and skills, both technically and aesthetically.

- Know that astrophotography is a very demanding discipline requiring knowledge, patience and perseverance.

- Pursue the excellence.

- Like complexity and diversity.

- Like analyzing and solving problems.

- Are open to different ways of solving problems.

- Are more interested in the process than in the end result.

- See image processing as an enjoyable task.

- Want to learn something new each time you process an image.

- Want to respect the original data to the maximum possible extent.

- Want to solve image processing problems algorithmically.

- Want to have full control on every aspect of your image processing.

- Want to make your own decisions, including the possibility to make mistakes.

- Are not scared about following long paths to find solutions to complex problems, including intensive trial/error work.

- Like science, technology and development.

- Value professional, multi-platform, innovative and comprehensive software development projects.

As my favorite writer about image processing software would say -

"Every image - an adventure"

 

[vicent_peris]

But when I compare for example how it is difficult to test some parameters in noise reduction (even if I can limit those tests to previews) to how easy it is to apply a noise reduction with a mask in LR. Well, it's no contest. So yes, for sure the LR noise reduction is for cave-men (although...), there must be a way to make it more user friendly.

Hi,

Try TGVDenoise. First, you should consider to work in the RGB color space or the CIE L*a*b. You'll need to work in the CIE L*a*b when your lightness is very clean and your chrominance is very noisy. Then, you'll find most of the time a good solution simply by changing the "Edge protection" value. This tool is pretty easy to use.

V.

[ruediger]

I think that there is not too much freedom at least when working with linear data and for every picture I find myself repeating the same dull tasks that could easily be simplified.

Examples:

1 removing satellite trails
To maximize SNR and raise sigma values in ImageIntegration I started to remove satellite trails from the calibrated images. The best way I know to accomplish this task is: enlarge picture, scroll, move mouse to 1st point of satellite trail, memorize x,y, enter x,y in a PixelMath expression based on d2line, scroll, move mouse to 2nd point, memorize, enter x,y, execute PixelMath expression.
-> why is there no easy way to draw a line with simply clicking on start and end point in the picture?

2 Deconvolution
Open DynamicPSF, enlarge picture, scroll through picture, click on 60 to 70 stars manually, sort the list, select best 50 stars, generate PSF, open Deconvolution
-> why do I have to pick stars? PixInsight should know better where suitable stars are located for generating the PSF. Imagine the picture is plate solved, then I would expect a high level function like "select all stars with magnitude 14 to 18" or something.
The Deconvolution needs a DeringingMask -> another boring task which could be automated, followed by trial&error of other settings like the NoiseThresholds where I would expect good starting points are generated from picture data automatically.

3 Using TGVDenoise in linear stage
A powerful tool, but seriously: Is anyone there using it in a different way than the following which was shown here by forum member Philippe B:
- define a rectangle with only background, measure avgdev, memorize the value, enter value *0.1 as edge protection value
- open STF setting, copy shadowing clipping, close STF setting (it's a modal dialogue), use as TGVDenoise local support shadows, same steps for midtones setting
- generate a 50% mask with PixelMath
- execute TGVDenoise
-> especially the "50% mask trick" is so useful, but so inconvenient. I would like to have an easy way to execute a process and afterwards reduce it's strength comfortably with a slider. Doing it manually (cloning image, enter PixelMath, trial&error of blending strength) is another dull task.

I don't see these steps (and some others) as "enjoyable tasks".

Once leaving linear stage, it's getting more enjoyable


Rüdiger

[rockyraccoon]

Requests for more documentation have been coming in for years, and even casual browsing of astrophotography forums will reveal that the biggest "gripe" associated with PI is a steep learning curve and undocumented features and workflows.
At this point it's clear that the dev team has no intention to strategically address this shortcoming, and let the user community help itself by publishing tutorials (both free and paid), cheatsheets, and forum threads.
Personally I think some of the processes in PI could use demystification - especially those where more than two parameters need to be adjusted at the same time to values that do not immediately make much sense to anybody who hasn't had the chance to look "under the hood".
One example is the "TGVDenoise" tool. Clearly very powerful - but does it really need 4-5 adjustable parameters? and who knows why "edge protection" defaults to 0.002 (how does that number relate to anything else in the picture one is trying to denoise).
There is great value in hiding complexity from the user, and that does not have to come in the form of trivializing a tool.

In the announcement thread, I posted my disagreement with the approach that would seem to be taking PI in the direction of becoming an all-inclusive vehicle for hardware control and data acquisition, as well as for processing. If I sounded churlish in doing so, my apologies.

I appreciate there are tutorials on how to use PI . . . and even a small industry that produces same. I have watched many, and subscribe to one. I am grateful for all of them!

That said, I have not found what I feel really should be available. The tutorials presently available are very "tool-oriented" as opposed to "project-oriented" as I see it. What I feel is needed is an advanced series of tutorials that take a data set for a specific type of DSO and process it through from beginning to end. As I see it, about four are required, one each for galaxies, emission nebulae, globulars and galactic open clusters. Each of these objects, while similar in demands in many aspects, have some specifics more important to one than the others.

The ideal would be that all who desire to do so, be able to attend a PI workshop. I am sure I speak for many when I say "ain't gonn'a happen" for me.

Why could not these workshops be put on-line for those of us unlikely to ever see a workshop even in our own country, let alone our local area.
Money is not the issue. Charge for them by all means! I can't be the only individual more than willing to pay to access such.

I've been using PI for years, but still feel I have not scratched the surface of what could be done with my data by a true master of the software.

[mmirot]

I think that there is not too much freedom at least when working with linear data and for every picture I find myself repeating the same dull tasks that could easily be simplified.

Examples:

1 removing satellite trails
To maximize SNR and raise sigma values in ImageIntegration I started to remove satellite trails from the calibrated images. The best way I know to accomplish this task is: enlarge picture, scroll, move mouse to 1st point of satellite trail, memorize x,y, enter x,y in a PixelMath expression based on d2line, scroll, move mouse to 2nd point, memorize, enter x,y, execute PixelMath expression.
-> why is there no easy way to draw a line with simply clicking on start and end point in the picture?

2 Deconvolution
Open DynamicPSF, enlarge picture, scroll through picture, click on 60 to 70 stars manually, sort the list, select best 50 stars, generate PSF, open Deconvolution
-> why do I have to pick stars? PixInsight should know better where suitable stars are located for generating the PSF. Imagine the picture is plate solved, then I would expect a high level function like "select all stars with magnitude 14 to 18" or something.
The Deconvolution needs a DeringingMask -> another boring task which could be automated, followed by trial&error of other settings like the NoiseThresholds where I would expect good starting points are generated from picture data automatically.

3 Using TGVDenoise in linear stage
A powerful tool, but seriously: Is anyone there using it in a different way than the following which was shown here by forum member Philippe B:
- define a rectangle with only background, measure avgdev, memorize the value, enter value *0.1 as edge protection value
- open STF setting, copy shadowing clipping, close STF setting (it's a modal dialogue), use as TGVDenoise local support shadows, same steps for midtones setting
- generate a 50% mask with PixelMath
- execute TGVDenoise
-> especially the "50% mask trick" is so useful, but so inconvenient. I would like to have an easy way to execute a process and afterwards reduce it's strength comfortably with a slider. Doing it manually (cloning image, enter PixelMath, trial&error of blending strength) is another dull task.

I don't see these steps (and some others) as "enjoyable tasks".

Once leaving linear stage, it's getting more enjoyable


Rüdiger

I agree there  some things that should be addressed to make the work flow easier.

The STF needs improvements.

1. The auto STF sliders that are revealed on control click should be available for real time adjustments.
2. Send the current STF to other images in the work space.
3. Apply the STF to the histogram tool (creating a nonlinear image) for multiple selected images at a single go.
4. If we apply a histogram stretch,  the resulting image should deactivate the STF automatically. 

Personally, I don't type very well.  So I have often thought that some of Pixelmath's operations could be done without typing.
For example adding or subtracting images with a  weighting factor could be done with just a few clicks. The time involved typing adds up if I misspell an image name.
I find myself changing names to "A and "B" just to prevent this. That should not be necessary.  Improvements along these lines are only beneficial and should not be seen as dumbing down the platform.   

Setting some of the default setting to a value closer to the range where they are often applied is a well deserved tweak. Perhaps even order the sliders such that uncommonly used setting are on the bottom.

Auto selection of stars in dynamic PSF is good idea too.

Max

[RickS]

Personally, I don't type very well.  So I have often thought that some of Pixelmath's operations could be done without typing.
For example adding or subtracting images with a  weighting factor could be done with just a few clicks. The time involved typing adds up if I misspell an image name.
I find myself changing names to "A and "B" just to prevent this. That should not be necessary.

Have you tried using the PixelMath Expression editor, Max?  You can select the image names from a drop down menu on the top right hand side.  I also copy and paste image names (double click on the Window name, Ctrl-C or non-Windows equivalent to copy the text, cancel the dialog then paste image name into PixelMath.)

Cheers,
Rick.

[jkmorse]

Many of us came to PI from other, unnamed and often outrageously expensive, alternatives for the reasons Juan cites, most particularly for me:

- See image processing as an enjoyable task.

- Want to learn something new each time you process an image.

- Want to respect the original data to the maximum possible extent.

- Want to have full control on every aspect of your image processing.

- Want to make your own decisions, including the possibility to make mistakes.

Of these the most critical for me is being true to the data.  Like the Mikes and others, I have a set workflow that I use for 90% of my processing, but I always know I can turn to PI's incredibly powerful tools to solve some unique problem or to pull out the extra bit of data that makes an image special.  A "cookie-cutter" approach would, in my mind, go against everything PI stands for, especially in regards to being true to the data.  I have any number of images that I am still working on and my workbook that I share with others free of charge is now up to Rev39a, but I wouldn't have it any other way because every minute Juan and his team spend writing tool descriptions is a minute they aren't writing code.  I  want them writing code and I will take care of figuring out how to use what they produce, admittedly with lots of trial and error.  I think all the people writing the workbooks, cheat sheets and producing tutorials show the love the PI community has for this product and our contributions are a way to give back to the community of PI users and to share some of the burden Juan and his team have taken on. 

Just don't fool yourself into thinking this is an easy road.  That's what programs like LR are for and we all start there.  But eventually it just isn't enough for what we want to achieve and so we come here.  Its a steep climb and many of us are much closer to the bottom than the top, but, IMHO, that's what the science of astrophotography is all about.

For what it's worth.

Best,

Jim