Newbie amazement... and frustration

[calan]

After a couple hundred hours of watching tutorials, reading, and experimenting with countless processes and parameters (I live alone and have no life ), I'm absolutely amazed at the power of this software. I've been able to get some very surprising images out of data that I assumed was completely unusable.

With that said...

Making a simple star mask is mind-numbingly complex and frustrating. I've messed around with the star mask tool until I have nightmares about it, and I still have no clue what it's doing. Every parameter seems to affect every other parameter, and it's completely unpredictable. Just when I think I have it figured out...the next time I use the saved process, it's nowhere close to what I expected.

Setting a large scale of 1 more often than not gives me ugly square stars...unless I change some other parameter. Then suddenly large scale doesn't seem to do anything at all. Truncation works great to solidify the star centers; until it unpredictably changes the overall size. Sometimes adjusting the midtones can be used to filter out smaller stars...until it doesn't because of some other parameter. The only parameters I've gotten to work predictably are the noise level and smoothness, and I have never been able to get compensation and small scale to make any noticeable difference at all.

Why all the complexity and interaction between parameters? How about just clicking any two stars in an image, and saying "give me a mask of all stars between these two diameters". Add in a truncation parameter that doesn't affect the star's size (diameter), keep the smoothness parameter as it is, and you're pretty much there. (Being able to click on stars to exclude or include them on the fly before finalizing the mask, similar to the way DBE works, would be a nice bonus). Three mouse clicks and you pull out a mask that is basically a duplicate of a range of selected stars, or adjust truncation and smoothness as desired first. Done.

The range mask is nice, and I'm learning quite a few other tricks for mask creation using some of the MS processes, luminance masks, etc. But making a simple, accurate star mask is still incredibly tedious and WAYYYY more time consuming than it needs to be. Granted I've barely scratched the surface of PI's capabilities, but I probably spend 80% of my processing time trying to make masks, and the other 20% on color calibration, deconvolution, detailing, noise reduction, and everything else.

Ok.. rant off. This software is amazing and I'm seriously impressed every time I use it. But please, please give us a better star mask tool. 

[Niall Saunders]

First - don't worry, you are not having a rant (and, anyway, rant are encouraged here - providing they have justification, as in your case)

Second, you are not spending nearly enogh time trying to learn PixInsight - I assume you are still wasting precious PI time doing nonsensical things like eating and sleeping. Given that you have wisely already eliminated friends and family, perhaps you might try focusing just a little bit harder, and eliminate eating and sleeping as well (you will also find that toilet-breaks become les necessary as well) 

I know it is hard, I have been a PixInsight user for years and years, but ill-health forced me away from the PC for about a year, and I have found that - like you - I am more or less starting from the beginning all over again.

What I cannot unfortunately help you with is star-masks!! I have managed to avoid these - but maybe that is why my images still leave something to be desired. You are doing the right thing though (required by many of PixInsight's more complex processes) and that is you are slowly, and carefully, experimenting with all the options to try and determine the connection between them, and their resulting effect on your image.

Would it be easier if the processes were less complex? Of course it would, but would you then still have the same level of control?

One thing that perhaps wasn't clear in your post was whether you were expecting a star-mask process that 'works' on one image to then 'work' in the same way on an entriely different image. Often as not this will not be the case (hence the reason the processes are so often 'complex')

Keep trying - I am ten days (and 160 hours) into post-processing an image at the moment, and yesterday eventually decided I was chasing around in circles and shut down PixInsight, clearing the entire workspace, ready for a fresh start later today.

So, you are not alone!

[calan]

First - don't worry, you are not having a rant (and, anyway, rant are encouraged here - providing they have justification, as in your case)

Second, you are not spending nearly enogh time trying to learn PixInsight - I assume you are still wasting precious PI time doing nonsensical things like eating and sleeping. Given that you have wisely already eliminated friends and family, perhaps you might try focusing just a little bit harder, and eliminate eating and sleeping as well (you will also find that toilet-breaks become les necessary as well) 

I know it is hard, I have been a PixInsight user for years and years, but ill-health forced me away from the PC for about a year, and I have found that - like you - I am more or less starting from the beginning all over again.

What I cannot unfortunately help you with is star-masks!! I have managed to avoid these - but maybe that is why my images still leave something to be desired. You are doing the right thing though (required by many of PixInsight's more complex processes) and that is you are slowly, and carefully, experimenting with all the options to try and determine the connection between them, and their resulting effect on your image.

Would it be easier if the processes were less complex? Of course it would, but would you then still have the same level of control?

One thing that perhaps wasn't clear in your post was whether you were expecting a star-mask process that 'works' on one image to then 'work' in the same way on an entriely different image. Often as not this will not be the case (hence the reason the processes are so often 'complex')

Keep trying - I am ten days (and 160 hours) into post-processing an image at the moment, and yesterday eventually decided I was chasing around in circles and shut down PixInsight, clearing the entire workspace, ready for a fresh start later today.

So, you are not alone!

I know what you mean by going in circles. I usually make small incremental improvements as I discover some new tool or procedure, but I will also spend hours on redoing something, and then when I compare it to the previous version it's actually worse.  LOL

On the star mask process...  I don't expect it to be directly transferable; I understand that every image has it's own personality. I just wish the tool itself was more predictable. It is always a complete guessing game as to what you'll (I'll) get with it, unless you never change anything but one parameter. And even then it's not predictable. 

[chris.bailey]

One of the 'problems' with StarMask is that it is quite slow so adjusting settings by trial and error is a time consuming business. One thing that can speed up the process is to make a preview that include the range of star size in your image and drag the preview tab onto the desktop to create a new image. Do your trials on this and when happy, apply to the full image.

Chris

[msmythers]

I made a little video about making a simple star mask from a linear image. This is not a mask made from a stretched clone but the actual linear image. This is a bit of a different approach as I make use of the readout cursor and Statistics to set the Mask Preprocessing parameters.

The video has several examples using different images. In the video I will use the star masks to lower the brightness of the stars in the linear image. This video does not have audio as I have problems with speech. I did annotate some of the video at the beginning.

As Chris mentioned trial and error is a big part with learning the Star Mask tool. Using a small representative preview is a great way of testing parameters. I have been using PI since 2012 and just now feel like I have a good grasp on this wonderful tool. 

One more thing if the video seems blurry just pause and restart the video. Please feel free to download the video.


https://drive.google.com/file/d/0B13r3kYqTf8Nb3VnSVhHOVRabGc/view?usp=sharing


Mike

[Juan Conejero]

Just for the record, I agree completely with your criticism. It is a proven fact that StarMask needs an urgent revamp. This tool requires a complete redesign/rewrite, actually. It is too slow, given its complexity, and I have now much more accumulated knowledge and resources to write a better tool (the original StarMask implementation is from 2006 or 2007 if I remember well). This is definitely in the to-do list, but don't hold your breath because I have a huge amount of pending work.

Mike, very good video. I like the way you define StarMask parameters based on image statistics. The masks you've generated are very accurate.

[calan]

Thanks Mike, I'll check it out.

I've made literally hundreds of masks while experimenting. I feel like I have a good idea of what the parameters do...the problem is how they interact. It's just extremely frustrating to get a predictable set of stars at a given size, brightness, and smoothness.

I also use the readout to measure stars and noise level, etc., and that works well for setting the noise and shadow levels. But after that, all bets are off once you try to start narrowing in on a certain opacity or star size. It rapidly turns into a recursive convoluted process of using sub-masks, pixel math, clone stamp, and range mask to hopefully end up with what you were shooting for, and by the time you do it is so complicated that you can never repeat it. And then you start the whole process all over again from scratch when something in the image changes or for different images of course.

My current workflow involves a "mask building session" right after I do a rough deconvolution to get precise star centers. I then spend time building a generic map and as many masks as I think I'll need. Sometimes it works out well, sometimes not. Either way it is tedious and all the trial and error isn't a lot of fun. 

EDIT:

Thanks Juan. I hope it moves up the list; so far I haven't found anything while processing a typical DS image that is more important than being able to build a reliable star mask. 

I'm really enjoying PI and will be purchasing in the next couple of weeks, and I eagerly look forward to whatever you come up with.

[msmythers]

Thanks Juan.

Since I shoot a lot with a small focal length camera lens learning how to make an accurate star mask was a must. Using the Statistics tool and the readout is so easy. You and the team have given us tools that can be used with great precision compared to black box, one size fits all solutions.

Regards


Mike

[ecorm]

I'm also finding star masks to be extremely difficult, tedious, and frustrating. It's probably the main reason why I have a year's worth of unprocessed data sitting on my hard drive - I'm dreading having to deal with star masks.

PixInsight already knows how to compute the PSF of stars. It can also automatically detect stars in the StarAlignment process. Why then do we need to do all this manual tweaking of knobs that can never seem to converge on an ideal star mask? It wouldn't be so bad if StarMask wasn't so darn slow to run, even on small previews.

Shouldn't development priority go towards improving things that make users waste the most of their time? Star masks are THE single most time-consuming thing I have to do when post-processing images.

[Juan Conejero]

I understand your frustration, and as I said two years ago, StarMask is one of the tools in urgent need of a complete redesign/reimplementation. I am conscious of this problem and will try to solve it as soon as other priority tasks allow me to do so.

Shouldn't development priority go towards improving things that make users waste the most of their time?

Unfortunately, things are much more complex than what can be seen on the surface by looking at specific problems. There are many aspects in which PixInsight is weak and must be made much stronger to survive, now and in the medium-long term. PixInsight is a very difficult project and there is a thin line between being and not being viable. My daily work is so that this line does not disappear, or at least to make it very difficult for the people trying to erase it.

That said, star masks are not so difficult if you use the appropriate tools beyond StarMask in PixInsight. For example, you can use multiscale analysis tools such as MultiscaleLinearTransform to generate star masks much more efficiently in many cases. You can also use HDRMultiscaleTransform to preprocess an image before star mask generation, which simplifies the StarMask task by solving the implicit dynamic range problem. MorphologicalTransformation and HistogramTransformation can be used to refine an existing star mask, adapting it to specific requirements. All of these tools and techniques can make the star mask generation task much more tractable and efficient, instead of trying to use just the StarMask tool alone. On the other hand, people often tend to rely on star masking excessively in my opinion; a different way to understand the image, and a change of focus and/or goals may also help many times.

If you have a specific image where you are having trouble with star masks, I'll be glad to take a look and make suggestions if you upload the relevant data. I am sure other forum users will also be glad to help you with this.

[ecorm]

Thank you for the reply and for the suggestions. I have tried pre-processing with MultiscaleLinearTranform and HistogramTransformation, and that made things easier (still not easy).

Have you considered a different approach where each detected star is modeled by PSF parameters (the same way it does via DynamicPSF)? You could combine this with whatever technique you use for star detection in StarAlignment.

Once the PSF is calculated for each star, users can then specify by how much the star mask should extend over star, and how quickly it should fade away. There could be a preview for stars of various sizes.

The intermediate data containing the PSF paramaters for every detected star could be saved in a special file. This would allow the user to rapidly generate different star masks based on the same star data.

Please forgive me if the above suggestion sounds naive.

[pfile]

with all the new astrometry stuff in PI it seems like one could even generate a star mask from PPMXL, but i guess you'd have to do some optics modeling to come up with the right FWHM, and impose magnitude limits.

rob

p.s. a couple of other star mask ideas - you can try AdvStarmask.js from https://www.skypixels.at/pixinsight_scripts.html. also one technique is to somehow generate a starless image and take the difference between the original and starless image to get an image of just the stars. additionaly the python version of starnet produces a star mask image.