StarNet - new module for star removal

[mmisiura1]

Hi, everyone!

For I while now (more than a year) I have been working on a new process for star removal. It is a specially trained neural network that is supposed to remove all the stars in one simple step and does not require any parameters to tune. It just works (or sometimes it does not ).

Anyway, I released it a while ago and it was shown to work pretty well for most users. I was getting a lot of good feedback and also was asked a lot for a PI module version of the program.

So here is the link to download the new module: https://sourceforge.net/projects/starnet/files/PixInsight_module/

I consider it to be a beta release, so there might be some bugs. Also, one downside for now is that PI freezes and does not reply while the code is running. So keep that in mind and try on a small image at first.

Please read README.txt file for all the instructions and caveats. You can find some more information, discussion and examples of images if you search 'StarNet' on AB or in my releases on AB:

https://www.astrobin.com/409279/
https://www.astrobin.com/339099/


Let me know how it works for you! Also, it would be nice if someone could give me some idea on how to make PI not to freeze while the code is running!

Thank you!

[dld]

Hello Nikita,

I did some experiments with the command line tool and now with the module. It works well with my widefield images. The equation image = starless + stars seems to hold, and can be exploited to separate and treat stars and nebulosity differently. For that reason, I believe StarNet is a solution to the nebula/stars dynamic range problem and not for creating starless images

Thank you for this great tool!

[Bernie]

Hello Nikita

Great job and thank you so much for your work!

- Bernie

[Juan Conejero]

Hi Nikita,

Thank you so much for porting your work to PixInsight. This is a very interesting development project and a great contribution to our platform. Welcome to PixInsight development.

I am here to help and assist you with everything you need to develop this new module. Just a few remarks to help get this project running:

- To help you we need your C++ source code. Assuming that your want to release this module as an open-source project (as you already have done with the Python version), I have invited you to join us at our official GitLab repositories. You should upload your module implementation as a new merge request to the PCL repository.

- Any officially supported PixInsight module must be available on Linux, macOS and Windows at least. Note also that the reference implementation of PixInsight is the Linux version, so a Linux version of your module is necessary as soon as possible. Although I haven't seen your code, the only problem I can detect for this to happen with your module is its strong dependency on TensorFlow. Fortunately, TensorFlow is available as a stable C API on all required platforms, so this problem has an obvious solution. Since it has been released under Apache License, we can integrate the required parts of TensorFlow with our standard PCL distribution, just as a new third-party library. I assume that TensorFlow manages all GPU code and its associated dependencies internally on each platform, so we really don't need to care about this directly—let me know if I am wrong about this (we might have problems in such case).

- I see that only nonlinear (stretched) images can be processed. Is this an unavoidable limitation of the implemented algorithms, or just the result of lack of network training? The ability to work with linear images would be a very important feature of this process (example: deconvolution). If this is an algorithmic limitation I can devise some ways to circumvent it, which would be relatively easy to implement.

Congratulations for making possible a very exciting and useful project. Thank you again for sharing it with the PixInsight community.

[dld]

- I see that only nonlinear (stretched) images can be processed. Is this an unavoidable limitation of the implemented algorithms, or just the result of lack of network training? The ability to work with linear images would be a very important feature of this process (example: deconvolution). If this is an algorithmic limitation I can devise some ways to circumvent it, which would be relatively easy to implement.

For a reasonable invertible stretch function, and assuming StarNet preserves the bit depth of its input, maybe something can be done

I'll experiment on this later today

EDIT/UPDATE:

After stretching a linear grayscale image using a simple MTF function, splitting into stars/background using StarNet, applying the inverse MTF separately at stars and background image, and then adding the result together, I noticed a loss in highlights. The most probable reasons for this are:  It is not easy to find non-trivial functions f for which f(x) + f(y) = f(x+y) (think f being the inverse MTF and x,y the stretched background and stars respectively). My lack of knowledge on imaging processing fundamentals (albeit the documentation on HistogramTransformation was very helpful).

I'll end this here since it's the wrong place for this, and I'll patiently wait for your thoughts on the subject.

[paulsartory]

Love to try this out but I keep getting the "missing checkpoint file" error. Downloaded the zip file and reinstalled the files as per instructions several times but to no avail?

[paulsartory]

I managed to use the command line exe (outside of PI) without a problem and it is very effective

[paulsartory]

Further clarification...the standalone exe is from the windows application not the PI process module..

[paulsartory]

Update....StarNet worked in a new PI "instance" but not within an existing project...I use a default project setup when starting PI for processing.

[paulsartory]

Yet another update !!!....Fired up my Default project setup and it now works fine...not sure why I had to launch and close a new instance for it to work within a project?

[paulsartory]

I think I spoke to soon...seems you still get the "missing checkpoint file" error whenever you run it within a saved project file

[mmisiura1]

I think I spoke to soon...seems you still get the "missing checkpoint file" error whenever you run it within a saved project file

Thank you for your messages, your experiments can be very helpful for me! Could you please give me a recipe how to reproduce this problem? I never used saved project files and not sure what you mean by 'running within saved project file'?

I tried saving project file and then loading it back, and StarNet still worked fine...

Thank you!

[mmisiura1]

- I see that only nonlinear (stretched) images can be processed. Is this an unavoidable limitation of the implemented algorithms, or just the result of lack of network training? The ability to work with linear images would be a very important feature of this process (example: deconvolution). If this is an algorithmic limitation I can devise some ways to circumvent it, which would be relatively easy to implement.

For a reasonable invertible stretch function, and assuming StarNet preserves the bit depth of its input, maybe something can be done

I'll experiment on this later today

EDIT/UPDATE:

After stretching a linear grayscale image using a simple MTF function, splitting into stars/background using StarNet, applying the inverse MTF separately at stars and background image, and then adding the result together, I noticed a loss in highlights. The most probable reasons for this are:  It is not easy to find non-trivial functions f for which f(x) + f(y) = f(x+y) (think f being the inverse MTF and x,y the stretched background and stars respectively). My lack of knowledge on imaging processing fundamentals (albeit the documentation on HistogramTransformation was very helpful).

I'll end this here since it's the wrong place for this, and I'll patiently wait for your thoughts on the subject.

What you describe is actually one approach I was thinking of, but didn't get to try yet, so your input can be really useful here. Not sure why you would want to add back stars, however?) I thought that you just stretch the image, remove stars and then 'destretch' result back into its original form. Then, assuming that all the nebulosity is dim, you would get good image.

Anyway, I didn't think too much about it for now, but definitely something I should try and find out.

Thank you very much for your input! Feel free to let me know about your further experiments, if you want!

[mmisiura1]

Juan,

Thank you vary much! I am glad to be helpful for the community! So far working with PCL was a real pleasure, BTW, so writing the existing code took not that much effort as I thought it would initially. And I am not too much of a C++ programmer really, this is basically my first code in C++ ever.

Compiling for different platforms is a biggest issue for now. I wonder, how do you guys do it? Do you have a few separate machines with different OS and compile binaries using those? Are there no shortcuts?

As far as I understand, tensorflow should not be a problem, it handles everything internally as you said. Don't see any problems here.

The fact that current implementation does not support linear images is that I could not get training data for those. As you probably know, to train a good neural network, you need a hefty amount of training data. In this case I had to spent countless hours removing stars in Photoshop by hand. Those were obviously stretched images. I don't know how to do this for linear data. Note, that using some other methods rather than 'by hand' will most likely leave artifacts and NN will learn to reproduce those artifacts. In other words, results generated by NN will always be worse than training data, so to get good results in this way, the training data should be next to perfect. This is one limitation I know of, there should be no technical limitations, but I can not say for sure right now.

[paulsartory]

Hello Nikita,

The error occurs if you launch Pixinsight by clicking on the project file rather launching Pixinsight first.

Cheers
Paul