StarNet - new module for star removal

To Nikita and all users interested in this module,

Sorry so much for not being able to invest the required time and resources to put this module in the standard tool set. Nikita already uploaded his source code to our GitLab repository, and I have been taking a look. The module requires substantial work, but this is not difficult, just time-consuming. Perfectly normal for a new PixInsight tool.

The real problem here is TensorFlow. This is an extremely complex and huge software library (undoubtedly one of the most complex ones I've ever seen). There are several problems. Once of them is building TensorFlow on all platforms supported by PixInsight, which is not a trivial task. It is a very complex task that requires time and care. Unfortunately, the prebuilt libraries provided by Google don't work, so the only way to get this working is rebuilding everything.

Another problem is TensorFlow's dimensions. If included in PixInsight's standard distribution, it would increase the size of a PixInsight installation by about 300 MB. A fixed footprint like that is questionable to support a single module. A good solution to this problem is creating a dedicated update repository for StarNet on our official software distribution server, so the StarNet module would be an optional component. For example, the same will happen with the new Gaia star databases that we are preparing for version 1.8.7 of PixInsight.

Please bear with me while I can find the required time to perform all of these tasks as required. Currently we have a lot of priorities that require all of my attention and concentration. The new version of PixInsight comes with very important improvements (A rewritten StarAlignment tool with a completely new distortion correction algorithm, important new features in ImageIntegration and DrizzleIntegration, just to name a few) on which I have been working very hard during July, August and September. I ask for patience...
 
Juan Conejero said:
...The real problem here is TensorFlow. This is an extremely complex and huge software library (undoubtedly one of the most complex ones I've ever seen). There are several problems. Once of them is building TensorFlow on all platforms supported by PixInsight, which is not a trivial task. It is a very complex task that requires time and care. Unfortunately, the prebuilt libraries provided by Google don't work, so the only way to get this working is rebuilding everything.

Another problem is TensorFlow's dimensions. If included in PixInsight's standard distribution, it would increase the size of a PixInsight installation by about 300 MB. A fixed footprint like that is questionable to support a single module. A good solution to this problem is creating a dedicated update repository for StarNet on our official software distribution server, so the StarNet module would be an optional component. For example, the same will happen with the new Gaia star databases that we are preparing for version 1.8.7 of PixInsight.
....
Click to expand...
Juan,

Yes, this is a real problem, and I have encountered it elsewhere as well. If you try to do everything in-process (i.e. inside the PixInsight binary), then each and every library that you use needs to be binary compatible, i.e. has to be built with a compatible compiler, and the library itself and all the libraries that it depends on need to be compatible with PixInsight and its libraries. For Tensorflow (and Python, and numpy, and scipy, and astropy, and ....) the number of dependent libs is huge.

A minimal installer for Tensorflow with Python on Linux is something like 320MB, and if you add the usual scientific stuff, you are at 640MB (compressed). Installed you are at 3.2GB.

A possible solution: Run Tensorflow et. al as an external process, and communicate with them via some RPC mechanism (like gRPC, Python pickle or the mechanism you designed for communication between PI instances). Let users install Tensorflow as an addon (for instance via the Anaconda distribution https://www.anaconda.com/distribution/, very liberal license and rather complete), which also removes from you the burden to compile all this stuff with PI.

Georg
 
I just installed PI Linux module and if I use StarNet++ process to image I get an error message:


Restoring neural network checkpoint...
*** Error: Checkpoint file not found!


I got it working after executing following command in ProcessConsole first:

cd /opt/PixInsight/bin

Then Stanet++ process woks.
 
georg.viehoever said:
A possible solution: Run Tensorflow et. al as an external process, and communicate with them via some RPC mechanism (like gRPC, Python pickle or the mechanism you designed for communication between PI instances). Let users install Tensorflow as an addon (for instance via the Anaconda distribution https://www.anaconda.com/distribution/, very liberal license and rather complete), which also removes from you the burden to compile all this stuff with PI.
Click to expand...

This is a good idea, I personally like it, but unfortunately majority of users will hate it. Tensorflow does not come with Anaconda by default, so people will have to install it, which will be problematic for many. Plus we will have possible issues with a version control, which will cause problems for some people, I am sure.

I already tasted all this with my Python version of StarNet, many people just don't get all these things and don't wanna mess with it.
 
Juan Conejero said:
To Nikita and all users interested in this module,

Sorry so much for not being able to invest the required time and resources to put this module in the standard tool set. Nikita already uploaded his source code to our GitLab repository, and I have been taking a look. The module requires substantial work, but this is not difficult, just time-consuming. Perfectly normal for a new PixInsight tool.

The real problem here is TensorFlow. This is an extremely complex and huge software library (undoubtedly one of the most complex ones I've ever seen). There are several problems. Once of them is building TensorFlow on all platforms supported by PixInsight, which is not a trivial task. It is a very complex task that requires time and care. Unfortunately, the prebuilt libraries provided by Google don't work, so the only way to get this working is rebuilding everything.

Another problem is TensorFlow's dimensions. If included in PixInsight's standard distribution, it would increase the size of a PixInsight installation by about 300 MB. A fixed footprint like that is questionable to support a single module. A good solution to this problem is creating a dedicated update repository for StarNet on our official software distribution server, so the StarNet module would be an optional component. For example, the same will happen with the new Gaia star databases that we are preparing for version 1.8.7 of PixInsight.

Please bear with me while I can find the required time to perform all of these tasks as required. Currently we have a lot of priorities that require all of my attention and concentration. The new version of PixInsight comes with very important improvements (A rewritten StarAlignment tool with a completely new distortion correction algorithm, important new features in ImageIntegration and DrizzleIntegration, just to name a few) on which I have been working very hard during July, August and September. I ask for patience...
Click to expand...

Juan,

Thank you for looking into it! This is not a trivial task indeed!
 
this has been built and released for OSX... check nikita's sourceforge page for the zipfile.

rob
 
Thank you nikita.

This is just what i'm hoping for :)
Working with tensorflow outside of PI on OSX was possible but this module is very convenient.
 
paulsartory said:
Update....StarNet worked in a new PI "instance" but not within an existing project...I use a default project setup when starting PI for processing.
Click to expand...

I can confirm I have the same problem. The StarNet++ module only works for me if I completely close out of PI and open a new instance, then open my image and finally StarNet++. If I try to use the module within my default project I get the "Checkpoint file not found" message. Hopefully this can be fixed in a future release because it is a very useful tool!

Thanks
 
Hi, I am currently using StarNet on a windows10 machine with Pixinsight but when I run an image processing, I don't see the progress ratio in the console window but only the file name loaded and the result of processing after the starnet processing ended. However if I use StarNet in a windows terminal command I see the progress ratio. Is it normal? That would be great to see the progress ratio under the pixinsight's console as sometimes it takes a long time to process (especially if the tile size is low, for example 32).
 
yes it is one of the "TODO" items - nikita left a note in the source code saying that it would be nice if StarNet did not lock up PI while it was running.

i took a look at it and having never written a PCL module, i'm not entirely sure how this would be accomplished. probably you would need to spawn the starnet work as a thread which presumably could then be interrupted by the console.

rob
 
Hi everyone, I hope you can help me. I copied the files to the right directory but when I try to install it, find the plugin but I get this error message "Unsupported API version 162 (expected <= 160).: Module load error". Do you have any idea how I can solve it?
Please help me, thanks
 

Attachments

  • Schermata 2019-11-24 alle 19.17.24.png
    Schermata 2019-11-24 alle 19.17.24.png
    447 KB · Views: 83
Hey guys,

I have this working fine in PI and on a PC.  I have been helping a person use Starnet for his PC after installing the redistributatble C++ library he gets (which is what I get- but no image output!):

>>>>>>>>>>>>>>>>>>

C:\Users\Ron\Documents\Starnet\Starnet_Win>rgb_starnet++.exe rgb_test 5.tiff  .tif  rgbtest5_s.tif 64 Reading input image...  Done!
Bits per sample:  16
Samples per pixel:  3
Height:  712
Width  1048
Restoring neural network checkpoint... Done!
2019-12-10  10:39:58.662584:  I tensorflow/core/platform/cpu_feature_guard.cc:141)
Your cpu supports instructions that this TensorFlow binary was not compiled to use:  AVX2 Total number of tiles:  204 100% finished

Done!

C:\Users\Ron\Documents\Starnet\StarNet_Win>pause
Press any key to continue . . .

Computer is a Hewlett Packard 64 bit, 8.00 GB RAM, i3-4130 CPU.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Anyone have any ideas?  This is not the most powerful machine, but it still seems to be executing the code. 

JY
 
looks like an extra argument here:

rgb_starnet++.exe rgb_test 5.tiff  .tif  rgbtest5_s.tif 64

should be

rgb_starnet++.exe rgb_test_5.tiff rgbtest5_s.tif 64

note that you can't have spaces in filenames in a command line without escaping them somehow, so you should rename the input file to not have spaces for ease...

rob
 
KenR said:
paulsartory said:
Update....StarNet worked in a new PI "instance" but not within an existing project...I use a default project setup when starting PI for processing.
Click to expand...

I can confirm I have the same problem. The StarNet++ module only works for me if I completely close out of PI and open a new instance, then open my image and finally StarNet++. If I try to use the module within my default project I get the "Checkpoint file not found" message. Hopefully this can be fixed in a future release because it is a very useful tool!

Thanks
Click to expand...

Same way for me with Pix last update 1.8.8 and Win10 in new or existing project("Checkpoint file not found"). Before with Pix 1.8.7 StarNet work.
Thanks
 
You must log in or register to reply here.
Back
Top