PixInsight Forum

Software Development => New Scripts and Modules => Topic started by: cameronleger on 2017 November 05 13:57:26

Title: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 05 13:57:26
Foreword
First, I must thank Mike Schuster for the original version of this fantastic PixInsight Script. Second, of course, the PixInsight team and their continuing open-source initiative, without which this Module would be near impossible to complete. The recent release of 1.8.5 is also worth mentioning because two very important inclusions made this Module significantly easier to make.

I'm sure that most users love the SubframeSelector. It seemed such a worthy item in the pre-processing toolkit that I believed it deserved a more 'standard' method of use that fits inside PixInsight. My primary motivator for this was that Scripts 'block' PixInsight from other tasks. With nearly every use, I singled out a few frames to check out; why are the statistics so different for this frame, and how 'bad' is my worst 'statistically significant' frame? This involved remembering the file name, finding it in a file explorer, and checking it out in another image editor. My secondary motivator was speed. By far this is the slowest part of my pre-processing workflow. Less importantly, I prefer prefixes and this Script only supported postfixes.

Initial Release Notes
This Module has become stable and includes nearly every feature as the SubframeSelector Script. I work in Linux, and while I've provided a Windows build, I've only opened it in Windows to ensure that it loads fine. I do not currently plan on including more features, but I am absolutely welcome to feedback and will continue to develop this as needs arise.

Notable Differences
Of course, this is a native PCL Module instead of a PJSR Script. That alone offers many conveniences, including a huge performance boost. With the addition of threading for the measurement stage, this Module ran 12 frames in the time it takes the original Script to run 2.

The Output Files section is based off the common section between many other PixInsight Modules.

I've ported the recent PJSR StarDetector library to PCL and am using that exclusively instead of the StarAlignment tool. Some of the Star Detector parameters are slightly different because of this.

I've ripped the PSF Fitting portion of DynamicPSF. This plus my port of the StarDetector means this Module is more self-contained and does not call on any other PixInsight Modules or Scripts.

For most measurements, I'm using the Median and Mean Deviation from the Median for dispersion calculations. Star-related paramters, such as FWHM, are now a weighted average of the best-fit stars instead of a simple image-wide average.

The Approval and Weighting expressions are mostly the same, but behind-the-scenes are completely different. I've renamed a few parameters, so be sure to check them if you're copying your existing expressions. These expressions are now run as JavaScript code in the PJSR, thanks to recent updates. This vastly simplified the work involved for parsing expressions. Although I've limited the allowed characters, there are more possibilities with expressions. For example, you can use standard JavaScript functions like 'Math.abs()'.

The Graphs are now a WebView which utilizes the Dygraphs JS library also recently introduced. They are a little more interactive now, and display more information. In lieu of the extra line items in the Measurement Table, the Median and Mean Deviation are displayed on the graph and in tooltips that appear as you hover over points. Subframe Weights are a secondary axis, and the dispersion is displayed with a gradation. Clicking frames still toggles the Approved status, and shift-clicking unlocks the frame.

An additional Graph is also included which shows a histogram of the selected property with a line representing the Empirical Distribution Function. While this graph is not as interactive as the previous, it helps to identify ranges within your distribution of data that are best to include or exclude.

A few items are now placed in a 'Routines' dropdown: Measurement, Output, and Star Detector Preview. First, you must add your frames and measure them. Then, you can use the Output Routine. The Star Detector Routine runs only the Star Detector on the first frame that's supplied, and outputs relevant images for this process and the final structure map used to fit stars against. This helps you when tweaking Star Detector parameters.

(https://raw.githubusercontent.com/cameronleger/PCL/SubframeSelector/src/modules/processes/SubframeSelector/Screenshot.jpg)

To-Do

Release Information
My code is available in my forked branch of the PCL repository: https://github.com/cameronleger/PCL/tree/SubframeSelector
You can download a Linux and Windows build at this releases page of the repository: https://github.com/cameronleger/PCL/releases/tag/01.04.01.0001
OS X 10.12 build provided by Rob: https://drive.google.com/file/d/1xWKzGr4YKXbNaaeui5nH8RZR7olD1bIj/view?usp=sharing

I welcome any and all feedback. Thank you and enjoy!
Title: Re: Subframe Selector PCL Module
Post by: Geoff on 2017 November 05 15:41:23
Looks good but in my trial case I got completely different results using the script  vs the module.  I am using Win10.
Geoff
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 05 17:20:20
Hi Geoff,
Thanks for trying this out and providing those screenshots.

For the FWHM and Eccentricity differences, I do expect them but I'd like feedback as to if it's 'better' or not. The original SubframeSelector averages those values for all stars it finds. This does an average too, but it weights them by the MAD of the PSF Fit compared to the best PSF Fit in the image, so that it gives less preference to stars that weren't fit as well. This seemed to be recommended in the PixInsight documentation.

It may also be that the PSF Fitting settings in Star Detector Parameters is different than your setting for the Script. Can you also provide how many stars each version matched?

The SNR Weight appears the same, and I expect the Median and other image-related statistics to be extremely similar since it looks like you're looking at monochrome images. I've seen differences in those numbers myself, but I'm using debayered DSLR images and I suspect that's because the ConvertToGrayscale process that the Script uses comes up with a slightly different result than the getImageIntensity method that I'm using.
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2017 November 05 22:35:40
i have built this for OSX, but against OSX 10.11 (El Capitan). i am not sure if it would work against 10.12 (Sierra) or 10.13 (High Sierra) but for what it's worth, here it is:

https://drive.google.com/file/d/0B6cW4ae2KyMJcXI5aThGUDZId2s/view?usp=sharing

rob
Title: Re: Subframe Selector PCL Module
Post by: avastro on 2017 November 06 02:05:52
Looks good too, I've done a comparison, using the same expression, FWHM seem better with PCL.
PC with Win10. I utilize Subframe selector all the time and appreciate the PCL evolution. Thank you for all the contributors.
Antoine
Title: Re: Subframe Selector PCL Module
Post by: Juan Conejero on 2017 November 09 01:38:15
Hi Cameron,

As it happens every time a developer publishes a new module or script on this forum, you have realized the main reason why I started and have continued my work on the PixInsight project during all of these years. When this happens, the only thing I can say is thank you, so thank you very much for this excellent work, ad welcome to PixInsight development. This module is especially valuable for me because you have shown that all the effort I am putting in version 1.8.5 to strengthen PixInsight as a development platform is worthwhile. So thanks again.

Also as you have noted in your initial post, here we owe our gratitude to Mike Schuster for the SubframeSelector script, which is one of the most useful and beautiful resources available in PixInsight. Thank you so much Mike for making this piece of art possible and available to all PixInsight users.

Needless to say, I want this released as an official module as soon as possible. I have invited you to join our team of contributors to our official GitHub repository. Please submit a pull request to include your code in the PCL master branch. I want to discuss a few things with you (mainly regarding GUI topics and a few items related to high-level parallel processing) but I think the module is basically ready for release.
Title: Re: Subframe Selector PCL Module
Post by: dld on 2017 November 09 04:17:37
Great news!

Users who take lots of frames with low-end mounts (like myself), or experimenters on lucky imaging using cooled CMOS cameras, should benefit a lot after this development!

I have only one wish; histogram plots or other data exploration plots. So far I plot the results outside of PI in order to judge from the distribution of the measured values where to set an approval/rejection threshold. I am a novice Python/Numpy/Pandas programmer and I had in mind to write a Jupyter Notebook for the task, i.e., importing the SubframeSelector .csv output and do my plots there and get rid of the closed-source tools I use nowadays :D

Thanks again!
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 09 04:49:14
Hi Juan,
I'm happy to do this. I'll also look into forking Reference-Documentation and providing proper documentation to the standard of the Script.

dld,
Please, let's discuss more on this. From what you've said, I would think it's already in this Module, because every column of data in the CSV Output is also available as a Graph. Could you give an example of some plots you'd like to see? If there's more meaningful data that can measured and weighted/approved against, I want it! I was unsure if by 'histogram plot' you meant a plot of each histogram of the image channels or histogram as a graph of any numerical data ;)
Title: Re: Subframe Selector PCL Module
Post by: dld on 2017 November 09 06:39:51
dld,
Please, let's discuss more on this. From what you've said, I would think it's already in this Module, because every column of data in the CSV Output is also available as a Graph. Could you give an example of some plots you'd like to see? If there's more meaningful data that can measured and weighted/approved against, I want it! I was unsure if by 'histogram plot' you meant a plot of each histogram of the image channels or histogram as a graph of any numerical data ;)

A histogram as a graph of any numerical data. Suppose our column data is FWHM. I would be happy to be able to plot a histogram of the FWHM values (example: the first attached plot). In my example I'll approve images with FWHM < 3.0 since images with FWHM > 3.0 lies at the bad "tail" of the distribution.

A rather more complicated plot is the second attachment (scatter_matrix) which (may) combine smoothed histograms (diagonal of the matrix) with scatter plots. This is called a "scatterplot matrix". Such plots allows us to visualize correlations between our variables. For each cell (except the diagonal) we plot the column variable as x and the row variable as y. For example, at the top row the second plot from the left plots Eccentricity (y-value) against FWHM (x-value) . Probably such plots are not that important but may help us 1) find and reject outlier frames (example: sometimes a frame has a low eccentricity value due to passing clouds = low StarSupport and maybe high Median)  2) choose what is the most meaningful combination of variables for an approval formula.

Cumulative probability plots may also be useful (no example attached  :-( ).
Title: Re: Subframe Selector PCL Module
Post by: kor on 2017 November 14 18:52:51
Wow Cameron!

One of my most important tools just got better!  Thank you.  It used to take about an half and hour to measure my frames.  Now it takes a couple of minutes. 

A couple of suggestions:
1. Add the "Approved # of #" label somewhere--it's convenient to know how many frames pass the Approval expression.

2. Is there any way to add the ability to minimum and maximum on a Weighting symbol.  So something like
Code: [Select]
(FWHM - 3.642) / (8.449 - 3.642) would become something like
Code: [Select]
(FHWM - Math.min(FWHM)) / (Math.max(FWHM) / (Math.min(FWHM))Use case:  I add frames over time and remeasure to see if I have enough "good" ones yet.  When I do, I have to redo a bunch of minimum and maximum values in my Weighting expression.  Not a huge deal, but it would be nice to just let the tool take care of that.

Thanks again - Shane
Title: Re: Subframe Selector PCL Module
Post by: dld on 2017 November 15 05:32:54
Cameron, I have just installed and run your Module. It runs FAST! Thanks again for your efforts!
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 15 13:47:36
Shane,
Thank you for the feedback. I was already thinking of doing the first item, and the second item is a good suggestion. I figure I can add a Median, Min, and Max just like the Sigma is currently handled, and that would do what you want I think. At first, I didn't want to implement a 'file cache' feature because of the complexity versus perceived usage of the feature. However, with what you've said and my recent experience with multi-night shoots, I think this would be another great addition.

dld,
That's great news. I also put histograms on the roadmap. I'm still toying with it in my head, but I also think that allowing the choice for the X axis and generating a scatter plot would also handle some of the other plots you have described. I hadn't thought to get that technical with the graphs, but I can see its usefullness now.
Title: Re: Subframe Selector PCL Module
Post by: kor on 2017 November 15 20:05:09
Cool.  Great idea about the Median, Min, and Max.  I looked through the code today and kind of figured that would be the best way to do it.  My "multi-night" shoots last months and the file cache, if it survives PI shutdown and startup would save a lot of time too.

Once again, thanks for your work on this module.

- Shane
Title: Re: Subframe Selector PCL Module
Post by: dld on 2017 November 16 05:00:53
Hello Cameron,

Something like this https://en.wikipedia.org/wiki/Empirical_distribution_function (https://en.wikipedia.org/wiki/Empirical_distribution_function)  should be useful too. It helps to visualize what is the percentage of images accepted for a given acceptance threshold. I think people doing planetary imaging are using something similar.

Anyway, let's keep it simple :) My "matrix" example was way too complicated. I am already a happy user :) And I don't feel comfortable suggesting features while someone else does the hard work :)

Thanks and clear skies!
Title: Re: Subframe Selector PCL Module
Post by: vicent_peris on 2017 November 17 03:31:26
Hi,

Congratulations for your new module. I want to add a couple ideas:

- I also think the cumulative probability graph can be very useful to establish a threshold.

- Since this is a PCL module, I think it would be really useful if your tool is able to monitor a directory to analyze newly saved images. To would be open the possibility to monitor in real time the image quality while acquiring data (inside or outside PixInsight).


Best regards,
Vicent.
Title: Re: Subframe Selector PCL Module
Post by: frant on 2017 November 18 00:44:03
Fantastic work Cameron, really appreciate your hard work on this.

I think may have found a small issue. When outputting approved subs, the FITS Keyword is not written into whichever is the last/bottom sub in the Measurements Table list. It does seem to be related to whichever is the last sub in the list, as when I reorder the list on a different metric (for example Weight, or FWHM) and therefore change which is the last in the table, then that sub becomes the one that doesn't get the Keyword written in the output file.

Thank again, a great addition to the PixInsight toolbox  :)
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 19 09:30:20
I appreciate all your feedback. I'm not ready to provide a new build yet, but I hope to very soon.

Shane,
I've got the Median, Min, and Max special variables working. It offered me a great chance to refactor how some of that was working into much simpler code.

dld & Vicent,
I've got a histogram working and am working on a cumulative function. I feel that the Empirical Distribution Function makes the most sense given that we have the empirical data, but I've hit a hurdle. I want to show both of these on the same distribution-related graph to save space and since they're strongly related, but this graphing library prefers multiple series to have the same independent axis. As such, a cumulative histogram (https://en.wikipedia.org/wiki/Histogram#Cumulative_histogram) seems attractive, except that it would limit the 'resolution' of the cumulative function to the number of bins used in the histogram.

An alternative could be to 'zip' the two series together and provide 'blanks' where either series does not align with the other, it just seems difficult when comparing floating-point numbers and they may not align nicely. Thoughts? I'm just looking for a value or priority on the solutions to compare complexity/benefit of them.

Edit: I've got the union working properly, so an EDF + Histogram is going well.

Frant,
Thank you for finding that. I'll definitely look into it. I already found another critical bug where a memory leak could happen during the exports. It's in the PCL repository, so if this is released by PTeam before my next update you'll have it then.
Title: Re: Subframe Selector PCL Module
Post by: kor on 2017 November 19 13:16:33
Great Cameron!  I look forward to the release.  (I've got a large subframe set I'll test it with!)

- Shane
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 20 16:45:00
Fantastic work Cameron, really appreciate your hard work on this.

I think may have found a small issue. When outputting approved subs, the FITS Keyword is not written into whichever is the last/bottom sub in the Measurements Table list. It does seem to be related to whichever is the last sub in the list, as when I reorder the list on a different metric (for example Weight, or FWHM) and therefore change which is the last in the table, then that sub becomes the one that doesn't get the Keyword written in the output file.

Thank again, a great addition to the PixInsight toolbox  :)

Hey frant,
Unfortunately, I don't have a clue as to why this is happening and I haven't been able to reproduce this. I looked at files generated by earlier and current versions of this module and I did not see this happening between batches of 4 - 300. It's confusing because the output routine loops the 'base' set of measured subframes and not the 'sorted' set that's generated for the table view, so I'm not sure how they could be affecting each other. I was checking the output files by opening them and looking at the 'FITS Header' menu entry under the 'File' menu. Do you have any more hints?
Title: Re: Subframe Selector PCL Module
Post by: frant on 2017 November 21 11:04:40
Hey frant,
Unfortunately, I don't have a clue as to why this is happening and I haven't been able to reproduce this. I looked at files generated by earlier and current versions of this module and I did not see this happening between batches of 4 - 300. It's confusing because the output routine loops the 'base' set of measured subframes and not the 'sorted' set that's generated for the table view, so I'm not sure how they could be affecting each other. I was checking the output files by opening them and looking at the 'FITS Header' menu entry under the 'File' menu. Do you have any more hints?

I don't at the moment, but I'll be using it again this evening, so I'll see if I can come up with a reproducible test case.
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 22 06:38:01
Based on your feedback and more usage on my part, I have a new update ready for release. I've updated the original post and I'll include a link here too for simplicity: https://github.com/cameronleger/PCL/releases/tag/01.03.01.0003
Title: Re: Subframe Selector PCL Module
Post by: frant on 2017 November 22 11:30:33
I'll see if I can come up with a reproducible test case.

Well I couldn't :)
That is to say, I used the tool 4 times during the integration of L,R,G and B subs and it worked flawlessly each time, it just isn't doing it now.
I'm a bit puzzled as to what I was doing wrong before, but happy to say it was probably finger trouble on my part, rather than a bug.
Just installed the new version too, keep up the great work.

Cheers,
Richard
Title: Re: Subframe Selector PCL Module
Post by: dld on 2017 November 22 13:47:15
Fantastic! I just noticed that the measurements are cached! No more anxiety on loosing the measurements by accidentally pushing the wrong button (been there, done that, several times in the past...) Thanks again Cameron!
Title: Re: Subframe Selector PCL Module
Post by: kor on 2017 November 22 18:08:26
Way Cool, Cameron!

You just reduced my workload and shortened by processing time even further!  The Min/Max stuff works great.  The number of files accepted is very useful.  The histogram adds yet another important visualization of the quality of the entire subframe set.  And the cache--yet another time saver when I've already measured 130 frames and have five more to add!

One more (very simple) suggestion:  Add the index number to the black rectangle on the graph hover.  Some of us end up with quite a few subframes and it's difficult to tell which one the hightlighted dot and hover information represents.

Thanks again for your great tool - Shane
Title: Re: Subframe Selector PCL Module
Post by: jshortt on 2017 November 25 12:11:58
Hi, I'm working on a project with about 250 subs that I'd like to check this out on. I can download it from GitHub but I don't know how to install it into my PI setup. Can anyone provide me with a quick set of instructions?

Update: I think I've figured it out. There's an item under the Process menu to install a new module

John
Title: Re: Subframe Selector PCL Module
Post by: tdgm on 2017 November 25 20:41:30
Hi

Just installed the new SFS Module and have been looking at it everything seems to work (Mac running High Sierra) but I can't get a probability graph /histogram to come up. Is that agraph with the Mac or am I missing something.


Thanks for the new module
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2017 November 25 20:58:23
it could be that i need to rebuild it? i just did git pull and i can see a lot of changes, though the tag seems to be the same (1.03.01.0003)

rob
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 26 18:29:43
Rob, can you rebuild it? Your original build was against 01.01.01.0001, and I updated the link in the original post to point to the latest except for the OS X version.
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2017 November 26 19:33:37
ok will do

rob
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2017 November 26 20:17:59
ok here is a new build for osx...

https://goo.gl/X3c9Wc

i noticed a couple of things:

1) when double-clicking a point in the graph, the viewport containing the file list does not always scroll properly to reveal the file in the list. i was not able to detect a pattern. this happened in the previous version as well...
2) if i right-click the graph pane, i get a contextual menu. one of the options is "view source". if i select this i get a dialog that the process segfaulted and from that point on the graph pane no longer works. not sure how to fix it without restarting PI.

rob

Title: Re: Subframe Selector PCL Module
Post by: tdgm on 2017 November 27 09:53:21
Still looking for an answer have installed module on my Mac running High Sierra  and module seems to run fine except noted 2 things pf mentioned above but I still can't get a graph That is mentioned in the original post (An additional Graph is also included which shows a histogram of the selected property with a line representing the Empirical Distribution Function. While this graph is not as interactive as the previous, it helps to identify ranges within your distribution of data that are best to include or exclude),.

Please help


Steve
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2017 November 27 10:25:11
this is with the latest build i just made - i see the histogram on the right...

did you load the latest build?

rob
Title: Re: Subframe Selector PCL Module
Post by: tdgm on 2017 November 27 11:08:36
Thank Rob

Since the downloads file has the same name I was not sure what version it was but I did switch the files to the latest download on this post and now it is present thanks again.


Steve
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2017 November 27 19:56:43
Thanks, Rob, I updated the OP with the new link.

Re: left-click scrolling, I think a single-click works as well, but that's not the problem. I did not see a method to scroll to an item in the TreeBox, so I'm scrolling to a position I gather from the item at that index. It's possible that I'm using the wrong coordinates; I noticed similar behavior but most of the time it worked. I'll look into this.

Re: view-source, while I don't get the segfault on Linux, it can produce an unrecoverable state. I'm not exactly sure what causes this, but I hoped no one would try! I presume it's because I'm directly setting HTML content instead of pointing to a source file, yet it may have to do with my event handling. I'm hoping I can disable this right-click menu. Otherwise, it's a method to get up a creek without a paddle.
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 January 28 10:40:39
Hi Cameron,

i'm a late comer here, but just installed and tried this PCL module. it looks very promising! i love the addition of math, which eliminates the extra steps and need for the spreadsheet tool i have been using (posted elsewhere here) to create the weighting value

I may be running into a bug? I was able to run it once successfully, so i know i can get it to work

I was entering various weighting formulas, seeing the red 'x' indicating a formula error, and fixing syntax until i got the green check mark.

however now when I run measure, it errors out with "nan not defined". in addition, i get the red x for some pretty basic approval expressions. I can't get beyond this point

Thanks

Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 January 30 13:53:12
Hi Cameron,

i'm a late comer here, but just installed and tried this PCL module. it looks very promising! i love the addition of math, which eliminates the extra steps and need for the spreadsheet tool i have been using (posted elsewhere here) to create the weighting value

I may be running into a bug? I was able to run it once successfully, so i know i can get it to work

I was entering various weighting formulas, seeing the red 'x' indicating a formula error, and fixing syntax until i got the green check mark.

however now when I run measure, it errors out with "nan not defined". in addition, i get the red x for some pretty basic approval expressions. I can't get beyond this point

Thanks

Brian

Hey Brian,
I'm sorry to hear that. Would you mind using the "Edit Instance Source Code" button at the bottom right of the Module and sending me the contents, perhaps via pastebin? That will help me see all the parameters and measurements to try to track this down.

I cannot tell much so far, other than measurements seem to have been taken but the graph view is not visible, so that might be the source of the JS error. I'm wondering if some parameters were not measured correctly, and the translation from PCL into PJSR produces "nan" for non-numbers instead of the "NaN" that JS would read correctly - not that a NaN is a good thing though.

Also, a quick update. I had done the documentation for this Module, and most of the recent changes were not pushed back to Juan. I was unsure how long his process would take and if he would make changes himself, so I wanted to wait to push those afterwards so that I could handle the merging of code instead of giving him more work. Sadly, I haven't had much time to use this since then. I thought winter would be *awesome* for astrophotography, but it turns out in Atlanta winter means few clear skys and very poor seeing!
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 January 30 13:58:54
sure - i'll get that together tonight and PM you with those contents

Thanks

Brian
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 January 30 18:26:54
here you go:

https://pastebin.com/i90pcZtJ

two attachments show

1. before state, entered the approval and weighting formulas, they both 'check out'

2. after loading data and pressing measure, get the error
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 January 31 18:57:42
Thanks Brian,
I can confirm the issue is with the Weighting Expression, not the Approval Expression, although that one can display the X for some reason.

The 'plain' variables such as FWHM and Eccentricity only represent the current subframe being measured. So, Math.min(FWHM) and Math.max(FWHM) will return the same thing as FWHM, and subtracting these values will result in 0. The 'nan' error appears when it's dividing by 0.

The simple fix here is to use the 'special' variables such as FWHMMax and FWHMMin which are calculated before each subframe and represent the global max and min of all subframe FWHMs. Here's that expression modified to work for me:

Code: [Select]
35*(1-(FWHM - FWHMMin) / (FWHMMax - FWHMMin)) + 7*(1-(Eccentricity - EccentricityMin) / (EccentricityMax - EccentricityMin)) + 18*((SNRWeight - SNRWeightMin) / (SNRWeightMax - SNRWeightMin)) + 40

That said, I will definitely look into 'nicer' or more obvious messages when 'nan's are involved, and try to make the X's work better in these cases.
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 January 31 19:58:55
btw there are no logs that i can see, so i don't know how to usefully report these bugs, but i've had a couple of instances where the SFS module hangs up and can't be quit, necessitating a force quit of the whole PI platform. i've taken to saving my project before running it.

i believe these occurred while measuring subframes, but they also may have occurred while interacting with the UI. sorry i can't be of more help...

rob
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 January 31 21:32:17
Thanks Brian,
I can confirm the issue is with the Weighting Expression, not the Approval Expression, although that one can display the X for some reason.

The 'plain' variables such as FWHM and Eccentricity only represent the current subframe being measured. So, Math.min(FWHM) and Math.max(FWHM) will return the same thing as FWHM, and subtracting these values will result in 0. The 'nan' error appears when it's dividing by 0.

The simple fix here is to use the 'special' variables such as FWHMMax and FWHMMin which are calculated before each subframe and represent the global max and min of all subframe FWHMs. Here's that expression modified to work for me:

Code: [Select]
35*(1-(FWHM - FWHMMin) / (FWHMMax - FWHMMin)) + 7*(1-(Eccentricity - EccentricityMin) / (EccentricityMax - EccentricityMin)) + 18*((SNRWeight - SNRWeightMin) / (SNRWeightMax - SNRWeightMin)) + 40

That said, I will definitely look into 'nicer' or more obvious messages when 'nan's are involved, and try to make the X's work better in these cases.

Great thanks. Is there a place where all the special variables are documented? these are the obvious ones I need for now :)

Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 February 01 19:57:52
Hey Rob,
I haven't had anything like that; you're on OS X right? I'll look into debugging/logging information.

Hey Brian,
They're currently in the tooltips on those fields. I think it lists the variables, and that each of them have a few special versions and lists those types. I also have it in the documentation with some examples, but that's not released yet. I could look into distributing that before the formal release, but I think it only works in PI.
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 February 01 20:41:37
Hey Brian,
They're currently in the tooltips on those fields. I think it lists the variables, and that each of them have a few special versions and lists those types. I also have it in the documentation with some examples, but that's not released yet. I could look into distributing that before the formal release, but I think it only works in PI.

no sweat, you showed me the main ones i need for now, so I am patient for the full release. Thanks again so much, you saved me i don't know how much time.

B
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 February 01 20:58:58
Hey Rob,
I haven't had anything like that; you're on OS X right? I'll look into debugging/logging information.


yes, OSX. thanks.

rob
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 February 03 11:57:56
Hi Cameron

that works beautifully.

i have one more request - not a huge one, but is there a way to save the weighting formula from session to session or through application restarts?

i typically use the same formula, so i have to either type it in (painfully) every time or copy it from another place.

or maybe there's a way to save and restore different formulas?

minor request, I don't want to look a fabulous gift horse in the mouth ;)


Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 February 03 18:04:43
Hey Brian,
One of the reasons I wanted to make this as a Module was to gain Module-specific features, such as saving Process Icons and their included settings. I recommend you do that once you have the formula you want. I think I saw in your screenshots that you had tweaked other parameters too; that will of course save all of them as well.

Another thing to note about that: it will also save all loaded subframes and their measurements. I also find that really handy to go back and quickly check weights for other processes where I'd want a really good frame. I have a couple of these icons saved for different camera setups, and I keep one around in the project for the last measurements I took just in case I have to pause and come back later.
Title: Re: Subframe Selector PCL Module
Post by: paulsartory on 2018 February 14 21:45:35
Hi,

I am probably missing something really obvious here...but how do I output only the approved frames to a separate directory......after measuring the sub-frames I change to "output Sub-frames but it outputs all of them included the rejected frames?

Paul
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 February 14 22:10:56
Hi Paul,

two things come to mind:

1. make sure you have rejected frames, there will be red 'x's on the output graph - this is the obvious one

2. The process window counts all frames, but actually only outputs the approved ones. This threw me too, but i checked the folder and sure enough it was only the approved frames

hth


Brian
Title: Re: Subframe Selector PCL Module
Post by: paulsartory on 2018 February 14 23:32:29
Thanks.....I guess that's the (not so obvious) thing I missed....P
Title: Re: Subframe Selector PCL Module
Post by: cfosterstars on 2018 August 06 16:21:20
For some reason all my graphs disappeared and I cant seem to get them back.
Title: Re: Subframe Selector PCL Module
Post by: astrovienna on 2018 August 08 05:59:00
A possible issue with expressions for weighting:  the script allowed exponential expressions [FWHM^2, SNRWeight^(0.5)], but I can't get  those to work in the process.  Is this intentional?  I'd like to use roots in some of my weighting expressions.

Kevin
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 August 08 07:12:26
@cfosterstars
Could you explain more or provide screenshots? Has a PixInsight or OS restart solved this issue?

@astrovienna
It is weird that it's accepting it; can you provide the whole expression for me? I tried what you sent like "FWHM^2" or even "2^3" and it shows the red X and didn't calculate the weights.

In any case, what you want is possible with the Math module. JavaScript doesn't have a quick symbol for powers, but the function Math.pow(base, exp) will do it. See this page for more details: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math
Title: Re: Subframe Selector PCL Module
Post by: astrovienna on 2018 August 08 08:25:27
Thanks for the quick reply.  I'm not at my PI machine now, but I'm almost certain that one of the expressions I tested was simply FWHM^2, and it successfully parsed in the script.  I'm trying to use a generic approach to weighting rather than one tied to a particular data set - something like SNRWeight/(FWHM^0.5) rather than relying on the min and max of SNR and FWHM in a given data set.

From the page you pointed to, it looks like the Java expression I need would be SNRWeight/Math.pow(FWHM, 0.5).  Correct?  Sorry for asking, but me using Java scripts is like a gorilla playing with hand grenades.  :)
 
Kevin
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 August 08 21:41:54
Looks like it should work to me!
Title: Re: Subframe Selector PCL Module
Post by: astrovienna on 2018 August 09 04:48:25
Yes, that worked.  Thanks for writing the module.  This is a great addition to the tool set.

Kevin
Title: Re: Subframe Selector PCL Module
Post by: gnewell on 2018 August 20 08:51:39
Feature Request:

Add the ability to include FITS header keywords as columns to aid in grading.

should be part of csv export file as well.

See the Fits Keyword script for a (scripted) example.

Thanks for considering!

 
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 August 20 09:06:36
gnewell

I don't know if this meets your need, but you can already add a FITS header keyword for the weighting when you output your frames. I don't think you can do more than one though


brian
Title: Re: Subframe Selector PCL Module
Post by: Lightpath on 2018 August 29 20:17:40
I'm really enjoying tinkering with this tool!  It's really well done.

My problem at the moment is that I don't have a good solid understanding of what makes a "good" sub.  Does anyone have a resource I could read that would help me come up with my own expression?

Thanks very much-

Mark.
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 September 20 18:19:31
https://github.com/cameronleger/PCL/releases/tag/01.04.01.0001

All,
There is a technically significant update available, however I only have a Linux build at-the-moment so I'd appreciate any volunteers on the other builds. It was hard to notice at first since it was already much faster than the PJSR version and clearly benefited from more threads, but I have a larger machine now and it was essentially choking on the higher core counts. Turns out, there's a setting that should've increased performance considerably, but it did the opposite :tongue: If you thought it was fast before... well let's keep going :police:

Mark,
I think this would be very relative and dependent on the context, but that's OK because a lot of the parameters can be used in a relative way! It will come down to your strengths, weaknesses, and preferences. Personally, I use this expression as a start on every batch:

(FWHMSigma / -3) + (EccentricitySigma / -3) + (SNRWeightSigma / 3)

This basically says: combine the relative values of FWHMs, Eccentricities, and SNRWeights, where a lower FWHM and Eccentricity are preferred and a higher SNRWeight is preferred. Those values are scaled by 3 sigma units (the negative sign is what's making it prefer lower numbers) as nearly all of my frames are within that range. For example, if a subframe's FWHM is 3 sigmas lower than the median, it will contribute 1, whereas if it were 1 sigma higher it would contribute -0.33. Sometimes it's necessary to add a constant afterwards like 5, so that my smallest weight is above 0. I think otherwise the further processes' weighting calculations are relative so it doesn't matter if your best is 500 or 5, it compares it as a percentage of the best.

Perhaps the original documentation here, https://pixinsight.com/doc/scripts/SubframeSelector/SubframeSelector.html, or my work-in-progress updates, https://github.com/cameronleger/Reference-Documentation/blob/master/tools/SubframeSelector/SubframeSelector.pidoc, will help you start your expressions. If you don't like reading my documentation as it is in that link (and why would you?), you can download that folder and use the 'Development' -> 'Documentation Compiler' Script to view it in PixInsight. Since this is not formally included with PI, it's a little funny to use.
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 September 20 21:42:17
i can try to build it for OSX but i think i'm maybe behind a couple of revs on Xcode and don't have the latest PCL installed so it may take me a couple of days.

rob
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 September 20 22:06:01
new update looks really nice - it was already performing nicely but i'll take faster performance any day!

i hope someone does windows soon ;)

thanks again

Brian
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 September 21 00:22:57
i can try to build it for OSX but i think i'm maybe behind a couple of revs on Xcode and don't have the latest PCL installed so it may take me a couple of days.

rob

well so i made the mistake of pulling the latest PCL from the pixinsight github, and i found that after i built the module that 01.08.05.1352 (which is the latest for OSX) complains that the API version is unknown (161) vs an expected of 160.

i guess i can rebuild PCL from your fork of the PCL tree which i assume is going to have api version 160... but something is funny here. either OSX core is a release behind the PCL release, or the PCL API version has been bumped without a corresponding release of the core application on every platform... (?)

i also have an older build of PCL that i used to build the last SFS so maybe i can just point to that instead.

rob
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 September 24 11:56:35
so i'm going to need some guidance here -

the release called PCL-01.04.01.0001 on your github does not seem to contain the SubframeSelector module source code.

the branch called SubframeSelector does contain the source code, however, that branch contains the API version 161, so i can't build in that tree.

if i build PCL in the PCL-01.04.01.0001 release, then build SubframeSelector from the same-named branch, PI gives a segmentation fault when attempting to load the module. not sure exactly what is wrong there.

there is one more thing i can try - i have an older build of the PCL libraries which should be api version 160. but i feel like the SubframeSelector source should have been in your PCL-01.04.01.0001 release zipfile, and it's apparently not.

rob
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 September 24 12:34:04
Sorry this is so confusing, Rob.

My SubframeSelector branch, https://github.com/cameronleger/PCL/tree/SubframeSelector, is based off the latest PCL commit and has all the SS code up-to-date. I merged the latest PCL at the end of this development cycle so that a Pull Request would be clean.

The tag for my release was automatically added when I did a 'release' in GitHub, but that appears to have automatically applied to the 'master' branch which is actually the plain PCL repository from a while back... It also appears that the automatic zip of the code has the same issue. I'll see about moving that tag to the correct location, and that's on me for not understanding what GitHub does automatically.

You are right that PCL sets an API version 161, that happened six months ago here: https://github.com/cameronleger/PCL/commit/97e83fd6747bcf5655a4caf395a6109c348c79e9.

Comments indicate it applies for 1.8.5.1357, and yet I'm running 1.8.5.1352 also and had no issues with this build or loading the module. For some reason, the Linux 1352 doesn't care about the API version change while the OS X 1352 does.

For what you want to try, it should work. Since I haven't modified any files outside of the SubframeSelector folder, you can probably checkout any PCL commit you want, copy that folder in, and try building. I can't imagine that I'm using any extremely new portions of the PCL.

Aside from fixing the 'release' on my GitHub, let me know if you'd prefer I do anything else for you. While I cannot do the OS X build myself, I can do whatever magic in the structure/code/files that makes it easier for you.

EDIT: The tag is in the correct place now, and the source zips were automatically updated with that as well. Looks like the same issue applies to previous tags as well.
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 September 24 12:46:24
i got it built against the older v160 PCL tree that i had hanging around, and SFS loads and runs OK in PI. juan confirms that 161 corresponds to a future release of PI, so that problem will take care of itself eventually.

at this point the only open question is if building against the 10.14 SDK is kosher for running this module against older versions of OSX. since i am running 10.13 and the latest XCode contains only the 10.14 SDK tree, it seems like the answer is probably yes. hopefully someone who is running 10.12 can try my build.

one other question - when opening the module in PI, besides the main interface window, a 2nd (blank) window opens behind it. any idea what that is about?

thanks,

rob

https://drive.google.com/file/d/1xWKzGr4YKXbNaaeui5nH8RZR7olD1bIj/view?usp=sharing


Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 September 24 13:11:15
I've seen the blank window too. It used to appear and disappear, and sometimes it would stick, but lately it's been sticking around. This inconsistency happened without code/build changes also! I fear that it has something to do with the QTWebKit views that PI is using and/or the JS graphing library. I noticed the new PhotometricColorCalibration doing it too, but it's much quicker and disappears every time.

Eventually, I'd love to build a native-drawing graphing control instead of relying on this embedded WebView; that's great for quick setups but the interaction code I had to write with it is pretty quirky. I'll look for any hints in the PCC code if it's available.
Title: Re: Subframe Selector PCL Module
Post by: dld on 2018 September 24 13:13:57
one other question - when opening the module in PI, besides the main interface window, a 2nd (blank) window opens behind it. any idea what that is about?

I have the same minor issue but I thought it was something I've done wrong.
Title: Re: Subframe Selector PCL Module
Post by: mijoschille on 2018 October 26 16:48:32
i can try to build it for OSX but i think i'm maybe behind a couple of revs on Xcode and don't have the latest PCL installed so it may take me a couple of days.

rob

Sorry for the noobie question - but how do you install the module at the Linux level? I know once it's installed I need to go to the Manage Modules feature in PI to ensure it's checked.

Thanks.

Mike
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 October 26 16:50:35
Hey Mike,
That's basically it. Wherever you have it and installed it from, it will keep looking for it there. If you'd like, you can move it somewhere that makes sense to you e.g. somewhere in the PixInsight folder, but that's purely maintenance and shouldn't affect it.
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 October 26 17:12:18
was there ever a windows compile done on the latest version?

thanks

Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 October 26 17:13:46
Sorry, no, I'll look into it this weekend and post it here :)
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 October 27 12:43:47
Sorry, no, I'll look into it this weekend and post it here :)

fantastic! that would be great, i appreciate it

Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 October 27 12:50:10
Hey Brian,
I literally got this latest reply as I was uploading the release to GitHub. It's ready :) https://github.com/cameronleger/PCL/releases/tag/01.04.01.0001

I simply pulled the latest changes, built the solution, and opened the module in PixInsight to make sure I didn't have the same OS X API version issues. I wasn't able to run it, but the latest version is built and opens at the least.
Title: Re: Subframe Selector PCL Module
Post by: mijoschille on 2018 October 27 15:56:03
Hey Mike,
That's basically it. Wherever you have it and installed it from, it will keep looking for it there. If you'd like, you can move it somewhere that makes sense to you e.g. somewhere in the PixInsight folder, but that's purely maintenance and shouldn't affect it.

Worked like a charm. Thanks Cameron.

Mike
Title: Re: Subframe Selector PCL Module
Post by: mijoschille on 2018 October 27 17:14:16
Successfully ran (and LOVE) the process! A few questions I hoping someone can help me with. Below is a screen shot of my FWHM weights after entering the subframe expression. I guess the first question I have is can we use the same formula structure as we use with the script? Here's the formula that I entered:

(10*(1-(FWHM-2.528)/(2.528-4.482)) + 10*(1-(Eccentricity-0.497)/(0.847-0.497)) + 30*(SNRWeight-1.1167)/(1.1479-1.1167)) + 50

As I understand it the lower the FWHM number the better the image. My next set of  questions relate to the following data:
-Index Image #3 has a FWHM of 2.602 (worst number), and a weighted score of 92.772 (highest score)
-Index Image #6 has a FWHM of 2.265 (second worst number) and a weighted score of 74.103 (lowest score)
-Index Image #7 has a FWHM of 2.257 (best number) and a weighted score of 88.084 (second best score)

Why would image #3, which has the worst FWHM number, have the best weighted score? And image #6 has the second worst FWHM number yet it has the lowest weighted score? If I am interpreting the data properly, the worst scored FWHM image (#3) will carry the most weight (92.772) into the final product rather than the best FWHM image (#7).

Lastly, to underscore my complete confusion and ignorance, highlighting my need for help - image #7 has the best FWHM score, has a better weighted score than image #6, but not the best score that I would expect it to have. Someone please point me in the right direction...

Title: Re: Subframe Selector PCL Module
Post by: RoMe on 2018 October 27 17:34:15
(10*(1-(FWHM-2.528)/(2.528-4.482)) + 10*(1-(Eccentricity-0.497)/(0.847-0.497)) + 30*(SNRWeight-1.1167)/(1.1479-1.1167)) + 50

You have inverted the FWHM value in your term. It should be (FWHM-2.528)/(4.482-2.528). I notied this because I just made the same mistake.  ::)
Title: Re: Subframe Selector PCL Module
Post by: mijoschille on 2018 October 27 19:04:09
(10*(1-(FWHM-2.528)/(2.528-4.482)) + 10*(1-(Eccentricity-0.497)/(0.847-0.497)) + 30*(SNRWeight-1.1167)/(1.1479-1.1167)) + 50

You have inverted the FWHM value in your term. It should be (FWHM-2.528)/(4.482-2.528). I notied this because I just made the same mistake.  ::)

Wow - what a bonehead - here I am typing all this detail... Thanks!
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 October 28 12:56:39
Hey Mike,
I think in some cases the old expressions may be used, but in this case I think you'll benefit from reading the tooltips on the expression inputs. For example, you can replace a lot of your constant numbers with smart parameters like FWHMMin, EccentricityMax, and so on.
Title: Re: Subframe Selector PCL Module
Post by: mijoschille on 2018 October 28 13:43:54
Hey Mike,
I think in some cases the old expressions may be used, but in this case I think you'll benefit from reading the tooltips on the expression inputs. For example, you can replace a lot of your constant numbers with smart parameters like FWHMMin, EccentricityMax, and so on.

Sounds great. Where can I find the expression inputs? I've looked and unable to find. Thanks for all the help!
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 October 28 13:49:59
Sorry, I meant 'expression inputs' as the text area inside the SS where you are typing the Weighting or Approval Expressions. Hover over them for a little bit and you'll get a large tooltip explaining the things you can type into them. I think that most things in the SS will have these.

'Inputs' is a common term for a UI developer to say something that receives user information, like a box to type in text, or a checkbox, or a file browser icon.
Title: Re: Subframe Selector PCL Module
Post by: mijoschille on 2018 October 29 20:43:50
Sorry, I meant 'expression inputs' as the text area inside the SS where you are typing the Weighting or Approval Expressions. Hover over them for a little bit and you'll get a large tooltip explaining the things you can type into them. I think that most things in the SS will have these.

'Inputs' is a common term for a UI developer to say something that receives user information, like a box to type in text, or a checkbox, or a file browser icon.

Got it. So if I understand the documentation the formula would read:
(10*(1-(FWHM-FWHMmin)/(FWHMmax-FWHMmin)) + 10*(1-(Eccentricity-Eccentricitymin)/(Eccentricitymax-Eccentricitymin)) + 30*(SNRWeight-SNRWeightmin)/(SNRWeightmax-SNRWeightmin)) + 50

Btw I love the fact that the data points are cached. I forgot to identify the best weighted image among my LRGB files - what a breeze with that feature!

Mike
Title: Re: Subframe Selector PCL Module
Post by: mmirot on 2018 November 04 19:33:05
Where can I get the latest version for windows?

Max
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 November 04 20:01:56
https://github.com/cameronleger/PCL/releases/tag/01.04.01.0001

Although, the *latest* version will be released soon :)
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 November 12 13:07:10
https://github.com/cameronleger/PCL/releases/tag/01.04.01.0001

Although, the *latest* version will be released soon :)

any update for the release of the latest version on windows? thanks again!
B
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 November 12 13:34:22
The changes that I've listed in this thread are in that link. The *latest* version will come with the next version of PixInsight.
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 November 12 14:06:03
great thanks. I looked on github and it looked like the date was september, but i'll download the latest, and await the PI update ;)

Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 November 12 14:27:10
I think that's because I setup the release for Linux then, and simply updated it with the Windows build after I stopped procrastinating. It wasn't a new release and it doesn't show an "edited" timestamp.
Title: Re: Subframe Selector PCL Module
Post by: CharlesW on 2018 November 13 18:43:35
A person mentioned in a related post that some Weighting expressions are built in. I’ve looked all over SFS and can’t find them. Is that possibly not the case, or where might they be located?
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 November 13 19:50:39
If you're thinking of whole preset Weighting Expressions, then no that's not the case. Maybe they were trying to talk about the "variables" that are pre-defined for use in the Weighting Expression, such as "EccentricitySigma" or "FHWMMax", which would be replaced with the values of the Eccentricity in Sigma units of the whole set or the maximum FHWM value of the whole set, for example. Hover over the expression text boxes for a tooltip that describes them.

Once you come up with one you like, if you save the instance as an icon on your workspace it will be there for you next time you open it.
Title: Re: Subframe Selector PCL Module
Post by: souls33k3r on 2018 November 19 12:32:44
I've just stumbled across this and it looks awesome but i've hit a brick wall here.

I downloaded the Windows version of the zip file from https://github.com/cameronleger/PCL/releases/tag/01.04.01.0001 but i'm not sure how to install/add this script to my PI.

Sorry for being thick.
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 November 19 13:06:53
http://pixinsight.com.ar/en/docs/36/pixinsight-install-modules.html

rob
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 November 19 13:36:44
this may help

http://pixinsight.com.ar/en/docs/36/pixinsight-install-modules.html
Title: Re: Subframe Selector PCL Module
Post by: souls33k3r on 2018 November 19 14:54:30
awesome stuff. Cheers fellas. I knew i had to install it as a module but i was pointing at the .zip folder extracted contents. That‘s where i was going wrong. :)
Title: Re: Subframe Selector PCL Module
Post by: Subaru on 2018 December 05 07:55:35
I tried to output the subframes with overwrite existing files enabled and a blank postfix, but encounter the following error.

I just want to add into the FITS header SSWEIGHT, as there are drizzle files attaching to the original registered files.

Using the new files cannot reuse the drizzle files.

This issue does not exist with the Subframe selector script.

Can you help?
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 December 05 09:58:10
is it possible that file is locked because it's open in another process or program?

Brian
Title: Re: Subframe Selector PCL Module
Post by: pfile on 2018 December 05 10:23:59
if i understand what is being done, the file is locked because SFS has it simultaneously open for reading and writing.

sounds like you'll have to write the files to another directory then move the files back to the original directory (thus overwriting the originals) by hand.

the script version of this tool might be reading the whole image into memory, closing the file, then writing the file from memory to disk. the SFS process i guess works differently. cameron would have to comment.

rob
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 December 05 11:25:33
Rob is correct, but there's more information than just the image to transfer, and basically all of it is directly from the input file to the output file. I never imagined someone trying to replace the input images, so perhaps it needs to check for an empty pre/post-fix before attempting the process.

I'm not certain that reworking the read/write to support this use case is the best idea. At least in my processing and the other workflows I've seen, using the Subframe Selector would come before registration (and thus before any drizzle files exist). The most important reason for this is that, depending on how much the images are shifted from registration, you can massively skew the metrics it uses and improperly weight the images. Another less important reason is that you might want to use the process to determine which image you're registering to as a reference.
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 December 05 11:29:48
oh I see what's going on.

yeah, i have to bite the bullet and create a new set of image files when doing SFS. i do this prior to registration, which avoids your issue. it's extra disk space and hassle, but it works for me


hth


Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 December 05 11:50:30
Rob's suggestion of using a different directory and moving them over the originals will work as well.

Overwriting files, especially the input files, is scary! It's unfortunate that it will take disk space, but basically every process will do this so it's not abnormal to me.

With a small drive, I'm frequently deleting the previous few steps' data if I'm comfortable running their processing again if I mess up, in order to be able to run most of my processing.
Title: Re: Subframe Selector PCL Module
Post by: Igor on 2018 December 19 08:17:53
Hello Cameron,

I just used the SubframeSelector process in the last 1.8.6 release of PixInsight. It works very well. Great job thanks :-)

Nevertheless Output Files are .XSIF. Is threre any way to output files in .FIT (even if FITS is a deprecated file format) ?

Thanks

Regards

Igor
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2018 December 19 08:39:19
yeah - nice job on the final implementation. the floating panels will take some getting used to but it continues to be integral to my processing workflow


Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 December 19 08:50:07
Hello Igor,
Thanks to Juan it now follows a pattern similar to most processes; that is initially hidden from the user but if you Edit the Instance's Source code with the icon at the bottom right, there's a parameter in there that you can change.
Title: Re: Subframe Selector PCL Module
Post by: Igor on 2018 December 19 09:00:18
Very good ! Thanks Cameron  :)
Title: Re: Subframe Selector PCL Module
Post by: samlising on 2018 December 19 13:25:06
Thanks Cameron for the great update to subframe selector.

I really like having both prefix and postfix available now. In the previous version on the script I hacked it so I could add a property like FWHM or Weight to the prefix or postfix of a file. Very useful for a quick sort of files. I'm wondering is there a way to add property variables to the postfix or prefix in the Subframe Selector PCL Module?

Thanks.
Sam
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2018 December 19 13:27:00
There is not, but that sounds like a good idea to me.
Title: Re: Subframe Selector PCL Module
Post by: samlising on 2018 December 19 13:29:54
Would be great if you could add this feature :)
Title: Re: Subframe Selector PCL Module
Post by: Robert-Poelzl on 2018 December 26 07:01:52
Hello at all,

Im used the old sfs for a long time and now the new subframeselector process.
Gratulation for the new tool !!!!

Before some days i did the update to the new PI Version 1.08.06.1448
Sometimes the tool worked well but most of time (and at the moment allways) an issue appears - see attachment.

I dont know what i can do....

Thanks for an idea.

kind regards
Robert
Title: Re: Subframe Selector PCL Module
Post by: Juan Conejero on 2018 December 26 09:05:04
Hi Robert,

This is a known problem with the current version. We are working to fix it, sorry for the inconvenience.

However, when this happens, please close PixInsight and run it again. The script execution error only happens occasionally, or at least this is what we have seen so far.
Title: Re: Subframe Selector PCL Module
Post by: Robert-Poelzl on 2018 December 26 10:55:41
Hello Juan,

Thanks for your info. No worries
Good look for your work.

kind regards
Robert
Title: Re: Subframe Selector PCL Module
Post by: STAstro on 2018 December 30 14:07:04
Am I missing something here, in your screenshots it shows the Subframe Selector and the Measurements window combined, in my 1.8.6 on Windows 10 it shows them as two separate windows

Title: Re: Subframe Selector PCL Module
Post by: astrosam on 2019 January 03 09:25:00
Hello,

I've been using the module with 1.8.5 and I am now trying with 1.8.6. The graph error appears almost all the time.

I'm using a laptop with high density pixels under windows 10, so that there is scaling of the screen and the pixinsight UI. Could it be related ????

Thanks,
Title: Re: Subframe Selector PCL Module
Post by: astrosam on 2019 January 03 14:45:28
Tried this evening on my other PC which has a "regular" display (no scaling) still under windows 10. No error appeared during manipulations :)

Other differences can exist (this is a fresh windows install, while the previous laptop has been running for 2 years), but this still is a hint.

Edit : I made another test on the "scaled" screen and it works with a different screen configuration. I forgot to mention that I'm working on multiple screen configuration, which seems to be an issue with Pixinsight : the auxiliary screen cannot be "above" the main screen. So I changed the screen positions to level both screen vertically, and the graphs are working properly. So that may be another cause of the issue, instead of scaling :).

Cheers,
Title: Re: Subframe Selector PCL Module
Post by: jruuth on 2019 January 05 14:03:43
New SubframeSelector process is great, now I can automate it from my script.

My problem is that it leaves Measurements and Expressions windows open. Is there a way not to open them, or close them from a script?

Another issue is that my script dialog window goes empty when I use SubframeSelector inside my script. All buttons and text is blank, I see only one frame in the script dialog window. This could be my use of Dialog but it did not happen until I started calling SubframeSelector.

Thanks,
Jarmo
Title: Re: Subframe Selector PCL Module
Post by: Robert-Poelzl on 2019 January 05 14:13:48
Hello at all,

I hope somebody can help me....
Im seeking for a "Approved" formula.

For me it wouldt be fine: FWMW<2.5    Eccentricity < 0.55      and SNRWeight < 17
I hope thats possible to combine this.
 
I tried but cant get no red crosses.
Screenshot is attached.

Thank everybody !
Robert
Title: Re: Subframe Selector PCL Module
Post by: RickS on 2019 January 06 00:56:26
Try using the boolean AND operation, for example: "FWHMSigma<2.5 && Eccentricity<0.55 && SNRWeight>17"  This will approve subs that meet all three criteria.  I turned around your SNRWeight expression.  For that parameter bigger is better :)
Title: Re: Subframe Selector PCL Module
Post by: PepeChambo on 2019 February 12 12:46:42
Graphs not works for me in new Subframe Selector.
Some clue why?

Title: Re: Subframe Selector PCL Module
Post by: PepeChambo on 2019 February 13 07:07:28
I have PI installed on two computers, and tried that graph works on Windows 7 but not in Windows 8.1 (or perhaps some library missing or unupdated or ...)  :sad:
Title: Re: Subframe Selector PCL Module
Post by: bvalente on 2019 February 13 14:03:57
Is it possible you didn't have any weighting expression?

Brian
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2019 February 13 18:41:01
Pepe, do you see the graphs from the newer PCC module? Previously, I was simply setting the graph view directly with the generated HTML and web scripts for the graph. However, Juan discovered an interesting bug in QT that was giving some users the crashing graphs issues. So, he reworked it to be like those, which technically write their HTML to a temporary file and load that into the view. It should work the same between these two.

I'm not familiar with the temporary directories on Windows, but checking the PCC and where these temporary files go would be my first troubleshooting steps.
Title: Re: Subframe Selector PCL Module
Post by: PepeChambo on 2019 February 14 07:02:38
Brian:
Graphs works in my PC W7 without any data, as you can see. Thanks.

Cameron:
Thanks for your response, but I'm sorry, I'm not a computer programmer ( only an astrophotographer  :D ) and do not understand these concepts. I know only that SubframeSelector process not works in my W8.1 laptop, and I have to return to old Mike Schuster's "SubframeSelector" in the "Script" Menu  :-\
Title: Re: Subframe Selector PCL Module
Post by: PepeChambo on 2019 February 15 09:43:50
Cameron, finally I found a solution for me.

Uninstalling process 01.04.04.0029 version integrated on PI, and downloading and installing your 01.04.01.0001 version.
It works fine ;)

Thanks Cameron.
Title: Re: Subframe Selector PCL Module
Post by: sharpie78 on 2019 February 15 13:12:16
I have installed the latest version of PixInsight 01.08.06.1457 and like the new SFS process (Thank-you Cameron) but it is giving me different results to the SFS script that I'm used to using......

I ran the new process and got FWHM results of 7-8. Knowing I had reasonable data that would normally give lower FWHM results, I decided to run the script instead.....FWHM came out around 4-5, which from past experience is typical for good data using my setup. Tbf all the measurements are a little different. I have the System Parameters all set the same in both.

Could someone please help me understand why this is, and if it's just a case of getting used to knowing what numbers represent good data, when using the new SFS process.

I've attached screenshots of both the script and process....if any of my settings look wrong please point me in the right direction  :D :D

.....the graph stopped working but thanks to this thread a reboot of PI sorted it.  :D

Thanks in advance....
Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2019 February 16 10:24:25
Have you used the same or similar numbers under System Parameters? Those directly affect the way you see those numbers. Also, this module uses DynamicPSF to calculate the values, whereas the older script has its own methods for calculations.
Title: Re: Subframe Selector PCL Module
Post by: sharpie78 on 2019 February 16 15:26:13
Thanks for getting back to me. I did use the same values under system parameters so from what you're saying I would assume it's the different method for calculating values that is giving me the differences.

Would you suggest just accepting this and adjusting my approval of subframe values to adjust for this increase ?
eg. 4.5 FHWM was a good sub using the script and the new module measures the same sub a 7.9 FWHM. Maybe set my approval FWHM as 8.5 ?

Title: Re: Subframe Selector PCL Module
Post by: cameronleger on 2019 February 16 15:56:25
Yes, updating it would be fine if you have tested those values and determined that in your case they are ideal numbers. If you change lenses, focal lengths, or cameras, I think it's better to look into the FWHMSigma, FWHMMedian, etc. values so that you could use the same expression for any case. That'd make the expression relative to the data-set instead of fixed values, though, so it's more like "best of this batch" and not "best ever".
Title: Re: Subframe Selector PCL Module
Post by: sharpie78 on 2019 February 18 14:36:07
Nice one. Thanks again. Whist it wasn't halting my processing, it was making me wonder. I can keep going forward and it looks like I've got some more learning to do......what FWHMsigma and median are. lol
Title: Re: Subframe Selector PCL Module
Post by: dragracingdan on 2019 February 18 16:06:03
When I 'Measure' my subs, some of them have a weight of 'inf'

What's going on here?

Thanks,
Dan
Title: Re: Subframe Selector PCL Module
Post by: CraigNZ on 2019 February 25 13:12:02
In the original SFS, output section, there were two sets of drop down boxes: approved, rejected.  In my case where I process batch streams of 200+ images, most of the images are okay and the two or three bad ones I want moved to a "Rejected" folder.  So I leave the approved action as "None" and the rejected action as "Move".  I then end up with approved images where they were and a new folder "Rejected" containing the bad images.  This makes it a lot easier when coming back to this job later and analyzing what was wrong with the rejected images.  It also means it is easy to move the rejected images back into the main folder to rerun SFS with a different set of parameters.

Can we get this same functionality added to the new PCL SFS?

Craig
Title: Re: Subframe Selector PCL Module
Post by: niccoc1603 on 2019 March 23 14:48:44
hi, I am using the module on 1090 subframes.
But when I get to about 800-900 frames analyzed (in the console) the PC crashes
Title: Re: Subframe Selector PCL Module
Post by: pk825 on 2019 June 04 01:51:33
Thanks Brian,
I can confirm the issue is with the Weighting Expression, not the Approval Expression, although that one can display the X for some reason.

The 'plain' variables such as FWHM and Eccentricity only represent the current subframe being measured. So, Math.min(FWHM) and Math.max(FWHM) will return the same thing as FWHM, and subtracting these values will result in 0. The 'nan' error appears when it's dividing by 0.

The simple fix here is to use the 'special' variables such as FWHMMax and FWHMMin which are calculated before each subframe and represent the global max and min of all subframe FWHMs. Here's that expression modified to work for me:

Code: [Select]
35*(1-(FWHM - FWHMMin) / (FWHMMax - FWHMMin)) + 7*(1-(Eccentricity - EccentricityMin) / (EccentricityMax - EccentricityMin)) + 18*((SNRWeight - SNRWeightMin) / (SNRWeightMax - SNRWeightMin)) + 40

That said, I will definitely look into 'nicer' or more obvious messages when 'nan's are involved, and try to make the X's work better in these cases.

Could you help me understand if is it possible to get the min or max of variables such as FWHM , Eccentricity or SNRWeight of the analysed frames?
Title: Re: Subframe Selector PCL Module
Post by: pk825 on 2019 June 05 07:40:35
Thanks Brian,
I can confirm the issue is with the Weighting Expression, not the Approval Expression, although that one can display the X for some reason.

The 'plain' variables such as FWHM and Eccentricity only represent the current subframe being measured. So, Math.min(FWHM) and Math.max(FWHM) will return the same thing as FWHM, and subtracting these values will result in 0. The 'nan' error appears when it's dividing by 0.

The simple fix here is to use the 'special' variables such as FWHMMax and FWHMMin which are calculated before each subframe and represent the global max and min of all subframe FWHMs. Here's that expression modified to work for me:

Code: [Select]
35*(1-(FWHM - FWHMMin) / (FWHMMax - FWHMMin)) + 7*(1-(Eccentricity - EccentricityMin) / (EccentricityMax - EccentricityMin)) + 18*((SNRWeight - SNRWeightMin) / (SNRWeightMax - SNRWeightMin)) + 40

That said, I will definitely look into 'nicer' or more obvious messages when 'nan's are involved, and try to make the X's work better in these cases.

Could you help me understand if is it possible to get the min or max of variables such as FWHM , Eccentricity or SNRWeight of the analysed frames?

I understand how it works and I have to use variables FWHMmax e FWHMmin.