WBPP 2.0 released

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
There's two features I've asked about in the past and I'm curious if they were added or planned to be added:

Custom weighting options? Would be amazing to be able to load our own weighing formula. Like using Stars instead of or in addition to median.
yes it would be nice but the implementation is currently far from this possibility....

Saving the weighing keyword into the fits header without going past the registration step? Iirc, WBPP would run its weighting but not actually save anything into the files at that stage, which was frustrating for those of us who like to look at SFS manually.
This is in place :) indeed when you enable the generation of the weights the image measurements are stored into the measured file's FITS header along with the weighs. If you run again the WBPP using these images (or using the registered version) measurements will be taken from the FITS header and images measurement will be skipped. This speeds up quite a lot the execution of the WBPP applied on registered images only in order to integrate with different weights (as an example).
In case you would ignore the measurements in the FITS header and measure the images again, you can check the option "force images measurement".
 
Last edited:
  • Like
Reactions: aatdalt

physiks_guy22

New member
Feb 1, 2019
2
0
How do I install this update? Sorry, I did try everything (or just about) and can't figure it out. I went to updates inside PixInsight and it says there is nothing to update. I went to the pixinsight website and there's nothing about WBPP 2.0, please help. (I even googled it...)
 

pfile

PTeam Member
Nov 23, 2009
6,638
253
How do I install this update? Sorry, I did try everything (or just about) and can't figure it out. I went to updates inside PixInsight and it says there is nothing to update. I went to the pixinsight website and there's nothing about WBPP 2.0, please help. (I even googled it...)
are you running the latest version of pixinsight? (1.8.8-7) you need to update the PI application itself separately by downloading the new version from the distribution center. core application upgrades are not done via the update system.

rob
 
  • Like
Reactions: physiks_guy22

physiks_guy22

New member
Feb 1, 2019
2
0
are you running the latest version of pixinsight? (1.8.8-7) you need to update the PI application itself separately by downloading the new version from the distribution center. core application upgrades are not done via the update system.

rob
Thanks, Rob! I'll do that! Clear skies!
 

pfile

PTeam Member
Nov 23, 2009
6,638
253
Script > Batch Processing > Weighted Batch Preprocessing

if it is not there, try Script > Feature Scripts, then click the "regenerate" button.

rob
 

aatdalt

Member
Feb 24, 2020
8
0
I understand the helpfulness of auto-populating master files based on filenames and keywords but would it be possible to add a way to designate a file as a master file (flat, dark, bias) from within the WBPP console? This seems like something missing.
 

sn2006gy

Well-known member
Jun 10, 2020
53
19
Is it possible to add a tuneable to how many compute descriptor tasks can run in parallel? Running on a 32core or 64 core machine, this is now my longest-running task. My cpu load never gets much above 7-9 with, it seems like I have room to double or triple the workers for this task potentially cutting this time down by 1/2 or more... I can see its threaded because every cpu gets a bit of work... but its certainly not IO bound, nor is it memory bound and it seems like one could brute force this quite a bit more :)

for example, right now I'm running this on 1440 subs and it will take 4-5 hours to complete with 32cpus and 512gb ram on nvme storage (roughly 9-10 seconds a sub - asi 2600mc pro 26mp subs)
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
Hi @sn2006gy,
for the moment the images are measured sequentially by the Javascript code and the JS engine is single-threaded so there is no way to execute multiple measurements in parallel. What is under investigation is to have the measurements to be parallelized by means of a dedicated internal process which will take advantage of multi-CPUs but for the moment there is nothing concrete.

In any case, this is definitely the main bottleneck of the whole process so it's with to put some effort in that direction.

Robyx
 

sn2006gy

Well-known member
Jun 10, 2020
53
19
Hi @sn2006gy,
for the moment the images are measured sequentially by the Javascript code and the JS engine is single-threaded so there is no way to execute multiple measurements in parallel. What is under investigation is to have the measurements to be parallelized by means of a dedicated internal process which will take advantage of multi-CPUs but for the moment there is nothing concrete.

In any case, this is definitely the main bottleneck of the whole process so it's with to put some effort in that direction.

Robyx
cool, thanks for following up! glad it's being looked into. I put a whopper of a machine into integrating and this is just the last bit that seems to not beat it up :)

thanks!
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
I understand the helpfulness of auto-populating master files based on filenames and keywords but would it be possible to add a way to designate a file as a master file (flat, dark, bias) from within the WBPP console? This seems like something missing.
Hi @aatdalt , the step towards this automatic approach follows a pattern that in programming is called "convention over configuration". In a short, this principle is about reducing as much as possible the time spent on configuring things by following proper conventions. In the WBPP context, this helps to reduce the number of features and the complexity of the interface (and the code behind) at the minimum while requiring a reasonably small effort in adopting some standard conventions that limit the number of different cases to be managed due to "custom" conventions adopted by users or any other software.

Indeed, the solution to simply add "master" to the file name can be easily addressed by anyone and, moreover, follows a good practice regarding properly naming the name for what it is: a master file. That's not mandatory, it's just a best practice that is reasonable to follow.

From this perspective, allowing a non-conventional master file to be used as a master represents a case out of the scope for WBPP, moreover, master files produced by WBPP follow this convention, so they are 100% compatible.

My suggestion is to rename your master file properly or if you need to use a non-master file as master, just create a copy and name it as it would be a master.
 
  • Like
Reactions: aatdalt and dld

MartinB2444

New member
Nov 4, 2019
1
0
Sorry for the seemingly dumb question but where do I find the WBPP2 script to install. I can't see it in the repository.
 

clabille

Member
Aug 23, 2019
5
0
Script > Batch Processing > Weighted Batch Preprocessing

if it is not there, try Script > Feature Scripts, then click the "regenerate" button.

rob
Hi. I'm running version 1.8-8-7 and have also tried this "regenerate" process. It did not do anything that I can tell. I still only see WBPS v1.5.2.
 

pfile

PTeam Member
Nov 23, 2009
6,638
253
Hi. I'm running version 1.8-8-7 and have also tried this "regenerate" process. It did not do anything that I can tell. I still only see WBPS v1.5.2.
but have you successfully run the update and restarted PI before trying this?

rob
 

aacc

New member
Feb 17, 2021
1
0
Looks great. I already tried, and like it much better than the previous version.

Quick question. I don't see a way to Drizzle. I can see that it can generate the _d.txt file, but there isn't a clear way to actually generate a drizzled file. Maybe I am missing something.
 

sbenz

Member
Jul 8, 2017
16
0
I ran the script yesterday on a set of lights, darks, flats and flat darks.
It really worked great!

Two observations though:
- The script would not detect the "flat darks" automatically to be darks (despite having "flat dark" in the name), but would sort them to be flats. No big deal, just have to add them manually as darks. Calibration works fine.
- After running the script I find a master flat file in the folder, but the script seems to neither store the master dark nor the master flat dark, at least I could not find them. Did I miss something?

Thanks
Stefan
 

freed

Well-known member
Mar 6, 2017
107
5
A request for a future version. Provide the ability to set instances of cosmestic correction per filter. In practice, I find that a need different options on cc for braodband and narrowband.
Thank you
 

Djibi

Member
Jun 11, 2020
12
4
Thanks a lot for the great work that has already been performed up to now.

My usage of WBPP is so limited that I know I am not using it to its full potential.

I have one question, I don't know if it is a feature I am missing or a feature that is not integrated in the current version of WBPP: is there or do you plan to integrate the possibility to perform an Automatic Background Extraction for each individual frame?

This could be useful to add a "simple" ABE at this level: if it were for me , I would try to use it with a function degree limited to 1 (but of course this would be a parameter).

thanks again for your woirk :)
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
Hi @Djibi,

the ABE is a step that strongly belongs to the post-processing phase, indeed WBPP remains a tool focused on PRE-processing i.e. data calibration, the fact that light frame weighting, registration and integration have been added is simply to provide a comfortable way to "preview" the result of your session.
Indeed, frame inspection, weighting, registration and integration need some fine tuning iterations to really get the best out of your data and ensure that no artifacts are produced.

From this perspective, ABE is not only "out of scope" but more dangerously it can't be applied blindly since it's pretty common to perform some iterations before getting to a good result with this tool, at each iteration you have to inspect the outcome of ABE and adapt some thresholds and other parameters that can hardly be tuned with an automated process. Moreover, consider that ABE rarely produces a result that can be acceptable without running DBE as an extra refinement.

So, it's not "impossible" by itself but this addition would produce just a partial and an unreliable result.

Robyx
 
  • Like
Reactions: Djibi

Djibi

Member
Jun 11, 2020
12
4
Thanks @robyx
Indeed, I have learnt (I have been discovering imaging and post processing since less than a year !) the "danger" of gradient removal when it is not performed adequatly.
I understand the mathematical modelisation and assumptions that lie behind this process so I am aware that if you are not cautious, you may remove some "true" signal

But, without the knowledge and experience that come with the time, I have seen some articles where this is adviced as a first step that is useful for a more precise gradient removal during post processing. Of course you must be very cautious, in particular limiting the "function degree" of the interpolation polynom is very important; personnaly I would never go above 1 at this stage.
And you must take other preacutions : for example you would not use such a feature for a frame with too many nebulosity in it.

When I read these articles, this was making sense to me : you can test on a single frame if it is relevant and decide to use or not, I can not judge the statements that say "performing this action at this stage will make the full gradient removal easier at a later stage"

I am not using the fact that some software are having this feature during the integration process (I am thinking of APP), it is very instructive for me to see that on such a subject, there are various vision of what should be done and what should be avoided

I will take the time in the future to look at these statements :unsure:, food for thoughts !

Thanks for your explanation

Jean-Baptiste alias Djibi