first, the new dialog box for StarMask seems like a step back. the older one was much more compact.
but the real problem is, it seems to be broken. with default parameters against a very pedestrian image, i get the following - screenshot attached. the default noise threshhold is probably too small for the image but adjusting it just reduces the # of detected stars as you'd expect. something's weird about the size of the stars in the mask (too huge) and it seems to not detect stars at the top of the frame properly...