The StarMask tool uses a multiscale algorithm to isolate significant image structures (this is the 'structure detection' phase). Unfortunately, a drawback of many multiscale processes is that they are not fully previewable. The reason is that these algorithms are strongly dependent on large-scale features of the image as a whole, which vary when they work on a partial preview. There are cases —as the one you are encountering, probably—, where the differences can be quite significant.
Another example of non-previewable multiscale process is HDRWaveletTransform. In general, you can use a partial preview consistently only if it covers the whole range of brightness present in the image; otherwise the preview results are not guaranteed to be the same obtained for the whole image.
We have these problems in mind and always try to find ways to overcome them, but it is difficult. Anyway, the next version of these tools will provide a significant performance improvement (along with all wavelet-based tools in PixInsight), so this will be less of a problem.
To prevent these problems, define a complete preview (one covering the whole parent image) and work with it. At least, in this way you can get the benefits derived from the volatile nature of previews (no disk swap files and automatic undo), especially with your L-sized RAM
