you apply this thru a mask, right?
i guess what's happening is that when you don't use "replace image" you get a copy of the original image, which is correct. if you were to use replace image without a mask, you'd overwrite the whole target image with the source image. but since you are using a mask, only those pixels exposed by the mask are written by pixelmath.
i think if you want a new image of just the changes, you'd have to use the pixelmath expression star_mask*rgb_stars (or maybe ~star_mask*rgb_stars).
rob