This often works for me. First, extract the luminance and then use histogram transformation to brighten th DSO and darken the background. Then create star mask from your original image. Finally, subtract the star mask from the luminance mask using pixelmath. Applying the expression, $T - star_mask to the luminance mask will subtract the stars. Now the luminance mask will have the DSO selected, but no stars or background.
Cheers,
Jason