Hi Sara,
I think I try to do TOO much, but can't seem to get anyway near anything decent, and in trying harder it all just gets worse.
Your analysis is correct IMO. The main problem is that you're trying to get too much from the data. You have data to make a decent image, but not enough signal to achieve the result you want, especially in terms of color saturation.
Below is my try applying very basic steps, plus a final trick (I am against tricky processing in general, but I've made an exception this one time).
Step 1. Alignment of the individual RGB channel images.
Step 2. Combine the three channels as a RGB color image. Use STF AutoStretch to inspect the linear image.
Step 3. Gradient removal with AutomaticBackgroundExtractor (ABE).
Step 4. Fix residual gradients with DynamicBackgroundExtraction (DBE).
Step 5. Neutralize the background.
Step 6. Set a plausible white balance to maximize information representation (aka color calibration). The white reference is the integrated light from the main galaxy.
Step 7. Make a noise reduction mask: Extract a brightness component (I channel of HSI in this case) and make it nonlinear.
Step 8. Noise reduction with MultiscaleMedianTransform, with the mask of step 7 enabled. Trick: don't stretch the image too much with STF. Just be realistic and stretch it moderately with low signal data.
Step 9. Adjust the mask to control noise reduction.
The problem: With low-signal data noise reduction has to be applied at large scales; otherwise the background gets filled with the typical 'blotches', as a result of surviving large-scale noise structures. However, too strong of a noise reduction leads to a 'washed' background with an artificial plastic look.
The real solution: Gather more data.
A tricky solution: Force a minimum mask value with PixelMath. In this way you get the noise only partially removed at all scales (i.e. noise reduction,
not noise suppression).
Step 10. Nonlinear stretch.
Step 11. Now the really tricky part: Selective noise addition.
Important note: I don't like or recommend this procedure, in general. I have implemented it in this example because it yields a good result from an aesthetic point of view, given the lack of signal in the original data.
This procedure is borderline of bad practice. The idea is simple: add a moderate amount of synthetic Gaussian noise on the background (with mask protection) to replace artificial looking noise with a 'nice and smooth' noise distribution.
Step 12. Color saturation enhancement. A lightness mask has been used to protect the background.
Step 13. The final result after applying SCNR with default parameters to remove residual green noise.
I think the result is very decent. After just a few easy steps (with the exception of the noise addition trick), The image is colorful, the noise is under control, and some nice details can be seen in the galaxies. Not bad for that orange Olocau sky
I hope this contributes to remove most of your disheartenment