Hi Julian,
What are the exposure times for each channel? Please tell us the sub-exposures and total number of subs for each channel.
My processes are based on Harry's video tutorial (
http://www.harrysastroshed.com/pixinsighthome.html ) and PixInsight video tutorial (
http://www.pixinsight.com/videos/NGC1808LRGB-vperis/en.html ). I will try my best to explain what I did. My processing skill level is beginner/intermediate.
1) Combine Sii/Ha/Oiii using LRGB Combination. Uncheck "L" box.
2) Crop the image to remove black borders using Dynamic Crop.
3) DBE. This one is tricky because nebulosity was all over the image. It was hard to find background not covered by nebulosity. I had to make lots of guesses. I am not sure if this is a good way. The image looks better if you un-link RGB channel in STF after clicking on "A" (autostretch). But I linked RGB channel and it was easier to find darkest areas this way. I don't think this is the best way to do it. You could try Automatic Background Extractor (ABE) and use default values.
4) Background Neutralization. Create a small preview at area with little to no nebulosity coverage.
5) Color Calibration. Use same preview as #4 for Background reference. Create a new preview at Thor's helmet area for White reference. Uncheck Struture Detection. There are several ways to do Color Calibration, this is the most common way.
6) Multi Scale Median Transform (MMT) to reduce noise while image is still linear. Use Harry's tutorial at
http://www.harrysastroshed.com/at%20noise.html . The tutorial uses Atrous Wavelet but it's very close to MMT. Watch it and you will get the idea.
7) Histogram Transformation. I use iterative process. I start with middle slider and move to left, apply to image. Then use left slider and move it to right (don't clip it!!!!) and apply to image. Repeat the process until it looks good. Don't get too greedy and over stretch too much.
8 ) Create star mask for HDR and Morphological Transformation. Increase threshold to remove noise, you only want to see stars. Sometimes you need to increase to 0.500. Also you may need to increase the scale if big stars don't show up on the map. Apply star mask to image.
9) HDR. Make sure that you invert the mask so that the stars are red in order to protect the stars. Test HDR with several different layers for HDR. I think 4 or 5 layers are good. Either use Linear Interpolation or B3 Spline (5) for Scaling functions.
10) Invert the mask again to protect everything except the stars. It's up to you to make the stars smaller using Morphological Transformation. I use Circular Structure, 2 iterations and adjust the amount to between 0.4 and 0.8. The less amount the less reduction.
11) Remove star mask. Extract Luminance and apply Luminance mask to image. Run Local Histogram Equalization. Adjust Contrast Limit to your taste. Play with between 0.2 and 2.5. I used 2.0. Remove Luminance mask.
12) ACDNR. I don't adjust Luminance mask for this. For Lightness, I check Lightness Mask, Std Dev. to 2.0. For Chrominance, Lightness Mask is checked and Std Dev. to 3.0. Apply to image.
13) Curves Transformation using "S". Extract Luminance and apply Luminance mask to image. Use preview. Click and drag at the center to upper left direction until it's to your taste. Apply to image. Remove Luminance mask.
14) Curves Transformation using "RGB/K". Create a "S" shaped curve. It's usually very small adjustment.
15) If the image looks too soft, you could try Unsharp Mask. I just discovered this recently. Play with Std Dev and Amount. Go very easy on this. I think it works better if you apply Luminance mask first. I normally do not use this tool because they can look too fakey.
I hope this is clear. I follow the video tutorial and not necessarily understand why it works. As time goes on, you will slowly learn why they work.
Attached is an updated image using Unsharp mask.
Peter