Jeff,
thanks for providing the images. So far I have had a look at the luminance files (those with an L in the name), and I did not have a lot of trouble getting decent reults. Let me outline the steps:
1. Create reference mosaic: This mosaic is later being used to aligned the images for GMM. Use StarAlignment with Registration Model "2D Surface Splines" and Distortion Correction enabled (to adjust for slight distortions between the images). Set mode to "Register/Union Mosaic" and Reference View to one of the images, so that StarAlignment will produce a merge between reference image and the image you apply it to. Activate Frame Adaption so images get similar brightness. Apply this to one of the neighbors of the Reference View, make the result the reference view and repeat with the next image until to have a combination of all images. The original image and StarAlignment settings can be seen in screenshot 1, result in screenshot 2 (next post) left. The result is not bad, but we clearly see the borders between images.
2. Align images to reference mosaic: Use the same settings in StarAlignment as in step 1, but in Register/Match Images mode. Settings in screenshot 2 right. Result in images of screenshot 3 (next post). Tje aligned images show the parts of the image in the correct position
3. First GMM attempt with aligned images: Use the default GMM settings, see screenshot 3. The result is in screenshot 4. The result is not bad, except for a few artifacts, the most prominent once marked with previews. They are caused by stars that are just on the border of one of the images with slightly different brightness values to the corresponding pixels in the other image. This does not matter a lot for faint structures, but for bright once the differences are huge, especially in a linear image.
Workaround is to stamp out those stars with the CloneStamp tool. Set radius to 14 or so, and strictly speaking you should also set Softness to 0 to get a sharp change to black (I forgot that during my work - it was not harmful). Do GMM again with the modified image. Settings in screenshot 4 (next post).
4. The result in screenshot 5 (next post) no longer has the artifacts - there are some more if you look very carefully, but they can be handled the same way.
Let me know if this procedure solves your problems. It should be the same for the other channels as well. It would be interesting for me to know what you did differently, and why this caused the problems you see.
Georg