I was playing around with this very thing (again) last night, and found exactly the same problem. Red/Ha is very bright. I also struggle figuring out how the iterations work, because taking the new RedSynthetic (as you call it) and dividing again by Ha gives you exactly the same continuum C as you just had - unless I'm missing something in that step.
I have seen this question asked many times, and have asked it myself several times, and have never seen a conclusive clarification on the method. Invariably you just get pointed back to the NBRGB (formerly HaRGB) scripts.
http://pixinsight.com/forum/index.php?topic=4120.msg28692#msg28692http://pixinsight.com/forum/index.php?topic=3446.msg29566#msg29566http://pixinsight.com/forum/index.php?topic=5644.msg38594#msg38594Where I ended up last night was exactly where you did with that PixelMath solution with nonlinear (stretched) images:
R=max($T,k*(_Ha-($T-_Ha)*5/100))
G=$T
B=max($T,BAL*_Ha)
k=0.9,BAL=0.15
The 5/100 is for my Astrodon filters. The BAL is to put some Ha in B for bit of balance, but not necessary. k is a factor to play with for balance too. The formula for R was from here:
http://pixinsight.com/forum/index.php?topic=2298.msg15353#msg15353, but don't believe it is truly faithful to "Vicent's Method" as I understand it.
The results were pleasing enough, and it was late, so I left it there for now. Would like to investigate further if anyone can shed some light here. This has to be one of the most common questions - how to blend Ha (or any NB) with RGB - and "Vicent's Method" gets referred to a lot, yet haven't seen any further clarification or posts on it since the original was written. I'd be happy to be proven wrong and pointed to posts that do.