Hi Sander,
I fully agree with Display instead of Mode, or "mask display mode", depending on the context. Thanks for the suggestion, I'll make the change in the next version.
why anyone would want to replace the red channel with a mask or use any of the other colors.
The red channel isn't replaced on the screen. The mask is displayed as an "overlay" —just as if the mask were a monochromatic, translucent film layer placed on top of the image.
An overlaid mask representation has some important advantages over the classic multiplicative representation (classic in PI, of course). With an overlaid mask, you can see the true protection effect of the mask on all parts of the image. With a multiplicative representation, dark or bright areas cannot be represented correctly, depending on the mask used. For example, this is a big problem with deep-sky images using noninverted masks, mainly because low values (the sky background) multiplied by low values (the sky in the mask) are much lower values. With inverted masks, the same happens for bright areas. This never happens with overlaid masks.
The different colors are there because depending on the image, a particular mask color can be useless. For example, a red mask representation is bad for an image where the red color dominates. The particular colors offered as mask display modes (red, orange, etc.) cover virtually all possible cases, and can be implemented with very efficient code by PI's screen rendering engine.