Scaled video modes work perfectly on our iMac machines with 1.8.4.1195, so this is really strange. I don't know why this happens, just that it cannot happen, in theory. Most likely this is a Qt bug, which doesn't surprise me too much on OS X, actually.
Fortunately, you have found a workaround by changing to a scaled display mode while the application is running, so stick with it!
Or you can try the following: EDIT > Global Preferences > Core UI Resources > Automatic UI scaling (disable option). Make sure that UI scaling factor = 1, then press F6 to execute globally. Exit PixInsight, select a "more space" video mode, and run it again. Perhaps this will work.