The theoretical maximum image dimensions are (231-1)x(231-1) pixels, since image coordinates are internally represented as 32-bit signed integers.
You're experiencing a 2 GB file size limit. Assuming you're working with an NTFS filesystem, this shouldn't happen on Windows. On 64-bit Linux, I've just written a 32-bit floating point image with 13500x13500 pixels to a FITS file without problems, on a laptop with 8 GB of RAM. The same on Mac OS X.
So the problem must be with CFITSIO, which is probably using old Win32 APIs for FAT32 compatibility. I'll investigate it.