Since Apple officially supports X11 now, I am going to use it in what follows.
Apple's X11 for MacOS X
The X11 package comes with 10.3 stored as /Applications/Installers/X11User.pkg or in Panther Installation Disk 3. Double-click the package icon and install it if you haven't. In addition, you need to install the X11 developer package stored as /Applications/Installers/Developer Tools/Packages/X11SDK.pkg or in Xcode Tools CD. If you haven't installed it, install it.
In addition, if you want to install RPM packages that depend on X11's dynamic shared libraries, you need to install a dummy X11 package as:
# rpm -Uvh <somewhere>/X11-4.3.0-1a.ppc.rpm
in order to resolve unsatisfied dependencies (this packages does not install anything but simply registers the X11-related shared libraries into the RPM data base).
You may also need to install the Xaw3d library as:
# rpm -Uvh <somewhere>/Xaw3d-1.5-19a.ppc.rpm
if you want to install packages such as gv, etc.
Customization of X11
Optionally, you can add more fonts as you need: in my case, I added small fonts for my PowerBook's LCD.
# cd <somewhere>/fonts
6x12rk.pcf.gz fonts.alias.75dpi.added s5x8.pcf.gz
# cp *.gz /usr/X11R6/lib/X11/fonts/misc/
# mkfontdir /usr/X11R6/lib/X11/fonts/misc/
# cat fonts.alias.75dpi.added >> /usr/X11R6/lib/X11/fonts/75dpi/fonts.alias
# cat fonts.alias.misc.added >> /usr/X11R6/lib/X11/fonts/misc/fonts.alias