Here’s what I did (based on a combination of info at http://h8300-hms.sourceforge.net/ and The legOS HOWTO installation page) to build GNU binutils, gcc, and brickOS and associated tools on Mac OS X Tiger 10.4.3 with Xcode 2.2:

My gcc setting in case it matters:

$ gcc_select 
Current default compiler:
gcc version 4.0.1 (Apple Computer, Inc. build 5247)

First, I setup a directory just for brickOS tinkering:

mkdir brickos
cd brickos/

binutils

wget 'ftp://ftp.gnu.org/gnu/binutils/binutils-2.9.1.tar.gz'
# (tried with binutils-2.16.1.tar.gz, but ran into 
# "invalid operand/opcode" problems later when building BrickOS)
tar -xzf binutils-2.9.1.tar.gz
cd binutils-2.9.1/
mkdir objdir
cd objdir/
../configure --prefix=/usr/local --target=h8300-hms
make CFLAGS="-O2 -fomit-frame-pointer" all
sudo make install

gcc

wget 'http://easynews.dl.sourceforge.net/sourceforge/h8300-hms/h8300-hms-gcc-3.1-1.patch'
wget 'ftp://ftp.gnu.org/pub/gnu/gcc/gcc-3.2.1/gcc-3.2.1.tar.gz'
wget 'ftp://sources.redhat.com/pub/newlib/newlib-1.14.0.tar.gz'
tar -xzf gcc-3.2.1.tar.gz
cd gcc-3.2.1/
ln -s ../newlib-1.14.0/newlib .
patch -p1 < ../h8300-hms-gcc-3.1-1.patch
cd ..                            
# gcc docs recommend that you don't put your 
# objdir inside of the gcc directory
mkdir build_gcc_h8300-hms
cd build_gcc_h8300-hms
../gcc-3.2.1/configure \
        --prefix=/usr/local --target=h8300-hms \
        --enable-languages=c,c++ --with-newlib
make CFLAGS="-O2 -fomit-frame-pointer" all
sudo make install

Building brickOS

wget 'http://easynews.dl.sourceforge.net/sourceforge/brickos/brickos-0.9.0.tar.gz'
tar -xzf brickos-0.9.0.tar.gz 
cd brickos-0.9.0/

The firmdl3 utility wouldn't build for me on Mac OS X without the following patch, although it's sort of a moot point because as you'll see later, I couldn't get firmdl3 to work with my USB IR tower and had to resort to using the improved lejosfirmdl utility in leJOS (another RCX firmware for Java programming).

--- rcx_comm.c.orig     2006-01-06 22:40:05.000000000 -0800
+++ rcx_comm.c  2006-01-06 22:40:19.000000000 -0800
@@ -102,7 +102,7 @@
 {
     char *bufp = (char *)buf;
     int len = 0;
-#if defined(LINUX) | defined(linux)
+#if defined(LINUX) | defined(linux) | defined(__APPLE__)
    int count;
    fd_set fds;
    struct timeval tv;

Other than that, building brickOS worked without a hitch:

./configure 
make

Building lejosfirmdl

I tried to use the firmdl3 utility that comes with brickOS to download brickOS via my USB IR transmitter, but I couldn't get it to work. I tried setting RCXTTY to "usb" and "/dev/usb/lego0" and even installing libusb from Fink but it wouldn't work. Glancing at the code, it looks to me like the legOS firmdl USB support only works with Linux (using LegoUSB) and Cygwin? Anyway, a search on the Web brought me to the version of firmdl that comes with leJOS and I saw that the leJOS firmdl source code had #ifdef \_\_APPLE\_\_ sections with Mac OS X IOKit USBLib code inside them. So I downloaded leJOS so I could build the lejosfirmdl utility in there:

wget 'http://easynews.dl.sourceforge.net/sourceforge/lejos/lejos_2_1_0.tar.gz'
tar -xzf lejos_2_1_0.tar.gz 
cd lejos_2_1_0/tools/firmdl/

In order to build leJOS without errors, I needed this patch. The other route is to try to make just lejosfirmdl, but that also requires a patch (a hack really) due to the way the Makefiles are structured:

In tools/firmdl:

--- Makefile.orig       2006-01-07 05:03:23.000000000 -0800
+++ Makefile    2006-01-07 05:13:50.000000000 -0800
@@ -26,7 +26,7 @@
 
 CC=$(shell ../../cctest.sh)
 
-OS=$(shell echo $$OSTYPE)
+OSTYPE=$(shell echo $$OSTYPE)
 
 CFLAGS = -O2 
 

Then:

make

to build ../../bin/lejosfirmdl.

When I fired up lejosfirmdl, I saw the light on the USB IR transmitter momentarily light up, a very good sign:

~/lego/brickos/brickos-0.9.0$ lejosfirmdl boot/brickOS.srec
segment.length[0] = 11336 (12k)
Total image size=11336 (12k)
RCXTTY undefined. Using: usb
/usr/local/bin/lejosfirmdl: no response from rcx

Looking much better... Time to go get the RCX and see if I could download brickOS...

(Hmmm. I wonder if the brickOS folks are planning to merge these Mac OS X USB changes to their own version of firmdl, or perhaps a better way is to do USB access through libusb, which I believe will abstract out the underlying OS implementation - the source code is definitely from the same origin although they seemed to have forked off from each other and things are perhaps a bit messy right now).

Downloading brickOS to the RCX

This part was very straightforward and it took only a couple of minutes to complete.

~/lego/brickos/brickos-0.9.0$ lejosfirmdl boot/brickOS.srec
segment.length[0] = 11336 (12k)
Total image size=11336 (12k)
RCXTTY undefined. Using: usb
Transferring "boot/brickOS.srec" to RCX...
100%        

The RCX counts up on the LCD display while it's downloading. When it's done, you can check that brickOS was successfully downloaded by pressing the Prgm or Run button on the RCX and verifying that the LCD reads "None" (indicating that no programs have been downloaded to the RCX yet).

The Catch

Even though I was able to use lejosfirmdl to download the brickOS firmware to the RCX, I have no way yet of downloading my own programs to the RCX from the Mac, because the dll utility in brickOS also doesn't work with USB on the Mac as far as I can tell and dll is not present in leJOS. Bummer.

It would be nice to write new versions of brickOS firmdl3 and dll that use libusb or at least that incorporate the IOKit USBLib code in lejosfirmdl.

I posted a link to this document on lugnet.robotics.rcx.legos