Easier way to build _bsddb for OS X

I blogged previously about how to build _bsddb for Python 2.6 by hacking the Python source code to accommodate changes in berkeley-db.

Here’s an easier way that doesn’t require hacking source code.

Install berkeley-db using Homebrew:

brew install berkeley-db

Install the bsddb3 module, pointing it at an installation of berkeley-db:

sudo \
    BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/5.3.28 \
    pip-2.6 install bsddb3

Simply copy it into the appropriate place as _bsddb.so. This is super hacky and will fail if the interfaces of these two ever change, but for now for my limited purpose, it seems to work:

cp \
    /Library/Python/2.6/site-packages/bsddb3/_pybsddb.so \
    /Library/Frameworks/Python.framework/Versions/2.6/lib \
      /python2.6/lib-dynload/_bsddb.so

Enjoy:

$ python2.6
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
>>> dir(bsddb)
['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']

Building _bsddb for Python 2.6 on OS X 10.8.5

It’s well-known that the Python that comes with OS X machines typically doesn’t have _bsddb.

I downloaded Python 2.6.8 and attempted to build it and it wouldn’t build the _bsddb module.

Here’s what I did to fix this.

I used Homebrew to install berkeley-db4. For me, this grabbed version 4.8.30.

brew install berkeley-db4

Edited Modules/_bsddb.c and replaced all instances of DB_XIDDATASIZE with DB_GID_SIZE. I removed lines that had DB_XA_CREATE.

Compiled and linked with:

gcc -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/berkeley-db4/4.8.30/include/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c /Users/marca/src/Python-2.6.8/Modules/_bsddb.c -o _bsddb.o
gcc -bundle -undefined dynamic_lookup _bsddb.o -L/Library/Frameworks/Python.framework/Versions/2.6/lib -L/usr/local/lib -ldb -o _bsddb.so -Wl,-search_paths_first

This created the file Modules/_bsddb.so.

At this point, I can import if I run python in the Modules directory where I built _bsddb.so:

$ python2.6
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
>>> dir(bsddb)
['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']

Now you can copy _bsddb.so into your lib-dynload directory — e.g.:

sudo cp _bsddb.so /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload

(above is for the system Python 2.6 that comes with OS X)

or

sudo cp _bsddb.so sudo cp /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload

(above is for a Python 2.6 installed from a .dmg from Python.org or built from source as a framework build)

References:

Hacking OS X’s Python dbhash and bsddb modules to work

By default on my Leopard system, the dbhash and bsddb Python modules cannot be loaded.

marc@hyperion:~$ python
Python 2.5.1 (r251:54863, Oct  5 2007, 21:08:09)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbhash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.5
        /lib/python2.5/dbhash.py", line 5, in <module>
    import bsddb
  File "/System/Library/Frameworks/Python.framework/Versions/2.5
        /lib/python2.5/bsddb/__init__.py", line 51, in <module>
    import _bsddb
ImportError: No module named _bsddb

I managed to get it to work though by installing bsddb3 from the pybsddb site and hacking the file /System/Library/Frameworks/Python.framework/Versions/2.5
/lib/python2.5/dbhash.py
:

marc@hyperion:~$ diff -u dbhash.py.orig dbhash.py
--- dbhash.py.orig      2007-11-28 10:16:33.000000000 -0800
+++ dbhash.py   2007-11-28 10:36:52.000000000 -0800
@@ -2,7 +2,7 @@

 import sys
 try:
-    import bsddb
+    import bsddb3 as bsddb
 except ImportError:
     # prevent a second import of this module from spuriously succeeding
     del sys.modules[__name__]

Here’s the patch.

Now it works:

marc@hyperion:~$ python
Python 2.5.1 (r251:54863, Oct  5 2007, 21:08:09)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbhash
>>> dir(dbhash)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', 'bsddb',
'error', 'open', 'sys']