Easier way to build _bsddb for OS X

I blogged previously about how to build _bsddb for Python 2.6 by hacking the Python source code to accommodate changes in berkeley-db.

Here’s an easier way that doesn’t require hacking source code.

Install berkeley-db using Homebrew:

brew install berkeley-db

Install the bsddb3 module, pointing it at an installation of berkeley-db:

sudo \
    BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/5.3.28 \
    pip-2.6 install bsddb3

Simply copy it into the appropriate place as _bsddb.so. This is super hacky and will fail if the interfaces of these two ever change, but for now for my limited purpose, it seems to work:

cp \
    /Library/Python/2.6/site-packages/bsddb3/_pybsddb.so \
    /Library/Frameworks/Python.framework/Versions/2.6/lib \
      /python2.6/lib-dynload/_bsddb.so

Enjoy:

$ python2.6
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
>>> dir(bsddb)
['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']

Building _bsddb for Python 2.6 on OS X 10.8.5

It’s well-known that the Python that comes with OS X machines typically doesn’t have _bsddb.

I downloaded Python 2.6.8 and attempted to build it and it wouldn’t build the _bsddb module.

Here’s what I did to fix this.

I used Homebrew to install berkeley-db4. For me, this grabbed version 4.8.30.

brew install berkeley-db4

Edited Modules/_bsddb.c and replaced all instances of DB_XIDDATASIZE with DB_GID_SIZE. I removed lines that had DB_XA_CREATE.

Compiled and linked with:

gcc -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/berkeley-db4/4.8.30/include/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c /Users/marca/src/Python-2.6.8/Modules/_bsddb.c -o _bsddb.o
gcc -bundle -undefined dynamic_lookup _bsddb.o -L/Library/Frameworks/Python.framework/Versions/2.6/lib -L/usr/local/lib -ldb -o _bsddb.so -Wl,-search_paths_first

This created the file Modules/_bsddb.so.

At this point, I can import if I run python in the Modules directory where I built _bsddb.so:

$ python2.6
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
>>> dir(bsddb)
['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']

Now you can copy _bsddb.so into your lib-dynload directory — e.g.:

sudo cp _bsddb.so /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload

(above is for the system Python 2.6 that comes with OS X)

or

sudo cp _bsddb.so sudo cp /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload

(above is for a Python 2.6 installed from a .dmg from Python.org or built from source as a framework build)

References:

Make PyCrypto work with PyPy

I was trying to pip install PyCrypto in a PyPy virtualenv and kept getting build errors, mostly related to PyLongObject.

I worked around the problem by uninstalling the gmp library.

I’m on OS X so I did:

brew uninstall gmp

(If you’re on a different platform, replace “brew” with your package manager — apt-get, yum, etc.)

Then things worked:

(pypy-pycrypto)marca@marca-mac:~/python/virtualenvs$ pip install PyCrypto
Downloading/unpacking PyCrypto
  Real name of requirement PyCrypto is pycrypto
  Using download cache from /Users/marca/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpycrypto%2Fpycrypto-2.6.1.tar.gz
  Running setup.py egg_info for package PyCrypto

Installing collected packages: PyCrypto
...
Successfully installed PyCrypto
Cleaning up...

Voila!

Ag: The Silver Searcher

I don’t know if I’ve mentioned “ack” before — it’s been my favorite tool for searching code for a while (it’s better than grep, because it’s fairly speedy because it knows how to search only files that are code and how to ignore a lot of cruft plus it does nice output with coloring, paging, etc.).

Anyway, I just found an even better tool:

https://github.com/ggreer/the_silver_searcher

It’s more or less a drop-in replacement for “ack”; but it’s written in C (“ack” is Perl) and it’s faster. Looks like this:

Screen Shot of ag (the silver searcher)

Screen Shot of ag (the silver searcher)

And you can use it in your favorite editor probably:

https://github.com/leonid-shevtsov/SearchInProject_SublimeText2
https://github.com/rking/ag.vim
https://github.com/mileszs/ack.vim

Cython bug?

I have this Cython code:

cdef struct MyStruct:
    int x
 
def byte_bugginess(x=1):
    cdef char val = 1
    cdef MyStruct foo
 
    print("1 - val = %d" % val)
 
    if x in (3, 4):
        print("2 - val = %d" % val)
        val = 2
        print("3 - val = %d" % val)
    else:
        print("4 - val = %d" % val)
        val = foo.x
        print("5 - val = %d" % val)
 
    print("6 - val = %d" % val)

and the output is:

marca@marca-mac:~/dev/git-repos/cython-test$ make
python setup.py build_ext --inplace
running build_ext
gcc-4.2 not found, using clang instead
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.macosx-10.6-intel-2.7
clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c hello.c -o build/temp.macosx-10.6-intel-2.7/hello.o
clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-2.7/hello.o -o /Users/marca/dev/git-repos/cython-test/hello.so
python -c 'import hello; hello.byte_bugginess(1)'
1 - val = 1
4 - val = 1
5 - val = 0
6 - val = 2

How could val be equal to 2? This implies that the false clause of the if statement is getting evaluated?

Weird.

Full code is at this gist.

I posted about this to the cython-users Google group. I am curious to see what they say.

Compiling FreeTDS on Windows

This is how I got FreeTDS to compile on Windows 7 Professional with Visual Studio 2008.

1. Install Visual Studio 2008 (a.k.a.: Visual Studio 9.0)
2. Download the FreeTDS 0.91 source tarball and extract it somewhere. I chose C:\Users\marca\src\freetds-0.91.
3. Open a Visual Studio 2008 Command Prompt.
4. Change to the directory with FreeTDS:

cd C:\Users\marca\src\freetds-0.91

(or whatever you chose)
5. Give this a shot — it will fail probably:

nmake -fNmakefile -nologo apps PLATFORM=win32 CONFIGURATION=debug

6. You will get a bunch of errors that look like this:

c:\users\marca\src\freetds-0.91\include\tds_sysdep_private.h(239) :
fatal error C1083: Cannot open include file: 'inttypes.h':
No such file or directory

7. Fix the inttypes.h error — edit win32/config.h and comment out this line (line 92 for me):

#define HAVE_INTTYPES_H 1

8. If you repeat the above nmake command, then you’ll get a new error:

c:\program files (x86)\microsoft visual studio 9.0\vc\include\stdio.h
(358) : error C3163: '_vsnprintf': attributes inconsistent 
with previous declaration
        c:\program files (x86)\microsoft visual studio 9.0\vc\
        include\stdio.h(350) : see declaration of '_vsnprintf'

9. Fix the _vsnprintf error by editing include/tds_sysdep_private.h and commenting out the second instance of this line (line 96 for me):

#define vsnprintf _vsnprintf

10. Repeat the nmake command and this time it should succeed and you will end up with a tsql.exe file.

C:\Users\marca\src\freetds-0.91>nmake ^
More? -fNmakefile -nologo apps ^
More? PLATFORM=win32 CONFIGURATION=debug
...
C:\Users\marca\src\freetds-0.91>dir src\apps\win32\debug\tsql.exe
 Volume in drive C has no label.
 Volume Serial Number is D0C5-FA7C

 Directory of C:\Users\marca\src\freetds-0.91\src\apps\win32\debug

07/18/2013  04:19 PM           847,872 tsql.exe
               1 File(s)        847,872 bytes
               0 Dir(s)  19,728,400,384 bytes free

11. Give it a spin:

C:\Users\marca\src\freetds-0.91>src\apps\win32\debug\tsql ^
More? -H <server> -p 1433 ^
More? -U <username> -P <password> -D <database>
...
1> SELECT @@version
2> GO

Microsoft SQL Server 2008 R2 (SP2) - 10.50.4279.0 (X64)
        Mar 26 2013 17:33:13
        Copyright (c) Microsoft Corporation
        Developer Edition (64-bit) on Windows NT 6.1  
       (Build 7601: Service Pack 1) (Hypervisor)

(1 row affected)

A few useful links: