Author Archives: Marc

Easier way to build _bsddb for OS X

Posted on by
2

I blogged previously about how to build _bsddb for Python 2.6 by hacking the Python source code to accommodate changes in berkeley-db.

Here’s an easier way that doesn’t require hacking source code.

Install berkeley-db using Homebrew:

brew install berkeley-db

Install the bsddb3 module, pointing it at an installation of berkeley-db:

sudo \
    BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/5.3.28 \
    pip-2.6 install bsddb3

Simply copy it into the appropriate place as _bsddb.so. This is super hacky and will fail if the interfaces of these two ever change, but for now for my limited purpose, it seems to work:

cp \
    /Library/Python/2.6/site-packages/bsddb3/_pybsddb.so \
    /Library/Frameworks/Python.framework/Versions/2.6/lib \
      /python2.6/lib-dynload/_bsddb.so

Enjoy:

$ python2.6
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
>>> dir(bsddb)
['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']

Building _bsddb for Python 2.6 on OS X 10.8.5

Posted on by
Reply

It’s well-known that the Python that comes with OS X machines typically doesn’t have _bsddb.

I downloaded Python 2.6.8 and attempted to build it and it wouldn’t build the _bsddb module.

Here’s what I did to fix this.

I used Homebrew to install berkeley-db4. For me, this grabbed version 4.8.30.

brew install berkeley-db4

Edited Modules/_bsddb.c and replaced all instances of DB_XIDDATASIZE with DB_GID_SIZE. I removed lines that had DB_XA_CREATE.

Compiled and linked with:

gcc -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/berkeley-db4/4.8.30/include/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c /Users/marca/src/Python-2.6.8/Modules/_bsddb.c -o _bsddb.o
gcc -bundle -undefined dynamic_lookup _bsddb.o -L/Library/Frameworks/Python.framework/Versions/2.6/lib -L/usr/local/lib -ldb -o _bsddb.so -Wl,-search_paths_first

This created the file Modules/_bsddb.so.

At this point, I can import if I run python in the Modules directory where I built _bsddb.so:

$ python2.6
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
>>> dir(bsddb)
['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']

Now you can copy _bsddb.so into your lib-dynload directory — e.g.:

sudo cp _bsddb.so /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload

(above is for the system Python 2.6 that comes with OS X)

or

sudo cp _bsddb.so sudo cp /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload

(above is for a Python 2.6 installed from a .dmg from Python.org or built from source as a framework build)

References:

Make PyCrypto work with PyPy

Posted on by
5

I was trying to pip install PyCrypto in a PyPy virtualenv and kept getting build errors, mostly related to PyLongObject.

I worked around the problem by uninstalling the gmp library.

I’m on OS X so I did:

brew uninstall gmp

(If you’re on a different platform, replace “brew” with your package manager — apt-get, yum, etc.)

Then things worked:

(pypy-pycrypto)marca@marca-mac:~/python/virtualenvs$ pip install PyCrypto
Downloading/unpacking PyCrypto
  Real name of requirement PyCrypto is pycrypto
  Using download cache from /Users/marca/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpycrypto%2Fpycrypto-2.6.1.tar.gz
  Running setup.py egg_info for package PyCrypto

Installing collected packages: PyCrypto
...
Successfully installed PyCrypto
Cleaning up...

Voila!

Ag: The Silver Searcher

Posted on by
Reply

I don’t know if I’ve mentioned “ack” before — it’s been my favorite tool for searching code for a while (it’s better than grep, because it’s fairly speedy because it knows how to search only files that are code and how to ignore a lot of cruft plus it does nice output with coloring, paging, etc.).

Anyway, I just found an even better tool:

https://github.com/ggreer/the_silver_searcher

It’s more or less a drop-in replacement for “ack”; but it’s written in C (“ack” is Perl) and it’s faster. Looks like this:

Screen Shot of ag (the silver searcher)

Screen Shot of ag (the silver searcher)

And you can use it in your favorite editor probably:

https://github.com/leonid-shevtsov/SearchInProject_SublimeText2
https://github.com/rking/ag.vim
https://github.com/mileszs/ack.vim

Cython bug?

Posted on by
Reply

I have this Cython code:

cdef struct MyStruct:
    int x
 
def byte_bugginess(x=1):
    cdef char val = 1
    cdef MyStruct foo
 
    print("1 - val = %d" % val)
 
    if x in (3, 4):
        print("2 - val = %d" % val)
        val = 2
        print("3 - val = %d" % val)
    else:
        print("4 - val = %d" % val)
        val = foo.x
        print("5 - val = %d" % val)
 
    print("6 - val = %d" % val)

and the output is:

marca@marca-mac:~/dev/git-repos/cython-test$ make
python setup.py build_ext --inplace
running build_ext
gcc-4.2 not found, using clang instead
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.macosx-10.6-intel-2.7
clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c hello.c -o build/temp.macosx-10.6-intel-2.7/hello.o
clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-2.7/hello.o -o /Users/marca/dev/git-repos/cython-test/hello.so
python -c 'import hello; hello.byte_bugginess(1)'
1 - val = 1
4 - val = 1
5 - val = 0
6 - val = 2

How could val be equal to 2? This implies that the false clause of the if statement is getting evaluated?

Weird.

Full code is at this gist.

I posted about this to the cython-users Google group. I am curious to see what they say.

Compiling FreeTDS on Windows

Posted on by
3

This is how I got FreeTDS to compile on Windows 7 Professional with Visual Studio 2008.

1. Install Visual Studio 2008 (a.k.a.: Visual Studio 9.0)
2. Download the FreeTDS 0.91 source tarball and extract it somewhere. I chose C:\Users\marca\src\freetds-0.91.
3. Open a Visual Studio 2008 Command Prompt.
4. Change to the directory with FreeTDS:

cd C:\Users\marca\src\freetds-0.91

(or whatever you chose)
5. Give this a shot — it will fail probably:

nmake -fNmakefile -nologo apps PLATFORM=win32 CONFIGURATION=debug

6. You will get a bunch of errors that look like this:

c:\users\marca\src\freetds-0.91\include\tds_sysdep_private.h(239) :
fatal error C1083: Cannot open include file: 'inttypes.h':
No such file or directory

7. Fix the inttypes.h error — edit win32/config.h and comment out this line (line 92 for me):

#define HAVE_INTTYPES_H 1

8. If you repeat the above nmake command, then you’ll get a new error:

c:\program files (x86)\microsoft visual studio 9.0\vc\include\stdio.h
(358) : error C3163: '_vsnprintf': attributes inconsistent 
with previous declaration
        c:\program files (x86)\microsoft visual studio 9.0\vc\
        include\stdio.h(350) : see declaration of '_vsnprintf'

9. Fix the _vsnprintf error by editing include/tds_sysdep_private.h and commenting out the second instance of this line (line 96 for me):

#define vsnprintf _vsnprintf

10. Repeat the nmake command and this time it should succeed and you will end up with a tsql.exe file.

C:\Users\marca\src\freetds-0.91>nmake ^
More? -fNmakefile -nologo apps ^
More? PLATFORM=win32 CONFIGURATION=debug
...
C:\Users\marca\src\freetds-0.91>dir src\apps\win32\debug\tsql.exe
 Volume in drive C has no label.
 Volume Serial Number is D0C5-FA7C

 Directory of C:\Users\marca\src\freetds-0.91\src\apps\win32\debug

07/18/2013  04:19 PM           847,872 tsql.exe
               1 File(s)        847,872 bytes
               0 Dir(s)  19,728,400,384 bytes free

11. Give it a spin:

C:\Users\marca\src\freetds-0.91>src\apps\win32\debug\tsql ^
More? -H <server> -p 1433 ^
More? -U <username> -P <password> -D <database>
...
1> SELECT @@version
2> GO

Microsoft SQL Server 2008 R2 (SP2) - 10.50.4279.0 (X64)
        Mar 26 2013 17:33:13
        Copyright (c) Microsoft Corporation
        Developer Edition (64-bit) on Windows NT 6.1  
       (Build 7601: Service Pack 1) (Hypervisor)

(1 row affected)

A few useful links:

Python context manager for redirected stdout and stderr

Posted on by
8

I was working on the setup.py for pymssql and was annoyed by the distutils.ccompiler.CCompiler.has_function function — because it checks whether a C function exists by creating a very simple C program and invoking the compiler and linker and the compiler and linker may emit warnings and errors if the function doesn’t exist and then this shows up in the output of python setup.py install, etc. so it can confuse users and make users think that something is wrong when there really isn’t. Here’s an example of what I’m talking about:

$ python setup.py install
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:1:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
main (int argc, char **argv) {
^~~~
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:2:5: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
    clock_gettime();
    ^
2 warnings generated.
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I don’t want users to see linker errors when they install the package and think that something is wrong. This is perfectly normal that -lrt didn’t work — this is on Mac OS X and it doesn’t have an rt library and we just detected that so that we don’t try to use it. The user doesn’t need to see these errors though, as it could confuse them.

So I looked for a way to not have the compiler errors displayed on stderr.

I found:

It seemed silly to have to reimplement logic in distutils for compiling code when all we want to do is just suppress error output by redirecting stderr. I thought one could simply call the distutils stuff inside a context manager that redirects stderr.

This is what I came up with — a context manager called stdchannel_redirected:

Here’s how I use it:

    with stdchannel_redirected(sys.stderr, os.devnull):
        if compiler.has_function('clock_gettime', libraries=['rt']):
            libraries.append('rt')

I used this in the setup.py of pymssql — see this commit.

I suspect that this probably exists out there in some other form, so I tweeted about it. Let me know if you know a better way…