Make PyCrypto work with PyPy

I was trying to pip install PyCrypto in a PyPy virtualenv and kept getting build errors, mostly related to PyLongObject.

I worked around the problem by uninstalling the gmp library.

I’m on OS X so I did:

brew uninstall gmp

(If you’re on a different platform, replace “brew” with your package manager — apt-get, yum, etc.)

Then things worked:

(pypy-pycrypto)marca@marca-mac:~/python/virtualenvs$ pip install PyCrypto
Downloading/unpacking PyCrypto
  Real name of requirement PyCrypto is pycrypto
  Using download cache from /Users/marca/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpycrypto%2Fpycrypto-2.6.1.tar.gz
  Running setup.py egg_info for package PyCrypto

Installing collected packages: PyCrypto
...
Successfully installed PyCrypto
Cleaning up...

Voila!

Ag: The Silver Searcher

I don’t know if I’ve mentioned “ack” before — it’s been my favorite tool for searching code for a while (it’s better than grep, because it’s fairly speedy because it knows how to search only files that are code and how to ignore a lot of cruft plus it does nice output with coloring, paging, etc.).

Anyway, I just found an even better tool:

https://github.com/ggreer/the_silver_searcher

It’s more or less a drop-in replacement for “ack”; but it’s written in C (“ack” is Perl) and it’s faster. Looks like this:

Screen Shot of ag (the silver searcher)

Screen Shot of ag (the silver searcher)

And you can use it in your favorite editor probably:

https://github.com/leonid-shevtsov/SearchInProject_SublimeText2
https://github.com/rking/ag.vim
https://github.com/mileszs/ack.vim

Cython bug?

I have this Cython code:

cdef struct MyStruct:
    int x
 
def byte_bugginess(x=1):
    cdef char val = 1
    cdef MyStruct foo
 
    print("1 - val = %d" % val)
 
    if x in (3, 4):
        print("2 - val = %d" % val)
        val = 2
        print("3 - val = %d" % val)
    else:
        print("4 - val = %d" % val)
        val = foo.x
        print("5 - val = %d" % val)
 
    print("6 - val = %d" % val)

and the output is:

marca@marca-mac:~/dev/git-repos/cython-test$ make
python setup.py build_ext --inplace
running build_ext
gcc-4.2 not found, using clang instead
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.macosx-10.6-intel-2.7
clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c hello.c -o build/temp.macosx-10.6-intel-2.7/hello.o
clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-2.7/hello.o -o /Users/marca/dev/git-repos/cython-test/hello.so
python -c 'import hello; hello.byte_bugginess(1)'
1 - val = 1
4 - val = 1
5 - val = 0
6 - val = 2

How could val be equal to 2? This implies that the false clause of the if statement is getting evaluated?

Weird.

Full code is at this gist.

I posted about this to the cython-users Google group. I am curious to see what they say.

Compiling FreeTDS on Windows

This is how I got FreeTDS to compile on Windows 7 Professional with Visual Studio 2008.

1. Install Visual Studio 2008 (a.k.a.: Visual Studio 9.0)
2. Download the FreeTDS 0.91 source tarball and extract it somewhere. I chose C:\Users\marca\src\freetds-0.91.
3. Open a Visual Studio 2008 Command Prompt.
4. Change to the directory with FreeTDS:

cd C:\Users\marca\src\freetds-0.91

(or whatever you chose)
5. Give this a shot — it will fail probably:

nmake -fNmakefile -nologo apps PLATFORM=win32 CONFIGURATION=debug

6. You will get a bunch of errors that look like this:

c:\users\marca\src\freetds-0.91\include\tds_sysdep_private.h(239) :
fatal error C1083: Cannot open include file: 'inttypes.h':
No such file or directory

7. Fix the inttypes.h error — edit win32/config.h and comment out this line (line 92 for me):

#define HAVE_INTTYPES_H 1

8. If you repeat the above nmake command, then you’ll get a new error:

c:\program files (x86)\microsoft visual studio 9.0\vc\include\stdio.h
(358) : error C3163: '_vsnprintf': attributes inconsistent 
with previous declaration
        c:\program files (x86)\microsoft visual studio 9.0\vc\
        include\stdio.h(350) : see declaration of '_vsnprintf'

9. Fix the _vsnprintf error by editing include/tds_sysdep_private.h and commenting out the second instance of this line (line 96 for me):

#define vsnprintf _vsnprintf

10. Repeat the nmake command and this time it should succeed and you will end up with a tsql.exe file.

C:\Users\marca\src\freetds-0.91>nmake ^
More? -fNmakefile -nologo apps ^
More? PLATFORM=win32 CONFIGURATION=debug
...
C:\Users\marca\src\freetds-0.91>dir src\apps\win32\debug\tsql.exe
 Volume in drive C has no label.
 Volume Serial Number is D0C5-FA7C

 Directory of C:\Users\marca\src\freetds-0.91\src\apps\win32\debug

07/18/2013  04:19 PM           847,872 tsql.exe
               1 File(s)        847,872 bytes
               0 Dir(s)  19,728,400,384 bytes free

11. Give it a spin:

C:\Users\marca\src\freetds-0.91>src\apps\win32\debug\tsql ^
More? -H <server> -p 1433 ^
More? -U <username> -P <password> -D <database>
...
1> SELECT @@version
2> GO

Microsoft SQL Server 2008 R2 (SP2) - 10.50.4279.0 (X64)
        Mar 26 2013 17:33:13
        Copyright (c) Microsoft Corporation
        Developer Edition (64-bit) on Windows NT 6.1  
       (Build 7601: Service Pack 1) (Hypervisor)

(1 row affected)

A few useful links:

Python context manager for redirected stdout and stderr

I was working on the setup.py for pymssql and was annoyed by the distutils.ccompiler.CCompiler.has_function function — because it checks whether a C function exists by creating a very simple C program and invoking the compiler and linker and the compiler and linker may emit warnings and errors if the function doesn’t exist and then this shows up in the output of python setup.py install, etc. so it can confuse users and make users think that something is wrong when there really isn’t. Here’s an example of what I’m talking about:

$ python setup.py install
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:1:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
main (int argc, char **argv) {
^~~~
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:2:5: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
    clock_gettime();
    ^
2 warnings generated.
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I don’t want users to see linker errors when they install the package and think that something is wrong. This is perfectly normal that -lrt didn’t work — this is on Mac OS X and it doesn’t have an rt library and we just detected that so that we don’t try to use it. The user doesn’t need to see these errors though, as it could confuse them.

So I looked for a way to not have the compiler errors displayed on stderr.

I found:

It seemed silly to have to reimplement logic in distutils for compiling code when all we want to do is just suppress error output by redirecting stderr. I thought one could simply call the distutils stuff inside a context manager that redirects stderr.

This is what I came up with — a context manager called stdchannel_redirected:

Here’s how I use it:

    with stdchannel_redirected(sys.stderr, os.devnull):
        if compiler.has_function('clock_gettime', libraries=['rt']):
            libraries.append('rt')

I used this in the setup.py of pymssql — see this commit.

I suspect that this probably exists out there in some other form, so I tweeted about it. Let me know if you know a better way…

OS X Homebrew: Build FreeTDS from HEAD (gitorious)

Here’s a pull request I just submitted to homebrew to make it possible to build FreeTDS from master on its gitorious repo:

https://github.com/mxcl/homebrew/pull/21309

Example of using it:

$ brew install freetds --HEAD --universal
==> Cloning https://git.gitorious.org/freetds/freetds.git
Updating /Library/Caches/Homebrew/freetds--git
==> autoreconf -i
==> ./configure --prefix=/usr/local/Cellar/freetds/HEAD --with-openssl=/usr/bin --with-tdsver=7.1 --mandir=/usr/local/Cellar/freetds/HEAD/share/man
==> make
==> make install
🍺  /usr/local/Cellar/freetds/HEAD: 63 files, 3.6M, built in 67 seconds

$ brew test freetds --HEAD -v
Testing freetds
==> /usr/local/Cellar/freetds/HEAD/bin/tsql -C
/usr/local/Cellar/freetds/HEAD/bin/tsql -C
Compile-time settings (established with the "configure" script)
                            Version: freetds v0.92.dev.20130718
             freetds.conf directory: /usr/local/Cellar/freetds/HEAD/etc
     MS db-lib source compatibility: no
        Sybase binary compatibility: no
                      Thread safety: yes
                      iconv library: yes
                        TDS version: 7.1
                              iODBC: yes
                           unixodbc: no
              SSPI "trusted" logins: no
                           Kerberos: no

Pretty simple:

commit 3d786f23bc2ff5d73a22474f9256330a17f01e23
Author: Marc Abramowitz 
Date:   Thu Jul 18 09:10:33 2013 -0700

    freetds.rb: Enable support for building FreeTDS from HEAD (gitorious)

diff --git a/Library/Formula/freetds.rb b/Library/Formula/freetds.rb
index 0a700e6..362b411 100644
--- a/Library/Formula/freetds.rb
+++ b/Library/Formula/freetds.rb
@@ -5,10 +5,19 @@ class Freetds < Formula
   url 'http://mirrors.ibiblio.org/freetds/stable/freetds-0.91.tar.gz'
   sha1 '3ab06c8e208e82197dc25d09ae353d9f3be7db52'

+  head 'https://git.gitorious.org/freetds/freetds.git'
+
   depends_on "pkg-config" => :build
   depends_on "unixodbc" => :optional

+  if build.head?
+    depends_on :automake
+    depends_on :libtool
+  end
+
   def install
+    system "autoreconf -i" if build.head?
+
     args = %W[--prefix=#{prefix}
               --with-openssl=/usr/bin
               --with-tdsver=7.1

Here’s the commit on GitHub.

I also added support for a test and for doing a universal build.

Compiling a universal library

A note for my future self on how to compile a library as universal binary in Mac OS X.

For a while, I’ve been getting warnings like these when building pymssql:

ld: warning: ignoring file /usr/local/lib/libsybdb.dylib, 
file was built for unsupported file format 
( 0xcf 0xfa 0xed 0xfe 0x 7 0x 0 0x 0 0x 1 0x 3 0x 0 0x 0 0x 0 0x 6 0x 0 0x 0 0x 0 ) 
which is not the architecture being linked (i386): 
/usr/local/lib/libsybdb.dylib
ld: warning: ignoring file /usr/local/lib/libct.dylib, 
file was built for unsupported file format 
( 0xcf 0xfa 0xed 0xfe 0x 7 0x 0 0x 0 0x 1 0x 3 0x 0 0x 0 0x 0 0x 6 0x 0 0x 0 0x 0 ) 
which is not the architecture being linked (i386): 
/usr/local/lib/libct.dylib

I finally decided to do something about them. I rebuilt FreeTDS from source as follows:

./configure \
    CFLAGS="-arch i386 -arch x86_64" \
    CXXFLAGS="-arch i386 -arch x86_64" \
    LDFLAGS="-arch i386 -arch x86_64" \
    --disable-dependency-tracking
make
make install

Goodbye warnings!

This Stack Overflow post was helpful: Stack Overflow: How to compile universal libraries on Mac OS X?