Python context manager for redirected stdout and stderr

I was working on the for pymssql and was annoyed by the distutils.ccompiler.CCompiler.has_function function — because it checks whether a C function exists by creating a very simple C program and invoking the compiler and linker and the compiler and linker may emit warnings and errors if the function doesn’t exist and then this shows up in the output of python install, etc. so it can confuse users and make users think that something is wrong when there really isn’t. Here’s an example of what I’m talking about:

$ python install
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:1:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
main (int argc, char **argv) {
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:2:5: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
2 warnings generated.
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I don’t want users to see linker errors when they install the package and think that something is wrong. This is perfectly normal that -lrt didn’t work — this is on Mac OS X and it doesn’t have an rt library and we just detected that so that we don’t try to use it. The user doesn’t need to see these errors though, as it could confuse them.

So I looked for a way to not have the compiler errors displayed on stderr.

I found:

It seemed silly to have to reimplement logic in distutils for compiling code when all we want to do is just suppress error output by redirecting stderr. I thought one could simply call the distutils stuff inside a context manager that redirects stderr.

This is what I came up with — a context manager called stdchannel_redirected:

Here’s how I use it:

    with stdchannel_redirected(sys.stderr, os.devnull):
        if compiler.has_function('clock_gettime', libraries=['rt']):

I used this in the of pymssql — see this commit.

I suspect that this probably exists out there in some other form, so I tweeted about it. Let me know if you know a better way…

Compiling a universal library

A note for my future self on how to compile a library as universal binary in Mac OS X.

For a while, I’ve been getting warnings like these when building pymssql:

ld: warning: ignoring file /usr/local/lib/libsybdb.dylib, 
file was built for unsupported file format 
( 0xcf 0xfa 0xed 0xfe 0x 7 0x 0 0x 0 0x 1 0x 3 0x 0 0x 0 0x 0 0x 6 0x 0 0x 0 0x 0 ) 
which is not the architecture being linked (i386): 
ld: warning: ignoring file /usr/local/lib/libct.dylib, 
file was built for unsupported file format 
( 0xcf 0xfa 0xed 0xfe 0x 7 0x 0 0x 0 0x 1 0x 3 0x 0 0x 0 0x 0 0x 6 0x 0 0x 0 0x 0 ) 
which is not the architecture being linked (i386): 

I finally decided to do something about them. I rebuilt FreeTDS from source as follows:

./configure \
    CFLAGS="-arch i386 -arch x86_64" \
    CXXFLAGS="-arch i386 -arch x86_64" \
    LDFLAGS="-arch i386 -arch x86_64" \
make install

Goodbye warnings!

This Stack Overflow post was helpful: Stack Overflow: How to compile universal libraries on Mac OS X?

Back to setuptools

Browsing PyPI yesterday, I was surprised to see a new version of setuptools. I almost thought it was a bug. For years, I’ve been drinking the distribute Kool-Aid

pip/distribute Public Service Announcement

and automatically using distribute instead of setuptools. Well, I guess that things have changed and now distribute has been merged into setuptools and distribute is deprecated and setuptools is the thing to use. It may take me a little while to unlearn my old habits and it might make explaining Python packaging a little bit more difficult (not that it was easy :-)).

Continuation of mod_wsgi setup

My previous setup worked until I wanted to add another Django instance and then it didn’t work, because I had two WSGIScriptAlias entries under different URLs but it would launch whichever app it wanted to regardless of the URL. A little searching suggested that it had to do with mod_wsgi process pools. I eventually ended up with something like this:

WSGIPythonHome /www/python/pypi.venv
WSGIPythonPath /www/python/django
WSGIPassAuthorization On
WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>


    DocumentRoot /www/python

        WSGIDaemonProcess djangoplayground_wsgi user=apache python-path=/www/python/django
        WSGIScriptAlias /djangoplayground /www/python/django/djangoplayground/
        <Location /djangoplayground>
            WSGIProcessGroup djangoplayground_wsgi

        WSGIDaemonProcess djangopypi_wsgi user=apache python-path=/www/python/django
        WSGIScriptAlias /djangopypi /www/python/django/djangopypisite/
        <Location /djangopypi>
            WSGIProcessGroup djangopypi_wsgi


I had been working on DjangoPyPI and trying to fix some tests related to XML-RPC that were failing in versions of Python < 2.7, because the testing was relying on specifics of the implementation of xmlrpclib in Python 2.7. I found some code at this blog post from Forest Bond that showed a better way to test XML-RPC views in Django, namely, by creating a special XML-RPC transport object that uses the Django test client. I extracted this out and tweaked it slightly to add tests and better compatibility across Python versions and published it on PyPI as…


Build Status

Test Django XML-RPC views using the Django test client. Because you’re using the Django test client, you’re not actually sending HTTP requests and don’t need a server running.

Example usage:

from djangotestxmlrpc import DjangoTestClientXMLRPCTransport

class TestXmlRpc(django.test.TestCase):

    def test_list_package(self):
        pypi = xmlrpclib.ServerProxy(
        pypi_hits = pypi.list_packages()
        expected = ['foo']
        self.assertEqual(pypi_hits, expected)

Supported Python versions

  • Python 2.5
  • Python 2.6
  • Python 2.7
  • PyPy 1.9
  • Python 3.1
  • Python 3.2

or says tox:

~/dev/git-repos/djangotestxmlrpc$ tox
  py25: commands succeeded
  py26: commands succeeded
  py27: commands succeeded
  pypy: commands succeeded
  py31: commands succeeded
  py32: commands succeeded
  congratulations :)

You also can check the latest Travis CI results, but Travis doesn’t build all of the above platforms.


Send your bug reports and feature requests to

Some tips for setting up Apache and mod_wsgi

I recently set up an internal PyPI server at work for hosting some of our internal Python packages. I used djangopypi and that was pretty easy to set up and get it working in the Django runserver. Most of the difficulty came in getting it to run in Apache with mod_wsgi. I always find mod_wsgi a little difficult to set up, so I thought I’d jot down a few notes on things I ran into and how I solved them, for future readers (including my future self! :-))

Prerequisite: Which Python are you going to use?

I’m going to briefly cover this, because getting Python installed on your OS of choice could be a blog post in itself. Are you going to use the system Python? This is easiest if your OS has a recent enough version that you want to use. Mine, CentOS 5.5, in this case comes with Python 2.4.3, which is pretty long in the tooth, so I wouldn’t bother with it. Even if it was a modern Python, I tend to shy away from doing much with the system Python, because it’s the system python, and I don’t want to do things and break my system. You might be OK with using a virtualenv of your system Python to get some isolation or you might do like I did in this case and simply install a fresh new version of Python with no connection whatsoever to the system Python. In fact, I installed a new Python and created a virtualenv in it specifically for the Django stuff I was doing.

Install mod_wsgi

Depending on your OS and package repositories, it might be an apt-get install or yum install away. For me it wasn’t and I downloaded and built it from source. Make sure that mod_wsgi is compiled against the Python version that you actually want to use. It’s pretty easy for it to link against your system Python or some other Python you don’t intend to use and that will cause problems. I built mine like this:

PATH=/usr/local/bin:/usr/bin:/bin ./configure --with-python=/usr/local/bin/python
LD_RUN_PATH=/usr/local/lib make
sudo make install

I needed the LD_RUN_PATH stuff so that could link with /usr/local/lib/ at runtime. This installed as /usr/lib64/httpd/modules/ (which is also available at /etc/httpd/modules/ because I already had a symlink from /etc/httpd/modules to /usr/lib64/httpd/modules.

Turn on mod_wsgi in the Apache config

For me, I had to do:

$ cat /etc/httpd/conf.d/wsgi.conf 
LoadModule wsgi_module modules/

Important: Turn off mod_python!

This is sort of commonly-known wisdom for people who are in the know about mod_wsgi, but I don’t set up mod_wsgi very often and so I forgot about it. And it caused great pain, because it caused a very mysterious failure. Basically, I had a state where a “Hello World” WSGI app worked just fine, but when I tried to use the of my Django site, it would crap out silently. No errors in the Apache error log even with LogLevel set to debug. Through some ugly hacking of the and some files in Django itself, I could see that it was silently dying inside Django when it got to the statement:

from threading import Lock

I did a bunch of searching and found posts of people having similar troubles but I didn’t find an answer.

And then I remembered to try turning off mod_python and that worked! Which was good, but it burned a lot of my time and psychic energy. All I had to do was comment out the following line in /etc/httpd/conf.d/python.conf:

# LoadModule python_module modules/

I wonder if mod_wsgi could be modified to detect the presence of mod_python and if it finds it, blast the logs with warnings? Frankly, I don’t know enough about Apache modules to know if this is possible.

Set up an Apache config that points to the

Mine looks roughly like this:

WSGIPythonHome /www/python/pypi.venv
WSGIPythonPath /www/python/django
WSGIPassAuthorization On

<VirtualHost *:80>
    DocumentRoot /www/python
    WSGIScriptAlias /djangopypi /www/python/django/djangopypisite/

This of course will have to be heavily customized depending on whether you’re using a virtualenv and where it is and where your app is located, etc.

The WSGIPassAuthorization On line is something that I didn’t have at first and only later ran into problems and ended up adding it…

Test it out…

At this point, I had the Django app working more or less the same as it was working in the Django runserver except…

Make sure directories are writable by Apache, etc.

If you set up everything initially as your own user and then moved it over to mod_wsgi, then you might have files that belong to your user and which are not writable by Apache. Log directories, SQLite databases, directories for uploaded files (typically called “media” in Django paralance), etc.

Setting up static serving

The Django runserver makes things easy by serving static files for you. Django when deployed via mod_wsgi won’t serve static files by default. There are probably hacks to make it do that, but if you’re already running Apache it makes more sense to just have Apache serve those static files. That is certainly what the Django guys encourage. I set the MEDIA_ROOT and MEDIA_URL settings in and then did some symlinking and Apache configuration so that Apache could serve the files out of /www/python/media. And I created a directory /www/python/static for stuff like CSS and JS files and collected them from the apps into this directory using python collectstatic.

WSGIPassAuthorization On: The last piece of the puzzle

At this point, I had done a lot of hacking and got it mostly working. The one missing piece was that I could not upload packages to my custom PyPI server (i.e.: with python register -r chegg sdist upload -r chegg). Actually it worked when I had my ~/.pypirc pointed to the Django runserver; it just didn’t work with the mod_wsgi version — it failed with Upload failed (401): UNAUTHORIZED. This took a while to figure out, but I eventually found the reason and answer here. Basically this is because djangopypi does checking of the HTTP Authorization header in order to see if you’re authorized, and mod_wsgi by default filters out this header as a security precaution, on the assumption that you would let Apache do the authorization check and not involve the WSGI apps in this. I imagine that this is something useful for providers of shared web hosting, but it obviously will not work if you have WSGI apps doing their own authorization, as I had in this case with djangopypi. The answer was to turn on WSGIPassAuthorization by adding WSGIPassAuthorization On to my Apache vhost config.

That’s all folks!

So now it’s working although it took annoyingly long to set up.

The temptation was to move on as quickly as possible to something else now since I spent more time than I wanted to on this. Surely I’m now a smarter, wiser person who learned and won’t run into these problems next time… On second thought, realistically I may not have to do this again for months…or years…and I know myself well enough to know that I probably won’t remember much of this in a year. So I decided to invest just a little more time in writing down some of the high-level points so that I (or someone else) can refer to them later. And now because I blogged it, a few months from now, I will only have to remember to search my blog to find this post… 🙂


A while back, I mentioned here and here that OmniFocus maintains some kind of cache of your data in a SQLite database.

I just hacked together a quick little prototype Python module for reading OmniFocus data:

Please feel free to fork it and send pull requests! Or file bugs. Or send ideas for improvements.

This could be used as part of a larger solution to pull all the data out of OmniFocus and publish it as some consumable format for the web (e.g.: XML, JSON, HTML, etc.). Or to have an OmniFocus CLI, which is what I’m most interested in.