I wanted to see a bit about virtualenvs when I activate them with workon from virtualenvwrapper.
I added the following to $VIRTUALENVWRAPPER_HOOK_DIR/postactivate:
#!/bin/bash # This hook is run after every virtualenv is activated. python -V easy_install --version pip --version
Here’s how it looks:
$ workon pip Python 2.7.6 setuptools 3.6 pip 6.0.dev1 from /Users/marca/dev/git-repos/pip (python 2.7)
For more information on customizing with hooks, read the docs.
https://github.com/Supervisor/supervisor/pull/477
Let’s you do stuff like:
$ supervisorctl cat:0 RUNNING pid 57305, uptime 0:00:07 cat:1 RUNNING pid 57304, uptime 0:00:07 cat:2 RUNNING pid 57307, uptime 0:00:07 cat:3 RUNNING pid 57306, uptime 0:00:07 cat:4 RUNNING pid 57308, uptime 0:00:07 dog:0 RUNNING pid 57300, uptime 0:00:07 dog:1 RUNNING pid 57299, uptime 0:00:07 dog:2 RUNNING pid 57302, uptime 0:00:07 dog:3 RUNNING pid 57301, uptime 0:00:07 dog:4 RUNNING pid 57303, uptime 0:00:07 supervisor> help signal signal <signal name> <name> Signal a process signal <signal name> <gname>:* Signal all processes in a group signal <signal name> <name> <name> Signal multiple processes or groups supervisor> signal 1 dog:3 dog:4 dog:3: signalled dog:4: signalled supervisor> signal HUP dog:3 dog:4 dog:3: signalled dog:4: signalled supervisor> signal HUP dog:* dog:1: signalled dog:0: signalled dog:3: signalled dog:2: signalled dog:4: signalled supervisor> signal USR1 dog:1 dog:2 dog:1: signalled dog:2: signalled
Also, if you can’t wait for supervisor to support this, the mr.laforge package supplies a supervisor plugin that can be used to send signals to processes:
$ supervisorctl kill HUP nginx
That said, it would be nice to have this built into supervisor…
I blogged previously about how to build _bsddb for Python 2.6 by hacking the Python source code to accommodate changes in berkeley-db.
Here’s an easier way that doesn’t require hacking source code.
Install berkeley-db using Homebrew:
brew install berkeley-db
Install the bsddb3 module, pointing it at an installation of berkeley-db:
sudo \
BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/5.3.28 \
pip-2.6 install bsddb3
Simply copy it into the appropriate place as _bsddb.so. This is super hacky and will fail if the interfaces of these two ever change, but for now for my limited purpose, it seems to work:
cp \
/Library/Python/2.6/site-packages/bsddb3/_pybsddb.so \
/Library/Frameworks/Python.framework/Versions/2.6/lib \
/python2.6/lib-dynload/_bsddb.so
Enjoy:
$ python2.6 Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import bsddb >>> dir(bsddb) ['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']
It’s well-known that the Python that comes with OS X machines typically doesn’t have _bsddb.
I downloaded Python 2.6.8 and attempted to build it and it wouldn’t build the _bsddb module.
Here’s what I did to fix this.
I used Homebrew to install berkeley-db4. For me, this grabbed version 4.8.30.
brew install berkeley-db4
Edited Modules/_bsddb.c and replaced all instances of DB_XIDDATASIZE with DB_GID_SIZE. I removed lines that had DB_XA_CREATE.
Compiled and linked with:
gcc -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/berkeley-db4/4.8.30/include/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c /Users/marca/src/Python-2.6.8/Modules/_bsddb.c -o _bsddb.o gcc -bundle -undefined dynamic_lookup _bsddb.o -L/Library/Frameworks/Python.framework/Versions/2.6/lib -L/usr/local/lib -ldb -o _bsddb.so -Wl,-search_paths_first
This created the file Modules/_bsddb.so.
At this point, I can import if I run python in the Modules directory where I built _bsddb.so:
$ python2.6 Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import bsddb >>> dir(bsddb) ['MutableMapping', '_DBWithCursor', '_DeadlockWrap', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_bsddb', '_checkflag', '_db', '_iter_mixin', '_openDBEnv', 'absolute_import', 'btopen', 'collections', 'db', 'dbutils', 'error', 'hashopen', 'os', 'ref', 'rnopen', 'sys']
Now you can copy _bsddb.so into your lib-dynload directory — e.g.:
sudo cp _bsddb.so /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
(above is for the system Python 2.6 that comes with OS X)
or
sudo cp _bsddb.so sudo cp /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
(above is for a Python 2.6 installed from a .dmg from Python.org or built from source as a framework build)
References:
I was trying to pip install PyCrypto in a PyPy virtualenv and kept getting build errors, mostly related to PyLongObject.
I worked around the problem by uninstalling the gmp library.
I’m on OS X so I did:
brew uninstall gmp
(If you’re on a different platform, replace “brew” with your package manager — apt-get, yum, etc.)
Then things worked:
(pypy-pycrypto)marca@marca-mac:~/python/virtualenvs$ pip install PyCrypto Downloading/unpacking PyCrypto Real name of requirement PyCrypto is pycrypto Using download cache from /Users/marca/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpycrypto%2Fpycrypto-2.6.1.tar.gz Running setup.py egg_info for package PyCrypto Installing collected packages: PyCrypto ... Successfully installed PyCrypto Cleaning up...
Voila!
I have this Cython code:
cdef struct MyStruct:
int x
def byte_bugginess(x=1):
cdef char val = 1
cdef MyStruct foo
print("1 - val = %d" % val)
if x in (3, 4):
print("2 - val = %d" % val)
val = 2
print("3 - val = %d" % val)
else:
print("4 - val = %d" % val)
val = foo.x
print("5 - val = %d" % val)
print("6 - val = %d" % val)
and the output is:
marca@marca-mac:~/dev/git-repos/cython-test$ make python setup.py build_ext --inplace running build_ext gcc-4.2 not found, using clang instead cythoning hello.pyx to hello.c building 'hello' extension creating build creating build/temp.macosx-10.6-intel-2.7 clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c hello.c -o build/temp.macosx-10.6-intel-2.7/hello.o clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-2.7/hello.o -o /Users/marca/dev/git-repos/cython-test/hello.so python -c 'import hello; hello.byte_bugginess(1)' 1 - val = 1 4 - val = 1 5 - val = 0 6 - val = 2
How could val be equal to 2? This implies that the false clause of the if statement is getting evaluated?
Weird.
Full code is at this gist.
I posted about this to the cython-users Google group. I am curious to see what they say.
I was working on the setup.py for pymssql and was annoyed by the distutils.ccompiler.CCompiler.has_function function — because it checks whether a C function exists by creating a very simple C program and invoking the compiler and linker and the compiler and linker may emit warnings and errors if the function doesn’t exist and then this shows up in the output of python setup.py install, etc. so it can confuse users and make users think that something is wrong when there really isn’t. Here’s an example of what I’m talking about:
$ python setup.py install
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:1:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
main (int argc, char **argv) {
^~~~
/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/clock_gettimeZdbS70.c:2:5: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
clock_gettime();
^
2 warnings generated.
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I don’t want users to see linker errors when they install the package and think that something is wrong. This is perfectly normal that -lrt didn’t work — this is on Mac OS X and it doesn’t have an rt library and we just detected that so that we don’t try to use it. The user doesn’t need to see these errors though, as it could confuse them.
So I looked for a way to not have the compiler errors displayed on stderr.
I found:
hasfunction function that implements a simplified version of distutils.ccompiler.CCompiler.has_function. But it’s limited — for example you can’t pass a libraries parameter to it, which happens to be a feature I needed.It seemed silly to have to reimplement logic in distutils for compiling code when all we want to do is just suppress error output by redirecting stderr. I thought one could simply call the distutils stuff inside a context manager that redirects stderr.
This is what I came up with — a context manager called stdchannel_redirected:
Here’s how I use it:
with stdchannel_redirected(sys.stderr, os.devnull):
if compiler.has_function('clock_gettime', libraries=['rt']):
libraries.append('rt')
I used this in the setup.py of pymssql — see this commit.
I suspect that this probably exists out there in some other form, so I tweeted about it. Let me know if you know a better way…