Category Archives: Python

Patch for gitpoller accepted into Buildbot

Posted on by
Reply

I use Buildbot for a (Django) project at work. When we switched from svn to git, I had problems getting Buildbot to pull down the code from our private GitHub repo. Rather, I was getting this error:

Failure: twisted.internet.utils._UnexpectedErrorOutput: got stderr: 
"fatal: ambiguous argument 'master..origin/master': unknown revision 
or path not in the working tree.\nUse '--' to separate paths from revisions\n"

It turned out that the problem was that the git subprocesses weren’t inheriting my environment which has SSH_AGENT_PID and SSH_AUTH_SOCK environment variables to make ssh to GitHub work.

The effect of the patch is that when calling external git commands, the environment is passed so that crucial environment variables like SSH_AGENT_PID and SSH_AUTH_SOCK get propagated. Otherwise, Buildbot will not be able to make use of ssh keys with passphrases that are cached by ssh-agent.

Added Selenium 2 support for collective.recipe.seleniumrc

Posted on by
Reply

I wanted to use the collective.recipe.seleniumrc buildout recipe to deploy Selenium RC, but quickly discovered that it didn’t work with recent Selenium 2 versions, because the recipe assumes that downloads are .zip files with embedded .jar files but new versions of Selenium RC have the recommended “standalone” package come as just .jar files.

So I made the changes to support Selenium 2 and published them here:

https://github.com/msabramo/collective.recipe.seleniumrc/tree/selenium2_support

I sent an email to the maintainer so that hopefully this support will get merged in but until that happens, you can use my recipe by using a buildout.cfg like this:

[buildout]
find-links = http://github.com/msabramo/collective.recipe.seleniumrc/tarball/selenium2_support#egg=collective.recipe.seleniumrc-0.6dev
parts = seleniumrc

[seleniumrc]
recipe = collective.recipe.seleniumrc>=0.6dev

Python txLoadBalancer

Posted on by
3

The idea of a Python-based load balancer, txLoadBalancer, piqued my interest. It took a bit of futzing around to actually get it running, because the distribution doesn’t specify its dependencies or include a sample config file.

$ mkvirtualenv --no-site-packages txLoadBalancer-test
$ bin/pip install txLoadBalancer twisted pycrypto pyasn1 pyopenssl
$ mkdir etc
$ curl -L http://bit.ly/iYWhWp > etc/config.xml
$ bin/twistd -noy ./bin/txlb.tac

Aside from the web interface, you can connect to the program with ssh:

$ ssh -p 7002 admin@localhost

This gives you a Python though I don’t yet know how to do anything useful with it 🙂

Python tool of the day: yolk

Posted on by
Reply

I just stumbled upon yolk.

It does a bunch of things related to package management, but the simplest and arguably most useful is to just list installed packages.

marcabramowitz@Marc-Abramowitzs-MacBook-Pro:~$ yolk -l
Django          - 1.3          - active development (/usr/local/lib/python2.7/site-packages)
Python          - 2.7.1        - active development (/usr/local/Cellar/python/2.7.1/lib/python2.7/lib-dynload)
distribute      - 0.6.16       - active development (/usr/local/lib/python2.7/site-packages)
ipython         - 0.10.2       - active development (/usr/local/lib/python2.7/site-packages)
meld3           - 0.6.7        - active development (/usr/local/lib/python2.7/site-packages)
mercurial       - 1.8.2        - active development (/usr/local/lib/python2.7/site-packages)
pip             - 1.0.1        - active development (/usr/local/lib/python2.7/site-packages)
pyodbc          - 2.1.8        - active development (/usr/local/lib/python2.7/site-packages)
setuptools      - 0.6c11       - active development (/usr/local/lib/python2.7/site-packages)
supervisor      - 3.0a10       - active development (/usr/local/lib/python2.7/site-packages)
virtualenv      - 1.6          - active development (/usr/local/lib/python2.7/site-packages)
virtualenvwrapper - 2.7.1        - active development (/usr/local/lib/python2.7/site-packages)
wsgiref         - 0.1.2        - active development (/usr/local/Cellar/python/2.7.1/lib/python2.7)
yolk            - 0.4.1        - active development (/usr/local/lib/python2.7/site-packages)

Although now that I think about it, you could get almost the same info from pip:

marcabramowitz@Marc-Abramowitzs-MacBook-Pro:~$ pip freeze
Django==1.3
distribute==0.6.15
ipython==0.10.2
meld3==0.6.7
mercurial==1.8.2
pyodbc==2.1.8
supervisor==3.0a10
virtualenv==1.6
virtualenvwrapper==2.7.1
wsgiref==0.1.2
yolk==0.4.1

though yolk can do some other nice things like display dependencies:

marcabramowitz@Marc-Abramowitzs-MacBook-Pro:~$ yolk -d supervisor
supervisor 3.0a10
  meld3>=0.6.5

and entry points:

marcabramowitz@Marc-Abramowitzs-MacBook-Pro:~$ yolk --entry-points paste.paster_create_template
paste.deploy.paster_templates
   paste_deploy = paste.deploy.paster_templates:PasteDeploy

paste.script.templates
   basic_package = paste.script.templates:BasicPackage

pylons.util
   pylons_minimal = pylons.util:MinimalPylonsTemplate

pylons.util
   pylons = pylons.util:PylonsTemplate

and querying for the latest goodies on pypi:

(yolk)marcabramowitz@Marc-Abramowitzs-MacBook-Pro:~/python/virtualenvs/yolk$ yolk --latest-releases=4
django-dynamic-choices 0.1.6
pydub 0.4.2
collective.portlet.calendar 0.5
sslsnoop 0.7.1
TexDO 0.1.1
haystack 0.7
thumbor 0.7.12
webapp2 1.2
django-thumbnail-works 0.2.1
Beacon 1.3-alpha

Fun with zc.buildout

Posted on by
Reply

I’ve been exploring zc.buildout (another link) lately. In the past, I’ve typically gone the simpler route of using virtualenv and pip, but I knew that buildout could do some powerful things and I’ve long wanted to kick the tires.

I’m using buildout for a Django project at work and it’s working out pretty well, though I sense that I might just be scratching the surface. Tonight, I wanted to play with some of the other buildout recipes.

The result is this educational buildout gist that demonstrates a recipe for building and deploying Varnish and a recipe for installing Supervisor.

The supervisor manages two processes — a simple one-line Python “origin server” on port 8080 and a Varnish reverse proxy on port 8000.

PyObjC and how to parse Preview.app bookmarks with Python

Posted on by
Reply

I wrote a Python program that parses Preview.app‘s bookmarks. The program itself is probably of dubious use, but it illustrates some interesting techniques like using class-dump to figure out how programs written in Objective-C work and how to use PyObjC to write Cocoa-style code with Python.

https://github.com/msabramo/print_os_x_preview_app_bookmarks

PyObjC allows you to write Python code that access the Cocoa libraries in a fairly intuitive way. The trickiest part of writing the program was figuring out how the data archived in the plist file was archived, so that I could write code to unarchive it. How did I figure out what the data looks like?

1. Install class-dump.

$ brew install class-dump

2. Run it (The awk is just to filter the output to show just the PVBookmark class).

$ class-dump /Applications/Preview.app | awk '/@interface PVBookmark/, /@end/ { print; }'
@interface PVBookmark : NSObject
{
    NSString *_UUID;
    NSString *_parentUUID;
    PVFileReference *_file;
    NSDate *_fileModDate;
    NSString *_label;
    int _pageIndex;
}

- (id)initWithFilePath:(id)arg1 label:(id)arg2 pageIndex:(unsigned long long)arg3;
- (void)dealloc;
- (id)initWithCoder:(id)arg1;
- (void)encodeWithCoder:(id)arg1;
- (id)description;
- (id)UUID;
- (id)filePath;
- (id)fileModificationDate;
- (id)label;
- (void)setLabel:(id)arg1;
- (unsigned long long)pageIndex;
- (id)targetExists;
- (BOOL)targetIsOnNetworkVolume;
- (id)displayPath;
- (unsigned long long)pageNumber;

@end