TarColor: color the output of tar tvf similarly to the way GNU ls would

One day I was looking through a tar file and looking for an executable. When I use ls, I can do this at a glance by just looking for green filenames, as I always have my ls configured to show executable files in green. It occurred to me that it would be nice to see the same kind of coloring when looking at the contents of tar files with tar tvf. Thus, TarColor was born. It was a quick Perl script to scratch my own itch. I thought others might find it useful, so I put it on GitHub and CPAN.

Usage

You can use tarcolor manually like this:

$ tar tvf some_tarball.tgz | tarcolor
... colored output ...

There is also a bundled shell script (for bash and zsh) that makes tar automatically pipe its output through tarcolor:

$ source /usr/local/etc/tarcolor/tarcolorauto.sh
$ tarcolorauto on
$ tar tvf some_tarball.tgz
... colored output ...
$ tarcolorauto off
$ tar tvf some_tarball.tgz
... normal uncolored output ...

Customization

Colors can be customized using the LS_COLORS or TAR_COLORS environment variables:

$ export LS_COLORS='di=01;34:ln=01;36:ex=01;32:so=01;40:pi=01;40:bd=40;33
:cd=40;33:su=0;41:sg=0;46'

The format for LS_COLORS and TAR_COLORS is the same format used by LS_COLORS (used by GNU ls). So if you use GNU ls and have your LS_COLORS set, then tarcolor will use similar colors as ls. If you want to tweak your colors and don’t want to edit LS_COLORS manually, you might try this online LS_COLORS generator.

Links

Pull request for Python pip installer accepted

https://github.com/pypa/pip/pull/374

The pull request addresses a UnicodeDecodeError bug in pip. It affects pip in Python 3 and it’s triggered when a package install fails and emits Unicode characters to the console. A classic case of this are gcc error messages which use exotic quotes.

It adds a test for the issue and then fixes it. Fixing this also required making a fix to Ian Bicking’s ScriptTest package, which was incorporated in ScriptTest 1.2.

The pip fix has not been included in an official pip release yet.

Don’t be alarmed by super large VSIZE on OS X

(Also at https://github.com/urbancoding/jenx/issues/32)

Don’t freak out too much if 64-bit processes on OS X have a VSIZE of 10 GB or more.

    marca@SCML-MarcA:~$ cat macruby_test.rb 
    print "What's your name? "
    name = gets.chop
    puts "Hi, #{name}!"

    marca@SCML-MarcA:~$ psgrep macruby
    USER    PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
    marca 76551   0.0  0.3 10899176  12272 s004  S+    9:09PM   0:00.06 macruby macruby_test.rb

    marca@SCML-MarcA:~$ top -pid 76551 -l 1 | tail -2
    PID    COMMAND %CPU TIME     #TH #WQ #PORTS #MREGS RPRVT  RSHRD RSIZE VPRVT VSIZE PGRP  PPID  STATE    UID       FAULTS COW  MSGSENT MSGRECV SYSBSD SYSMACH CSW PAGEINS USER 
    76551  macruby 0.0  00:00.06 1   0   40+    131+   5640K+ 15M+  12M+  323M+ 10G+  76551 72846 sleeping 346486362 3266+  189+ 234+    115+    254+   157+    35+ 0       marca

What this shows is a ridiculously simple Ruby program (running under MacRuby) having a VSIZE of 10 GB.

OTOH, Activity Monitor shows up a much more sane “Virtual Mem” of 315.0 MB for this process:

http://cl.ly/3F3N0P2M0l2O3q0b280Y

Also, vmmap seems to indicate that most of the virtual memory comes from “reserved VM address space (unallocated)” and it seems to consider this as unimportant:

    marca@SCML-MarcA:~$ vmmap 76551
    Virtual Memory Map of process 76551 (macruby)
    Output report format:  2.2  -- 64-bit process
    
    ...
    
    ==== Summary for process 76551
    ReadOnly portion of Libraries: Total=68.5M resident=59.0M(86%) swapped_out_or_unallocated=9792K(14%)
    Writable regions: Total=8.1G written=5068K(0%) resident=5164K(0%) swapped_out=0K(0%) unallocated=8.1G(100%)
    
    REGION TYPE                      VIRTUAL
    ===========                      =======
    MALLOC                            286.3M        see MALLOC ZONE table below
    MALLOC (reserved)                   7.8G        reserved VM address space (unallocated)
    ...
    ===========                      =======
    TOTAL                               8.2G
    TOTAL, minus reserved VM space    463.3M

                                         VIRTUAL ALLOCATION      BYTES
    MALLOC ZONE                             SIZE      COUNT  ALLOCATED  % FULL
    ===========                          =======  =========  =========  ======
    auto_zone_0x100bbd000                 256.0M      10635       680K      0%
    DefaultMallocZone_0x100b5b000          29.2M      28569      3207K     10%
    DispatchContinuations_0x100bfe000      1024K          1         32      0%
    ===========                          =======  =========  =========  ======
    TOTAL                                 286.3M      39205      3887K      1%

This issue, by the way, is not specific to MacRuby (my initial suspicion). I saw a similar pattern with Xcode, which also has a VSIZE of around 11 GB. I did not see this pattern with MRI Ruby [ruby 1.9.3p0 (2011-10-30 revision 33570)], Mail.app, Finder, iTerm, Alfred, Safari, or mysqld — these apps all have a much lower VSIZE of around 3 or 4 GB each. I have no idea why.

OS X appears to “reserve” a large amount of virtual memory for 64-bit processes, but they are not necessarily using it just because it’s been reserved for them.

phpserialize on Python 3

I was playing around with phpserialize and found out that phpserialize doesn’t work on Python 3.

However, I put together a GitHub fork that works with Python 3 (and also has tests, especially for Unicode which is the real gotcha when porting from Python 2.x to Python 3.x):

To use this, I put

-e git+git://github.com/msabramo/phpserialize.git@python3.x#egg=phpserialize

in my pip requirements file.

For Tox (a recent find and a very nice tool), this needs to be tweaked because it uses ConfigParser to parse its ini file and ConfigParser swallows the stuff after the # as a comment (I could find no way to escape the #; others have noticed this too). So in my tox.ini I used:


https://github.com/msabramo/phpserialize/tarball/python3.x

for the deps setting.

Full tox.ini for the curious:

More pull requests accepted into South

I had a few more of my contributions accepted into South:

Bitbucket pull request #10: Make `schemamigration` output more readable and PEP8-compliant
Bitbucket pull request #11: Print the failing SQL when a query fails
Bitbucket pull request #12: For manage.py migrate –list –verbosity=2, print the datetime that migrations were applied
Bitbucket pull request #13: Add ability to migrate forwards and backwards easily one by one