projects.skurfer.com

The Problem

Normally, you could install python-ldap like this:

sudo pip install python-ldap

That will appear to work fine, but then when you try to use the ldap module, you get this:

% python -c 'import ldap'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/ldap/__init__.py", line 22, in <module>
    from _ldap import *
ImportError: dlopen(/Library/Python/2.7/site-packages/_ldap.so, 2): Symbol not found: _ldap_create_assertion_control_value
  Referenced from: /Library/Python/2.7/site-packages/_ldap.so
  Expected in: flat namespace
 in /Library/Python/2.7/site-packages/_ldap.so

In a nutshell, /usr/include/ldap.h is a lie. It's the header for OpenLDAP 2.4.23, which is what comes bundled with Lion. All the binaries like ldapsearch, slapd, etc. are also at this version, but one thing was overlooked: the libraries.

% otool -L /usr/lib/libldap_r.dylib
/usr/lib/libldap_r.dylib:
    /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.2.0)
    /usr/lib/libsasl2.2.dylib (compatibility version 3.0.0, current version 3.15.0)
    /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 46.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 633.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 55010.0.0)
    /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 395.6.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.0.0)

That's OpenLDAP 2.2.0, which is God knows how old. WTF, Apple?

The Work-around

One way around this is to use the older 2.3.13 version of python-ldap, but who wants that, right?

Install the OpenLDAP Libraries

What I did was set up Homebrew to install only the libraries and headers from OpenLDAP 2.4.23, then build python-ldap against them. If you don't use Homebrew for some reason, I'm sure you can figure out how to do things by hand after looking at the example.

Here's the formula for Homebrew. Save it to /usr/local/Library/Formula/openldap-libs.rb.

require 'formula'

class OpenldapLibs < Formula
  url 'ftp://ftp.OpenLDAP.org/pub/OpenLDAP/openldap-stable/openldap-stable-20100719.tgz'
  homepage 'http://www.openldap.org/'
  md5 '90150b8c0d0192e10b30157e68844ddf'
  version '2.4.23'

  def install
    system "./configure", "--disable-debug", "--prefix=#{prefix}",
                          "--disable-slapd", "--disable-slurpd"

    # empty Makefiles to prevent unnecessary installation attempts
    makefile = "all:\ninstall:\n"
    unwanted_paths = ['clients', 'servers', 'tests', 'doc']
    unwanted_paths.each do |upath|
      File.open(Dir.getwd + '/' + upath + '/Makefile', 'w') {|f| f.write(makefile)}
    end

    system "make install"
    File.rename("#{prefix}/etc/openldap/ldap.conf", "#{prefix}/etc/openldap/ldap.conf.backup")
    File.symlink('/etc/openldap/ldap.conf', "#{prefix}/etc/openldap/ldap.conf")
  end
end

Then install it:

% brew install openldap-libs
==> Downloading ftp://ftp.OpenLDAP.org/pub/OpenLDAP/openldap-stable/openldap-stable-20100719
==> ./configure --disable-debug --prefix=/usr/local/Cellar/openldap-libs/2.4.23 --disable-sl
==> make install
/usr/local/Cellar/openldap-libs/2.4.23: 19 files, 1.5M, built in 49 seconds

Things to be aware of:

• This does some hacky things to prevent installation of duplicate client tools and man pages. The ones you have bundled with the OS are fine. We just need the libraries.
• This symlinks /usr/local/etc/openldap/ldap.conf to /etc/openldap/ldap.conf. That way, you don't have to duplicate the settings that you've likely already put there. If you happened to have something in /usr/local/etc/openldap/ldap.conf already, it'll get backed up as ldap.conf.backup. (I originally tried to just have the libraries point to the existing system's config files, but it attempts to install them, which is unnecessary and requires root privileges which Homebrew typically shouldn't have.)

Install python-ldap

You need to modify the default setup.cfg, so installing python-ldap is a manual process. Download the latest tarball from http://pypi.python.org/pypi/python-ldap/. Extract it and modify the _ldap section of setup.cfg to match what's shown below.

[_ldap]
library_dirs = /usr/local/lib
include_dirs = /usr/include/sasl
extra_compile_args = -g -arch x86_64
extra_objects = 
libs = ldap_r lber sasl2 ssl crypto

Now build and install:

% python setup.py build
extra_compile_args: -g -arch x86_64
extra_objects: 
include_dirs: /usr/include/sasl
library_dirs: /usr/local/lib
libs: ldap_r lber sasl2 ssl crypto
running build
running build_py
…blah blah blah…
% sudo python setup.py install
…blah blah blah…

You should have a working module at this point.

% python -c 'import ldap; print ldap.__version__'
2.4.3

And for what it's worth, I'm assuming something similar should work for Snow Leopard.

I'm quickly becoming a fan of the Python-Markdown implementation (to which I am now a contributor). I have a lot of code blocks containing LDIF, especially in documentation at work, but Pygments didn't support the syntax, so I've done my best to remedy that.

Here it is highlighting the output from ldapsearch sn=mcbroom.

SASL/GSSAPI authentication started
SASL username: rmcbroom@EMPLOYER.COM
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <dc=employer,dc=com> (default) with scope subtree
# filter: sn=mcbroom
# requesting: ALL
#

# rmcbroom, users, employer.com
dn: uid=rmcbroom,ou=users,dc=employer,dc=com
objectClass: top
objectClass: inetOrgPerson
objectClass: posixAccount
cn: Rob McBroom
displayName: Rob McBroom
gidNumber: 1000
homeDirectory: /home/rmcbroom
loginShell: /bin/tcsh
mobile: 800-555-1212
o: Employer
ou: users
pagerMail: 8005551212@vtext.net
sn: McBroom
givenName: Rob
uidNumber: 1000
uid: rmcbroom
mail: rmcbroom@employer.com
userPassword:: SSBkaWRuJ3QgcHV0IG15IGFjdHVhbCBwYXNzd29yZCBoZXJlLCBqYWNrYXNzLg==
sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1Z4uWN3tsCVp7ptNjFw69HxP4vEr
 VAK1h3zYHRETM5hK9YqQAQu+ZW+xrJSrWrQVdj7/KLqMHbnHS/0NaJLHne+N5SKGwWUTbKhKIUvEU
 YuMIfqwNpYU85tFkQ+HT29CDEvl/vEHXOO3ZCynGdbntShXDIplfbnmEs1IQJEH3aGQGtyfxsI5ee
 fK8BfY1RSd1S9x+NmtITPWUN0MacPWNt9QoLY/fZG3jmmCPOWpijPdjJZ0V3fVqwcyFHvGg1UD2BQ
 0ONGRc5fxMQpK6vV4G/vc9SdCOnXGv3OR0VKdIizIKg4sC1zLlTDAXNRU3rv8CpagHRhSgEEi+Y8r
 v6l9/w==

# search result
search: 5
result: 0 Success

# numResponses: 2
# numEntries: 1

It handles comments, attributes, values, multi-line values and even some things that aren't LDIF at all, like the authentication info at the top and the search result section at the bottom. I've even tested it on the hoary beast you get from querying Active Directory and it all looks good.

The lexer itself is pretty simple, so I'll show it here for the curious.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
    pygments.lexers.ldif
    ~~~~~~~~~~~~~~

    Pygments lexer for LDAP Data Interchange Format.

    :copyright: (c) 2011 by Rob McBroom.
    :license: LICENSE_NAME, see LICENSE_FILE for more details.
"""

from pygments.lexer import RegexLexer, bygroups
from pygments.token import *

class LdifLexer(RegexLexer):
    """Pygments lexer for LDAP Data Interchange Format."""
    name = 'LDAP Data Interchange Format'
    aliases = ['ldif', 'LDIF']
    filenames = ['*.ldif']
    tokens = {
        'root': [
            # authentication noise (not LDIF, but sent to STDOUT)
            (r'^SASL.*$', Text),
            # comments and in betweens
            (r'^#.*(\n .*){1,}$', Comment.Multiline),
            (r'^#.*$', Comment.Single),
            (r'^-$', Punctuation),
            (r'^(search|result|ref):\s.+$', Text),
            # attributes
            (r'^(add|replace|delete|replica|changetype)(?=:)',
             Keyword.Reserved),
            (r'^dn(?=:)', Name.Attribute, 'dn'),
            (r'^\w+(?=:)', Name.Attribute),
            # multiline values
            (r'(?<=:<\s).*(\n .*){1,}$', Name.Namespace),
            (r'(?<=::\s).*(\n .*){1,}$', Number.Hex),
            (r'(?<=:\s).*(\n .*){1,}$', Name.Variable),
            # values
            (r'(?<=changetype:\s)(add|modify|delete)$', Keyword.Reserved),
            (r'(?<=:\s)\d{14}(\.\d)?Z$', Number.Integer),
            (r'(?<=:<\s)\S.*$', String.Doc),
            (r'(?<=::\s)\S.*$', Number.Hex),
            (r'(?<=:\s)\S.*$', Name.Variable),
            # in-line separators
            (r'(?<=:)\s', Whitespace),
            (r'(?<=:<)\s', Whitespace),
            (r':[:<]?', Operator),
        ],
        'dn': [
            (r'(:)(\s)', bygroups(Operator, Whitespace)),
            (r'(?<=:\s).*(\n .*){1,}$', Name.Class),
            (r'(?<=:\s).*$', Name.Class),
        ],
    }

I've asked the Pygment's folks about including it by default, so hopefully before long it'll be widely available. Enjoy.

Side Note

I also created the LDIF bundle for TextMate. I know I'm not the only person in the world using LDAP. My theory is that I'm just the only person in the world using LDAP in conjunction with nice, modern tools.

There's the latest, and then there's the latest. That is to say, there are frequently some cool updates and new features in pull requests that haven't been merged yet. What if you want to try them out? Here's what I do.

Things I'm assuming you've already done

1. Fork the Quicksilver repository

2. Clone the fork to your machine

   % git clone git@github.com:username/Quicksilver.git
   

3. Add a remote for the official repository¹

   % git remote add upstream git://github.com/quicksilver/Quicksilver.git
   

Getting the Latest

Now, to get the good stuff, make sure your master branch is up to date.

% git checkout master
Switched to branch 'master'
% git pull --rebase upstream master
From github.com:quicksilver/Quicksilver
 * branch            master     -> FETCH_HEAD
Current branch master is up to date.

Then start browsing the pull requests. Open one you're interested in and look toward the top. You'll see something like

devuser wants someone to merge 5 commits into quicksilver:master from devuser:featurebranch

You'll want to create a new branch to contain this untested code. GitHub uses the convention of naming this local branch otheruser-branchname, but use whatever makes sense to you if that seems too long.

% git checkout -b devuser-featurebranch master
Switched to a new branch 'devuser-featurebranch'
% git pull https://username@github.com/devuser/Quicksilver.git featurebranch
... stuff comes down ...

Repeat this process for any other pull requests.

Now, create a new branch where you'll merge everything together and start merging things in from the other branches.

% git checkout -b mine master
Switched to a new branch 'mine'
% git merge devuser-featurebranch
... merge stuff ...
% git merge otheruser-otherfeature
... merge stuff ...

Now, while still on the "mine" branch, open the Quicksilver project in Xcode, clean all targets, and build your Frankenstein. (You have backups, right?)

Additional Tips

I make a new "mine" branch almost daily. It's easy to start fresh.

% git checkout master
Switched to branch 'master'
% git branch -D mine
Deleted branch mine (was d6d61ac).
% git checkout -b mine master
Switched to a new branch 'mine'

If additional commits are added to a pull request you were testing, you can update it without starting from scratch. Just switch to that branch and run the same pull command.

% git checkout devuser-featurebranch
Switched to branch 'devuser-featurebranch'
% git pull https://username@github.com/devuser/Quicksilver.git featurebranch
... stuff comes down ...

You can merge this with "mine" again to get the updates there.

Last but Not Least

As long as you're going to the trouble of trying out cutting-edge stuff, please take the time to report and problems by commenting on the pull request on GitHub.

Everyone with a SSD in their Mac has read this, right? I thought the RAM disk was going a bit far and I never implemented it, but since I've been building Quicksilver many times per day lately, I thought I should give it another look.

What I've done isn't that different from the nullVision solution, but I had a couple of reasons to change things around.

1. I don't need all of /tmp to be on a RAM disk, just /tmp/QS.
2. My wife is not building Quicksilver on a regular basis. This drive only needs to be available when I log in.
3. A StartupItem? In 2011? I didn't know they still worked.

So what I wanted was a 256MB RAM disk mounted at /tmp/QS. The first thing you need is a script to create the disk. This'll work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

#!/bin/bash

RAMDisk() {
    mntpt=$1
    rdsize=$(($2*1024*2))
    echo "Creating RAMdisk for $mntpt"
    mkdir $mntpt
    # Create the RAM disk.
    dev=`hdid -drivekey system-image=yes -nomount ram://$rdsize`
    # Successfull creation…
    if [ $? -eq 0 ] ; then
        # Create HFS on the RAM volume.
        newfs_hfs $dev
        # Mount the RAM disk to the target mount point.
        mount -t hfs -o union -o nobrowse $dev $mntpt
    fi
}

RAMDisk /private/tmp/QS 256

I called the script qsdevdisk.sh. To make the disk available when you log in, create ~/Library/LaunchAgents/com.qsapp.devdisk.plist and put this in it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.qsapp.devdisk</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/qsdevdisk.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Log out and log back in and it should be there. The only drawback to this (besides the obvious memory it consumes) is that logging out seems to take a long time. I'm guessing the disk appears to be in use and the logout has to wait for a timeout before forcing it to unmount. Anyone know a solution to this?

For the past few months, a lot of people have been talking about the return of Quicksilver. I'm pretty involved in the community, and I have to say that I didn't understand where it was coming from. All the same bugs and work-arounds from the past year and a half were still in effect. But I wasn't about to question the good publicity.

Anyway, as of today, it's true. Quicksilver is back! Details can be found over at LoveQuicksilver, but the most important news isn't going to be on a list of features.

Upgrading. Finally.

A lot of people, including me, have been limping along on version β54, which was the last version from Alcor, the original developer. I'm pretty sure it was released on Mac OS X 10.4. It's old. Today's release is version β59. This is the first version I'll be running since it went open source, and I'm not alone. This, to me, is the true significance of this release.

I no longer feel like I'm hovering on the outside, hoping for things to get fixed. Now, not only are a lot of things fixed, but by using the latest version, I feel more invested. I can report on outstanding issues. I can stop trying to help users figure out which version they want and just recommend the latest without reservation. (I can also modify my preferences again, which is nice.)

Toot! Toot!

Another major feature of this release is that it includes code I wrote. I never thought I'd be saying that. Makes me feel like part of history. This is, after all, the most important application on Mac OS X.

I should also mention that there's a lot of momentum right now. There are a lot of us and we're finding and fixing bugs almost daily. B59 is a big step, but don't expect it to be the only one.