Digressions of a Drive-by Hacker

Resolved: GDAL on AWS GPU Instances

2015-01-27T00:00:00Z

Cross-posted from openterrain.tumblr.com.

Specifically, g2.2xlarge instances with Nvidia GRID K520 GPUs running Amazon Linux.

As we kicked off the new Knight News Grant, it was clear early on that we were going to be processing quite a lot of raster data. Given that, I wanted to ensure that we'd be able to benefit from GDAL's OpenCL-accelerated warping (for reprojection and scaling).

Ubuntu is typically my weapon of choice for these sorts of things, however I wanted to minimize hardware-related compatibility problems, and Amazon publishes an Amazon Linux AMI with NVIDIA GRID GPU Driver on the AWS Marketplace.

Straightforward, right? Sadly, no.

It seemed thoroughly unlikely that GDAL from yum would include OpenCL support (rightly so), so I went about compiling GDAL from source, omitting everything I didn't care about (basically everything except GeoTIFF, zlib, curl, and OpenCL support):

# enable EPEL (for proj-devel)
sudo yum-config-manager --enable epel
sudo yum -y update
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel

cd /tmp
curl -L https://download.osgeo.org/gdal/1.11.1/gdal-1.11.1.tar.gz | tar zxf -
cd gdal-1.11.1
./configure  --prefix=/opt/local \
            --with-threads \
            --with-ogr \
            --with-geos \
            --without-libtool \
            --with-libz=internal \
            --with-libtiff=internal \
            --with-geotiff=internal \
            --without-gif \
            --without-pg \
            --without-grass \
            --without-libgrass \
            --without-cfitsio \
            --without-pcraster \
            --without-netcdf \
            --without-png \
            --without-jpeg \
            --without-gif \
            --without-ogdi \
            --without-fme \
            --without-hdf4 \
            --without-hdf5 \
            --without-jasper \
            --without-ecw \
            --without-kakadu \
            --without-mrsid \
            --without-jp2mrsid \
            --without-bsb \
            --without-grib \
            --without-mysql \
            --without-ingres \
            --without-xerces \
            --without-expat \
            --without-odbc \
            --without-sqlite3 \
            --without-dwgdirect \
            --without-idb \
            --without-sde \
            --without-perl \
            --without-php \
            --without-ruby \
            --without-python \
            --with-hide-internal-symbols \
            --with-opencl \
            --with-opencl-include=/opt/nvidia/cuda/include
sudo make install

# allow GDAL to find necessary libraries
export LD_LIBRARY_PATH=/opt/local/lib:$LD_LIBRARY_PATH
export PATH=/opt/local/bin:$PATH

My first discovery was that GDAL wouldn't even try to use the GPU, even when explicitly requested to (using -wo "USE_OPENCL=TRUE"). Running as root solved the problem, allowing subsequent non-root invocations (without explicitly requesting OpenCL) to also work.

watch nvidia-smi is a good way to see whether tasks are being offloaded to the GPU.

Once I got it to start using the GPU, I immediately ran into a problem:

ERROR 1: Error: Failed to build program executable!
 Build Log:
 :55:20: error: cannot decrement value of type 'float
 __attribute__((address_space(1)))'
 dstPtr[iDstOffset] --;
 ~~~~~~~~~~~~~~~~~~ ^

 ERROR 1: Error at file gdalwarpkernel_opencl.c line 2325:
 CL_BUILD_PROGRAM_FAILURE
 ERROR 1: OpenCL routines reported failure (-11) on line 3250.
 ERROR 1: Error: Failed to build program executable!
 Build Log:
 :55:20: error: cannot decrement value of type 'float
 __attribute__((address_space(1)))'
 dstPtr[iDstOffset] --;
 ~~~~~~~~~~~~~~~~~~ ^

 ERROR 1: Error at file gdalwarpkernel_opencl.c line 2325:
 CL_BUILD_PROGRAM_FAILURE
 ERROR 1: OpenCL routines reported failure (-11) on line 3250.

Fortunately, this turned out to be a quick fix for Even Roualt (thanks!!), though it did reinforce my fear of GPU compatibility headaches (<thing>-- vs <thing> = <thing> - 1):

 Index: alg/gdalwarpkernel_opencl.c
 ===================================================================
 --- alg/gdalwarpkernel_opencl.c (révision 28173)
 +++ alg/gdalwarpkernel_opencl.c (copie de travail)
 @@ -593,7 +593,7 @@
          "if (dstPtr[iDstOffset] == dstMinVal)\n"
              "dstPtr[iDstOffset] = dstMinVal + 1;\n"
          "else\n"
 -            "dstPtr[iDstOffset] --;\n"
 +            "dstPtr[iDstOffset] = dstPtr[iDstOffset] - 1;\n"
      "}\n"
  "}\n"
  "#endif\n"

Once patched, it works like a dream, happily churning through jobs at an impressive clip.

This fix will be included in GDAL-1.11.2.

The requirement to run GDAL as root to initialize the GPU turned out to be a configuration issue in the Amazon AMI. Running modprobe nvidia_uvm to load the kernel driver for the GPU on boot solved the problem, but only after adding a udev rule (/etc/udev/rules.d/99-nvidia.rules) to create the necessary device nodes:

# /etc/udev/rules.d/99-nvidia.rules
KERNEL=="nvidia_uvm", RUN+="/bin/sh -c '/bin/mknod -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/devices | cut -d \  -f 1) 0'"

make For Data Using Makefiles

2015-01-07T00:00:00Z

Mr. Mather recently spotted the Makefile we've been using for the updated version of Toner and asked for a walk-through.

Like many other people who wrangle data for a living, I've been in on-and-off pursuit of a make-like approach to processing and transforming data. Specifically, one that allows me to idempotently bootstrap and process data without starting from zero each time. Also, one that makes it easy enough to script experiments pro-actively rather than putting replicability off.

Common Idioms

We use a number of sh / bash idioms that are helpful to recognize as such.

Use of Exit Codes

When a process exits, it does so with a numeric code that indicates success or failure (in the latter case, often with a value that can be used to determine why). 0 is considered success; anything else is a failure of some sort. These codes aren't directly visible, but make uses them to determine whether a target was successful (and whether execution should continue). $? can be used to surface the code in a shell context:

$ echo hi; echo $?
hi
0

There are a few cases where we want to shield make from failures (to prevent execution from stopping). For example:

carto -l $< > $@ || (rm -f $@; false)

This runs carto; if it fails, it deletes the output ($@). However, if there was no output, rm will return successfully, but the task actually failed. To work-around that, we execute rm in a subshell (using parentheses) and explicitly return false (false) so that make ceases execution.

`||` and `&&`

|| and && are used in conjunction with exit codes to conditionally execute subsequent parts of a composite command. || is used when you want to execute a command only if the first returns an error (non-zero exit code); && when you want to execute a command only if the first was successful.

`> /dev/null 2>&1`

POSIX processes are provided 3 file descriptions (handles to files or file-like things) by default. stdin (content from a file or other source like a keyboard) is file descriptor 0, stdout is 1, and stderr is 2. Thus, echo hi > /dev/null 2>&1 says to redirect stdout (the default output) to /dev/null (into the abyss) and then stderr (2) to the same place as stdout (as a reference: &1). In this context, we use this to squelch output, since we generally just care about the exit code (either directly, or after greping through it or counting lines (wc -l)).

`psql` Checks

At various points, we want to short-circuit tasks if a resource already exists. In a traditional Makefile, these resources would be files and that short-circuiting is built-in. However, since we're since working with database tables (etc.) and there are no file equivalents, we need to implement equivalent functionality.

We use psql -c for practically all existence checks, as it allows us to construct a SQL command and use it to query PostgreSQL. Unfortunately, different checks result in varying output and exit codes.

Looking for Relations

\d <relation> will check for the existence of a "relation" (which could be a table, view, etc.). It displays the name, type, and owner of the relation and exits with 0 only if it matched something.

Looking for Extensions

\dx <extension> will check for extensions matching the provided name. If present, it will display various information about it. Unfortunately, it exits with 0 regardless of whether anything has been found or not, so we need to grep the output for the presence of the name.

Looking for Functions

\df <name> will check for functions matching the provided name. When a function has multiple signatures, it will be displayed multiple times. Like \dx, it exits with 0 no matter what, so we need to grep (case-insensitively) for the function we're looking for.

Implicit `make` Variables

make includes many implicit variables, most of which are intended for use in an environment where source files are being compiled and linked into binaries. However, there are still a few we make use of:

$@ - the name of the target (the thing before the :).
$< - the name of the first prerequisite. Convenient.
$^ - all prerequisites, space-delimited. Can be combined with $(word <n>,$^) to select the _n_th one.

`make` Functions

Since many of the targets we're working with are synthetic, we need to extract relevant components of their names. Database-related functionality is grouped under the db/ "path", so we primarily use subst to remove irrelevant components. We also use word to refer to components within space-delimited values.

`make` Patterns

There are many resources that follow patterns when we work with data. Natural Earth's filenames and source URLs are a good example of this. In keeping with the DRY principle ("don't repeat yourself"), we fold these into a smaller number of targets using make patterns. These are strings that include % anywhere text may vary.

The convenient bit is that prerequisites can also use the % syntax and the value of the pattern in the target name will be substituted. Thus, %: %.mml will convert toner to toner: toner.mml and gives us the behavior we're looking for.

Annotated `Makefile`

Without further ado, here's an annotated snapshot of the Makefile-driven approach we've been using. Suggestions are most welcome, especially if they simplify or clarify.

SHELL := /bin/bash

Use bash for sub-shells, allowing use of bash-specific functionality.

PATH := $(PATH):node_modules/.bin

Add npm-installed binaries to the PATH.

define EXPAND_EXPORTS
export $(word 1, $(subst =, , $(1))) := $(word 2, $(subst =, , $(1)))
endef

Define a macro that expands (splits on =) and exports (makes available to sub-shells) key-value arguments, e.g. DATABASE\_URL=postgres:///db.

# wrap Makefile body with a check for pgexplode
ifeq ($(shell test -f node_modules/.bin/pgexplode; echo $$?), 0)

Check whether pgexplode has been installed (it's effectively a prerequisite for the entire Makefile); test doesn't output anything, so the return code ($?) needs to be echo'd for comparison with 0 (and escaped with an extra $ to be passed through to the shell call).

# load .env
$(foreach a,$(shell cat .env 2> /dev/null),$(eval $(call EXPAND_EXPORTS,$(a))))

Read .env (squelching error messages if one doesn't exist) and pass each environment pair to EXPAND\_EXPORTS to make it available to commands in targets.

# expand PG* environment vars
$(foreach a,$(shell set -a && source .env 2> /dev/null; node_modules/.bin/pgexplode),$(eval $(call EXPAND_EXPORTS,$(a))))

Use pgexplode to expand DATABASE\_URL into libpq-compatible environment variables. This will read from the environment ($DATABASE\_URL) if one isn't present in .env (or .env doesn't exist).

define create_relation
@psql -c "\d $(subst db/,,$@)" > /dev/null 2>&1 || \
        psql -v ON_ERROR_STOP=1 -qX1f sql/$(subst db/,,$@).sql
endef

define create_extension
@psql -c "\dx $(subst db/,,$@)" | grep $(subst db/,,$@) > /dev/null 2>&1 || \
        psql -v ON_ERROR_STOP=1 -qX1c "CREATE EXTENSION $(subst db/,,$@)"
endef

Macro definitions that will strip db/ from targets' $@ (target name) and use it as the name of a SQL file or PostgreSQL extension ($(subst db/,,$@)). The first command (psql -c "\d $(subst db/,,$@)" > /dev/null 2>&1) checks for the presence of a relation in the database specified by DATABASE\_URL and only evaluates SQL commands if it fails. \d <name> checks for the presence of a relation of any kind, \dx for loaded extensions, etc.

ON\_ERROR\_STOP=1 is used to abort as soon as an error occurs, -q is "quiet", -X ignores any psqlrc files, -1 runs the command in a single transaction, -f provides a file containing commands, and -c tells it to use the provided command.

Commands are prefixed with @ to prevent make from printing them (since they're unnecessarily complicated due to the need to check for the existence of non-file resources).

define register_function_target
.PHONY: db/functions/$(strip $(1))

db/functions/$(strip $(1)): db
        @psql -c "\df $(1)" | grep -i $(1) > /dev/null 2>&1 || \
                psql -v ON_ERROR_STOP=1 -qX1f sql/functions/$(1).sql
endef

$(foreach fn,$(shell ls sql/functions/ 2> /dev/null | sed 's/\..*//'),$(eval $(call register_function_target,$(fn))))

Generate targets for each file in sql/functions, callable as db/function/<name> and depending on the db target (keep reading).

# Import PBF ($2) as $1
define import
.PHONY: db/osm-$(strip $(word 1, $(subst :, ,$(1)))) db/$(strip $(word 1, $(subst :, ,$(1))))

db/$(strip $(word 1, $(subst :, ,$(1)))): db/osm-$(strip $(word 1, $(subst :, ,$(1)))) db/shared

db/osm-$(strip $(word 1, $(subst :, ,$(1)))): db/postgis db/hstore $(strip $(word 2, $(subst :, ,$(1))))
        @psql -c "\d osm_roads" > /dev/null 2>&1 || \
        imposm3 import \
                --cachedir cache \
                -mapping=imposm3_mapping.json \
                -read $(strip $(word 2, $(subst :, ,$(1)))) \
                -connection="$${DATABASE_URL}" \
                -write \
                -deployproduction \
                -overwritecache
endef

Macro definition for importing OSM extracts. When called, it generates targets like db/<place> (which depends on both the import and db/shared (see below)) and db/osm-<place>, which does the actual import. osm\_roads is assumed to be created by imposm3 in this case. $${DATABASE\_URL} is escaped because this is a macro (so it's evaluated at runtime) and uses braces to use the environment variable.

Now begins what looks like a more conventional Makefile.

# default target
default: toner

# symlink into TileMill's project folder (if TM is installed)
link:
        @test -e ${HOME}/Documents/MapBox/project && \
                test -e ${HOME}/Documents/MapBox/project/toner || \
                ln -sf "`pwd`" ${HOME}/Documents/MapBox/project/toner

# clean up derivative files
clean:
        @rm -f *.mml *.xml

# create a default .env file with a sensible(?) default
.env:
        @echo DATABASE_URL=postgres:///toner > $@

%: %.mml
        @cp $< project.mml

A pattern, which obviates the need to declare multiple redundant targets. If make toner is run, it will depend on toner.mml (see below) and will silently copy the output ($< is the expanded name of the first dependency) to project.mml for TileMill to read.

mml: $(subst .yml,.mml,$(filter-out circle.yml,$(wildcard *.yml)))

xml: $(subst .yml,.xml,$(filter-out circle.yml,$(wildcard *.yml)))

Defines targets that will make MML and XML files corresponding to all .yml files except circle.yml (which is a control file for CircleCI, not a style).

.PRECIOUS: %.mml

%.mml: %.yml map.mss labels.mss %.mss interp js-yaml
        @echo Building $@
        @cat $< | interp | js-yaml > tmp.mml && mv tmp.mml $@

Builds *.mml by interpolating environment variables into a Mustache-templated YAML file (interp) and converting to JSON (js-yaml). tmp.mml is used so that mv can atomically move the file into place (without doing this, TileMill periodically chokes when reading partially-written files).

This depends on <style>.yml, map.mss, labels.mss, and <style.mss> so that it will be considered stale when any of those are modified. interp and js-yaml are explicitly called out as dependencies so that they can be installed if necessary (see below).

This is marked as .PRECIOUS so that artifacts won't be deleted when called as an intermediate target (i.e. from an XML target).

.PRECIOUS: %.xml

%.xml: %.mml carto
        @echo
        @echo Building $@
        @echo
        @carto -l $< > $@ || (rm -f $@; false)

Builds *.xml from <style>.mml (declared above). || (rm -f $@; false) is included because carto may leave behind invalid XML files when it fails (and because we want to pass the failure through and terminate the current make invocation.

.PHONY: carto

carto: node_modules/carto/package.json

.PHONY: interp

interp: node_modules/interp/package.json

.PHONY: js-yaml

js-yaml: node_modules/js-yaml/package.json

node_modules/carto/package.json: PKG = $(word 2,$(subst /, ,$@))
node_modules/carto/package.json: node_modules/millstone/package.json
        @type node > /dev/null 2>&1 || (echo "Please install Node.js" && false)
        @echo "Installing $(PKG)"
        @npm install $(PKG)

node_modules/interp/package.json: PKG = $(word 2,$(subst /, ,$@))
node_modules/interp/package.json:
        @type node > /dev/null 2>&1 || (echo "Please install Node.js" && false)
        @echo "Installing $(PKG)"
        @npm install $(PKG)

node_modules/js-yaml/package.json: PKG = $(word 2,$(subst /, ,$@))
node_modules/js-yaml/package.json:
        @type node > /dev/null 2>&1 || (echo "Please install Node.js" && false)
        @echo "Installing $(PKG)"
        @npm install $(PKG)

node_modules/millstone/package.json: PKG = $(word 2,$(subst /, ,$@))
node_modules/millstone/package.json:
        @type node > /dev/null 2>&1 || (echo "Please install Node.js" && false)
        @echo "Installing $(PKG)"
        @npm install $(PKG)

Artificial (.PHONY) targets for required commands along with file-based dependencies and checks for Node. PKG is defined as a make variable in preparation for future refactoring that turns this boilerplate into a macro.

package.json declares dependencies on these commands, but it also includes everything else to run a rendering node, so this is a lower-impact way of ensuring that they're installed.

.PHONY: DATABASE_URL

DATABASE_URL:
        @test "${$@}" || (echo "$@ is undefined" && false)

A target definition that can be used when one wants to ensure that a DATABASE\_URL was provided.

.PHONY: db

db: DATABASE_URL
        @psql -c "SELECT 1" > /dev/null 2>&1 || \
        createdb

Target to ensure that a database exists. If psql returns fall, createdb will be run (using libpq environment variables extracted from DATABASE\_URL) to create one.

.PHONY: db/postgis

db/postgis: db
        $(call create_extension)

.PHONY: db/hstore

db/hstore: db
        $(call create_extension)

Targets that create extensions (and require that a database exists) using the macros defined above.

.PHONY: db/shared

db/shared: db/postgres db/shapefiles

.PHONY: db/postgres

db/postgres: db/functions/highroad

Meta-targets that depend on both explicit (db/shapefiles) and implicit (db/functions/highroad) targets.

.PHONY: db/shapefiles

db/shapefiles: shp/osmdata/land-polygons-complete-3857.zip \
               shp/natural_earth/ne_50m_land-merc.zip \
               shp/natural_earth/ne_50m_admin_0_countries_lakes-merc.zip \
               shp/natural_earth/ne_10m_admin_0_countries_lakes-merc.zip \
               shp/natural_earth/ne_10m_admin_0_boundary_lines_map_units-merc.zip \
               shp/natural_earth/ne_50m_admin_1_states_provinces_lines-merc.zip \
               shp/natural_earth/ne_10m_geography_marine_polys-merc.zip \
               shp/natural_earth/ne_50m_geography_marine_polys-merc.zip \
               shp/natural_earth/ne_110m_geography_marine_polys-merc.zip \
               shp/natural_earth/ne_10m_airports-merc.zip \
               shp/natural_earth/ne_10m_roads-merc.zip \
               shp/natural_earth/ne_10m_lakes-merc.zip \
               shp/natural_earth/ne_50m_lakes-merc.zip \
               shp/natural_earth/ne_10m_admin_0_boundary_lines_land-merc.zip \
               shp/natural_earth/ne_50m_admin_0_boundary_lines_land-merc.zip \
               shp/natural_earth/ne_10m_admin_1_states_provinces_lines-merc.zip

Meta-target for processed Shapefiles.

# TODO places target that lists registered places
PLACES=BC:data/extract/north-america/ca/british-columbia-latest.osm.pbf \
       CA:data/extract/north-america/us/california-latest.osm.pbf \
       belize:data/extract/central-america/belize-latest.osm.pbf \
       cle:data/metro/cleveland_ohio.osm.pbf \
       MA:data/extract/north-america/us/massachusetts-latest.osm.pbf \
       NY:data/extract/north-america/us/new-york-latest.osm.pbf \
       OH:data/extract/north-america/us/ohio-latest.osm.pbf \
       sf:data/metro/san-francisco.osm.pbf \
       sfbay:data/metro/sf-bay-area.osm.pbf \
       seattle:data/metro/seattle_washington.osm.pbf \
       WA:data/extract/north-america/us/washington-latest.osm.pbf

$(foreach place,$(PLACES),$(eval $(call import,$(place))))

Define a list of places along with the local reference to their corresponding extract (data/extract/% and data/metro/% are patterns defined below) and generate import targets (e.g. db/BC for British Columbia).

.SECONDARY: data/extract/%

data/extract/%:
        @mkdir -p $$(dirname $@)
        curl -Lf https://download.geofabrik.de/$(@:data/extract/%=%) -o $@

.SECONDARY: data/metro/%

data/metro/%:
        @mkdir -p $$(dirname $@)
        curl -Lf https://s3.amazonaws.com/metro-extracts.mapzen.com/$(@:data/metro/%=%) -o $@

OSM extract patterns; i.e. anything under data/extract/ will be downloaded from Geofabrik, anything under data/metro/ from Mapzen's Metro Extracts).

These are marked as .SECONDARY to prevent them from being deleted (as they're "expensive" to create). Note: this isn't quite right; my intention is to keep them around but also to delete them if the target failed.

mkdir -p is used to ensure that the target directory exists. $$(dirname $@) is used to pass the literal $(dirname data/metro/<whatever>) to mkdir.

$(@:data/metro/%=%) substitutes <whatever> for data/metro/<whatever> in the target name.

curl's -f option is provided so that it will return with a non-zero exit code on failure and cause the target to fail.

.SECONDARY: data/osmdata/land_polygons.zip

# so the zip matches the shapefile name
data/osmdata/land_polygons.zip:
        @mkdir -p $$(dirname $@)
        curl -Lf https://data.openstreetmapdata.com/land-polygons-complete-3857.zip -o $@

shp/osmdata/%.shp \
shp/osmdata/%.dbf \
shp/osmdata/%.prj \
shp/osmdata/%.shx: data/osmdata/%.zip
        @mkdir -p $$(dirname $@)
        unzip -ju $< -d $$(dirname $@)

shp/osmdata/land_polygons.index: shp/osmdata/land_polygons.shp
        shapeindex $<

.SECONDARY: data/osmdata/land-polygons-complete-3857.zip

shp/osmdata/land-polygons-complete-3857.zip: shp/osmdata/land_polygons.shp \
        shp/osmdata/land_polygons.dbf \
        shp/osmdata/land_polygons.prj \
        shp/osmdata/land_polygons.shx \
        shp/osmdata/land_polygons.index
        zip -j $@ $^

Similar to the OSM extracts, but for a specific file with a non-matching source name. Some of this remains as a relic of when we were using different versions of the land polygons.

define natural_earth
db/$(strip $(word 1, $(subst :, ,$(1)))): $(strip $(word 2, $(subst :, ,$(1)))) db/postgis
        psql -c "\d $$(subst db/,,$$@)" > /dev/null 2>&1 || \
        ogr2ogr --config OGR_ENABLE_PARTIAL_REPROJECTION TRUE \
                        --config SHAPE_ENCODING WINDOWS-1252 \
                        --config PG_USE_COPY YES \
                        -nln $$(subst db/,,$$@) \
                        -t_srs EPSG:3857 \
                        -lco ENCODING=UTF-8 \
                        -nlt PROMOTE_TO_MULTI \
                        -lco POSTGIS_VERSION=2.0 \
                        -lco GEOMETRY_NAME=geom \
                        -lco SRID=3857 \
                        -lco PRECISION=NO \
                        -clipsrc -180 -85.05112878 180 85.05112878 \
                        -segmentize 1 \
                        -skipfailures \
                        -f PGDump /vsistdout/ \
                        /vsizip/$$



Macro definition that creates db/ne\_<whatever> and shp/natural\_earth/*
targets for Natural Earth sources. It
assumes that it's called with arguments in the form <name>:<source file>:[shapefile]. (If someone can help simplify the repeated $(strip $(word 1, $(subst :, ,$(1)))) declarations (used to extract the first
component), I'd appreciate it!)
The db/<whatever> target runs ogr2ogr with a set of options that we've
found to work well with the Natural Earth Shapefiles over the years.
shp/natural\_earth/*-merc.{shp,dbf,prj,shx} states that there will be
4 artifacts for each invocation of ogr2ogr (used to reproject here). Again,
we use options gathered over the years along with /vsizip to avoid needing
to unzip. If a single "file" (base name, really) exists in the zip, just the
name of the zip file is necessary, otherwise a path to the Shapefile within
the zip is necessary (the list of layers below takes advantage of that by
allowing [shapefile] to be optional. (This is necessary due to errors in
the packaging of some of the Natural Earth layers.)
Aggressive escaping is necessary in order for literal $s to be passed
through, e.g. $$$$(dirname $$@).
The shp/natural\_earth/*-merc.zip target uses $^ for the list of files to
compress, which contains the expanded names of all of the dependencies.
zip's -j option is used to "junk paths" and put everything in the root of
the zip file.

# ::[shapefile]
NATURAL_EARTH=ne_50m_land:data/ne/50m/physical/ne_50m_land.zip \
        ne_50m_admin_0_countries_lakes:data/ne/50m/cultural/ne_50m_admin_0_countries_lakes.zip \
        ne_10m_admin_0_countries_lakes:data/ne/10m/cultural/ne_10m_admin_0_countries_lakes.zip \
        ne_10m_admin_0_boundary_lines_map_units:data/ne/10m/cultural/ne_10m_admin_0_boundary_lines_map_units.zip \
        ne_50m_admin_1_states_provinces_lines:data/ne/50m/cultural/ne_50m_admin_1_states_provinces_lines.zip \
        ne_10m_geography_marine_polys:data/ne-stamen/10m/physical/ne_10m_geography_marine_polys.zip \
        ne_50m_geography_marine_polys:data/ne-stamen/50m/physical/ne_50m_geography_marine_polys.zip \
        ne_110m_geography_marine_polys:data/ne-stamen/110m/physical/ne_110m_geography_marine_polys.zip \
        ne_10m_airports:data/ne-stamen/10m/cultural/ne_10m_airports.zip \
        ne_10m_roads:data/ne/10m/cultural/ne_10m_roads.zip \
        ne_10m_lakes:data/ne/10m/physical/ne_10m_lakes.zip \
        ne_50m_lakes:data/ne/50m/physical/ne_50m_lakes.zip \
        ne_10m_admin_0_boundary_lines_land:data/ne/10m/cultural/ne_10m_admin_0_boundary_lines_land.zip \
        ne_50m_admin_0_boundary_lines_land:data/ne/50m/cultural/ne_50m_admin_0_boundary_lines_land.zip \
        ne_10m_admin_1_states_provinces_lines:data/ne/10m/cultural/ne_10m_admin_1_states_provinces_lines.zip:ne_10m_admin_1_states_provinces_lines.shp

$(foreach shape,$(NATURAL_EARTH),$(eval $(call natural_earth,$(shape))))

Define a list of Natural Earth layers along with their local file references
and (optional) Shapefile names and call natural_earth to generate targets.
As with the OSM extracts, patterns are used to match the conventions used by
the Natural Earth site.

define natural_earth_sources
.SECONDARY: data/ne/$(1)/$(2)/%.zip

data/ne/$(1)/$(2)/%.zip:
        @mkdir -p $$(dir $$@)
        curl -fL https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/$(1)/$(2)/$$(@:data/ne/$(1)/$(2)/%=%) -o $$@

.SECONDARY: data/ne/$(1)/$(2)/%.zip

data/ne-stamen/$(1)/$(2)/%.zip:
        @mkdir -p $$(dir $$@)
        curl -fL "https://github.com/stamen/natural-earth-vector/blob/master/zips/$(1)_$(2)/$$(@:data/ne-stamen/$(1)/$(2)/%=%)?raw=true" -o $$@
endef

Macro definition for meta Natural Earth patterns.

scales=10m 50m 110m
themes=cultural physical raster

$(foreach a,$(scales),$(foreach b,$(themes),$(eval $(call natural_earth_sources,$(a),$(b)))))

Generate targets for nested combinations of scales and themes.

# complete wrapping
else
.DEFAULT:
        $(error Please install pgexplode: "npm install pgexplode")
endif

Provide a default target (truly default in that it will execute for any
requested target) explaining what needs to be done for things to work
correctly.

Questions? Suggestions?



KVM, libvirt, and Ubuntu 14.04
2014-05-14T00:00:00Z
I recently upgraded one of the servers in the office to Ubuntu 14.04, the most
recent LTS release. While doing this, I decided that I would figure out a clean
way of managing virtual machines. This is what I found:
Canonical now publishes Ubuntu cloud images.
These match the official images that are used on EC2. It turns out that they
can also be used in a local environment, complete with the customizations that
user-data provide.
libvirt appears to be the cleanest abstraction of
KVM/QEMU, Xen, LXC, and others, so I took a stab at using that to manage my
VMs.
Installation was easy:
sudo apt-get install libvirt-bin
Creating VMs (well, importing existing images and running them) was another
matter. Over the last few years we've seen a proliferation of tools that
attempt to provision VMs without needing to create XML manifests (ultimately
culminating in OpenStack). virt-install, which comes with libvirt ended up
being the tool for me.
To start, I downloaded a qcow2 Ubuntu Cloud image into libvirt's images/
directory:
curl -LO https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
sudo cp trusty-server-cloudimg-amd64-disk1.img /var/lib/libvirt/images/
virsh pool-refresh default # tell libvirt to re-scan for new files
I also added an ethernet bridge device so that the VMs could co-exist on the
network with everything else in the office:
cat <<EOF | sudo tee -a /etc/network/interfaces
auto br0
iface br0 inet dhcp
  bridge_ports eth0
EOF
When booting one of the Ubuntu Cloud images on its own, it briefly hangs while
waiting for network connections to EC2's internal network that exposes
metadata. Rather than replicating their environment, the Cloud images have
a relatively secret super
power:
the ability to pull cloud-init
configurations off of a secondary mounted image.
To create an image containing a ``cloud-initconfiguration, create 2 files:meta-dataanduser-data. These are their minimal forms (meta-data` appears
to be ignored / fails to set the hostname):
cat <<EOF > meta-data
instance-id: iid-local01;
local-hostname: ubuntu
EOF

cat <<EOF > user-data
#cloud-config
EOF
Next, write these into an ISO and copy it into libvirt's images/ directory:
genisoimage -output configuration.iso -volid cidata -joliet -rock user-data meta-data
sudo cp configuration.iso /var/lib/libvirt/images/
virsh pool-refresh default
With this in place, it's now possible to boot a VM and avoid the network wait.
This will create a new VM ("domain" in libvirt parlance) with 1G RAM, 1 vCPU,
and bridged networking:
virsh vol-clone --pool default trusty-server-cloudimg-amd64-disk1.img test.img

virt-install -r 1024 \
  -n test \
  --vcpus=1 \
  --autostart \
  --memballoon virtio \
  --network bridge=br0 \
  --boot hd \
  --disk vol=default/test.img,format=qcow2,bus=virtio \
  --disk vol=default/configuration.iso,bus=virtio

virsh list
To see the generated XML for this domain, run:
virsh dumpxml test
To stop it:
virsh destroy test
To remove it:
virsh undefine test
virsh vol-delete --pool default test.img
Now, since we have the power of cloud-init, we can modify the initial boot
configuration to do some initial provisioning. To do that, update user-data:
cat <<EOF > user-data
#cloud-config

# upgrade packages on startup
package_upgrade: true

# install git
packages:
  - git

# create a user
runcmd:
  - [ useradd, -c, Seth Fitzsimmons, -u, 1001, -G, sudo, -U, -M, -p, $5$FVJ1C48Rlhy/$GOidCu4a0qTmngqhFMGT7z/N.8nYTuXaaGzEDPhfIL., -s, /bin/bash, seth ]
EOF
cloud-init supports user creation since 0.7.0 (trusty comes with 0.7.5), but
it does not appear to work locally and I'd like to be able to re-use these
configurations with Ubuntu 12.04 images (which ship with cloud-init 0.6.3), so I'm doing the same thing by hand withruncmd`.
So far (which hasn't been that long), this has been working well. One of the
next steps is to achieve a similar streamlined workflow for LXC / Docker,
similar to what Mike wrote up about LXC and
Virtualbox.


Binary Streaming with Hadoop (and Node.js)
2013-12-27T00:00:00Z
Manipulating binary data from Hadoop streaming jobs is a black art.
There are Python (dumbo,
pydoop and
hadoopy) and
R (rmr) tools to
facilitate streaming jobs, but all of them have abstracted the handling of byte
streams (using typed
bytes)
successfully enough that it's difficult to determine how they actually work.
Which, as it turns out, is important to know if you're not using them.
While researching this topic, I kept returning to a Stack Overflow
post
that contains more information on this topic in one place than anywhere I've
seen, but even then it doesn't fully explain what's going on. I found
additional useful information in
HADOOP-1722 and in Klaas
Bosteel's presentation on the
topic, but
eventually had to start over and work through it bit by byte.
Without further ado:
Create Some Typed Bytes
This is simple program that will output 2 pairs of key/value pairs as typed
bytes:
var prepare = function(typeCode, value) {
  value = new Buffer(value);
  var len = new Buffer(4);
  len.writeInt32BE(value.length, 0);

  return Buffer.concat([
    new Buffer([typeCode]),
    len,
    value
  ]);
};

// record 1
process.stdout.write(prepare(7, "key"));   // string
process.stdout.write(prepare(0, "value")); // bytes

// record 2
process.stdout.write(prepare(7, "key2"));  // string
process.stdout.write(prepare(0, "value")); // bytes
To use it to generate a file containing typed bytes:
node write.js > string_bytes.tb
The result looks like this:
00000000  07 00 00 00 03 6b 65 79  00 00 00 00 05 76 61 6c  |.....key.....val|
00000010  75 65 07 00 00 00 04 6b  65 79 32 00 00 00 00 05  |ue.....key2.....|
00000020  76 61 6c 75 65                                    |value|
Prepare a SequenceFile
SequenceFiles are one of Hadoop's solutions to the small file
problem. The
format is supposedly a bit Java-centric, but with streaming, we never need to
interact with them directly. For our purposes, you can think of them as
splittable wrappers for records represented as typed bytes.
To convert the typed bytes into
a SequenceFile
stored in HDFS, use loadtb:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.4.0.jar \
  loadtb string_bytes.seq < string_bytes.tb
The result looks like this:
00000000  53 45 51 06 2f 6f 72 67  2e 61 70 61 63 68 65 2e  |SEQ./org.apache.|
00000010  68 61 64 6f 6f 70 2e 74  79 70 65 64 62 79 74 65  |hadoop.typedbyte|
00000020  73 2e 54 79 70 65 64 42  79 74 65 73 57 72 69 74  |s.TypedBytesWrit|
00000030  61 62 6c 65 2f 6f 72 67  2e 61 70 61 63 68 65 2e  |able/org.apache.|
00000040  68 61 64 6f 6f 70 2e 74  79 70 65 64 62 79 74 65  |hadoop.typedbyte|
00000050  73 2e 54 79 70 65 64 42  79 74 65 73 57 72 69 74  |s.TypedBytesWrit|
00000060  61 62 6c 65 01 00 2a 6f  72 67 2e 61 70 61 63 68  |able..*org.apach|
00000070  65 2e 68 61 64 6f 6f 70  2e 69 6f 2e 63 6f 6d 70  |e.hadoop.io.comp|
00000080  72 65 73 73 2e 44 65 66  61 75 6c 74 43 6f 64 65  |ress.DefaultCode|
00000090  63 00 00 00 00 67 18 80  e0 41 fe df ee 0f 68 7a  |c....g...A....hz|
000000a0  d9 8f dd a5 d8 00 00 00  20 00 00 00 0c 00 00 00  |........ .......|
000000b0  08 07 00 00 00 03 6b 65  79 78 9c 63 60 60 e0 62  |......keyx.c``.b|
000000c0  00 02 d6 b2 c4 9c d2 54  00 06 ff 02 2d 00 00 00  |.......T....-...|
000000d0  22 00 00 00 0d 00 00 00  09 07 00 00 00 04 6b 65  |".............ke|
000000e0  79 32 78 9c 63 60 60 e0  66 00 02 b6 b2 c4 9c d2  |y2x.c``.f.......|
000000f0  54 23 00 09 71 02 61                              |T#..q.a|
Read and Write SequenceFiles
To run a job that only sees typed bytes but uses SequenceFiles to contain both
input and output, start it like so:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.4.0.jar \
  -D mapred.map.tasks=1 \
  -io typedbytes \
  -inputformat org.apache.hadoop.mapred.SequenceFileInputFormat \
  -outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat \
  -mapper "/usr/bin/tee /tmp/seq.debug" \
  -reducer org.apache.hadoop.mapred.lib.IdentityReducer \
  -input string_bytes.seq \
  -output hello-output.seq
-D mapred.map.tasks=1 tells Hadoop to only start a single mapper, which is
necessary to prevent the intermediate debug file (/tmp/seq.debug) from being
mangled while being written to by multiple processes.
-mapper "/usr/bin/tee -a /tmp/seq.debug allows us to see what was passed to
the mapper after it was run.
-io typedbytes tells Hadoop to use
org.apache.hadoop.typedbytes.TypedBytesWritable as key and value classes
(instead of org.apache.hadoop.io.Text).
-inputformat org.apache.hadoop.mapred.SequenceFileInputFormat and
-outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat tell Hadoop
that our input and output files are SequenceFiles.
The end result of this incantation is that the mapper is called with a stream
of typed bytes containing no delimiters. In this context, a record consists
of a pair of typed bytes.
This is the intermediate representation of the SequenceFile we prepared
above, as seen by our mapper. Look familiar? It matches the bytes we created
above:
00000000  07 00 00 00 03 6b 65 79  00 00 00 00 05 76 61 6c  |.....key.....val|
00000010  75 65 07 00 00 00 04 6b  65 79 32 00 00 00 00 06  |ue.....key2.....|
00000020  76 61 6c 75 65 32                                 |value2|
Since we didn't specify that compression should be used, the resulting
SequenceFile is slightly different than the one that was created with
loadtb:
00000000  53 45 51 06 2f 6f 72 67  2e 61 70 61 63 68 65 2e  |SEQ./org.apache.|
00000010  68 61 64 6f 6f 70 2e 74  79 70 65 64 62 79 74 65  |hadoop.typedbyte|
00000020  73 2e 54 79 70 65 64 42  79 74 65 73 57 72 69 74  |s.TypedBytesWrit|
00000030  61 62 6c 65 2f 6f 72 67  2e 61 70 61 63 68 65 2e  |able/org.apache.|
00000040  68 61 64 6f 6f 70 2e 74  79 70 65 64 62 79 74 65  |hadoop.typedbyte|
00000050  73 2e 54 79 70 65 64 42  79 74 65 73 57 72 69 74  |s.TypedBytesWrit|
00000060  61 62 6c 65 00 00 00 00  00 00 7d 88 7f 43 96 14  |able......}..C..|
00000070  6b 83 15 d9 dc aa d6 ec  99 1c 00 00 00 1a 00 00  |k...............|
00000080  00 0c 00 00 00 08 07 00  00 00 03 6b 65 79 00 00  |...........key..|
00000090  00 0a 00 00 00 00 05 76  61 6c 75 65 00 00 00 1c  |.......value....|
000000a0  00 00 00 0d 00 00 00 09  07 00 00 00 04 6b 65 79  |.............key|
000000b0  32 00 00 00 0b 00 00 00  00 06 76 61 6c 75 65 32  |2.........value2|
To dump the typed bytes contained in that SequenceFile, use dumptb:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.4.0.jar \
  dumptb hello-output.seq/part-00000 < hello-output.tb
Read Text and Write SequenceFiles
In some circumstances you may find yourself with text input (the output from
a Hive query, for example) and wish to produce binary output (images, say).
This works, but it's not obvious. The key is to not use -io typedbytes and
instead to be explicit about which stages of your workflow consume and produce
typed bytes using -D stream.<stage>.<input|output>=typedbytes. More on this
shortly.
First, create a text file containing tab-separated values and copy it to HDFS for use
as the input to your job:
echo -e "key\tvalue\nkey2\tvalue2" > hello.tsv
hdfs dfs -put hello.tsv
Write a mapper that outputs typed bytes. This is convert.js:
var split = require("split"),
    through = require("through");

var prepare = function(typeCode, value) {
  value = new Buffer(value);
  var len = new Buffer(4);
  len.writeInt32BE(value.length, 0);

  return Buffer.concat([
    new Buffer([typeCode]),
    len,
    value
  ]);
};

process.stdin
  .pipe(split())
  .pipe(through(function(line) {
    if (line.length === 0) {
      return;
    }

    var parts = line.split("\t"),
        key = parts.shift(),
        value = parts.shift();

    // output key as a string
    this.queue(prepare(7, key));

    // output value as bytes
    this.queue(prepare(0, value));
  }))
  .pipe(process.stdout);
Create a JAR containing the artifacts needed to execute the job and copy them
to HDFS (the cluster I'm using doesn't have Node installed, so I've added
bin/node to my working directory so it will be shipped alongside the code):
jar cf typedbytes.jar .
hdfs dfs -put typedbytes.jar
To run the job, we'll need a different set of arguments:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.4.0.jar \
  -archives hdfs:///user/cloudera/typedbytes.jar#typedbytes \
  -D mapred.map.tasks=1 \
  -D stream.map.output=typedbytes \
  -D stream.reduce.input=typedbytes \
  -D stream.reduce.output=typedbytes \
  -outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat \
  -mapper "typedbytes/bin/node typedbytes/convert.js" \
  -reducer org.apache.hadoop.mapred.lib.IdentityReducer \
  -input hello.tsv \
  -output hello.tsv.seq
In this case, typed bytes only encompass ¾ of the values we're passing around,
so we need to explicitly tell Hadoop that using -D stream.<stage>.<input|output>=typedbytes. If they were 100%, as above, we
could have used -io typedbytes.
hello.tsv.seq looks similar to the uncompressed output from earlier, although
if you look closely, there are some minor differences (which seem to be
differences in the SequenceFile's metadata):
00000000  53 45 51 06 2f 6f 72 67  2e 61 70 61 63 68 65 2e  |SEQ./org.apache.|
00000010  68 61 64 6f 6f 70 2e 74  79 70 65 64 62 79 74 65  |hadoop.typedbyte|
00000020  73 2e 54 79 70 65 64 42  79 74 65 73 57 72 69 74  |s.TypedBytesWrit|
00000030  61 62 6c 65 2f 6f 72 67  2e 61 70 61 63 68 65 2e  |able/org.apache.|
00000040  68 61 64 6f 6f 70 2e 74  79 70 65 64 62 79 74 65  |hadoop.typedbyte|
00000050  73 2e 54 79 70 65 64 42  79 74 65 73 57 72 69 74  |s.TypedBytesWrit|
00000060  61 62 6c 65 00 00 00 00  00 00 c7 91 8e c1 6a bc  |able..........j.|
00000070  e3 bb 3a 03 86 40 3a 01  72 8b 00 00 00 1a 00 00  |..:..@:.r.......|
00000080  00 0c 00 00 00 08 07 00  00 00 03 6b 65 79 00 00  |...........key..|
00000090  00 0a 00 00 00 00 05 76  61 6c 75 65 00 00 00 1c  |.......value....|
000000a0  00 00 00 0d 00 00 00 09  07 00 00 00 04 6b 65 79  |.............key|
000000b0  32 00 00 00 0b 00 00 00  00 06 76 61 6c 75 65 32  |2.........value2|
Output Binary Files
Before writing this, I had intended to produce a single logical file at the end
of my workflow (without a key) that could be fetched and used immediately.
However, when using MapReduce, the fundamental building blocks are
record-based, so all values must be accompanied by a key (unless one writes to
HDFS directly, presumably with hdfs dfs -put - dest to stream from stdin).
Working with smaller units of data (i.e. broken into records) facilitates the
use of subsequent jobs to perform further processing.
With that under consideration, the most sensible approach (and one I've seen
references to) appears to be the use of a SequenceFile as a virtual
filesystem using keys as filenames. dumptb will unpack an HDFS-hosted file
into one containing pairs of typed bytes which can be processed locally.
Once you've dumped some typed bytes to the local filesystem, you can unpack
records into individual files, named by key using unpack.js:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.4.0.jar \
  dumptb hello-output.seq/part-00000 < hello-output.tb
node unpack.js hello-output.tb
unpack.js looks like this:
"use strict";

var assert = require("assert"),
    fs = require("fs"),
    path = require("path");

// get the file containing typed bytes from the command line
var tb = path.resolve(process.argv.slice(2).pop());

var stats = fs.statSync(tb),
    fd = fs.openSync(tb, "r"),
    offset = 0,
    keyType = new Buffer(1),
    keyLength = new Buffer(4),
    key,
    valueType = new Buffer(1),
    valueLength = new Buffer(4);

while (offset < stats.size) {
  fs.readSync(fd, keyType, 0, 1, offset);
  offset++;

  assert.equal(keyType[0], 7);

  fs.readSync(fd, keyLength, 0, 4, offset);
  offset += 4;

  keyLength = keyLength.readInt32BE(0);
  key = new Buffer(keyLength);

  fs.readSync(fd, key, 0, keyLength, offset);
  offset += keyLength;

  fs.readSync(fd, valueType, 0, 1, offset);
  offset++;

  assert.equal(valueType[0], 0);

  fs.readSync(fd, valueLength, 0, 4, offset);
  offset += 4;

  valueLength = valueLength.readInt32BE(0);

  // pipe the subset of the file out
  fs.createReadStream(tb, {
    start: offset,
    end: offset + valueLength - 1
  }).pipe(fs.createWriteStream("./" + key));

  offset += valueLength;

  console.log("%s %d", key.toString(), valueLength);
}


DIY Fusion Drives: A Data Point
2013-09-24T00:00:00Z
In light of Apple having announced new iMacs today, I thought I'd throw out
a data point to the internet.
tl;dr: I've used 2 DIY Fusion Drives on 2 different Macs for the past year,
they're awesome, but you should use a Samsung SSD (I have an 830 and an 840
Pro).
Since last year's Fusion Drive
announcement
(and subsequent "reverse
engineering"),
I've been running 256MB SSD + 1TB HD Fusion drive setups on 2 Macs: a 2009 iMac
(with the SSD replacing the optical drive) and a 2012 Mac Mini (using the extra
bay after purchasing a second SATA cable). It's been awesome, especially after
sorting the kinks out. (No, no benchmarks, but it really does feel / act as
though it's an SSD-backed system, even when data needs to get shuffled around,
as in the case of processing a couple hundred gigs of GeoTIFFs.)
The kinks: I'd already upgraded the iMac with a 256GB Crucial M4 about a year
before, so I figured I would just back everything up and use that in
combination with the Apple-provided drive. Bad plan. I went through 2 M4s
(though probably not for the hardware problems they seemed to be manifesting)
before throwing in the towel and buying a Samsung SSD 840 Pro, which has been
completely problem-free since installing it. When I bought the Mini, I also
bought a Samsung 830 at the same time, which has been trouble-free since the
start (Apple uses, or used Samsung 830s at one point, so this seemed like the
best choice).
With the M4s, large files (100GB+ VMware VMs) would spontaneously corrupt
(presumably the filesystem passed the ~256GB threshold) and no amount of
monkeying with Disk Utility could successfully repair it, ultimately
necessitating Time Machine restores on multiple occasions. Things would be
peachy for a week or 2 and then, bam, time to reinstall. No good. One Crucial
rep I talked to when initiating an RMA suggested that the M4 wasn't getting
a chance to run its garbage collection routines, probably due to the I/O
patterns resulting from shifting data between disks.
Would I do it again? Definitely, though the move to PCIe SSDs complicates the
matter a bit. In the case of the Mini, it was borderline unusable (by my
standards, anyway) due to the I/O performance of the built-in 5400rpm drive,
even with an i7 and lots of RAM. With the Fusion drive? Incredibly pleasant
to use as my primary work machine for the last 9½ months.


Building Blocks of Real-Time Distribution
2010-10-27T00:00:00Z
Back in July, I gave a talk about the "Building Blocks of Real-Time
Distribution"
at the GeoWeb 2010 conference in Vancouver, BC.
It was pretty heavy on the buzzwords and the name dropping, so I created an
accompanying page with links and
definitions/background in order to make the slides a bit more coherent. The PDF
itself also has a rough transcript of what I said in the talk, so that should
help too.
In the talk, I attempted to cover the considerations and technology choices
when implementing real-time services in various environments with different
types of data. The talk focused on spatial data, but I think it remains
relevant, with the exception of one or two slides.
I continue to pitch XMPP because I still think it's a good, if
over-complicated, solution. Presence tracking and the ability to run ephemeral
clients are seriously awesome features. However, the real momentum these days
seems to be behind streaming HTTP APIs and PubSubHubbub. Julien and
SuperFeedr have done a great job adapting to the
changing landscape and shipping awesome features, so definitely check them out.
That is all.


The OS X Spatial Stack
2009-12-05T00:00:00Z
I've been doing a bunch of spatial analysis and exploration recently. Where I
previously would have used a Debian or Ubuntu server (or VM) to experiment, I'm
now using Snow Leopard exclusively.
However, it's not all roses. As with most things, there are many paths to
installing and configuring software, not all of them complementary. What
follows is my experience installing and configuring various tools and their
dependencies. I've attempted to install as few duplicate dependencies as
possible (preferring Apple-provided versions) and to simplify the overall
experience in order that individual elements may be upgraded independently.

Xcode
First, download and install Xcode
3.2.1.
The defaults are fine.
Homebrew
Next, install homebrew. I've found that
homebrew is significantly lighter and easier than
MacPorts for installing additional open source
software, as it re-uses system versions of libraries wherever possible.
Part of the homebrew philosophy is:

OS X is designed to minimise sudo use, you only need it for real root-level
stuff. You know your /System and /usr are as clean and pure as the day you
bought your Mac because you didn't sudo. Sleep better at night! ... Lets face
it; Homebrew is not installing anything system-critical. Apple already did
that.

Homebrew is typically installed in /usr/local, so:
sudo mkdir -p /usr/local
sudo chown -R `whoami` /usr/local
(If you already had MySQL installed in /usr/local, fix it: sudo chown -R mysql:mysql /usr/local/mysql.)
Now, install homebrew:
cd /usr/local
curl -L https://github.com/mxcl/homebrew/tarball/master | tar xz --strip 1 -C .
Homebrew-provided Libraries
While we're working with homebrew, let's install a dependencies that we'll
need later.
pkg-config is necessary to cleanly
compile PIL:
brew install pkg-config
You'll want to make sure that /usr/local/bin is in your $PATH so that
homebrew-installed binaries will get run.
KyngChaos Frameworks and Binaries
The KyngChaos Wiki is the holy grail for OS X
spatial downloads, as everything is cleanly packaged, up-to-date, and very well
organized.
From the Unix Compatibility
Frameworks page, you'll want to
download and install:

FreeType
GSL - The GNU Scientific Library
GDAL Complete (this includes UnixImageIO (notably jpeg and libpng), PROJ,
GEOS, SQLite3 (including Spatialite), and GDAL)

These will install to /Library/Frameworks and will set up pointers to their
corresponding Python packages in /Library/Python/2.6/site-packages/.
From the Qgis page, you'll want the
most up-to-date Qgis installer (non-standalone). If you intend to open
Spatialite data sources, you may need to install a more up-to-date version of
the SQLite3 Framework than is included in the GDAL Complete package (this is
generally applicable; if you want/need to live on the bleeding edge, install
frameworks individually).
From the PostgreSQL page,
download and install the newest versions of the following:

PostgreSQL (server + client)
PostGIS

Mapnik
Thanks to Dane Springmeyer and others, Mapnik is now
available as a framework.
This means fewer homebrew dependencies, a simple upgrade path, and
compatibility with 32-bit QGIS Python plugins. To install it, grab a Snow
Leopard build from the Mapnik OSX Downloads page
and run both the Mapnik Framework and Mapnik Python 2.6 System installers.
If you'd previously installed Mapnik from source, you should remove it from
your system before running the installer:
$ rm -rf /Library/Python/2.6/site-packages/mapnik
$ rm -rf /usr/local/lib/mapnik
$ rm /usr/local/lib/libmapnik.dylib
You can also remove boost and icu with homebrew if nothing else is using
them:
brew uninstall boost
brew uninstall icu
Quantumnik
Quantumnik is a QGIS
plugin that allows Mapnik to be used for rendering. In addition, it will map
many QGIS styles to Mapnik styles and provides an interface for editing them. In
short, it's an excellent way to prototype your maps before starting up a batch
rendering job.
To install Quantumnik, start up QGIS, enable the Plugin Installer in the
Plugin Manager, choose Fetch Python Plugins, add a new repository
(http://qgis.dbsgeo.com/), find it in the list, and check the box to enable it.
Python
We'll start with the easy stuff and easy_install the following:

Flickr.API - Flickr has good geo data (I'm biased)
IPython - Interactive Python
nik2img - Mapnik
command-line utility.
Nikweb - Mapnik-based
"static maps API" (though it requires GeoJSON input to determine appropriate
bounding boxes and additional features)
nose - Unit testing framework
numscons - scons for numpy/scipy--required for scipy installation
readline - causes IPython to use readline instead of libedit, solving all
sorts of problems
TileCache - WMS-C compliant tile server

easy_install Flickr.API
easy_install ipython
easy_install nik2img
easy_install Nikweb
easy_install nose
easy_install numscons
easy_install readline
easy_install TileCache
PIL
PIL is the Python Imaging Library.
Download the current source kit and extract it to /usr/local/src:
cd /usr/local/src
tar zxf ~/Downloads/Imaging-1.1.6.tar.gz
cd Imaging-1.1.6/
Since you previously installed pkg-config, building it is straightforward:
python setup.py install
matplotlib
matplotlib is a plotting library for
Python. It can be used both programmatically and interactively.
Download the source distribution from SourceForge and extract it to
/usr/local/src:
cd /usr/local/src
tar zxf ~/Downloads/matplotlib-0.99.1.2.tar.gz
cd matplotlib-0.99.1.1/ # why yes, this is a mistake in the pkg
Again, this is straightforward:
python setup.py install
matplotlib includes the Basemap
Toolkit, which makes it
easier to work with cartographic data.
Download the source distribution from
SourceForge
and extract it to /usr/local/src. While you're at it, download natgrid from
the same place.
cd /usr/local/src
tar zxf ~/Downloads/basemap-0.99.4.tar.gz
tar zxf ~/Downloads/natgrid-0.1.tar.gz
Basemap needs to know where GEOS was installed:
cd basemap-0.99.4/
GEOS_DIR=/Library/Frameworks/GEOS.framework/unix python setup.py install
natgrid is straightforward:
cd natgrid-0.1/
python setup.py install
numpy / scipy
numpy and scipy are the Pythonic chainsaws for
scientific computing. numpy will mostly install out-of-the-box, but we'll
install it from source.
Before we can build them, we need to install a Fortran compiler from AT&T
Research:
http://r.research.att.com/gfortran-4.2.3.dmg
Download source distributions of numpy and scipy from
SourceForge(numpy,
scipy) and extract them to
/usr/local/src:
cd /usr/local/src
tar zxf ~/Downloads/numpy-1.3.0.tar.gz
tar zxf ~/Downloads/scipy-0.7.1.tar.gz
Build and install numpy:
$ cd numpy-1.3.0/
$ LDFLAGS="-lgfortran -arch x86_64" FFLAGS="-arch x86_64" \
    python setup.py install
Build and install scipy:
$ cd scipy-0.7.1/
$ LDFLAGS="-lgfortran -arch x86_64" FFLAGS="-arch x86_64" \
    python setup.py install
Note: the GDAL installer from KyngChaos bundles a version of numpy in
/Library/Frameworks/GDAL.framework/Versions/1.6/Python/site-packages/numpy/,
so if you run into weirdness, it's probably safe to delete that one.
Note: scipy recommends installing fftw for fast Fourier transformations; I was
unable to get scipy to build when it was installed (using homebrew), so I'm
omitting that step.
Cascadenik
Cascadenik is a
pre-processor that converts CSS-like syntax into Mapnik XML configuration
files. It supports handy shorthand such as ".thing[zoom > 12] { ... }" for
targeting standard slippy zoom levels and generally makes it easier to maintain
map styles. nik2img supports Cascadenik styling natively, making it easier to
create and tweak styles (otherwise you'll need to run cascadenik-compile.py
manually).
First, check out Cascadenik from Google Code:
cd /usr/local/src
svn co http://mapnik-utils.googlecode.com/svn/trunk/serverside/cascadenik
Next, build and install it:
cd cascadenik
python setup.py install
It will automatically download and install cssutils for you before making
cascadenik-compile.py and cascadenik-style.py available in your $PATH.
OpenStreetMap
When working with data from OpenStreetMap,
osm2pgsql and
Osmosis are critical.
To install osm2pgsql, check it out from OSM's subversion repository and use
make to build and install it, linking against the KyngChaos framework:
$ cd /usr/local/src
$ svn co https://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
$ cd osm2pgsql
$ PATH=$PATH:/Library/Frameworks/GEOS.framework/unix/bin/ \
    CFLAGS="-I/Library/Frameworks/PROJ.framework/unix/include" \
    LDFLAGS="-L/Library/Frameworks/PROJ.framework/unix/lib/" \
    make
You'll have to install it by hand to /usr/local:
install -m 0755 osm2pgsql /usr/local/bin
install default.style /usr/local/share/osm2pgsql
default.style is the standard map of OSM tags to database columns; you'll
need it when you run an import, even if the defaults are fine.
Osmosis is a filter-driven tool
for processing large volumes of OSM XML.
To begin, download the latest version of
Osmosis.
Extract it to /usr/local:
cd /usr/local
tar zxf ~/Downloads/osmosis-latest-bin.tar.gz
Now, create a symlink to the binary:
ln -s /usr/local/osmosis-0.36/bin/osmosis /usr/local/bin/osmosis
PostgreSQL Additions
At present, the KyngChaos distribution of PostgreSQL does not include the
intarray module
(you can check /usr/local/pgsql/share/contrib to see if it's been added).
osm2pgsql uses this module to create updatable Postgres OSM mirrors (meaning
that you can stay up-to-date by applying planet
diffs).
This means that we need to build it ourselves. First, download the source
tarball corresponding to the version you already have
installed. Next, extract it to
/usr/local/src:
cd /usr/local/src
tar zxf ~/Downloads/postgresql-8.4.5.tar.bz2
Configure Postgres:
cd postgres-8.4.5
./configure
Change to the intarray contrib directory and make it:
cd contrib/intarray
export PATH=/usr/local/pgsql/bin:$PATH
export USE_PGXS=1
make
sudo make install
(This same process can be repeated for other extensions that you desire.)
With intarray now installed, you can enable the database of your choosing
(osm in this case):
sudo -u postgres psql -d osm -f /usr/local/pgsql/share/contrib/_int.sql
GUI Utilities
For graphical management of spatial data beyond what Qgis can handle, I usually
use Base (for SQLite databases) and
pgAdmin (for PostgreSQL databases).
Congratulations
You now have a working spatial stack on OS X! There are certainly things missed
here, but I'll try to keep this up-to-date as I discover new tools and simpler
methods (and in response to your comments, dear reader).


CSSpring Cleaning
2009-09-08T00:00:00Z
I was talking to some front-end devs a couple weeks ago about maintaining CSS
on an evolving website. In particular, how to figure out which rules are no
longer being applied (remnants from experiments / old designs / whatever) and
remove them.
I remembered seeing something a few months ago that claimed to do this. After
digging around on GitHub for a bit, I re-found
deadweight. Unfortunately, after
looking at it briefly, I realized that it made some pretty strong assumptions,
mainly about being run in the context of a Rails app.
Time for some hacking.
My fork can be installed with
RubyGems (0.0.5 is required for deflate and gzip content-encodings):
sudo gem install mojodna-deadweight -s https://gems.github.com/
The first step was to make it accessible as a command-line utility (ideally in
the spirit of the UNIX
Philosophy).
deadweight -s styles.css -s ie.css index.html about.html
(This will check styles.css and ie.css against index.html and
about.html, outputting unused rules.)
It also supports input from pipes, meaning that you can chain it together and
filter orphaned rules without writing them to a file (you'll have to configure
logging in order to limit output).
cat styles.css | deadweight index.html
Deadweight now contains an experimental -L argument that causes it to use
Lyndon if the lyndon executable is in
your $PATH (only possible on OS X with MacRuby installed). It was a bit
flaky in my testing, but I may be using an older version of MacRuby. Great
idea though. Check this post for more info on what it does / how it
works.
The next step was to expose it as an HTTP proxy. It uses 8002 by default for
no particular reason.
(oauth-proxy uses 8001.)
deadweight -l deadweight.log -s styles.css -w https://github.com/ -P
(This dumps log output to deadweight.log in $PWD and matches
https://github.com/* against styles.css.)
It's probably most useful as an HTTP proxy; you can start it with a list of
target stylesheets, configure your browser to load pages through it (sorry, it
gets really slow when requesting gzip- or deflate-encoded pages, which
is a lot of the time), and watch the output to see how many rules it's hit.
When you're done, hit ^C to stop it and (optionally) output the remaining
CSS rules into a file for you to examine (-o orphans.css will do this for
you when it shuts down).
If you're lucky, the file containing orphaned CSS will be empty. If not,
re-start the proxy with the orphaned CSS as the target and continue browsing
until you're ready to start looking for matches by hand (this is necessary, as
it won't catch classes applied by JavaScript).
Things to Improve
WEBrick's HTTP proxy implementation
leaves a lot to be desired (it's nowhere on the same level as
Twisted's), but it's the only game in town.
Ilya Grigorik has done some excellent work with
EventMachine and proxies, but it
doesn't appear that anyone has built a general-purpose HTTP proxy that easily
supports header modification. It's not a particularly hard problem and being
able to move deadweight's proxy implementation to such a base would provide
the ability to turn off compression and thus speed things up considerably.
One could also imagine a world in which the proxy server also contains a web
interface that displays unmatched CSS rules and updates on the fly.
EventMachine would do the trick here, too.


Exploring OAuth-Protected APIs
2009-08-21T00:00:00Z
From time to time I need to debug OAuth-protected APIs, checking response
headers and examining XML and JSON payloads. curl generally rocks for this
sort of thing, but when the APIs in question are protected with OAuth, things
break down. Likewise for benchmarking (ab, httperf, etc.) and
exploration--isn't it nice to browse APIs that return XML in Firefox?
This needn't to be the case.
Enter oauth-proxy
This is why I wrote
oauth-proxy. It does what it
says on the tin: it acts a proxy server that transparently adds OAuth headers
to requests.
There are 2 steps to using it. First, install it:
easy_install oauth-proxy
Then, start it:
$ oauth-proxy \
    --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    [--token <token>] \
    [--token-secret <token secret>] \
    [-p <proxy port>] \
    [--ssl]
If you're accessing a resource that only requires 2-legged OAuth, you can omit
--token and --token-secret. The proxy port defaults to 8001, and --ssl
is used if you're proxying connections to a provider that requires SSL (Fire
Eagle, for example).
Once it's been started, use curl to make requests through it:
curl -x localhost:8001 http://host.name/path
You can also benchmark your APIs through it using ApacheBench (ab, as it
includes support for HTTP proxies). Note that you are introducing additional
overhead by proxying the request, so your numbers may be a bit off.
ab -X localhost:8001 http://host.name/path
Firefox (and browsers in general) supports HTTP proxies, so you can add a
"Manual Proxy Configuration" and pass requests through oauth-proxy to
explore APIs in the comfort of your favorite browser. By default, all requests
that Firefox makes will go through the proxy and be signed, which may confuse
other websites you visit (and will inadvertently reveal your consumer key and
access token but not the corresponding secrets).
Access Tokens Sold Separately
If you're accessing a resource that requires 3-legged OAuth, you'll need a
token. You may already have one, but if you don't, you can use the OAuth
library for Ruby to obtain one.
First, install the gem (0.3.5 is current as of this writing):
sudo gem install oauth
Then, trigger the authorization process from the command-line:
$ oauth \
    --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    --access-token-url http://host.name/path/to/access_token \
    --authorize-url http://host.name/path/to/authorize \
    --request-token-url http://host.name/path/to/request_token \
    authorize
Follow the prompts, and voilà, an access token and secret that you can use
with oauth-proxy.
A Concrete Example
Twitter's popular, right? Let's use that.
First, register an application to get a
consumer key and secret. I registered as a "client" application, since the
command-line still doesn't have a callback url.
Once that's done, you'll get a set of credentials that can be used to initiate
the authorization process.
Let's authorize.
$ oauth \
  --consumer-key <consumer key> \
  --consumer-secret <consumer secret> \
  --access-token-url https://twitter.com/oauth/access_token \
  --authorize-url https://twitter.com/oauth/authorize \
  --request-token-url https://twitter.com/oauth/request_token \
  authorize
After following the prompts, you'll get something back that looks like this:
oauth_token_secret: [redacted]
oauth_token: [redacted]
user_id: [redacted]
screen_name: [redacted]
You can then use those values to start the OAuth proxy:
$ oauth-proxy \
    --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    --token <access token> \
    --token-secret <token secret>
Now we're set. Let's go exploring:
$ curl -sx http://localhost:8001 \
    https://twitter.com/statuses/friends_timeline.json | \
    jsonpretty | pygmentize -l js
You'll get exactly what you're expecting and you'll be using OAuth (this
is a partially contrived example since Twitter still supports HTTP Base Auth).
(jsonpretty rocks. pygmentize
(easy install pygments) makes it easier to make sense of the chaos.)
That's all. I use this stuff all the time and can't imagine debugging APIs
without it.


Subscribing to Wordpress.com with Switchboard
2009-07-21T00:00:00Z
Last week, Andy Skelton announced Real-time Wordpress.com
subscriptions
via XMPP. It does IM, but more interestingly for me, it supports PubSub. Had I
realized this when he posted that, I would have included it as an example in
switchboard : XMPP :: curl :
HTTP alongside
Fire Eagle and Superfeedr. What's done is done, however, so here's a quick
rundown of how to interact with Wordpress.com using Switchboard.
First, let's get a basic idea of the identities and features that
Wordpress.com advertises:
$ switchboard disco --target pubsub.im.wordpress.com info
=> Switchboard started.
Discovery Info for pubsub.im.wordpress.com
Identities:
  pubsub/service: Publish-Subscribe

Features:
  http://jabber.org/protocol/disco#info
  http://jabber.org/protocol/disco#items
  http://jabber.org/protocol/pubsub
  vcard-temp
  http://jabber.org/protocol/pubsub#access-authorize
  http://jabber.org/protocol/pubsub#access-open
  http://jabber.org/protocol/pubsub#access-presence
  http://jabber.org/protocol/pubsub#access-whitelist
  http://jabber.org/protocol/pubsub#auto-create
  http://jabber.org/protocol/pubsub#collections
  http://jabber.org/protocol/pubsub#config-node
  http://jabber.org/protocol/pubsub#create-and-configure
  http://jabber.org/protocol/pubsub#create-nodes
  http://jabber.org/protocol/pubsub#delete-items
  http://jabber.org/protocol/pubsub#delete-nodes
  http://jabber.org/protocol/pubsub#instant-nodes
  http://jabber.org/protocol/pubsub#item-ids
  http://jabber.org/protocol/pubsub#last-published
  http://jabber.org/protocol/pubsub#manage-subscriptions
  http://jabber.org/protocol/pubsub#member-affiliation
  http://jabber.org/protocol/pubsub#modify-affiliations
  http://jabber.org/protocol/pubsub#multi-subscribe
  http://jabber.org/protocol/pubsub#outcast-affiliation
  http://jabber.org/protocol/pubsub#persistent-items
  http://jabber.org/protocol/pubsub#presence-notifications
  http://jabber.org/protocol/pubsub#presence-subscribe
  http://jabber.org/protocol/pubsub#publish
  http://jabber.org/protocol/pubsub#publisher-affiliation
  http://jabber.org/protocol/pubsub#purge-nodes
  http://jabber.org/protocol/pubsub#retract-items
  http://jabber.org/protocol/pubsub#retrieve-affiliations
  http://jabber.org/protocol/pubsub#retrieve-default
  http://jabber.org/protocol/pubsub#retrieve-items
  http://jabber.org/protocol/pubsub#retrieve-subscriptions
  http://jabber.org/protocol/pubsub#subscribe
  http://jabber.org/protocol/pubsub#subscription-notifications
  http://jabber.org/protocol/pubsub#subscription-options
Shutdown initiated.
Lo, it supports PubSub! (But it doesn't support node discovery, so we can't
get a list of available nodes.) Fortunately, Andy's post includes a couple.
Let's subscribe:
$ switchboard pubsub --server pubsub.im.wordpress.com \
    --node /blogs/andy.wordpress.com subscribe
=> Switchboard started.
Subscribe successful.
Shutdown initiated.

$ switchboard pubsub --server pubsub.im.wordpress.com \
    --node /blogs/andy.wordpress.com/comments subscribe
=> Switchboard started.
Subscribe successful.
Shutdown initiated.

$ switchboard pubsub --server pubsub.im.wordpress.com --node \
     /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription/ \
    subscribe
=> Switchboard started.
Subscribe successful.
Shutdown initiated.
Switchboard claimed that that worked, but let's double-check and list our
subscriptions:
$ switchboard pubsub --server pubsub.im.wordpress.com subscriptions
=> Switchboard started.
Subscriptions:
4E039BC19E028: me@jabber.org => /blogs/andy.wordpress.com (subscribed)
4E039D2A2B691: me@jabber.org => /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription (subscribed)
4E039BC7E8A69: me@jabber.org => /blogs/andy.wordpress.com/comments (subscribed)
Shutdown initiated.
Actually, there was no need to subscribe to comments for "Real-time
Wordpress.com subscription" because we're also subscribed to the comments
feed. Let's unsubscribe and check our subscriptions again:
$ switchboard pubsub --server pubsub.im.wordpress.com --node \
     /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription \
    unsubscribe
=> Switchboard started.
Unsubscribe was successful.
Shutdown initiated.

$ switchboard pubsub --server pubsub.im.wordpress.com subscriptions
=> Switchboard started.
Subscriptions:
4E039BC19E028: me@jabber.org => /blogs/andy.wordpress.com (subscribed)
4E039BC7E8A69: me@jabber.org => /blogs/andy.wordpress.com/comments (subscribed)
Shutdown initiated.
Ok, all good. Now let's hope Andy posts something or gets a comment on
something so we can listen for it:
$ switchboard pubsub --server pubsub.im.wordpress.com listen
(sample post and comment <event /> stanzas to come)
For bonus points, let's implement a WordpressJack:
class WordpressJack
  def self.connect(switchboard, settings)
    switchboard.plug!(AutoAcceptJack, NotifyJack, PubSubJack)
    switchboard.hook(:new_post, :new_comment)

    switchboard.on_pubsub_event do |event|
      event.payload.each do |payload|
        payload.elements.each do |item|
          # grab the feed out of the payload
          feed = item.first_element("feed")

          # TODO determine whether it's a comment or a post
          # for now, we'll assume it's a post
          # invoke the :new_post hook with the feed subtree
          # (later: parse it with FeedTools or something)
          on(:new_post, feed)
        end
      end
    end
  end
end
And a quick-and-dirty consumer:
#!/usr/bin/env ruby -rubygems

require 'switchboard'
require 'wordpress_jack'

switchboard = Switchboard::Client.new
switchboard.plug!(WordpressJack)

switchboard.on_new_post do |post|
  puts "A new post was received:"
  puts post
end

switchboard.on_new_comment do |comment|
  puts "A new comment was received:"
  puts comment
end

switchboard.run!
There are a number of improvements that can be made here (this is the "making
blind assumptions about responses" version). Go wild.


Switchboard as a Framework
2009-07-19T00:00:00Z
While Switchboard is a useful tool for debugging and probing XMPP
services, it's
also a convenient and full-featured framework for building clients and
components.
A Simple Client
This is the simplest client (bot) I could think of. It listens for input and
replies with whatever was sent in the first place.
#!/usr/bin/env ruby -rubygems

require 'switchboard'

switchboard = Switchboard::Client.new
switchboard.plug!(AutoAcceptJack, EchoJack, NotifyJack)
switchboard.run!
Let's break it down.
#!/usr/bin/env ruby -rubygems

require 'switchboard'
Um, I hope this is pretty straightforward.
switchboard = Switchboard::Client.new
This instantiates Switchboard::Client with some default options, mainly
spin = true (this is the 2nd argument). spin means that #run! will cause
the process to run in a loop and not return immediately. ^C will interrupt
the process and shut it down cleanly. (^C a second time if it hangs.)
switchboard.plug!(AutoAcceptJack, EchoJack, NotifyJack)
This is the meat of it, even though it doesn't look like it at first glance.
Jacks are the basic units of shared functionality. More later, but the basic
rundown is that AutoAcceptJack auto-accepts (and reciprocates) roster
additions ("friend requests"), EchoJack does the heavy lifting of echoing
input back, and NotifyJack sends presence (i.e. "I'm online") notifications
to everyone (and everything) in your roster.
EchoJack (lib/switchboard/jacks/echo.rb) looks like this:
class EchoJack
  def self.connect(switchboard, settings)
    switchboard.on_message do |message|
      stream.send(message.answer)
    end
  end
end
I'll get to the details of self.connect shortly, but the gist of it is that
the EchoJack registers an on_message callback and uses Jabber::Message's
#answer method to create a response (inverting the sender and receiver)
before sending it back on the stream (client or component connection, as the
case may be).
Implementing the EchoBot as a jack is perhaps overkill, but the goal was to
demonstrate how short and modular Switchboard apps can be. The alternate
implementation looks like this:
switchboard.plug!(AutoAcceptJack, NotifyJack)
switchboard.on_message do |message|
  stream.send(message.answer)
end
Callbacks are executed in the context of the switchboard object; this is
important to remember, as variables defined in a different scope will be
unavailable.
switchboard.run!
This kicks off the process of connecting to the server. As XMPP is an
asynchronous protocol, the different hooks will be called in response to
varying inputs.
Jacks and Hooks
Jacks are extractions of standard functionality that are executed in the
context of the Switchboard core. Thus, references to stream, etc. call
methods implemented by Switchboard::Core and its subclasses rather than the
jack itself. If you need convenience methods, they should be defined on the
switchboard object provided as the 1st argument to connect:
def self.connect(switchboard, settings)
  def switchboard.helper_method
    # do something
  end

  switchboard.on_message do |message|
    helper_method
  end
end
connect is the entry point for all jacks. When the jack is plugged into
switchboard (using Switchboard::Core#plug!), connect is called with the
active Switchboard instance and its corresponding Switchboard::Settings. You
can modify the settings in the body of connect, but what you'll most often
be doing is adding additional functionality (via method definitions or
Modules) to the Switchboard instance.
PubSubJack (lib/switchboard/jacks/pubsub.rb) modifies the switchboard
object by extending a helper Module:
def self.connect(switchoard, settings)
  switchboard.extend(Switchboard::Helpers::PubSubHelper)

  switchboard.on_startup do
    # ...
  end
end
Like everything else, jacks have access to all hooks (lifecycle callbacks).
These are called using on_<hook name>. In general, these map to callbacks
defined by xmpp4r.

startup
shutdown
stream_connected
exception
stanza
message
presence
iq

Clients provide the following additional hooks:

roster_presence
roster_query
roster_subscription
roster_subscription_request
roster_loaded
roster_update

Strictly speaking, these hooks should probably be implemented by a
RosterJack and plugged in to Switchboard::Client by default.
Some jacks also introduce additional hooks (using hook(:name)), such as
PubSubJack:

pubsub_event

This hook is registed in Switchboard::Helpers::PubSubHelper
(lib/switchboard/helpers/pubsub.rb) with
Switchboard::Core.hook(:pubsub_event).
Standard Jacks
The following jacks are available from the standard distribution:

AutoAcceptJack - Auto-accepts and reciprocates roster requests.
DebugJack - Displays color-coded <message />, <presence />, and
<iq /> stanzas.
EchoJack - Example jack that echos inputs back to the sender.
NotifyJack - Sends online/offline presence to all roster items.
PubSubJack - Adds a pubsub_event hook and some PubSub convenience
methods.
OAuthPubSubJack - Adds a pubsub_event hook and some OAuth-enabled PubSub
convenience methods. The oauth gem must be installed for this to work.
RosterDebugJack - Like the DebugJack, but for roster events and
colorless.

Fire Hydrant includes a
FireEagleJack that introduces a location_update hook that yields a
FireEagle::User object, demonstrating that Switchboard apps can be
blissfully unaware of XMPP semantics when using the right jacks.
A More Complex Example, with Pandas
Bamboo Shooter is a
pseudo-realtime interface to the Flickr Panda
APIs.
It polls the APIs once per minute and dribbles the responses out over XMPP
over the course of the subsequent minute.
This represents a more complex example for several reasons. Firstly, it
coexists with EventMachine as a set of
additional threads (EventMachine is evented and thus single-threaded):
EM.run do
  Thread.new do
    # Bamboo::Shooter subclasses Switchboard::Component
    @shooter = Bamboo::Shooter.new(SETTINGS)
    @shooter.run!
  end

  # ...
end
Secondly, it implements a PubSub
service as a component, so it
subclasses Switchboard::Component rather than Switchboard::Client. A
side-effect of implementing a service as a component is that the logic for
everything that the server would ordinarily do has to be managed by your code
instead. Presence handling (implemented by Bamboo::Shooter#presence_handler)
is usually the first example of this.
PubSub requests and subscription handling are application-specific concerns,
but much of the implementation is essentially boilerplate.
Dovetail represents a start at
abstracting this, but it's incomplete at the moment. <subscribe /> and
<unsubscribe /> requests are the only operations currently supported; the fallback is to return a <feature-not-implemented /> response:
def iq_handler(iq)
  if iq.pubsub
    if subscribe = iq.pubsub.first_element("subscribe")
      node = subscribe.attributes["node"]

      puts "Subscription to #{node} requested by #{iq.from}"
      subscribers[node] ||= []
      unless subscribers[node].include?(iq.from.strip)
        subscribers[node] << iq.from.strip
      end

      resp = Jabber::Iq.new(:result, iq.from)
      resp.from = iq.to # TODO component.domain (elsewhere, too)
      resp.id = iq.id
      pubsub = Jabber::PubSub::IqPubSub.new
      subscription = Jabber::PubSub::Subscription.new(iq.from.strip, node)
      subscription.state = "subscribed"
      pubsub.add(subscription)
      resp.add(pubsub)

      deliver(resp)
    elsif unsubscribe = iq.pubsub.first_element("unsubscribe")
      node = unsubscribe.attributes["node"]

      puts "Unsubscription from #{node} requested by #{iq.from}"
      subscribers[node] ||= []
      subscribers[node].delete(iq.from.strip)

      resp = Jabber::Iq.new(:result, iq.from)
      resp.from = iq.to # TODO component.domain (elsewhere, too)
      resp.id = iq.id
      deliver(resp)
    else
      puts "Received a pubsub message"
      puts iq.to_s
      # TODO not-supported
      not_implemented(iq)
    end
  else
    # unrecognized iq
    not_implemented(iq)
  end
end
This is tightly tied to xmpp4r and REXML, but as the rest of Switchboard is
too, it's not a big deal for the time being.
Presence tracking is a somewhat tricky problem (particularly for large numbers
of consumers), so you'll notice that I completed punted and left it
unimplemented. This means that subscriptions remain active (and will attempt
to deliver photo payloads) even if a client has gone offline. For this reason,
the consumers of this service are written to subscribe on startup and
unsubscribe on shutdown. If the unsubscribe doesn't happen, there's a fair
chance that the bamboo-shooter process's memory usage will balloon. As a
result, there is currently no public instance of this service running.
Polling of the Panda APIs is done using EventMachine:HttpRequest, which is a
non-blocking, evented HTTP client. The result of each request (scheduled to
run once per minute per panda) is parsed using REXML (because Switchboard is
already using it via xmpp4r), split up, and scheduled for publishing using
EventMachine.add_timer:
EM.run do
  # ...

  check_pandas = lambda do
    params = {
      'api_key' => SETTINGS["flickr.key"],
      'method' => 'flickr.panda.getPhotos'
    }

    ["ling ling", "hsing hsing", "wang wang"].each do |panda|
      req = EventMachine::HttpRequest.new('https://api.flickr.com/services/rest/')
      http = req.get(:query => params.merge('panda_name' => panda))

      http.callback do
        begin
          doc = REXML::Document.new(http.response)
          doc.root.each_element do |rsp|
            total = rsp.attributes["total"].to_s.to_f
            panda = rsp.attributes["panda"].to_s
            interval = rsp.attributes["interval"].to_s.to_f
            interval = interval / total
            delay = 0.0

            puts "#{panda} found #{total} items with a #{interval}s delay."

            rsp.each_element do |node|
              EventMachine::add_timer(delay) do
                @shooter.publish("/flickr/pandas/#{CGI.escape(panda)}", node)
              end
              delay += interval
            end
          end
        rescue REXML::ParseException
        end
      end
    end
  end

  EventMachine::add_periodic_timer(61, &check_pandas)
  check_pandas.call
end
Consuming the Bamboo Shooter feed can either be done by running switchboard pubsub --server <server> listen or with code like this (earth.rb, which
will zoom Google Earth to the location where the photo was taken). Remember,
Wang Wang is the Panda who likes maps.
#!/usr/bin/env ruby -rubygems

begin
  require 'appscript'
rescue LoadError => e
  gem = e.message.split("--").last.strip
  puts "The #{gem} gem is required."
end

require 'cgi'
require 'switchboard'

node = "/flickr/pandas/#{CGI.escape(ARGV[0])}"

earth = Appscript.app("Google Earth")

DEFAULTS = {
  "resource" => "earth",
}

settings = YAML.load(File.read("bamboo_shooter.yml"))
switchboard = Switchboard::Client.new(settings.merge(DEFAULTS))
switchboard.plug!(AutoAcceptJack, NotifyJack, PubSubJack)

switchboard.on_startup do
  defer :subscribed do
    puts "Subscribing to #{node}"
    subscribe_to(node)
  end
end

switchboard.on_shutdown do
  puts "Unsubscribing from #{node}"
  unsubscribe_from(node)
end

switchboard.on_pubsub_event do |event|
  event.payload.each do |payload|
    payload.elements.each do |item|
      photo = item.first_element("photo")
      lat = photo.attributes["latitude"].to_f
      lon = photo.attributes["longitude"].to_f
      earth.SetViewInfo({:latitude => lat,
                         :longitude => lon,
                         :distance => (rand * 25000) + 5000,
                         :azimuth => rand * 360,
                         :tilt => (rand * 75)},
                        {:speed => 1})
    end
  end
end

switchboard.run!
Publishing is managed by the Switchboard component:
def publish(node, xml_node)
  event = Jabber::PubSub::Event.new
  items = Jabber::PubSub::EventItems.new
  items.node = node
  item = Jabber::PubSub::EventItem.new

  item.add(xml_node)
  items.add(item)
  event.add(items)

  (subscribers[node] || []).each do |subscriber|
    message(subscriber, event)
  end
end
Getting Bamboo Shooter running is a little tricky, as it requires an XMPP that
supports the component protocol (I use ejabberd with
the following configuration). More complicatedly, this server either needs to
be public (with DNS SRV records configured to support federation) or on a
private network with a second XMPP server running. When developing locally, I
use an Ubuntu VM with ejabberd running in component mode (using ejabberd.cfg
below) and a second ejabberd instance with the default configuration running
in OS X. bamboo-shooter.rb connects to Ubuntu, the client connects to OS X,
and the ejabberd instances figure out how to connect to one another with
ZeroConf (hostname.local; avahi-daemon on Ubuntu makes this possible).
{loglevel, 4}.
{hosts, ["localhost"]}.
{listen, [
{5222, ejabberd_c2s, [ {access, c2s}, {shaper, c2s_shaper}, {max_stanza_size, 65536} ]},
{5269, ejabberd_s2s_in, [ {shaper, s2s_shaper}, {max_stanza_size, 131072} ]},
{5288, ejabberd_service, [{host, "<hostname>", [{password, "<password>"}]}]}
 ]}.
{auth_method, internal}.
{shaper, normal, {maxrate, 1000}}.
{shaper, fast, {maxrate, 50000}}.
{acl, local, {user_regexp, ""}}.
{language, "en"}.
{modules, [
 ]}.
That's it for now
That's all I've got for now. It should be enough to get you started building
clients and components, but if you have any questions, post a comment below or
write to the Switchboard Google
Group.


switchboard : XMPP :: curl : HTTP
2009-07-16T00:00:00Z
Quick Start

Switchboard on GitHub:
https://github.com/mojodna/switchboard/tree
Switchboard Google Group:
https://groups.google.com/group/switchboard

Here's how to install and use Switchboard for a few basic use-cases:
$ # install Switchboard
$ sudo gem install switchboard

$ # list everyone on your roster (buddy list)
$ switchboard --jid client@example.com --password pa55word \
    roster list

$ # add a friend to your roster
$ switchboard --jid client@example.com --password pa55word \
    roster add friend@example.com

$ # listen for PubSub events
$ switchboard --jid subscriber@example.com --password pa55word \
    pubsub --server <pubsub server> listen
"curl for XMPP?"
curl is the ultimate Swiss Army Knife for HTTP. Switchboard aims to be the
same for XMPP.
HTTP is (relatively) easy. It's stateless and has fairly limited semantics.
However, if you're trying to debug a web service and want to determine what
certain ETags, additional headers, or specific content types respond with,
curl is your go-to.
XMPP is a bit more complicated; it's stateful, it (usually) requires login
credentials, it's asynchronous, and it has many extensions to the core
protocol that are in varying levels of use. Traditionally, in order to explore
an XMPP service, you'd have to delve into the advanced features of a client
like Psi or Synapse, often
dropping to the level of entering raw XML to see what happens in response.
Switchboard (and its command-line equivalent, switchboard) simplifies the
process of repeatedly making common requests (e.g. roster manipulation,
probing, and PubSub node operations) and is easily extended to support new
behaviors.
Switchboard background
Copying and pasting XML isn't the least error-prone process I can think of. In
addition, I found myself repeatedly testing the same types of stanzas (e.g.
PubSub requests) and attempting to
implement extensions for which no implementations existed (i.e. OAuth over
XMPP).
Diving into xmpp4r's functionality led me to
expose additional features to the command-line tool such as
PEP and Last
Activity to see how difficult it
would be.
Since all of its dependencies have been released and are installable via
RubyGems now, people who aren't me can finally use Switchboard without much
hassle. In honor of this milestone, Switchboard has been updated to 0.1.0.
Real mature, I know, but it is still quite useful.
People who are me have been using this tool since last November (in
fact, this is the project that inspired "My (public) Git
Workflow"). (Some people have been
using it to consume Fire Eagle's Location
Streams, but
it hasn't been for the faint of heart.)
Switchboard's primary strength is certainly the command-line interface, but it
turns out that it's a pretty good abstraction over xmpp4r for building
clients and servers (as components).
Here are a few projects I've worked on that use Switchboard:

bamboo-shooter - PubSub for
Pandas: an pseudo-realtime XMPP interface to the Flickr Panda APIs
fire-hydrant - A simple
library (as a Switchboard "jack") for consuming
Fire Eagle's Location Streams
mars and
jupiter - A matched pair of
experiments mapping REST to XMPP
dovetail - A start at a framework
for building PubSub servers (again, as components)

What to do?
Configuration
The first thing you'll probably want to do with Switchboard is to provide some
basic configuration so you won't have to constantly enter your login
credentials:
switchboard config jid jid@example.com
switchboard config password pa55word
To get the value of a setting, don't include a value:
switchboard config jid # => jid@example.com
Some additional useful settings to set defaults for are:

debug - Turn on verbose mode: good for debugging. (true / false)
resource - Override the default resource of /switchboard (don't include
the /). This is useful if you want to run multiple instances with the same
JID. This is equivalent to switchboard --resource <resource>. (String)
pubsub.server - Specify a default server to make PubSub requests against.
This is equivalent to switchboard pubsub --server <server>. (String)
oauth - Use OAuth when making PubSub requests. This
is equivalent to switchboard pubsub --oauth and requires that the
oauth gem be installed. (true /
false)
oauth.consumer_key - Specify a default OAuth consumer key. This is
equivalent to switchboard pubsub --oauth-consumer-key <key>. (String)
oauth.consumer_secret - Specify a default OAuth consumer secret. This is
equivalent to switchboard pubsub --oauth-consumer-secret <secret>.
(String)
oauth.token - Specify a default OAuth token. This is equivalent to
switchboard pubsub --oauth-token <token>. (String)
oauth.token_secret - Specify a default OAuth token secret. This is
equivalent to switchboard pubsub --oauth-token-secret <secret>. (String)

In general, anything that is referred to in the library in the form
OPTIONS["pubsub.node"] can be set as a Switchboard setting.
Roster Manipulation
Roster manipulation is something that can be easily handled by a desktop XMPP
client, but sometimes it's more convenient to be able to do it from the
command-line.
Rosters can be listed, added to, or removed from. I'll assume you've
configured Switchboard with some login credentials.
switchboard roster list
switchboard roster online
switchboard roster add friend1@example.org friend2@example.org
switchboard roster remove friend2@example.org enemy@example.org
Probing and Discovery
XMPP provides some pretty heady functionality when it comes to determining
what a server is capable of. A disco#info query is the first step to use
when determining capabilities:
switchboard disco --target jabber.org info
The response to this query includes http://jabber.org/protocol/disco#items,
which means that jabber.org supports disco#items queries, which allow you
to determine what top-level items (services) are available:
switchboard disco --target jabber.org items
This response includes conference.jabber.org. Let's list items available
there:
switchboard disco --target conference.jabber.org items
Whoa. A list of MUCs (multi-user chats) hosted on conference.jabber.org.
You can do the same thing to list available PubSub nodes if you're running a
local copy of ejabberd or another XMPP server that
supports it. (Note that you may need to prefix your hostname with pubsub in
order to see the nodes.)
PubSub
Switchboard supports more of
PubSub than any other extension,
mainly because that's been my primary focus of XMPP experimentation. To get a
full list of available PubSub commands:
switchboard pubsub
A basic sequence of events is to subscribe:
switchboard pubsub --server <server> subscribe --node <node>
List subscriptions:
switchboard pubsub --server <server> subscriptions
Listen for notifications:
switchboard pubsub --server <server> listen
Unsubscribe:
switchboard pubsub --server <server> unsubscribe --node <node>
Let's walk through a couple examples.
First, Superfeedr, which bills itself as "real-time
feed parsing in the cloud". To begin, you'll need to register and activate
your account. Once you've done that, set up a subscription a
Twitter search for "xmpp":
$ switchboard --jid <username>@superfeedr.com --password <password> \
    pubsub --server firehoser.superfeedr.com \
    subscribe --node "http://search.twitter.com/search.atom?q=xmpp"
We would next list subscriptions, but Superfeedr uses a non-standard mechanism
to do so. Instead, let's listen for new results:
$ switchboard --jid <username>@superfeedr.com --password <password> \
    pubsub --server firehoser.superfeedr.com listen
If you're lucky, you'll get an Atom payload or two. Here's one:
<event xmlns='http://jabber.org/protocol/pubsub#event'>
  <status feed='http://search.twitter.com/search.atom?q=xmpp' xmlns='https://superfeedr.com/xmpp-pubsub-ext'>
    <http code='200'>16933 bytes fetched in 0.600034s</http>
    <next_fetch>2009-07-22T17:26:22Z</next_fetch>
  </status>
  <items node='http://search.twitter.com/search.atom?q=xmpp'>
    <item chunk='1' chunks='1'>
      <entry xmlns='http://www.w3.org/2005/Atom'>
        <title>Great... trillian update has killed my ability to view my xmpp rosters</title>
        <summary>Great... trillian update has killed my ability to view my &lt;b&gt;xmpp&lt;/b&gt; rosters</summary>
        <link href='https://superfeedr.com/entries/tr5gfgstf8oqlcgr5opaeotxk39ovtos0oiat7h12mqfuoxdmgbjz1rnjtzswqvja2dqh8cgg31' rel='alternate' type='text/html'/>
        <published>2009-07-22T17:10:52Z</published>
        <id>tag:search.twitter.com,2005:2781212809</id>
      </entry>
    </item>
    <item chunk='1' chunks='1'>
      <entry xmlns='http://www.w3.org/2005/Atom'>
        <title>usando Tkabber: TKabber is Tcl/Tk Jabber-client with great functionality. It supports MUC, XMPP-statuses a.. http://bit.ly/AsFPg</title>
        <summary>usando Tkabber: TKabber is Tcl/Tk Jabber-client with great functionality. It supports MUC, &lt;b&gt;XMPP&lt;/b&gt;-statuses a.. &lt;a href=&quot;http://bit.ly/AsFPg&quot;&gt;http://bit.ly/AsFPg&lt;/a&gt;</summary>
        <link href='https://superfeedr.com/entries/w6cezbjniqqgga3bd79ruklxhu7f4a6qqpro76hgz50gzccuekgmehz39yb1zi1cclgo83s' rel='alternate' type='text/html'/>
        <published>2009-07-22T17:07:54Z</published>
        <id>tag:search.twitter.com,2005:2781162048</id>
      </entry>
    </item>
  </items>
</event>
It includes a Superfeedr-specific <status/> element with information on the
most recent fetch as well as standard Atom feeds contained within standard
<item/> elements. This means that you can extract the Atom elements from the
PubSub payload and hand it to a feedparser of some variety to work its magic.
Fire Eagle's Location
Streams also
use PubSub, but subscription management requires that requests are signed
using OAuth. This allows 3rd party applications to make requests on behalf of
specific users without having to obtain their actual credentials (in short,
it's exactly the same use-case for OAuth over HTTP).
Let's start with a subscriptions list request, since we have the credentials
("General Purpose Access Token") immediately after registering a "web"
application with Fire Eagle.
$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <general token> \
    --oauth-token-secret <general token secret> \
    --server fireeagle.com \
    subscriptions
Odds are, you'll have nothing there. Let's change that. Send yourself through
the authorization process in order to get a valid OAuth token and secret:
$ oauth --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    --access-token-url https://fireeagle.yahooapis.com/oauth/access_token
    --authorize-url https://fireeagle.yahoo.net/oauth/authorize
    --request-token-url https://fireeagle.yahooapis.com/oauth/request_token
    authorize
With that token and secret, subscribe to your Location Stream:
$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <token> \
    --oauth-token-secret <token secret> \
    --server fireeagle.com \
    subscribe --node "/api/0.1/user/<token>"
Now you'll have a subscription to query for:
$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <general token> \
    --oauth-token-secret <general token secret> \
    --server fireeagle.com \
    subscriptions
Listen for location updates:
switchboard pubsub --server fireeagle.com listen
Update your current location and
watch as the update rolls in. If you'd like to visualize updates with Google
Earth, check out fire-hydrant on
GitHub.
We're done, so we may as well clean up and unsubscribe:
$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <token> \
    --oauth-token-secret <token secret> \
    --server fireeagle.com \
    unsubscribe --node "/api/0.1/user/<token>"
PEP (Personal Eventing Protocol)
PEP is a specialized version of
PubSub, intended to allow individuals to associate data with their JIDs.
Switchboard supports publishing of User Tune and User Location.
Due to the ability of XMPP to allow multiple instances of the same account
online (identified with different resources), Switchboard can serve up User
Tune and User Location for the same account you're already online with. Not
every client supports displaying them, but they're fun to play with
regardless.
To publish User Tune, you need to be on a Mac, running iTunes, and have the
rb-appscript gem installed (sudo gem install rb-appscript). Once that's
done:
switchboard --resource switchtunes pep tune
To publish User Location, you need to be updating Fire
Eagle
(Clarke is an excellent background
updater for OS X) and have the fire-hydrant gem installed from GitHub (sudo gem install mojodna-fire-hydrant -s http://gems.github.com). Once that's
square:
switchboard --resource switchfire pep location
More
Switchboard supports more functionality than I've described above. To get a list of general switchboard commands (some of which may have sub-commands):
switchboard
Getting Help
In theory, if you want more information about a specific command, you can use
switchboard help <command>. For example:
switchboard help pubsub
For now, you'll notice that it's not particularly useful. If you'd like to
help rectify this, you can implement various help methods that are lying
around, such as Switchboard::Commands::PubSub.help.
Extending
Writing new Switchboard commands is really easy, assuming that the primary
application logic that you're depending exists elsewhere (i.e. in xmpp4r).
I was on a panel with Peter St. Andre and Jack Moffitt at the Glue Conference in Denver this Spring and we got to talking about tools like Switchboard. Jack wondered how hard it would be to implement something like grep for XMPP.
(Jack is one of the authors of a Python project similar to Switchboard named poetry).
I took a whack at it and it came out like this:
module Switchboard
  module Commands
    class Grep < Switchboard::Command
      description "Search for an XPath expression"

      def self.run!
        expr = ARGV.pop

        switchboard = Switchboard::Client.new
        switchboard.plug!(AutoAcceptJack, NotifyJack)

        switchboard.on_stanza do |stanza|
          # TODO doesn't handle default namespaces properly
          REXML::XPath.each(stanza, expr) do |el|
            puts el.to_s
          end
        end

        switchboard.run!
      end
    end
  end
end
Jacks (not Moffitt) deserve their own discussion, but the thrust of this
piece of code is the #on_stanza callback (which yields a REXML::Node
object) and the XPath expression. Note that for whatever reason, you can't
search for nodes that have a default namespace (e.g. <presence />); I'm
going to assume that this is a REXML quirk.
If you want to take a shot at implementing a Switchboard command,
Ping should be pretty simple.
Alternately, Switchboard doesn't support sending or receiving basic <message /> stanzas from the command-line. Supporting those would make it simple to
interact with a service like identi.ca.
How to Help
There are a few simple ways to help out with Switchboard's development:

explore the API
build out the docs
implement IM functionality
write up how to use it for a particular service / use-case
use it!

What's your favorite use for Switchboard?


Updating Ruby Consumers and Providers to OAuth 1.0a
2009-05-20T00:00:00Z
In a previous post, I did a quick run-through of the changes that were
introduced in OAuth
1.0a.
As promised, here's a rough guide to updating Ruby Consumers and Providers to
1.0a. Don't mind the pseudo code.
Updating Ruby OAuth Consumers to 1.0a
In order for things to work properly, you'll need to use a version of the
OAuth gem that's at least 0.3.4.1 (0.3.5 was released on 6/3/09). To install
it and check the version number:
sudo gem install oauth
oauth --version
Authorization code that once looked like this:
request_token = consumer.get_request_token
puts "Please visit the following URL to authorize this application:"
puts request_token.authorize_url(:oauth_callback => callback_url)
# wait for input
gets
access_token = request_token.get_access_token
Should now look like this:
request_token = consumer.get_request_token(:oauth_callback => callback_url)
puts "Please visit the following URL to authorize this application:"
puts request_token.authorize_url
# wait for input
puts "What's the value of `oauth_verifier`?"
oauth_verifier = gets.chomp
# `oauth_verifier` is extracted from the expanded callback URL or was displayed to the user
access_token = request_token.get_access_token(:oauth_verifier => oauth_verifier)
You can detect whether a Service Provider supports 1.0a:
request_token = consumer.get_request_token
puts "OAuth 1.0a detected" if request_token.callback_confirmed?
Updating Ruby OAuth Providers to 1.0a
As with the updates to consumer applications, you'll need to be running at
least 0.3.4.1. For the sake of simplicity (and additional laziness on my
part), I'll assume OAuth verification is baked into your Rails app rather than
as Rack middleware (where it probably belongs).
You'll need to add a pair of columns to your oauth_request_tokens table (or
whatever it's called): callback and verifier.
Accepting oauth_callback During the Request Token Phase
The first step to supporting OAuth 1.0a is to accept oauth_token parameters when issuing Request Tokens. To do this, you'll need to make the OAuth::RequestProxy::ActionControllerRequest available to methods that run later in a request's lifecycle:
def verify_oauth_signature
  valid = OAuth::Signature.verify(request) do |request_proxy|
    # make the request proxy available outside this block
    @_request_proxy = request_proxy

    # proceed normally...
  end
end

# accessor for the request proxy
def oauth_request_proxy
  @_request_proxy
end
Once this is set up, you'll need to modify your request_token method to associate the oauth_callback parameter with your Request token and set oauth_callback_confirmed to true:
def request_token
  request_token = new_request_token
   # request_proxy provides unified interface to params + headers
  request_token.callback = request_proxy.oauth_callback
  request_token.save

  render :text => "oauth_token=#{request_token.token}&" \
                  "oauth_token_secret=#{request_token.secret}&" \
                  "oauth_callback_confirmed=true"
end
Generating An oauth_verifier During the Authorization Phase
In addition to validating the Request Token during the authorization phase,
you'll want to generate an oauth_verifier value and return it to the
application via the pre-registered callback url (or display it to the user and
instruct them to enter it into their application).
def authorize
  # display the authorization page

  render and return unless request.post?

  # validate the token

  request_token = find_request_token(params[:oauth_token])
  request_token.validated = true
  # generate a verification code
  request_token.verifier = generate_verifier
  request_token.save

  if request_token.callback?
    # this was previously params[:oauth_callback]
    redirect_to request_token.callback + "?oauth_verifier=#{request_token.verifier}"
  else
    @verifier = request_token.verifier
    render # display the verification code to the user
  end
end
Verifying Access Token Exchanges
When exchanging a Request Token for an Access Token, you need to confirm that
the verification code provided by the consumer matches the one on file.
def access_token
  # this is a correctly signed request: oauth_token has already been loaded

  # compare the verifier from the request proxy to the one we generated
  if oauth_token.verifier == request_proxy.oauth_verifier
    # exchange the request token for an access token
    access_token = oauth_token.exchange
    render :text => "oauth_token=#{access_token.token}&" \
                    "oauth_token_secret=#{access_token.secret}"
  else
    raise OAuth::InvalidVerifier
  end
end
Summary
Obviously, there are cleaner ways to do this, but they're presumably very
specific to individual codebases. If you have questions, check out the
oauth-ruby mailing list.
Otherwise, patches can be submitted against
https://github.com/mojodna/oauth/tree/mergeable.
Good luck!


An Idiot's Guide to OAuth 1.0a
2009-05-20T00:00:00Z
I'm lazy. I don't usually enjoy re-reading things I've already read before
(Matt Ruff and Neil Stephenson books are an exception). So, as a service to
all 3 of my readers, I'll summarize the changes to the
OAuth specification for 1.0a (draft
3)
as I understand them. Finding changes in a large document is a pain--you have
better things to do with your time. Embrace your inner ignorance and leave
responsibility to the wind.
As an added bonus, I'll demonstrate how to update consumer and provider code
using the Ruby
library.
What Happened?
Much
has been written about this
elsewhere,
so I'll be brief. A session fixation attack was
discovered a little over a month ago.
Essentially, it affixes your sessions to someone else's. Imagine burrs or
inverted balls of duct tape with little teeny messages (on grains of rice)
attached to them.
Anyway. It was bad. It was one of those things that once you've seen, it's
impossible to un-see. There was no straightforward fix that could be deployed,
so you started seeing lots of BIG ANGRY warning notices explaining that your
car might be repossessed if you were bold enough to continue.
The longer-term solution was to change the
specification,
get providers on-board to change their code, and finally to reach
out to developers to update their client applications. All so the world
doesn't implode next time you turn right. DON'T TURN RIGHT. Ok?
So, What Changed?


Callbacks (to your site or application) have been supported in various
forms (with varying restrictions), but they have always been parameters
passed to the Service Provider's authorization page. Instead, the
oauth_callback parameter is now part of the Request Token phase. If your
client cannot accept callbacks, the value MUST be oob. SPs use this
to determine whether a client supports 1.0a.


oauth_callback_confirmed will be present (indeed, MUST according to
the spec) when Service Providers supporting 1.0a issue a Request Token.
(e.g. oauth_token=asdf&oauth_token_secret=qwerty&oauth_callback_confirmed=true)
Clients can use this to determine whether a server supports 1.0a.


An oauth_verifier parameter is provided to the client either in the
pre-configured callback URL or through the fingers of your users (the
aforementioned oob ("out of band") mechanism). This value MUST be
included when exchanging Request Tokens for Access Tokens.


The changes to the spec are limited to the Request Token ➡ Access Token
exchange, so once you have an Access Token, everything should behave as it did
before. For this reason, clients that only implement 2-legged OAuth are
unaffected.
Section 11.14. Cross-Site Request Forgery
(CSRF)
is also worth reading (and understanding) in order to further secure the use
of callback URLs.
Services Supporting OAuth 1.0a
At the time of this writing, the only service provider I'm aware of that
supports 1.0a is
Yahoo!.
This doesn't include Fire Eagle (unless you include my laptop), although we
intend for this to change in the next couple weeks. Google's OAuth endpoint
should support 1.0a soon, but I've yet to see any confirmation.


My (public) Git Workflow
2009-03-09T00:00:00Z
The problem: You want to be an effective contributor to a public project
that's using Git.
The solution: Follow conventions and keep clean branches to make them
easier to merge.
Let's Go
For your own projects, it doesn't really matter what you do to publish and
maintain your code, at least until other people start forking and attempting
to hack on your stuff. In any case, you should keep things simple. Use topic
branches for particular features or refactoring if you want, but no big deal.
Determining "Active" and "Authoritative" Forks
GitHub's search attempts
to surface active projects by displaying the number of forks, number of
watchers, and recent activity, but it doesn't tell the full story. The "root"
of the fork tree (which isn't really a tree, but is kinda useful to think of
that way) isn't necessarily the current focus of activity. Someone may have
published a project and lost interest, but a community may have formed around
one of the forks.
Alternately, a project may have been imported from Subversion by a wannabe
contributor prior to the original author joining GitHub and applying
community-created patches to his or her own fork (which, at this point, isn't
the root of the fork tree).
If there are a relatively low number of forks (say, one or two), you can
(mostly) assume that the fork with the most activity is authoritative. Other
contributors, with forks that come and go or that are generally out of date,
can be considered "outsiders."
For projects with a larger number of forks, there may be multiple authorities
(and potentially even multiple development directions), with many outsiders
periodically contributing patches.
All of this means that you, as a potential contributor, need to do some
additional research to determine which fork (or forks) is "active" and/or
"authoritative". They may not be one and the same; the "authoritative" fork
may focus on stability; you may need bleeding edge features developed by an
"authoritative" member of the community instead.
GitHub's Network
Graph is
a great tool for this purpose. From the interface, you can see which forks
have been the most active (hint: more •'s), most recently active
(•'s further right), or have had the most merges (more ⤴'s). Those don't
tell the full story, so you'll want to mouse-over individual •'s to see
who wrote the patch (and who committed it). That will help distinguish between
the forks where the patches originated and where they've merely been pulled
in. You'll also get a sense of the number and names of people working on a
project.
Some Rules of Thumb
When hacking on other peoples' stuff, your eventual goal is (probably) to get
those changes applied. Here are a few tips to make life easier for everyone
involved.
First rule of thumb: merge from authoritative contributors, cherry pick
from outsiders.
By merging (e.g. git merge user/master) from authoritative contributors, it
becomes easier for them to apply your changes if and when they deem them fit.
Merging from outsiders may force the authorities into accepting patches (or
series) that they may not be ready for (as those branches may include changes
unrelated to what you were working on).
GitHub's Fork Queue cherry picks (e.g. git cherry-pick 469df8) commits by
default, so that's a good way to get your tree in order before adding your own
changes. If there are changes to your fork's parent, you can fast forward
your fork before starting
work.
Second rule of thumb: keep your changes easily apply-able to the
authoritative repository.
For a single line of development, this is easy: rebase against the
authoritative master branch whenever possible. Feel free to rewrite history
(i.e. git push -f) as long as you're confident no one is treating you as
authoritative.
If others are treating you as an authoritative source, create a separate
branch that gets periodically rebased and make it clear that it's not suitable
for tracking. Others' changes will eventually need to be rebased against those
changes, but probably not until your changes have been applied by a real
authoritative contributor.
For changes that do many things, this means keeping multiple branches active.
I usually use a branch per feature, plus a separate branch for general
bugfixes and documentation. For minor changes, all branches can diverge from a
common commit (usually the tip of the authoritative repository's master
branch).
For more drastic changes, you'll need to decide which is either the most
important feature or the least controversial change. Use that as the first
patch series and rebase it against the authoritative repository's HEAD.
Rebase subsequent patch series against the next most important (or least
controversial).
Write down the order in which your branches should be merged. You can use your
project wiki or put it at the top of the README in your master branch.
(See my fork of xmpp4r to see how I've done
it.)
At this point, your master branch is just the default view of your fork and
shouldn't be considerably different from the branch of the authoritative copy
you're tracking (see the exception below).
In some cases, you'll have branches that aren't quite ready for prime-time.
Treat them the same as branches that are, but don't make apply-able branches
dependent on them (i.e. they should always be the last changes to be applied).
Include them in the summary in your README, but make it clear that they're
not ready yet.
Third rule of thumb: reset your history to match an authoritative
repository's history if their content is the same (or essentially the same).
If your history frequently diverges from an authoritative repository's
history, you're going to have more trouble applying others' changes and
synchronizing with authoritative repositories.
The simplest way is to delete your repository and re-fork. If you have minor
changes, generate a diff to apply once you've re-forked.
An Exception
Because GitHub will only build gems for changes on the master branch, if you
have changes that you want to publish (prepared neatly in topic branches, as
above), you'll need to break the rules somewhat.
To handle this scenario, start with your master branch matching the
authoritative branch you're tracking, then merge branches like the owner of
the authoritative branch would (this has the helpful side-effect of
discovering whether your changes will apply cleanly).
The tip of your master should always contain the most up-to-date gemspec (in
order to trigger GitHub's building and publishing of your fork) as well as a
README that contains information on the state of your repository (including
branches, as above, but also information about how you're treating the
master branch).
Maintaining a Project
Over the last couple months, I've become one of the authoritative forks for
the OAuth gem, indirectly resulting in this
document. I've attempted to be consistent in the management of my fork,
including publishing periodic releases for others to test functionality, but
it became clear that my methods weren't as transparent as they ought to be.
As a maintainer (or authoritative fork), you frequently find the need to merge
in work from other developers. GitHub's Fork
Queue is excellent for this, but
it's not the last word.
The Fork Queue is perfect for merging individual commits, but when someone
submits a patch series, it's time to make a decision: you can cherry-pick
(e.g. git cherry-pick 469df8) all of their commits (which is what the Fork
Queue does behind the scenes)--retaining authorship but setting the
committer field to yourself--or you can pull their branch down and perform a
merge (e.g. git merge user/master), retaining both author and
committer.
Cherry picking commits is easier--unless you want/need to make adjustments to
them (in which case the SHAs would change anyway)--and results in linear
history (no merges) with the downside of having repetitive commits (with
different committers) across the project network, thus confusing GitHub's
Network Graph (or anything that displays the history of multiple branches) and
making it more difficult to find un-applied patches (as the same patch may
exist with multiple SHAs).
Merging changes locally is more difficult, but the github
gem makes it easier by pulling in the
entire patch universe with gh network fetch. Once that's been done, the
combination of gh network commits, GitX, and the
Network Graph make it relatively easy to identify commits ripe for picking.
An advantage of cherry picking is that you have the freedom to amend (git commit --amend) commits before pushing them out as part of your tree. This is
a great opportunity to clean things up to match local coding standards (that
should already be the case, but YMMV) and to add additional context to the
commit message.
Note: You can achieve the same effect as amending commits by using git format-patch -1 469df8 to create a patch, applying it with git apply,
pausing to clean it up, and committing it with git commit -c 469df8. git
ready has a good article on picking out individual
commits.
The main problem with cherry picking becomes more acute when there are
multiple authoritative forks (which should maintain a consistent public state)
constantly passing work back and forth. If the participants periodically merge
others' work, you end up with a number of merge commits, but a relatively
coherent history. If one or more participants choose to cherry pick commits
instead, merging their branches back will include repeated commits with
different SHAs and committer fields. The resultant state of the tree is
fine, but the history gets somewhat polluted and becomes more difficult to
follow.
This can be mitigated by sending patches via email (or any mechanism by which
public repositories don't change) and repeatedly munging history locally, but
that's often not an option, given the level of coordination required.
Aside: Complexity and / or Maturity
I used to see complicated Network Graphs (with lots of merging and branching)
as a sign of project activity, but now I consider it more as a measure of
contributors' comfort with git (as individual contributors may be
responsible for multiple branches). I believe that the main reason that
projects end up with complicated history is due to (relatively) limited
coordination between collaborator and maintainer; if I had to guess, I would
expect closed projects using Git to have much more linear histories.
Ultimately, I think complicated histories are a side-effect of Git's merging
prowess promoting and supporting the drive-by contributor model. However, as
powerful as Git is, coordination between collaborators still lessens the
burden on the person doing the merging, thus the above manifesto.
Now What?
Get yourself out there, coding whatever and whenever, and contribute back.
You'll feel good and you will have made the world a little better, one patch
at a time.


Extending ActiveRecord Attributes
2007-06-14T00:00:00Z
David Black came to speak at the Boston Ruby
Group on Tuesday (along with Zed Shaw, making
it the most star-studded event since I started going). It was in one of the
lecture halls at the Harvard Science Center (which, if you follow the pattern
of every single other building on campus, is named after the "Science Center"
family; seems like it should be Merrill, or Morrill, or something
instead--here's your chance). I was expecting stand-up comedy, which turned
out to be an unreasonable expectation because it was in lecture hall A, not C
(which is where HSUCS (a group of stand-up comedians at Harvard) has had
events in the past). But I digress.
David's talk was about "Per Object Behavior in Ruby" and about treating it as
a prototype language (a logical follow-up to Dave Thomas' RailsConf keynote;
he didn't disappoint and provided classless code (except for the tests; that
would have been a nice touch)). I won't go much into the substance of the
talk, except to say that he managed to clarify the meaning and use of an
object's "singleton class" / "eigenclass" as well as further emphasizing that
class != type, but that a class is merely a template from which new objects
spring. He encouraged people to take advantage of Ruby's ability to extend
individual instances of objects with methods (generally via modules), as
that's a considerable part of what makes Ruby Ruby.
I was reminded that I had written some ActiveRecord code this spring that
allows "primitive" (String, Fixnum) attributes of AR objects to be
embellished by extending the attributes' objects before returning them from
the containing object.
In our case, we had a notion of FormattedText objects, which were
essentially Strings that could contain HTML. Unfiltered HTML, as it turned
out. Rather than filtering during the input stage, we decided that we would
store what the user provided and leave it to the output stage to handle the
filtering. Instead of using a helper method in each and every case (with the
downside of having to remember to use it), I decided that the filtering
functionality would fit better within the object (so it would be used
appropriately in the majority case without any extra thought). In this case,
by overriding String#to_s.
module FormattedText
  def to_s
    # filter and output
  end
end
The next step was making sure that only attributes known to contain formatted
text were extended by FormattedText (and that the internal representation
remained untouched, so that the filtered version wouldn't be saved back to the
database). A side-effect of this is that changing the value that was returned
and saving the object will not save the new value back to the database; to get
around this, you'll need to set the value on the AR object itself.
# Quick and dirty monkey-patch (should work with 1.1.x+)
class ActiveRecord::Base
  # Attribute extension
  cattr_accessor :extended_attributes
  @@extended_attributes = {}

  class << self
    # create the "extends" declaration
    def extends(attr_name, options)
      extended_attributes[attr_name.to_sym] = options[:with]
    end
  end

  alias_method :define_read_method_without_extends, :define_read_method
  def define_read_method_with_extends(symbol, attr_name, column)
    unless self.class.read_methods.include?(attr_name) ||
           self.class.extended_attributes.keys.include?(attr_name.to_sym)
      define_read_method_without_enhances(symbol, attr_name, column)
    else
      old_method_name = "#{symbol}_without_extends"
      define_read_method_without_extends(old_method_name.to_sym, attr_name, column)
      evaluate_read_method attr_name, "def #{symbol}; enhance(#{old_method_name}, '#{attr_name}'); end"
    end
  end
  alias_method :define_read_method, :define_read_method_with_extends

  # Override read_attribute for first call (pre-edge)
  # or when AR::Base.generate_read_methods = false
  alias_method :read_attribute_without_extends, :read_attribute
  def read_attribute_with_extends(attr_name)
    enhance(read_attribute_without_extends(attr_name), attr_name)
  end
  alias_method :read_attribute, :read_attribute_with_extends

  private

  # Enhance the attribute as appropriate
  def enhance(value, attr_name)
    mod = self.class.extended_attributes[attr_name.to_sym]
    if mod && !value.nil?
      value.dup.extend(mod)
    else
      value
    end
  end
end
An alternate way to implement this pattern would have been to use
composed_of and to create a FormattedText class that extend String. That
may be better for performance reasons (as it seems that it would simplify
method lookup), but strikes me as being slightly less Rubylike.
What do you think?


Teach Capistrano to Deploy From a Tag or Branch
2007-03-08T00:00:00Z
There comes a time in the life of many an application where it becomes more or
less stable. And once it becomes stable, it also becomes boring because the
changes that a developer really wants to make are bound to make it unstable,
and thus unsuitable for deployment.
An oft-used approach to this situation is to create a stable maintenance
branch (i.e. application/branches/1.0) and create tags for point releases
(i.e. application/tags/1.0.6), continuing on the trunk with potentially
destabilizing work.
Out of the box, Capistrano only supports deploying from the tip of the trunk.
The snippet below demonstrates how to deploy from a branch (to a QA or staging
environment) or tag (releases to production should always have tags).
Stick it in your config/deploy.rb.
set :base_repository, "http://svn.mojodna.net/repository/#{application}"
if variables[:tag]
  set :repository, "#{base_repository}/tags/#{variables[:tag]}"
elsif variables[:branch]
  set :repository, "#{base_repository}/branches/#{variables[:branch]}"
else
  set :repository, "#{base_repository}/trunk"
end
With this modification in place, you can now deploy from a branch:
RAILS_ENV=qa cap deploy -Sbranch=1.0
Or from a tag:
RAILS_ENV=production cap deploy -Stag=1.0.6
Go forth and revel in your instability!


Classloading in Rails
2007-02-12T00:00:00Z
Occasionally, I see code similar to the following used as a shortcut to load
all model objects. It's usually in a Rake task or somewhere that follows a
"global require" by looping through all available objects and doing something
with them. It's simple, straightforward, and a logical follow up to shorthand
like ["post", "user"].each { |m| require m }. However, it often sits out
there like a dormant volcano, with a wide variety of unpredictable behavior
waiting to erupt.
In our case, we'd been using said snippet to help test out
memcached; we would occasionally be handed
serialized objects of unrecognizable types; the source of the problem turned
out to be that the object's class definition had not yet been loaded. We also
discovered that finders in intermediate subclasses (we use STI extensively) would occasionally fail to find
instances of subclasses, again because the subclasses had not been loaded (and
the intermediate subclass was unable to provide a complete list of its
subclasses).
# load all models explicitly
Dir.glob(File.join(RAILS_ROOT,"app","models","*.rb")).each do |rbfile|
  require rbfile
end
There are a couple of problems with this approach. It doesn't guarantee that a
model will only be loaded once--classes may have already been loaded through
some external mechanism. Alternately, dependencies may be loaded while loading
a specific class. In both cases, the above code may cause classes to be loaded
multiple times (which can cause bizarre behavior). It also doesn't handle
namespaced models correctly.
# force loading (but not reloading) of all models by saying their name
# Adapted from PragDave's Annotate Models plugin.
model_path = File.join(RAILS_ROOT, "app", "models")
Dir.glob(File.join(model_path,"**","*.rb")).each do |m|
  class_name = m.sub(model_path + File::SEPARATOR, '').sub(/\\.rb$/, '').camelize
  klass = class_name.split('::').inject(Object){ |klass,part| klass.const_get(part) } rescue nil
end
This approach is considerably better; it supports namespaced models and defers
the actual classloading to Rails (note the lack of an explicit require),
which prevents classes from being loaded multiple times.
Anything to avoid time-eating debugging sessions, like the one described in
Rails Sometimes Eats Class Load
Errors. In our
case, it had to do with
ActiveMessaging and classes
silently failing to load outside the development environment. The fix wound up
being requiring config/messaging.rb before the block above. The lesson
learned is to make sure that any dependent resources for models are loaded
first and to look for them if things are acting strangely.
Update: The above primarily relevant to Rails 1.1. 1.2 exhibits similar
behavior, but it gets even weirder, as the class loading mechanism was
rewritten. There, you'll find that STI works properly the first time (because
all classes have been explicitly loaded), but if cache_classes is on (the
default in development mode), they'll be unloaded after the first request and
thus will fail to work on subsequent requests.
Our rather hackish solution is to trigger a before_filter in
ApplicationController when cache_classes is determined to be on (when
Dependencies.mechanism == :load):
before_filter do
  model_path = File.join(RAILS_ROOT, "app", "models")
  Dir.glob(File.join(model_path,"**","*.rb")).each do |m|
    class_name = m.sub(model_path + File::SEPARATOR,'').sub(/\\.rb$/,'').camelize
    klass = class_name.split('::').inject(Object){ |klass,part| klass.const_get(part) } rescue nil
  end
end if Dependencies.mechanism == :load


Searchable: Annotation-Driven Indexing and Searching with Lucene
2006-10-02T00:00:00Z
Overview
Searchable is a toolkit for Lucene
that harnesses the power of annotations to specify what properties to index
and how to treat them.
Basics
Searchable requires JDK 1.5 and Lucene
1.9+.
At its core, Searchable provides a set of annotations that can be used to
instruct its indexer how to deal with properties present in an annotated bean.
It can also be used without the annotations (by writing a custom
Indexer/Searcher combination). However, Searchable really shines when
custom Indexers and Searchers are combined with the annotations.
Searchable 0.8-SNAPSHOT can use Apache Solr
for indexing and searching via SolrIndexer and SolrSearcher
(SchemaGenerator will create an appropriate Solr schema for you; some
tweaks may be necessary).
Searchable is available on GitHub here:
https://github.com/mojodna/searchable
Do with it what you will.
Please see the examples below for pointers on usage.
Annotations
In order for the annotations to have any effect, they must annotate a class
that extends Searchable. With the exception of @DefaultFields, they must
all be placed on the reader method of a property (getXXX()). Unlike
traditional Annotation behavior, they may be placed on an interface method or
a method that is overridden.
@ID
This allows a property other than id to be specified as the id field. There
should only be one per class hierarchy.
e.g.:
@ID
public Integer getKey() { ... }
@Indexed
This specifies that a field should be indexed. The following attributes are
available:

aliases - Array of aliases to also use as field names. Default: none.
boost - Boost factor (as a float) for this field. Default: 1.0.
name - Field name to use for this property. Default: property name.
nested - Whether to index this field in a nested context (i.e. a
Searchable as a property of another Searchable). Default: false.
stored - Store this property in the index. Default: false.
storeTermVector - Store term vectors for this field. Default: false.
tokenized - Tokenize the value of this property before adding it to the index. Default: true.

e.g.:
@Indexed(boost=2.0F)
public String getName() { ... }
@Stored
This specifies that a field should be stored in the index. In typical usage,
this should not be necessary, although you may find that it comes in hand from
time to time. The following attributes are available:

aliases - Array of aliases to also use as field names. Default: none.
name - Field name to use for this property. Default: property name.
nested - Whether to store this field in a nested context (i.e. a
Searchable as a property of another Searchable). Default: false.

e.g.:
@Stored
public String getDescription() { ... }
@Sortable
This specifies that a custom Keyword field should be created to use for
sorting results in Lucene. The following attribute is available:

nested - Whether to create a sorted field for this property in a nested
context (i.e. a Searchable as a property of another Searchable).
Default: false.

e.g.:
@Indexed
@Sortable
public String getName() { ... }
@DefaultFields
This contains an array of field names that should be used as the default list.
If this annotation is not present and no fields are specified when searching,
the default is to use all fields present in the index.
Used during the search process.
e.g.:
@DefaultFields( { "name", "address.city" } )
public class Person implements Searchable { ... }
@Excerptable
This specifies that the annotated property should be used when creating a
search extract. NOTE: The resultant object must be hydrated in your
code before it can be excerpted. In many cases, this involves reloading the
object from Hibernate Session by using session.load().
e.g.:
@Indexed
@Excerptable
public String getDescription() { ... }
Extension Points
AbstractSearcher, AbstractMultiSearcher, AbstractIndexer, and
AbstractBeanIndexer are provided as abstract base classes with the majority
of necessary functionality provided as protected methods. AbstractSearcher
exposes multiple signatures for certain methods that allow alternate
implementations of Lucene _IndexReader_s and _Searcher_s.
AbstractMultiSearcher makes use of these to implement cross-index searching
(an example is provided below), but a similar approach could be used to
implement remote searching.
setAnalyzer(Analyzer) and setIndexPath(String) should be used to override
the default behavior of Searchable. Both are best called in your constructor.
getIndexReader(), getIndexModifier(), and getIndexSearcher() provide
shared access to _IndexReader_s, _IndexModifier_s, and _IndexSearcher_s over
the index specified using setIndexPath(String). In certain circumstances,
you may wish to override these methods to provide alternate implementations (a
MultiSearcher for example; if you wish to provide a RemoteSearchable, you
must define an additional method, as getIndexSearcher() returns a
Searcher, not a Searchable).
Searcher and Indexer are provided as interfaces that may be extended to
expose additional functionality to your application in a generic fashion.
Result and ResultImpl have been split into an interface and an
implementation in order to hide the add(Result) and replace(Result, Result) methods as well as to provide additional flexibility for alternate
implementations. One such alternate implementation would also implement
DisplayTag's PaginatedList.
AbstractResult is made available as a base class suitable for extension by
objects that implement Result and are not required to extend anything else.
Batch Indexing
BatchIndexer extends the Indexer interface by introducing three methods:
flush() (to be implemented by the indexer), setBatchMode(boolean), and
isBatchMode(), of which the latter two are provided by the indexing
infrastructure. flush() is executed during close() immediately before the
index is optimized. A typical implementation calls flushDeletes(). which
flushes any document deletions that had previously been queued (rather than
flushing them immediately, as in a non-batch indexer). The hybrid example
below demonstrates a BatchIndexer in action.
Limitations
AbstractSearcher does not yet support default field arrays as arguments to
the various doSearch() methods.
The @DefaultFields and @Excerptable annotations are only available on objects
that implement Searchable. Ideally, they would be available for any @Result,
as there's nothing that necessarily limits their use to _Searchable_s.
Fields that are stored in the index are returned in a storedFields Map
(specified in the Result interface) regardless of whether or not the names
correspond to properties on the object being returned. In the future, Lucene
could be used as a persistence tool of sorts by reconstituting the object as
much as possible. Thus, if all properties of an object were indexed (and
stored), the object could be fully reconstituted and no external persistence
mechanism would be necessary. The trade-off is index size, so this would not
be feasible for large datasets.
Examples
Simple Example
This example demonstrates how to use Searchable without using the annotations.
AddressIndexer.java:
/**
 * Indexes Addresses.
 */
public class AddressIndexer extends AbstractIndexer implements Indexer {
    public AddressIndexer() {
        // use /tmp/addresses as the index path
        setIndexPath("/tmp/addresses");
    }

    public void add(Address address) throws IndexingException {
        // create a document with "address" as the type
        Document doc = createDocument( "address", address.getId() );

        // add fields for each property of Address
        // implementation of Address is left to your imagination
        doc.add( Field.UnStored("street", address.getStreet() ) );
        doc.add( Field.UnStored("city", address.getCity() ) );
        doc.add( Field.UnStored("state", address.getState() ) );
        doc.add( Field.UnStored("zip", address.getZip() ) );

        // save the document to the index
        save( doc );
    }

    public void delete(Address address) throws IndexingException {
        delete( "address", address.getId() );
    }
}
AddressSearcher.java:
/**
 * Searches Addresses.
 */
public class AddressSearcher extends AbstractSearcher implements Searcher {
    public AddressSearcher() {
        // use /tmp/addresses as the index path
        setIndexPath("/tmp/addresses");
    }

    public ResultSet search(String query) throws SearchException {
        return doSearch( query );
    }
}
AddressIndexTest.java:
// ...
public void test() throws Exception {
    AddressIndexer indexer = new AddressIndexer();
    indexer.add( makeAddress() );
    indexer.close();

    AddressSearcher searcher = new AddressSearcher();
    ResultSet rs = searcher.search("city:Cambridge");
    Result result = rs.iterator().next();
    // Address does not implement Result, nor was it indexed with its
    // fully qualified class name, so the result is a GenericResult
    assertTrue( result instanceof GenericResult );
    GenericResult gr = (GenericResult) result;
    assertEquals( "address", gr.getType() );
}
Simple Annotation-Driven Example
This example demonstrates the basics of annotation-driven indexing.
SearchableBean.java:
/**
 * Implementation of Searchable to be indexed.
 *
 * Extends AbstractResult to avoid needing to implement Result methods
 * (inherited from Searchable).
 */
public class SearchableBean extends AbstractResult implements Searchable {
    private Integer id;
    private String name;

    public SearchableBean(Integer id, String name) {
        this.id = id;
        this.name = name;
    }

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    @Indexed
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}
IndexManager.java:
// ...
public void index(SearchableBean bean) throws IndexingException {
    BeanIndexer bi = new BeanIndexer();
    bi.add( bean );
    bi.close();
}

public ResultSet search(String query) throws SearchException {
    BeanSearcher bs = new BeanSearcher();
    // searching will attempt to reconstitute objects and set their ids
    // this allows the calling layer to know a) the type and b) the id
    // in order to refresh it properly
    return bs.search( query );
}
IndexManagerTest.java:
// ...
public void testSearch() throws Exception {
    IndexManager im = new IndexManager();
    im.index( new SearchableBean() );
    ResultSet rs = im.search("name:seth");
    assertEquals( 1, rs.size() );
    Result result = rs.iterator().next();
    assertTrue( result instanceof SearchableBean );
    assertEquals( 1, result.getId() );
    // name has not been set
    assertNull( result.getName() );
}
Hybrid Example
This example demonstrates how to use Searchable in a hybrid mode.
TeapotIndexer can operate in batch mode.
Teapot.java:
/**
 * A representation of a teapot.  When searching the Teapot index, only the
 * "name" and "description" fields will be searched by default (if
 * @DefaultFields were not specified, all indexed fields would be searched).
 */
@DefaultFields( {"name", "description" } )
public interface Teapot extends Searchable {
    /**
     * Name of the Teapot.  This serves as the id, is indexed, and can be
     * sorted by.  This is the only field that is indexed when a Teapot is
     * nested in another object (a TeaSet, for example).
     */
    @ID
    @Indexed
    @Sortable
    public String getName();

    /**
     * Color of the Teapot.  Aliased to "colour" for Brits.
     */
    @Indexed(aliases="colour", nested=false)
    public String getColor();

    /**
     * Material the Teapot is made of.  Indexed and stored.
    @Index(stored=true, nested=false)
    public String getMaterial();

    /**
     * The type of tea this Teapot contains.  Not searchable, but stored.
     */
    @Stored(nested=false)
    public String getTeaType();

    /**
     * Description.  Indexed and used when creating an excerpt.
     */
    @Indexed(nested=false)
    @Excerptable
    public String getDescription();
}
TeapotIndexer.java:
/**
 * Indexes teapots.
 */
public class TeapotIndexer extends AbstractBeanIndexer implements BatchIndexer<Teapot> {
    public TeapotIndexer() {
        // use /tmp/teapots as the index path
        setIndexPath("/tmp/teapots");
    }

    public void add(Teapot tp) throws IndexingException {
        Document doc = doCreate( tp );

        processBean( tp );
        postProcessTeapot( tp );

        // save the document to the index
        save( doc );
    }

    public void delete(Teapot tp) throws IndexingException {
        doDelete( tp );
    }

    public void flush() throws IndexException {
        // flush any pending deletes
        flushDeletes();
    }

    protected Document postProcessTeapot(Teapot tp) {
        // add an "owner" field not specified by the teapot
        doc.add( Field.UnStored("owner", "Nathan") );
    }
}
TeapotSearcher.java:
/**
 * Searches teapots.
 */
public class TeapotSearcher extends AbstractSearcher implements Searcher<Teapot> {
    public TeapotSearcher() {
        // use /tmp/teapots as the index path
        setIndexPath("/tmp/teapots");
    }

    public ResultSet<Teapot> search(String query) throws SearchException {
        return excerpt( refresh( doSearch( query ) ) );
    }

    protected ResultSet<Teapot> load(ResultSet<Teapot> results) {
        ResultSetImpl rs = (ResultSetImpl) results;
        for (Teapot tp : results) {
            // session is a HibernateSession
            // your actual implementation may involve a DAO
            // replaces stub Teapot with a fully loaded one
            // ResultSetImpl handles transferring of Result properties
            rs.replace( tp, (Teapot) session.load( tp ) );
        }
        return results;
    }
}
TeapotTest.java:
// ...
public void test() throws Exception {
    BatchIndexer<Teapot> indexer = new TeapotIndexer();
    // run TeapotIndexer in batch mode
    indexer.setBatchMode( true );
    indexer.add( makeTeapot() );
    indexer.close();

    Searcher<Teapot> searcher = new TeapotSearcher();
    ResultSet<Teapot> rs = searcher.search("material:china OR wireframe");
    Iterator<Teapot> i = rs.iterator();
    Teapot firstResult = i.next();
    Teapot secondResult = i.next();
    assertEquals( "china", firstResult.getMaterial() );
    assertTrue( secondResult.getDescription().contains("wireframe") );
    assertTrue( secondResult.getSearchExtract().contains("wireframe") );
}
Searching Multiple Indexes Example
This example demonstrates how to use Searchable to search multiple indexes for
different types of objects. This assumes a 1 class/index breakdown, but that's
not strictly necessary.
AddressAndTeapotSearcher.java:
/**
 * Searches Addresses and Teapots.
 */
public class AddressAndTeapotSearcher extends AbstractMultiSearcher implements Searcher {
    /**
     * Constructs this as a MultiSearcher for Addresses and Teapots.  The order and length
     * of both arrays must be equivalent.
     */
    public AddressAndTeapotSearcher() {
        super( new String[] { "/tmp/addresses", "/tmp/teapots"},
               new Class[] { Address.class, Teapot.class } );
    }

    /**
     * Searches the teapot index using default fields determined by @DefaultFields
     * annotation on the Teapot.  Also searches the address index using all available
     * fields as defaults.  If Address implemented Searchable and contained a
     * @DefaultField annotation, those fields would be used instead.
     */
    public ResultSet search(String query) throws SearchException {
        return doSearch( query );
    }
}


Sprout: Annotation-Powered Simplicity for Struts
2005-11-30T00:00:00Z
Overview
Sprout aims to significantly simplify development with Struts by reducing the
amount of configuration required through the use of annotations and sensible
defaults.
Basics
Sprout requires JDK 1.5. Without that, you're out of luck.
Sprout is an extension of a Struts MappingDispatchAction, which allows for
multiple actions to be defined within the same Action class. In this case,
the name of the method is mapped to the URL (after converting method names;
methodName is exposed as /method_name.do). Paths are determined based on
the package name of the action. For example,
net.mojodna.sprout.action.HomeAction corresponds to /, while
net.mojodna.sprout.action.example.ExampleAction corresponds to /example/.
Sprout also uses a custom RequestProcessor--SproutRequestProcessor, which
extends Spring's DelegatingRequestProcessor. This means that you can specify
dependencies within your actions using setter-injection.
Sprout is also completely backward-compatible with legacy Struts
applications.. It was built for use in a legacy Struts application; many of
the older Actions are untouched--new development is done using Sprout (I often
find myself removing action-mappings while adding new functionality).
Sprout is available on GitHub here:
https://github.com/mojodna/sprout Do with
it what you will.
Annotations
All annotations are optional.
@FormName
Allows the developer to override the name of the form-bean (defined in
struts-config.xml) used for this method. This is equivalent to setting the
name attribute within an action mapping.
Defaults to ${action-name}Form; e.g. for AdminAction the default
ActionForm name would be AdminActionForm.
@Forward
Specifies additional forwards. Multiple forwards may be specified by providing
arrays as arguments to name, path, and redirect. redirect defaults to
false.
A default redirect is provided; the key is Sprout.FWD_SUCCESS and the path
is the converted path + .jsp. E.g., AdminAction.methodName() corresponds to
method_name.jsp.
e.g. @Forward(name="failure", path="/failure.jsp" redirect="true")
@Input
This annotation is required if this action is validating the output from a
different action. In that case, the argument to @Input should be the path to
the JSP containing the form whose input is being validated.
e.g @Input("login.jsp") if this is not login() and the action that
initiated this request is login().
@Scope
Specifies the scope attribute for the generated action mapping. This exists
primarily for completeness; it is likely that you may never use this
annotation.
As with struts-config.xml, the default is request.
@Validate
Specifies the validate attribute for the generated action mapping. Set this
to true if you desire the output of this action to be validated. For this to
have any effect, you must have specified rules in validator-rules.xml.
Sprout does not contain anything to ease the actual validation process at
this time.
Example
src/java/net/mojodna/sprout/action/example/ExampleAction.java:
// URL should be /example/*
package net.mojodna.sprout.action.example;

// ...

public class ExampleAction extends Sprout {
  // overrides Sprout.index()
  public ActionForward index( ... ) {
    // do something

    // redirect to index.jsp
    return mapping.findForward( FWD_SUCCESS );
  }
}
src/java/applicationContext.xml:
...
<bean name="ExampleAction" class="net.mojodna.sprout.action.example.ExampleAction" singleton="true" />
...
src/web/WEB-INF/struts-config.xml:
...
<form-bean name="ExampleActionForm" type="org.apache.struts.validator.DynaValidatorForm">
  <form-property name="..." type="..." />
  ...
</form-bean>

<!-- No action-mappings!!! -->
<action-mappings />

<!-- Define an alternate RequestProcessor -->
<controller processorClass="net.mojodna.sprout.SproutRequestProcessor" />

<!-- Sprout plug-in -->
<plug-in className="net.mojodna.sprout.SproutAutoLoaderPlugIn" />
...
Shorthand
Index Actions
Sprout contains an index() method to speed up the process of getting
something working. To use this, subclass Sprout (no methods necessary),
register the action-bean in applicationContext.xml and create a
corresponding index.jsp. When you need to add logic to the action, override
index() in your Sprout sub-class and add it there.
DynaActionForms
Helper methods have been added to ease development using DynaActionForms.
String key = "foo";
String value = "bar";
// returns a String
f( key ) == ((DynaActionForm) form).getString( key );

// returns an Object
F( key ) == ((DynaActionForm) form).get( key );

// sets a value
s( key, value ) == ((DynaActionForm) form).set( key, value );
ActionMessage handling
Sprout contains adaptations to the traditional way Struts handles
ActionMessages. getMessages() and getErrors() have been modified to
store and retrieve messages from the session rather than the request. This
means that messages and errors will be displayed (and subsequently cleared) on
the next invocation of <html:messages /> or a variant (such as
<ui:notifications />), regardless of whether either of the get methods
have been called or if the invocation occurs during a separate request.
Sample message / error handling code (within an Action):
ActionMessages msgs = getMessages( request );
ActionMessages errors = getErrors( request );

// Add a message
msgs.add( ... );

// Add an error
errors.add( ... );

// Save messages and errors
saveMessages( request, msgs );
saveErrors( request, errors );
<ui:notifications /> (src/web/WEB-INF/tags/ui/notifications.tag) is an
alternative tag file that can be modified for your use. The primary difference
between <html:messages /> is that it will display both messages and errors
(and will discriminate between them, allowing you to style them differently
depending on your application's needs).
Servlets and Taglibs
Spring's Struts integration lacks the ability to autowire servlets and
taglibs. Sprout contains classes (Sproutlet and SproutTag) that can be
subclassed to provide auto-wiring capability. They will be wired during their
initialization process.
These support classes only support the byName auto-wiring mechanism.
Q + A

Q: Why the Spring dependencies?
A: I'm already using Spring. There's a good chance that you are as well.
Removing the dependency makes reflection upon the registered actions (or
finding them in the first place) significantly more difficult.



Introducing Sprout - Pure annotation goodness
2005-10-31T00:00:00Z
Summary: Sprout. Check it out.
I have a Struts application that I work on every
day. It has been a Struts application for about 3 years. It will likely
continue to be a Struts application for the foreseeable future (it's now 2009
and it's a Rails app). I imagine some of you may be in the same position.
Ever since I started working on web applications in Java, I've been poking
around the other web frameworks that are out there.
WebWork is interesting (especially now
that it's to merge with Struts). Spring MVC is
promising, if only because I like having my services wired together and to my
actions. Tapestry seemed too complex,
and SiteMesh, while not a framework,
gave me everything that Tapestry was promising (granted, I haven't looked at
Tapestry a whole lot and should probably have another look).
Wicket and especially
JSF, ick. The web is not a client
application and does not lend itself well to the Swing approach to components
and widgets (yet). JSF, in particular, strikes me as great for both sides of
the Toolbuilder / Application developer divide, but the overhead involved in
creating custom components (a requirement for me, as I still like reinventing
my HTML constructs) is simply too great. JSF in its optimal form also takes an
event-based view of the web, which just doesn't work so well yet (POSTs
everywhere!). This isn't about Java frameworks; this is about Struts.
Enter Ruby on Rails. Nifty. It fits the small
development team model much better (I am a development team of 1; Struts and
its ilk assume and thrive in an environment where tasks can and are divided
up, such as the one my app was originally written in). It was not created as a
framework, but more extracted from an application because it worked and made
its creator's life simpler. Convention over configuration (rather than
conventions AND configuration). Embedded opinions that just happen to jive
with my own view of the world. Looking at Rails and gaining an understanding
of how it fits together changed the way I think about web application
development. But this isn't about Rails.
I have a Struts application that I work on every day. I have a minor Rails
application that I rarely work on. I like Ruby. But I also use Java.
I recently spent some time focused on making Struts work for me rather than
accepting the tedium of its status quo. Out grew a
Sprout. The extract of something that
makes my life easier. It might do the same for yours. Let me know.
Update: Richard Harms took this work a couple years after I abandoned it
and maintained it for a year or so as a Google Code
project. It lives on GitHub for
posterity (and portfolio) and includes the work that Richard contributed.


Ongoing MBTA Work
2005-04-21T00:00:00Z
Update: I haven't worked on this in many moons and it has ceased to work.
Here is a currently working MBTA map.
The coverage area has increased to cover Boston, Charlestown, South Boston,
and Brookline (the 4 downtown maps). Rather than creating the slices for tiles
by hand in Photoshop, I whipped up a PHP script to do it for me -
slicer.php (requires GD w/ GIF
support, an increased memory_limit in php.ini, and a GIF version of full.psd
(same dimensions)).
I added zoom level 1 by importing the PDFs as 3500px wide images and providing
alternate coordinates to the slicer. Hint: it's much faster to work with them
after converting them to Indexed Color (via RGB, as they get imported as CMYK)
(and obviates the need to Export to Web).


Continuing to map the MBTA
2005-04-20T00:00:00Z
Update: I haven't worked on this in many moons and it has ceased to work. Here is a currently working MBTA map.
Ok, so it's not that useful right now, since it only
covers a small part of Cambridge at one zoom level. And since I don't really
have the time to keep working on it right now, I'll post what I've got so
anyone else can continue.

Maps in PDF
Tiles so far (x=5378, y=-740 (top left) to x=5386, y=-733 (bottom
right)) (these are 128x128
each)
Tile server
Greasemonkey script
(change mbta.baseURL to your own copy of the tiles)
Cambridge/Brookline/Charlestown/Boston, rasterized and stitched together
(12MB)

The 4 downtown maps work best when rasterized to a 1710px wide canvas (before
cropping) to create the tiles for zoom=2. I experimented with different sizes
and even figured out the pixel/mile ratio (more below), but that didn't help.
The PDFs are nice enough that it should be possible to create tiles for
multiple zoom levels.
The MBTA also has a nice Trip
Planner that could potentially be
scraped to provide transfers on the sidebar (where driving directions would
usually be placed). It appears to use lat/lon pairs internally, which is
helpful.


MBTA Maps
2005-04-19T00:00:00Z
Update: I haven't worked on this in many moons and it has ceased to work.
Here is a currently working MBTA map.
Following on the heels of Adrian Holovaty's work
with Chicago Transit
Authority maps, I dug
around on the MBTA's website (Boston's T) for some
decent maps.
These are very clean and to-scale, so they were ripe for a Boston version.
After some fiddling around, I discovered that the Cambridge / Somerville inset appears to correspond with Google's tiles when the PDF is imported into Photoshop with a width of 1710px. After some slicing and dicing, I give you:

This is a screenshot of the Greasemonkey
version (with
minor alterations to Adrian's).
Building on top of some experimentation I'd done last week with Google Maps
stand-alone, I made a
non-Greasemonkey version by adding in a hook that effectively runs the
Greasemonkey script. The default zoom-level is wrong, and my quick and dirty
tile-serving code uses redirects (slowing things down and not handling missing
tiles gracefully), so I'll post that later once I get a chance to clean it up.
Update: I don't have enough time to keep going on my own, so I've posted
my work and a basic knowledge
dump.
Update: Ok, so I found a bit more time and added zoom=1 and zoom=3
(the latter with the bigger map, thus covering the suburbs, though the North
Shore is a bit off). Adding additional zoom levels (zoom=4, zoom=5?) is
probably worthwhile to get more of an overview. Beyond that, what's next is
probably merging maps together to increase the coverage area and using the
insets for additional zoom levels.

Digressions of a Drive-by Hacker

Resolved: GDAL on AWS GPU Instances

make For Data Using Makefiles

Common Idioms

Use of Exit Codes

|| and &&

> /dev/null 2>&1

psql Checks

Looking for Relations

Looking for Extensions

Looking for Functions

Implicit make Variables

make Functions

make Patterns

Annotated Makefile

KVM, libvirt, and Ubuntu 14.04

Binary Streaming with Hadoop (and Node.js)

Create Some Typed Bytes

Prepare a SequenceFile

Read and Write SequenceFiles

Read Text and Write SequenceFiles

Output Binary Files

DIY Fusion Drives: A Data Point

Building Blocks of Real-Time Distribution

The OS X Spatial Stack

Xcode

Homebrew

Homebrew-provided Libraries

KyngChaos Frameworks and Binaries

Mapnik

Quantumnik

Python

PIL

matplotlib

numpy / scipy

Cascadenik

OpenStreetMap

PostgreSQL Additions

GUI Utilities

Congratulations

CSSpring Cleaning

Things to Improve

Exploring OAuth-Protected APIs

Enter oauth-proxy

Access Tokens Sold Separately

A Concrete Example

Subscribing to Wordpress.com with Switchboard

Switchboard as a Framework

A Simple Client

Jacks and Hooks

Standard Jacks

A More Complex Example, with Pandas

That's it for now

switchboard : XMPP :: curl : HTTP

Quick Start

"curl for XMPP?"

Switchboard background

What to do?

Configuration

Roster Manipulation

Probing and Discovery

PubSub

PEP (Personal Eventing Protocol)

More

Getting Help

Extending

How to Help

Updating Ruby Consumers and Providers to OAuth 1.0a

Updating Ruby OAuth Consumers to 1.0a

Updating Ruby OAuth Providers to 1.0a

Accepting oauth_callback During the Request Token Phase

Generating An oauth_verifier During the Authorization Phase

Verifying Access Token Exchanges

Summary

An Idiot's Guide to OAuth 1.0a

What Happened?

So, What Changed?

Services Supporting OAuth 1.0a

My (public) Git Workflow

Let's Go

`||` and `&&`

`> /dev/null 2>&1`

`psql` Checks

Implicit `make` Variables

`make` Functions

`make` Patterns

Annotated `Makefile`

Prepare a `SequenceFile`

Read and Write `SequenceFile`s

Read Text and Write `SequenceFile`s

Enter `oauth-proxy`

Accepting `oauth_callback` During the Request Token Phase

Generating An `oauth_verifier` During the Authorization Phase