GeoWeb 2010: Building Blocks of Real-Time Distribution
Hi. If you're reading this, there's a good chance that you saw my talk from
GeoWeb 2010 about the "Building Blocks of Real-Time Distribution." I made
reference to a number of pieces of technology where it may not have been
obvious where to look for more information. This should help that.
The presentation itself is here: Building
Blocks of Real-Time Distribution (PDF)
- My current employer. Our goal is to provide simple tools for those not
versed in traditional GIS to help manage data with associated location
- The venerable photo sharing site where I used to work.
- Fire Eagle
- Yahoo!'s platform for managing and brokering user location. I
Streams (XMPP PubSub) and helped specify OAuth-over-XMPP while serving
as the development lead.
- Provider of real-time transit information in partnership with local
- Location-based mobile social networking game that allows people to
"check in" to venues (bars, restaurants, parks, etc.), share information
about them, and compete for status.
- Similar to Foursquare.
- An instant messaging protocol designed for machines. Also known as
Jabber; this is what Google Talk uses. Well-specified for many uses, but
overwhelming. Its ability to track presence is one of its strongest
- IETF-defined XML syndication format with support for namespaces, etc.
Successor to RSS.
serializers in most languages. Frequently used in web applications due to
- The Geography Markup Language. XML-based and exceedingly thorough; many
uses of GML refer only to a constrained subset.
- Created for Google Earth, standardized for the rest of us. Both
represents data and defines rendering rules for it. XML-based.
- Intentionally stripped-down XML format for representing OGC Simple
Features, originally intended for use with RSS, but will work with any
- JSON equivalent of GeoRSS; even more simplistic (due to lack of support
for namespaces), yet still sufficient for representation of Simple
- Atom Streaming
- Publish Atom content over HTTP; connections remain open as long as
possible. Twitter and LiveJournal do this.
- XMPP PubSub
- Publish Atom (or other) content over XMPP using a publish-subscribe
model. “Nodes” can be published and subscribed to at will. Fire Eagle does
this (by the name of “Location Streams.”) SuperFeedr supports both
publishing and distribution.
- Web Hooks
- HTTP callbacks; when new data is available, it is POSTed to a
pre-configured URL. SimpleGeo supports this for re-publishing changes to
- Further clarification and definition of Web Hooks, including support
for deletion. Most Google services support this in some fashion. SuperFeedr acts as a hub and will
transform between different formats and protocols (XMPP).
- Flash Sockets
- Socket connections made available to a Flash application within a web
browser. Can be used to implement clients for multiple client-initiated
- Web Sockets
- Native socket connections for web browsers. Defined as part of HTML5
and only available in modern browsers (Firefox, Chrome, Safari).
- If you need a web server that will handle highly concurrent
long-lasting connections, Apache and IIS won’t cut it, as they have lower
limits on the number of clients connections they can hold open at one time.
Nginx tends to be faster in these circumstances.
excited about, based on Google’s V8 virtual machine (which powers Chrome).
Writing asynchronous network services is more straightforward than in other
- A lightweight HTTP server for Java, suitable for long-lasting
- An event-driven network framework for Python. It’s almost a world unto
itself, which ups the learning curve, although performance is quite
- Effectively the reference implementation for XMPP. Custom modules can
be built to support additional functionality, but they must be written in
- For our purposes, Prosody’s main differentiator is that it’s written in
Lua, a lightweight scripting language
often used in video games. This means that it can be extended fairly easily
to implement application-specific functionality.
- XMPP Component Protocol
- An alternative to connecting to an XMPP server as a client; allows
custom code to respond to XMPP events according to custom business
- OAuth-over-XMPP (XEP-0235)
- OAuth defined for XMPP PubSub subscription management. Used as an
alternative to trusting JIDs and associating them directly with
- Twisted Words
- An XMPP layer for Twisted. Can be run as a stand-alone server or as a
component with an external XMPP server.
- Wokkel / Idavoll
- A pair of enhancements to Twisted Words that make it easier to
implement PubSub services.
- An NIO (event-driven) network framework for Java.
- Event-driven networking framework for Ruby. Conceptually equivalent to
Node.js, Netty, and Twisted.
- C library for event-driven networking. Most of the frameworks for
higher-level languages use this.
- HTTP Authentication
- Traditional username / password style authentication for HTTP.
- Standardized token-based delegated authentication. Users can authorize
third parties to act on their behalf, allowing granular revocation if and
when necessary. Often used for securing web APIs these days.