Being away in India (more about that later) gave me time to think about my favorite tensions. Each is a pendulum -- every decade or so it seems like attitudes swing the other way.

Here are my top four tensions; have any other good ones?

1. Hardware vs. Software

   The mother of all tensions in our industry, no two moments exemplify the pendulum swing better to me than Transmeta's ballyhooed launch followed by Intel's crushing advancements in power-saving circuitry.

2. Web vs. Native

   This wonderful tension makes even the smartest of the large players schizophrenic: Microsoft devotes huge resources to advancing both IE and .NET and shows a meta-layer of confusion with Silverlight; Apple launched iPhone development web-app-only but eventually succumbed to native needs; GOOG has Android and Chrome OS to stir up internal confusion. HTML5 brings us to the point where some types of apps are just better as web apps; other app categories may never succumb to the web approach.

3. Thin vs. Thick Client

   The subset of cloud computing I think of as utility computing -- arbitrary compute resources available programmatically and on demand -- and the rise of mobile as the next decade's great charge makes me think that the pendulum is heading back towards thin, with the proviso that thin is a lot thicker than it used to be.

4. Centralized Service vs. Distributed Protocols

   Centralized services like Twitter and Facebook have all the traction; this should be no surprise given how well they reduce the complexity of getting started, and how quickly they provide network benefits. It's always tough, however, to balance the needs of users with the needs of business. I predict in several years we'll see a flourishing ecosystem of protocols for social presence -- right now, the pendulum is firmly in the centralized services' court.

So what happens when you want to generate an equivalent timestamp somewhere that you're not using UIEvent?

I did a little digging; here's what I came up with:

#import <mach/mach.h>
#import <mach/mach_time.h>

+ (NSTimeInterval)timestamp
{
    // get the timebase info -- different on phone and OSX
    mach_timebase_info_data_t info;
    mach_timebase_info(&info);

    // get the time
    uint64_t absTime = mach_absolute_time();

    // apply the timebase info
    absTime *= info.numer;
    absTime /= info.denom;

    // convert nanoseconds into seconds
    return (NSTimeInterval) ((double)absTime / 1000000000.0);
}

This will give you NSTimeIntervals that are meaningfully comparable to those collected from UIEvent instances.

1. Music For 18 Musicians (ECM)

   Generally considered Reich's first masterpiece, 18 Musicians broke Reich away from the strictly "minimal" and process-oriented pieces he had previously written.

2. Tehillim (ECM)

   A setting of Hebrew text to stunningly original orchestration, Tehillim is the piece that made me a life-long Reich devotee.

3. Electric Counterpoint

   Counterpoint is Reich's multi-layered masterpiece written for, and performed by, Pat Metheny. You might recognize it as an eternal source of samples, notably first with The Orb's Little Fluffy Clouds and later from several RJD2 backing tracks.

4. Different Trains

   On the same CD as the previous, Different Trains was one of Reich's earliest (and greatest) experiments with composing around the pitches present in human vowel sounds.

5. City Life

   A five-movement piece built around field recordings taken by Reich on the day of the original World Trade Center bombing, City Life is strangely beautiful and affecting.

6. The Desert Music

   Perhaps one of Reich's most difficult pieces, The Desert Music is complex, at turns claustrophobic and expansive, and worth many repeat listens.

7. You Are (Variations)

   Written while Reich was approaching 75 years of age, I think You Are is one of Reich's later masterpieces.

8. Early Works

   I've optimized this Reich ordering for easing into his music rather than studying him academically. That said, no Reich collection would be complete without a CD that highlights Reich's early philosophy of music as a gradual process.

Reich has written many wonderful pieces not included in this list, but I believe if you're serious enough about him to purchase these eight CDs you will probably end up like me and collect the rest. Enjoy!

Well, not quite the same. I've been a DreamHost customer for nearly a decade. I wanted something stupid cheap; I got something both cheap and stupid. To be fair, I'd been warned. The people behind DreamHost are friendly and responsive, so even though my MX records sometimes disappeared for a few hours, or my blog crumbled under even moderate load, or somehow an extra $7.5 million unexpectedly left customers' pockets... well, I stuck with it.

No more. Even for my modest business homepage I need a setup that is:

1. Reliable. My sites don't go down and they handle loads.
2. Repeatable. I can easily move between providers without complex setup or import/export processes.
3. Easy To Maintain. I have full source control over my sites and they use the simplest formats and tools possible.

As a result, I've moved my entire web presence into the "cloud." Specifically, I'm hosting all of my sites via git repositories; I'm hosting most of them via GitHub Pages.

For the majority of my sites, the move to GitHub Pages was painless. The exception, not surprisingly, was this blog: it is now built with Jekyll. Jekyll is a great tool and I highly recommend it. What hurt was exporting from WordPress. I ended up writing several custom python tools; Jekyll's built-in exporter was next-to-worthless. In the interest of time, I dropped the old comments (there weren't many.) Now that I've made the leap, I'm extremely excited to author my blog from emacs, in markdown. It's far easier for me than fighting the WordPress text entry area.

My remaining sites, like Go and WhereBeUs, are hosted via Google App Engine. (Of course, the code is managed in git, too.) Only my large MP3 files have escaped source control; they're hosted on Amazon S3. (Alas, git's performance with large binaries is not too hot.)

Finally, I've moved to Zerigo Managed DNS to handle the growing complexity of my domain name needs. In addition to having a great control panel for hand-editing DNS entries, Zerigo offers a killer REST API that is well-supported by both Python and Ruby. I couldn't be happier.

As for mail, I'm in the process of moving to hosted gmail -- if over the weekend you get a bounce, you'll know what's up.

Wondering when your friends are going to show up for coffee? Pop open WhereBeUs and you'll see exactly where they are right now.

There are similar applications, but Brian and I think we've built the simplest, most minimal such application. WhereBeUs gets your friends from Facebook and Twitter, so there's no need to create new accounts or set up new friendship connections. After you've signed in, you'll instantly see your other WhereBeUs-using friends.

This was a fun side project. It took a few weeks of spare time to build. We've decided to make it free and open source. The code is available on GitHub. Right now the front-end is iPhone only and the back-end is App Engine + Django. Despite the apparent simplicity, we've done a lot of work to ensure that the application scales to millions of users and tens of thousands of friends per user. We hope that our developer friends will contribute cool new features. We're especially excited about building cool new Android and Palm Pre front-ends.

So: check out WhereBeUs today and let us know what you think!

Favorite Music Released in 2009

A Certain Distance by Lusine. Lusine is a Seattle-based producer of what might commonly be termed minimalist micro-techno. In the past few years, however, his sonic palette has grown to encompass things warm, fuzzy, and beautiful. A Certain Distance is no doubt Lusine's breakout release; it is stunningly gorgeous and has been on heavy rotation for me since it arrived.

Central Market by Tyondai Braxton. This album is what happens when you take Copland-esque classical frivolity and mix it with any sound you can possibly imagine. Completely nuts. Zappa would approve.

Embryonic by The Flaming Lips. I know that the Lips' output is terribly inconsistent, but I'm a sucker for them nonetheless. This new release is a hefty two disc set and it is noisy, dark, brooding, and nasty. It's easily my favorite Lips release of the decade.

Historicity by Vijay Iyer. Iyer continues to bang away at the keys in time signatures you can't even count when sitting on your couch, let alone conceive of improvising on top of. There's a certain calculating coldness to Iyer's work which some might find off-putting; he has never quite struck balance with his emotional side. But with so much virtuosity on display, it's hard not to love and respect this impressive album.

Them Crooked Vultures by Them Crooked Vultures. I have no idea why this didn't make more year-end lists. What's not to love about Grohl, Homme, and JPJ getting together to rock the fuck out? They do so with raw power. This is Zep meets Queens Of The Stone Age meets 2009. Do not be afraid.

Vertical Ascent by Moritz Von Oswald Trio. In this album, minimalist composition (in the sense of Reich, Riley, and Glass) meets experimental jazz musicians. In some ways Vertical Ascent feels like a response to the work of Nik Bartsch's Ronin. It's great late-night music.

Favorite Music Discovered in 2009

Artifacts by Aether. This album is mixed as if the sound of compressor pumping is a gateway to Nirvana, but never-mind. Aether matches hip-hop influenced electronic grooves with music so beautiful-but-not-cheesy that it brings a tear to the eye.

Good & Evil Sessions by The Blue Series Continuum. This album was released many years ago but still lives in obscurity. It sounds to me as if acid jazz went underground for a few decades and re-emerged fully formed, much to the surprise of the rest of the jazz world.

Loud Louder Stop by The Neil Cowley Trio. Neil Cowley is a London-based pianist who works all day, every day, to put the rock back in staid jazz formats. Unlike other rock/jazz acts like The Bad Plus, Cowley actually knows how to play the piano. The tunes on this CD are in turns beautiful, whimsical, and just plain fun.

Whoops, I Forgot One.

Update on January 4, 2010: Whoops. I went back and looked at my iTunes play counts and, sure enough, I forgot one of my most listened-to albums of 2009:

Listening Tree by Tim Exile. Exile's latest effort is full of willfully crafted crazy-making. It evokes nothing so much as crack clowns and dancing bears in hell. Does that not help? Listening Tree has also been described as Exile's "likely failed" attempt at making a one-man, new-wave, abstract techno musical about groupthink and capitalism. Um, so, yeah. Just go listen to the samples on Amazon and admit that you've never heard anything like it, but you want to hear more.

The basic story is simple: you call an API that manufactures a POST URL. App Engine processes the POST, stripping out all file content and creating blobs (which are opaque) and BlobInfo objects (which are read-only datastore entities.) App Engine then calls your handler, substituting blob keys for the original file content in the POST data. When you're ready to serve a blob, you simply include the appropriate key in a response header; App Engine traps for this and serves your data as fast as it can.

This is about what I'd expect, with two exceptions:

1. There is no (straightforward) way to access blob contents from within your handlers. On the one hand, this protects naive developers who might attempt to serve blob bytes directly. On the other hand, this prevents developers from performing custom processing.

   In order to get at blob bytes (for example, to resize an image) it seems that you must URL fetch the blob, perform your transform, and URL POST back. This doesn't seem as straightforward as it could be; it also doesn't seem to be optimal from a performance perspective -- though this is hard to judge without knowledge of the blobstore's internal architecture.

2. I'm uncertain how cleanly the POST/redirect model will integrate with Django's forms model. It should be easy to create a new AppEngineBlobStoreField for forms, but the catch comes when some other part of the form fails validation. By the time your form handler is invoked, blobs have been created and stored. If the form does not pass validation, you'll have to manually remove the blobs and try again. (Or do something nicer, like keep the blobs around across multiple presentations of the form.)

All in all, this appears to be a promising start for the service. I look forward to getting those pesky screenshots out of City-Go-Round's datastore and into its newly minted blobstore!

City-Go-Round is a website where visitors can find public transit applications (apps) that work in their city. Visitors also see a list of which transit agencies make their data publicly available to software developers... and which agencies do not.

Did you know that there are 748 transit agencies in the U.S., only 84 of which provide their schedule data to software developers? Amongst the largest offending transit agencies are the MTA New York City, the New Jersey Transit Corporation, and the Metro-North Commuter Railroad Company in New York. These agencies collectively represent nearly 18 billion passenger miles of "unopened data."

We strongly believe that the next wave of innovation in transit will happen in software, and we believe that a key precondition for that wave to get rolling is for government agencies to make their data available to the public. To encourage openness, visitors to our web site can add their names to a petition for open public transit data in their city.

You can find out a lot more about City-Go-Round by visiting the site and perusing the about page; please check it out and let us know what you think!

Warning: Geek Talk Goes Here

As part of a push for openness in transit data and development, we've made the City-Go-Round web site 100% open source. It is licensed under the BSD license and the full source code is available at my CityGoRound GitHub repository. If you have improvements you'd like to make -- large or small, visual or back-end -- please fork our code and send pull requests my way. You might also want to join the Open Transit Data Google Group, where we discuss issues of open transit advocacy as well as specific technical issues with GTFS feeds and the City-Go-Round web site.

City-Go-Round is a Django + App Engine application. As I mentioned in an earlier post about Django and App Engine, picking the "right approach" for Django on App Engine can be a bit tricky. We decided to use the pre-installed version of Django 1.1 and we've been quite happy with that decision.

In my earlier post, I also mentioned that I had done work to support secure sessions and users on top of App Engine. I intend to write a full post about this code, but if you're curious about how it works you can find the code in the CityGoRound GitHub repository. Briefly, the key pieces of code include:

• securedictionary.py :: This code is a modified version of the Werkzeug Secure Cookie code. Instead of specifically dealing with cookies, my secure dictionary allows you to take a (presumably small) dictionary and turn it into a secure string that can't be edited without breaking a hash check.
• middleware.py :: This is where the magic happens. This file contains Django-style middleware for both secure sessions (AppEngineSecureSessionMiddleware) and basic user management (AppEngineGenericUserMiddleware.) Unlike standard Django users, you can essentially use any App Engine data store model you'd like to represent the user. Just make sure your model is imported as User into this file.
• context.py :: The appengine_user context processor ensures that the current user is part of all request contexts.

I'll write more about sessions and users on Django + App Engine soon, but hopefully this code will get you started.

I came to Dev Days with two expectations: that I would meet interesting and engaged software developers, and that perhaps half of the talks would stretch my brain in new directions.

Dev Days proved to be a great opportunity to catch up with engineering buddies I hadn't seen in a while. If you take just one thing away from this review, it should be that I had fun with people I already knew. However, the Dev Days format largely precluded meeting new kick-ass developers and entrepreneurs. The entire day was full of talks, with lunch the only real opportunity to connect with like-minded Seattleites. Awkwardly, lunch tables were "bucketed" into discussion topics. On the surface this seemed a clever idea, but at my table it artificially constrained an already strained conversation.

Talks proved to be Dev Days' greatest weakness -- and, given that Dev Days was all about talks, it's hard to feel positive about the full experience. It is very difficult, of course, to target a general audience with developer talks. By the end of the day, however, I realized that there was a format that could work for future Dev Days. Unfortunately, few speakers actually followed it. I believe the key to making a great Dev Days talk is to introduce a new set of abstractions and then prove their value via code and demo. Cory Lindley's jQuery talk did this to great effect; Steve Seitz's talk did too, though "code" for him was algorithmic intuition. Most other talks failed to meet this bar, or were structured as introductions to a topic. I don't believe introductions make a lot of sense at Dev Days. I'm happy to be challenged; even if I can't fully connect the dots, I'm excited to see what the state of the art of each language, framework, and algorithm can deliver.

Here's a brief review of each of the talks in turn.

Joel Spolsky kicked off the show with an entertaining discussion about simplicity and features in modern software. Unfortunately, his argument was broken on two levels: first, it made the implicit assumption that simplicity and power are at odds (they're not, as anyone who uses an iPhone or a Palm Pre can attest); second, he made numerous statements ( "if you add more features, you get more paying customers") for which he offered no direct evidence. Joel argued that "elegant" software finds the balance between these extremes, but later during the conference -- in an unwelcome bit of advertising I hadn't realized I'd paid for -- he inadvertently convinced me that FogBugz 7, while powerful, was anything butelegant.

Scott Hanselman stepped in with an overview of Microsoft's new ASP.NET MVC 2 framework, which seems to have grown by leaps and bounds since the last time I looked at it. I was happy to see him dive quickly into code, though I just as quickly realized that Microsoft still has a long way to go before it gets abstractions as clean as those found in, say, Django. The template approach he demonstrated seemed like bad architecture to me, though I'd have to dig a little deeper to pass true judgement. Scott has an entertaining stage presence, but he wore his status as ambassador from Microsoft awkwardly. What was with the passive agressive apologia-cum-fanboyism?

Rory Blyth's introduction to iPhone development started with a strong overview of principles, but fell off the rails the moment he left his powerpoint slides. It quickly became clear that he wasn't very experienced with XCode, Objective C ( "don't worry about the difference between methods and messages,") or Interface Builder. The talk was intentionally structured to make XCode look weak, and to then bring Mono Touch to the rescue. Now, I've never used Mono Touch -- and I am a huge C# and .NET fan -- but the continual crashes and ultimate inability to run his final demo made quick work of the argument I think Rory intended to deliver. Rory: the machismo-powered mower on your home page is genuinely hilarious. I worry it may have been turned off for an hour?

After lunch, Cody Lindley stepped in with what was, for me, the highlight of the day. I'm a prototype.js developer by training; learning jQuery has been on my to-do list for many moons. Cory gave an absolutely no-nonsense overview of the library. He clearly explained the core concepts. He gave strong arguments about jQuery's power and its aesthetically natural use of the Javascript language. He made no bones about the fact that Javascript developers should know their language and didn't bother to explain anonymous functions, closures, or Javascript's scoping rules. Cory also ran a great Q&A, which included a killer answer to a developer who asked why there was no class-like abstraction (as found in Prototype.js.) I was an instant jQuery convert.

Daniel Rocha from Nokia spoke well about Qt, the cross-platform UI framework. He showed some impressive demos of a single Qt app targeting Mac, Linux, Windows, and several smart phones. Unfortunately, most of the talk was a high-level architectural overview of Qt; there was little time devoted to getting to the metal. As someone who has played with Qt before, I was surprised that the words "signal" and "slot" never came up. I also think Qt has another hill to climb: the write-once, run-everywhere philosophy is fairly dead these days. Practice never matches promise. Rocha's talk said little here.

After a short break, Ted Leung came on stage to introduce the audience to Python. I've seen Ted give talks in town before, so I know he's an enthusiastic and influential supporter of the Python community. Unfortunately this didn't come across at all. As I mentioned above, I don't think that introductions are a good choice for Dev Days talks; Ted's talk was structured as a language overview of Python. I'm a day-in-day-out Python developer, but I've never been terribly fond of the syntax itself. There was no discussion about how Python is being used in new, cool, ways. Perhaps worst of all, Ted hadn't sanity checked his slides; keywords and strings in his colorized python listings were (literally) illegible.

Dan Sanderson of Google gave an overview of App Engine, a subject near and dear to my heart. Dan did a great job describing the benefits and limitations of App Engine as compared to roll-your-own web infrastructure. He dove into both Java and Python bootstrapping code, so it was easy to understand how the outermost interface between App Engine and developer applications functioned. Unfortunately, Dan didn't go into much detail about the technologies that make App Engine unique from a scalability perspective; no data store or task queue code was shown.

The day ended on a high note, with a fantastic talk by Steve Seitz from the University of Washington. Steve's research interest is in image feature extraction and matching algorithms; his work has led to Photosynth and, as he discussed during his talk, many new impressive projects. Steve's most ambitious current project involves taking every last photograph available on the Internet and reconstructing the entire world from them. There are many difficult algorithmic tasks associated with this ultimate goal, and Steve gave very high level intuition about a handful of the relevant algorithms. Steve's talk was a great way to end the day.

The underlying issue is that Django's relational data model is incompatible with App Engine's "BigTable" model. Features that depend on Django model APIs simply don't work on App Engine. This includes users, sessions, and the admin interface.

In this article I'll describe the three best approaches and then describe how I eventually made my choice.

Three forks in the road.

1. App Engine Patch.

   When you download and unzip App Engine Patch (AEP), you get a directory that contains all the config files and directory structure necessary to "start" a Python App Engine application. Included is a heavily customized Django 1.1 source tree. To keep your GAE application's static file count low, the Django sources are rolled up into a single zip file.

   The advantage of the AEP approach is that major Django features -- users, sessions, and admin -- work "out of the box" on App Engine. If you're a Django developer, you can move forward using the tools you know, including manage.py and unit tests.

   There appear to be a few disadvantages to AEP: (1) updates to mainline Django take a while to appear in AEP, (2) if you run into a bug, it is sometimes unclear whether Django or AEP is to blame, and (3) AEP uses the inefficient zipimport mechanism to load Django.

   In my opinion, zipimport performance is the real killer. AEP must unzip the Django source tree before it can handle a request. This is a slow process: it can take several seconds on the production App Engine infrastructure. The performance impact is partially mitigated by the fact that GAE caches the import for future requests. Unfortunately, it is impossible to predict how long the cache will last. If your app doesn't see many requests, GAE may decide to allocate zero CPUs to it until a new request comes in. As a result, "first" requests after a long pause are expensive. At the same time, if your traffic is spiking, GAE may decide to allocate several new CPUs to it -- each of which, presumably, will have to serve at least one "slow" request first.

   The AEP team is well aware of this issue. Unfortunately, there isn't a clean way forward just yet.

2. Google App Engine Helper For Django.

   Written by the App Engine team, this open source tool makes it easier to move a large Django-only codebase onto GAE. The key feature is the BaseModel class, which mimics Django's model API but is compatible with the GAE data store. There are also tools to get manage.py, unit tests, and so forth working. That said, you do not get admin, sessions, or users for free. If your application contains complicated queries -- and certainly if your application contains raw SQL -- then you'll have to do some work to re-factor your models regardless.

   As a booster while moving a Django app to GAE, the "helper" may provide some benefit. In my opinion, however, it doesn't confer much benefit to new applications.

3. Pre-Installed Django 1.x Libraries.

   Most people know that Django 0.96 is pre-installed with App Engine. Lesser known is that Django versions 1.0 and 1.1 are also available as part of the production GAE infrastructure. By default, if an App Engine application attempts to import django it gets 0.96. However, by calling App Engine's use_library(...) API, you can request the namespace get replaced with either version 1.0.2 or (recently, and not mentioned in all relevant places in the official documentation) 1.1.

   The great news about use_library(...) is that it is extremely fast. The bad news is that you're stuck with a build of Django where many features -- again: users, sessions, and admin -- simply don't work. Tools like manage.py don't work, either. Unfortunately, it doesn't seem possible to take the changes AEP has made in their custom source tree and "apply" them somehow as needed. You're better off working with what you're given.

What did I choose?

In order to choose, I decided to step back and ask myself why Django was so desirable on App Engine. The chief benefit of App Engine is that, with little engineering effort, developers can build highly scalable applications. The chief benefit of Django is that a lot of useful functionality works out of the box. Unfortunately, a lot of that functionality just isn't built in a fashion that's compatible with App Engine's intentional constraints. Trying to get "all of Django" running on App Engine felt a bit square-peg, round-hole to me. (In fact, I'd argue that admin on App Engine isn't terribly interesting, since admin is targeted to lower-scale CMS-style applications.)

So: if I don't get users, sessions, or admin -- well, what again is the point of Django on App Engine? There are still some pieces that work "out of the box," of course, including: forms, templates, URL routing, middleware, and context processors. That said, a few of those (templates, routing) work just fine under the lighter weight webapp framework.

In the end, there were two reasons why Django won out over webapp. First, I wanted to use the latest Django template syntax; webapp is locked to Django 0.96. Second, I wanted to use Django's newforms library. I tried to create a "frakenframework" that mixed webapp with newforms, but this proved impossible due to internal dependencies inside webapp.

The bottom line for me? I now use Django on App Engine via the use_library(...) API. In my next posts about Django on App Engine, I'll describe how to set up a use_library(...) app, and I'll describe the new code I've written to support both sessions and users on App Engine.