The title was AppEngine Nitty Gritty: Scalability, Fault-Tolerance, and Integrating With Amazon EC2. We gave high-level advice about how to design an application for high loads and how to architect an application that makes use of multiple cloud computing environments. We also went straight to the dirty grime and shed light on some issues we encountered while building the Walk Score API — issues like unpredictable data store contention and the failure of basic sharding techniques for certain high-scale features.

You can watch our presentation via YouTube. I hope you enjoy it!

First I write back-end code to define and manage my models. Then I write front-end code to do the same. The back-end code is in Python, or Ruby, or sometimes PHP. The front-end code is in JavaScript.

It seems absurd to have to define my models in two distinct languages. Worse, after I’ve done so, I have to write substantial glue code to round-trip changes from browser to server and back again.

Perhaps Google Web Toolkit (GWT) solves this problem for me? I haven’t dug deeply at all, but it seems as though I could simply define everything in Java and then let GWT handle the RPC work for me?

That would certainly be a plus in GWT’s favor. I’m not convinced it’s enough to balance out the two negatives I currently see: (1) you often have to “drop down” to HTML and JavaScript when tackling performance issues, and (2) you still have to use CSS; you still have to worry about cross-browser compatibility.

Some years ago, we would have followed TextMate to whatever end. Now, however, we see that TextMate was a false prophet. He led us down a garden path but abandoned us before we reached the promised land. In our desperation, we turned to the elder sages: vi and emacs. Alas, they seem curiously unaware that it’s 2009. You know: 2009. The year before the year we make contact.

Editor Gods, we don’t ask for much. We just want an editor that makes us stand up and shout: “Hey! The future is here, and I’ve got a text editor to prove it!”

While we’re praying, we might as well let you know exactly what our hearts yearn for.

We want an application designed from the ground up to be a real OSX application. We don’t want a complex port of code that was born three decades ago. We don’t want funny keyboard chording systems. We do want elegant control of windows and splits. We do want full-screen. We do want sane selection and clipboard behavior.

We want batteries included. We expect the most popular languages and source control systems to “just work” from minute zero. We care about dynamic scripting languages and promising new functional languages, and we care about distributed source control systems.

We want carefully designed extensibility. We want to write code to change much of the interactive behavior of the editor. Moreover, we want to write that code in a language of our choosing — not yours, so we’re probably looking for IPC rather than in-process extensibility. All this said, we don’t need too much extensibility. Our purpose is to edit code, not to surf the web or psychoanalyze-pinhead.

We want instant access to relevant documentation, regardless of the language or library we are using. Take us to the right web page and let us easily search our documentation sets.

We want true grammar-based language tools. Naive syntax highlighting can only take us so far. If we hover over an identifier, our editor should know that it is the name of a class and not just an arbitrary blob of text to color blue. We want an editor that gracefully handles files that contain PHP, JavaScript, CSS, and HTML all in one. We want intellisense that is aware both of our language and our libraries in context.

We want a navigation superstar. With deep language support comes potential for radical new navigation strategies. We want to bounce around our code with reckless abandon. How code is spread across files should not be important to us; file structure could potentially be a matter of policy.

We want large and remote file support. We work with enormous log files. We work with remote installations. Why is this so difficult in 2009? We want overall network awareness in our editor.

We want it to be open source, because we never again want to be abandoned by a false prophet.

Dear Editor Gods: that is all we ask. Is it so much? Will you hear our prayer?

Apple has a great sync story called MobileMe. It allows mail, calendar appointments, contacts, and a few other things to automatically sync via the cloud. Alas, Apple offers no public synchronization API. Apple’s home-grown apps sync up over MobileMe; Cocoa developers are left stranded, forced to “roll their own” sync solution.

Many Sync, Much Headache

As a result, the Mac software world is littered with disjoint sync strategies. Often, these hand-rolled strategies aren’t so great. To pick one example: Cultured Code makes a beautiful task management tool called Things which runs on both OSX and the iPhone. I use Things every day to manage my tasks and its sync behavior is my nemesis. In order to sync, I’ve got to remember to open up Things Touch before I leave home. Only if my phone and laptop are on the same network — and only if they’re both powered up and running Things — will my data get synchronized. If I run out the door for a meeting and forget to synchronize, I’m sunk.

I can understand why Cultured Code designed their sync system this way. It makes elegant use of Apple’s local area networking technologies and side-steps the problem of needing to maintain an intermediate service to pass data between app instances.

If you’re a die-hard Cocoa developer, the thought of building a scalable and reliable web back-end might just be daunting — daunting enough to cut it out entirely. I’ve noticed a strong resistance in the Cocoa community to building such things.

Think About Money

Enter AppEngine. AppEngine makes scalable + reliable back-end development extremely cost effective. You write some code; you deploy it in Google’s data centers; with little to no work on your part, you get a back-end that can handle far more traffic than you’ll ever see, and which will exhibit far better uptime than you could likely arrange for elsewhere.

Maintaining AppEngine applications is nothing at all like maintaining custom web deployments. You upload your code and check the AppEngine dashboard every once in a while to see how things are running. There is no need to play system administrator; no need to patch systems or tweak network configurations, etc.

Best of all, AppEngine has a freemium pricing model. Below a certain amount of usage, AppEngine is entirely free. After that, you pay as you go. For an indie OSX developer, this means that when you’ve just launched your product, your back-end fees will be $0 — a price that’s hard to argue with. My guess is that for many products — even moderately successful products — the monthly cost will remain $0.

If your product really takes off, back-end fees will scale with your user base. If you price your product carefully, you should have no problem covering your monthly AppEngine costs for many years to come. For example, let’s say Cultured Code stored all of my task data inside an AppEngine instance. My data weighs in at a hefty 13MB, which would cost Cultured Code exactly $0.02 per year to host. I think they can recover those costs, even if I remain a user for the next decade!

Think About Features

It’s not just economics: from a feature and technology standpoint, AppEngine dovetails well with the Cocoa sync problem.

In order to make scalability as simple as possible, AppEngine is a severely restrictive platform. Your AppEngine code must be strictly request/response. Individual responses must happen rapidly. You can’t use a relational database, but you can use a highly scalable property-value store. All transactions must run over HTTP.

It turns out that these restrictions are no obstacle to building a solid sync service; if anything, they’ll help you design it right the first time. It makes sense, especially in the iPhone world, to communicate with your back-end via HTTP (and in all probability, NSURLConnection.) It’s simple in both Python and Objective-C to work with JSON, so it should be simple to get your data transported across with a minimum of fuss. Once your data reaches AppEngine, you can dump it into the property store as a blob. There’s probably no reason for your back-end to need to know about the structure and format of that data.

What’s best is that your AppEngine back-end will always be there, ready to sync user content as soon as your front-end software requests it. And when iPhone OS 3.0 and push notification go mainstream, AppEngine will be the perfect environment from which to manage them, too.

The Missing Piece

All that’s missing, of course, is a library to actually perform the synchronization. This seems like an Open Source effort just waiting to happen. There would have to be three pieces:

1. A Python/AppEngine back-end that developers can easily deploy with little or no configuration on their part
2. A Objective-C front-end that would make it extremely easy to communicate NSData and various NSCollection objects with the AppEngine back-end. You would of course want to tag this data with unique IDs and you’ll need hooks for performing custom conflict resolution, but it’s probably not a terribly complex API.
3. A one-time-only mechanism for pairing disparate instances of the application together. An easy solution would be to let this take place over the user’s LAN, round-tripping the necessary tokens through the back-end service after pairing is complete. A more involved solution would be to actually have user accounts on your back-end service. I imagine the easy solution is good for 85% of the applications out there — it would certainly be sufficient for Things!

So there you have it. I admit this is nowhere near fully baked yet, but I believe that if you let your back-end have no knowledge of your data (beyond “here it is”) you can go a long way without too much complexity. It seems that Cocoa+AppEngine+sync is a beautiful world indeed. Anyone up for rolling up their sleeves and writing some code?

With apologies to Cultured Code, who make great products that I use every day and heartily recommend!

I’ll be there with my friends Jesse and Josh; we’re going to talk about building highly scalable applications using Google AppEngine and Amazon’s Web Services. Jesse, Josh, and I built the Walk Score API using AppEngine, EC2, and S3. The API gets tens of millions of daily hits from big-name customers like Zillow and Redfin. Best of all, it costs pennies to maintain.

We’re looking forward to meeting you there. We promise fireworks and/or leprechauns at our talk!

The iPhone’s default flick-to-scroll behavior is interesting. If you hold your finger down for a long time and move it in a variety of directions, you’ll see that only the very last direction is used to compute the scrolling motion.

If you try to build this code yourself, however, you’ll quickly discover that simply using the last two points to compute the direction of motion isn’t going to cut it. Instead, my code keeps a short history of touches and uses linear interpolation to determine where the touch “would have been” some small amount of time ago. This interpolated point is used as the basis for computing the motion vector. This leads to a very pleasing interaction.

The code is available on github, here: http://gist.github.com/100855. It is released under the BSD license.

As I build more sites, two important patterns have come to light:

1. I implement model classes on both the front and back ends. This allows me to manipulate my models wherever it’s most appropriate; for cool AJAX features, this is usually on the front-end.

As a result, I am forced to implement the same class — for example, a class that represents a user or the state of a game — in two different languages. I then have to do the work of marshalling model data between the back-end and front-end. Right now my marshalling is often one-off, but it occurs to me that this really shouldn’t be the case. It also occurs to me that it would be a lot more desirable to just implement each model once, in JavaScript.

2. Much the same: sometimes I generate large blobs of HTML on the server side; sometimes I need to generate the same blobs of HTML dynamically on the client side. 

A typical example is a contact list. Depending on the size, it might be desirable to generate HTML for known contacts on the server side. But when you add a contact you probably do so via AJAX calls, which means you need to dynamically generate that same HTML blob via JavaScript. There are plenty of approaches to this problem, of course — you might decide to always dynamically generate on the client side, or you might have the server return some HTML in response to an AJAX call (yuck!) — but the cleanest approach would certainly be: write the generating code once and use it everywhere.

Batteries

I am aware of some template languages that approximate a solution to my second problem. Most recently, the json-template library made a bit of a splash. None of the libraries I’ve evaluated so far seem to capture the full problem.

And my first and frankly more pressing problem? I’ve seen no solutions whatsoever.

These thoughts have led me to believe that JavaScript should be the next big language. The obstacle, of course, is that while JavaScript works pretty well inside a browser, outside of a browser it is largely a naked language. What is a scripting language without the ability to talk to a database, fetch a URL, access files on disk, or interoperate with web servers? Heck, what is a scripting language without a basic system for defining and importing modules?

For these reasons, JavaScript is currently a non-starter on the back-end. I’m hopeful that someone will take up the mantle and build both the batteries — the APIs that server-side JavaScript so desperately needs — and the standards for those batteries so that we have one language with one library, rather than many.

I should mention that I don’t think server-side JavaScript APIs need to look anything like client-side JavaScript APIs, though there would certainly be some crossover — DOM APIs belong on both sides, for example. I would look to the Python and Ruby worlds for inspiration about what sorts of APIs JavaScript needs before it can become the one language to rule them all.

Erik played the trombone with my jazz quintet — now, alas, a quartet. He also directed the Cascadia Jazz Big Band, an extremely talented and fun group of musicians here in the Seattle area. I miss his trombone’s sorrowful tones and his somewhat embarrassing taste for cheesy Bossa Novas.

If music was half of Erik’s life, biking was the other. In 2006, when I undertook the Seattle to Portland bike ride, it was Erik more than anyone else who encouraged (okay, forced) me to keep up with my training. I can’t count the number of Saturday mornings he called at 6AM and said “dammit, if you’re not on the trail yet, how are you and I going to to get 100 miles done in time for drinks?”

Erik, Me, Brent, Andrew, and Ian after a cold and wet bike ride around Bainbridge Island

In addition to being downright disrespectful of my weekend sleeping habits, Erik had a knack for mooching food whenever possible. If I brought dinner to Wednesday night jazz practice, Erik generally yoinked half of it while I wasn’t looking — unless, of course, it was fast food. He held his nose up at that.

One Wednesday night in early 2008, Erik came to jazz practice but sat it out. He said he was unusually short of breath. A week later, Erik said he had coughed up a little blood and didn’t feel terribly well. We encouraged (okay, forced) him to go see a doctor. Two weeks after that, Erik returned with the news: he had an unusual form of cancer that had already reached Stage IV. It had metastasized and spread just about everywhere in his body. There was apparently nothing that modern medicine could do.

Things moved quickly after that. The next time I saw Erik, he had to use portable oxygen tanks. I drove him to the Safeway to pick up a few things. When he caught a woman staring at him and his tanks in the checkout line, he turned around and barked in the sarcastic way that only he could muster: “What, you’ve never seen a dead man before?”

A few weeks later when I visited, Erik was bed-ridden. He had good care in his house and friends were visiting round-the-clock. But he was already clearly weak. We listened to his favorite latin jazz recordings and he told me something I’ll never forget, so out of character was it for him: “I’m okay with it now. I’m going to die. The world won’t miss me and it will get along just fine without me. In fact, I guess that’s the tragic secret to life. Each of us dies in turn, and it doesn’t matter. But somehow the whole game of life keeps moving forward.” Later he added: “Oh, if you see any religious types heading my way — turn ‘em back. I’ve got no need for that bullshit.”

Two weeks later I saw Erik for the last time. He didn’t answer when I knocked so I walked in to find him sleeping. I woke him up to say hi. He looked confused. We talked for a minute or two until his nurse arrived. Erik immediately sat up, pointed at me, and shouted. “This man has been in my house. He keeps talking to me. I don’t know him. Get him out. Get him out!” He looked terrified. The heavy dose of pain-relieving drugs he was taking had made him disoriented. At the nurse’s request, I left — a little shaken.

Erik’s funeral was exactly as it should have been. We gathered at the local church where his big band practiced. After saying a few words about Erik, most of them sarcastic, his band played for several hours. Our quintet also played his favorite tunes. After the service, many of the musicians stayed on to hold a late-night jam session.

This Saturday we’re going to get everyone together again to remember Erik. There will be music, food mooching, and Alaskan Ambers. I only wish he would be there to help us keep the party going.

By name, they were Ted Nelson, who penned and hand-published the still-prescient Computer Lib/Dream Machines; Mitch Kapor, who named his startup Lotus and later founded the Electronic Frontier Foundation; Ray Ozzie and Dan Bricklin, who saw artistry in software and conceived of the spreadsheet; Stewart Brand, who dazzled with his Whole Earth Catalog; Danny Hillis, who dreamed of Thinking Machines; Richard Stallman, who grasped the perils of intellectual property long before others; and perhaps even Steve Jobs, who drew inspiration from Buddhist philosophy and calligraphy.

These bohemians shared an excitement that software and inexpensive computer hardware could liberate the masses — that it could free the oppressed from tyranny, provide education and opportunity for the underprivileged, and connect disparate peoples as never before.

Fast forward to today. The industry grew up, and so did its bohemians. Through their work, many became luminaries and titans of industry. They crafted a new, nearly trillion dollar market sector. Along the way, a new generation — mine — arrived, born at a lucky time when software was young but some trails had been blazed. When we left college, money grew on trees. In early 2000, we started dot com after countless dot com. The bubble burst; we forgot the lessons; we created a new bubble with Web 2.0. Things are even harder now, but at least there’s iFart.

Have we forgotten our roots? This thing isn’t about get-rich-quick; it isn’t even about get-rich-slowly. [1] It’s about empowering people and improving their lives. It’s okay, of course, to do this in a small incremental fashion. But our progenitors in this industry weren’t interested in incrementalism. They saw the potential for a radical break from the past. In many ways, it doesn’t seem that their original vision has been met. Perhaps most sadly, some of the best of them ended up hawking beautiful but ultimately unimportant gadgets — selling sugar water when they wanted to make a dent in the universe.

This is why I’ve been excited to learn, lately, of a handful of startups in Seattle whose goal appears to be “improve people’s lives.” Some even have unusual corporate structures that combine the advantages of for-profit and non-profit organizations under one umbrella. One of my clients, Front Seat, has the stated goal of building Civic Software: software that enables or enhances aspects of neighborly life. Startups like Front Seat are not strictly green, but they’re qualitatively similar. They have an abiding belief, as did many of the industry’s original bohemians, that many of the problems we think of as social are in fact amenable to software solutions. With Walk Score, we learned that you don’t have to be an urban planner to have a positive impact on urban planning. Imagine what change might be unleashed if the hordes of “crap app” developers saw their industry and their world in this light.

——

[1] I do not have a problem with getting rich quick. I do, however, worry when vast numbers of people devote their time to getting rich quick who could otherwise have solved pressing problems and engendered positive long-term change. We’ve had gold rushes before, of course, and we’ll have them again. But, on average, the people who benefit most from a gold rush are the merchants who sell shovels.

I wrote the game partly for fun, and partly so that I’d have some meaty AppEngine example code to show people during my talk. It was a “weekend hack,” although I’ve tacked another ~12 hours onto that weekend to add support for chat and history views.

I chose a very restrictive license, the GNU Affero GPLv3. It turns out that my service is being used — a lot — and I want to make sure all similar services are forced to share their improvements with the world. I’m hoping that http://go.davepeck.org/ will remain the definitive version of the site. With a little work, perhaps it could become the definitive place to play Go on the Internet.