Mall Of America, of course!

I will admit to spending not one, not two, but three full days at the mall. I walked all three levels in both directions. The perimeter is approximately 6/10 of a mile, so I managed to get a minor (if culturally and hypsometrically flat) “hike” in during my visits. 

Fun and wet: riding the log chute in the amusement park. Fun and pricey: full-motion WWII dogfighting against my brother.

I have my limits, though. I will probably not want to set foot in another mall until, say, Christmas 2012. Mayans, beware.

Emacs scoffs at normal people.

In emacs, every language mode has its own peculiar notion of indentation. To make it possible to bind keys to indentation functions, indent-according-to-mode was introduced. This function looks for a buffer-local variable called indent-line-function and invokes it if found. Major mode authors know to point indent-line-function to their mode-specific indentation code.

At first blush, whoever built this mechanism appears to have neglected the equally important “outdent” feature. It’s not a surprise, though. Historically, in the world of emacs, mode-specific indent has been closer to “format this code nicely” than to “insert spaces or tabs.” Of course, we now have popular whitespace-dependent languages (Python) and markups (YAML) for which code formatting is less useful.

It’s certainly possible to hook the modes you use and manually set indent/outdent bindings for each. But I wanted to find a solution that worked independent of any specific major mode. Luckily, there’s a “convention” (of sorts) in the land of major modes. For the modes where it matters, it appears that C-c < gets bound to the mode-specific outdent function. So a simple bit of elisp is all we need to get ⌘-[ working. It’s a hack, yes, and hopefully someone will point out a better way:

(define-key osx-key-mode-map (kbd "A-[")
  (lambda () (interactive)
    (call-interactively (key-binding (kbd "C-c <")))))

Of course, you’ll have to pick the keymap appropriate for your specific emacs build. The above works for AquaMacs; for Carbon emacs, use mac-key-mode-map instead.

One last issue: this hack doesn’t do much if there is no underlying binding for C-c <. It might be useful to do something mode-agnostic in that case.

Like most developers who do web stuff, I sometimes need to edit files on a remote server. It turns out that this is a pain in the patookus. Or, at least, it was.

After the thirty day trial, I gave in to the power of ExpanDrive. Those thirty days convinced me that $40 was well worth the price of admission. It’s nifty to be able to mount your remote filesystems via SSH as if they were just network disks.

And yes, before I spent my bucks, I carefully evaluated using emacs tramp-mode, remote emacs via terminal, CyberDuck, Transmit, MacFuse with SSHFS, and MacFusion. I even looked at improving my source control policies and using Capistrano (yuck!) or Fabric (too immature!)

ExpanDrive is based on MacFuse, but not SSHFS; this makes all the difference. Whatever caching magic they do makes ExpanDrive scream when working with lots of small files in nested directories. Perhaps the SSHFS folks will catch up sometime soon, but at the moment the distance between the two is vast enough to justify the cost.

And I can’t wait for ExpanDrive’s planned S3 disk feature. Hot!

Back when I was in middle and high school, I did a lot of programming on the Mac. That’s back in the System 6 era! Dave Mark’s books, especially Macintosh Pascal Programming Primer and (later) Learn C On The Macintosh still sit on a bookshelf in my family’s house. The spines on my copies are so broken it’s a wonder they haven’t fallen apart.

Browsing the bookshelves at the local Barnes & Noble, I was excited to discover that Dave Mark is still at it. I picked up Beginning iPhone Development based on his name alone. While I’ve already done a fairly extensive amount of iPhone and Cocoa programming, Mark’s book doesn’t disappoint. It’s a solid and easygoing introduction to the platform. It doesn’t hit everything — it’s surprisingly silent on networking — but the topics it does cover leave little to be desired. If you’re thinking about getting in the iPhone game, it’s a good book to pick up.

I left to pursue life and career goals that didn’t fit well under the Microsoft umbrella. I missed and wanted to return to the world of startups and of self-directed effort. Which is to say, I think I left for the right reasons.

There was, however, a contributing factor — a “wrong” reason — that helped push me out the door. In my last year of employment, I harbored a vague suspicion that Microsoft was broken at a deep institutional level. Since leaving, I’ve encountered several “red flags” that help confirm my suspicion. These flags make the shareholder in me more than a twinge nervous.

From Where I Sat

Inside, Microsoft looks like bubbling chaos. There are endless new incubations attempting to breathe life into the next great idea. In theory, this isn’t such a bad organizational scheme. It’s inherently Darwinian: the best teams bubble the best products up to the top; everyone benefits. 

The reality, however, is that few of these teams end up building marketplace-successful technology. The key to any Darwinian system is its fitness function. It seems that Microsoft never explicitly designed its own incubation incentives; rather, as the company grew, a set of incentives — the wrong ones — arose without conscious effort.

The Gorilla At Work

In order for Microsoft to pursue a new opportunity, it has to be large. Launching hundreds of $10M businesses or even a handful of $100M businesses is not interesting. To propose an incubation, executives and engineers must think big. They must shoot for the moon. 

But it’s here that the institution’s incentives are very broken. Executives and their teams are rewarded for the pitch; they’re rewarded for a compelling mile-high description of their moonshot. Later on, they’re rewarded for providing demos that show they aren’t standing still. Teams are not, however, rewarded for quickly finding customers who need the technology. They’re not rewarded for explaining the pragmatic benefits of their technology; abstractions and generalized reasoning hold sway. Most importantly, teams are not rewarded for picking a small, tractable corner of their vision to pursue concretely and to completion. 

As a result of these poorly designed incentives, Microsoft is full of incubations at sail with no port in sight. And it’s easy to see the outcome play out in the broader marketplace.

Who Builds For The Builders?

Speaking with startups in the Seattle area, I’ve noticed a recurring trend: when choosing tools, startups rarely look at the Microsoft platform. It’s not that startup engineers harbor a grudge. Rather, Microsoft simply isn’t on their radar screen. And if Microsoft does appear, it’s often as a blip in the distance — too vague an object to warrant careful attention.

There’s a deeper issue: the majority of startups I’ve spoken with have technical problems for which Microsoft is silent. Seattle startups need tools to help write embedded software, design and deploy distributed algorithms, and scale-out complex CRUD web sites. It turns out that open-source efforts like Hadoop and Squid, along with cloud computing platforms like AWS and specialized hosts like Engine Yard, are the best games in town.

Microsoft loses out in a big way when it misses these key platform pieces. But no bulls-eye is no surprise given how its incentives are structured.

Sing The High Notes, Ignore The Low?

When and if customers finally do see the results of Microsoft’s labors, there is often a lot of head-scratching. At this year’s Professional Developers Conference, Microsoft unveiled two major initiatives: Windows Azure and Microsoft Oslo. Speaking with developers, my sense is that the value behind both of these launches could not have been less clear. This is tragic: there is some great technology lurking under the hood, if only it were easy to look.

Azure is a collection of all sorts of cloud-related ideas, only a few of which are likely to be important in practice. Trimming the fat wasn’t an option for the Azure team and so Microsoft’s potential customers have to wade through a pile of muck to find their diamonds. And there are diamonds to be found: the “web and worker” architecture, coupled with tight tool integration, potentially makes Azure the most appealing of the big three cloud services.

The Oslo initiative is similarly baffling. The Oslo team has thought deeply about how to represent and tool models. But there is scant evidence of pragmatic thinking about how models actually improve developers’ lives. As a result, Oslo allows developers to define models (in the sense of defining structures) but provides no runtime platform to access and manipulate them. Meanwhile, it’s simple to both define and manipulate models in platforms like Rails and Django. Once again, Oslo has some buried gems: parts of “M” are genuine game-changers for those in the business of designing languages and tools for languages. And “IntelliPad” is going to rock when it ships.

Even Gorillas Can Be Ignored

The upshot of all of this, I think, is that in many circumstances it’s simply easiest to ignore Microsoft. Wading through complex and poorly structured documentation takes time that startups are loathe to spend. Wading through poorly conceived and messily encumbered APIs takes longer and costs more.

Before I left Microsoft, there were encouraging signs that employees were aware of these problems. But changing institutional structure is not easy; there is too much at stake for employees under the current incentives.

I love my friends and colleagues from Microsoft. I had the opportunity to work with truly brilliant engineers, architects, communicators, and visionaries. I had the opportunity to work with lots of fun and energetic people. But in the end, I found fault with the institution we found ourselves in. I hope that Microsoft can see its way through these current problems and move onward to future greatness.

Late last week my client’s datastore indexes were corrupted. The failure was demonstrably AppEngine’s. Attempts to manually rebuild indexes failed. The AppEngine team has been working for several days to resolve the problem, which we appreciate.

But such failures are disheartening when you’re busy signing up paid customers.

Eight cores is the point where the memory wall causes a fall-off in performance on certain types of science and engineering workloads.

I’m sure we’ll surmount this obstacle, but the solution might not come from Intel. A few months ago, I attended a talk given by a senior Intel engineer where he described the many-core future as inevitable. He said nothing, however, about future memory architectures and was unable even to speculate when asked. Intel’s public road-map is silent on the problem, too.

Real-World Readiness. Of the three providers, only Amazon is out of beta, and only Amazon is actively supporting real customers at large scales across all of the services they offer. The AppEngine team and community are extremely responsive, and Google has published their post-beta road map, but major pieces still in progress — such as support for a second language — may disrupt their intended timeline. And Azure, of course, is barely open for public experimentation at the moment. It will be a year or two before Azure is a reasonable choice for deployment; expect significant growing pains and changes in that time.

Service-Level Agreements. Or, more specifically, lack thereof. Amazon is ahead of the curve with their S3 and EC2 SLAs; as a result, AWS is the only business-friendly cloud platform out there. AppEngine makes no guarantees about uptime, and in the beta period it is clear that Google is willing to tweak their services in ways that can break running applications for minutes to hours at a time. There seems to be a lot of skepticism in the community about Microsoft’s ability to deliver meaningful up-times, but I think this is hogwash. Microsoft has plenty of experience maintaining data centers — though, of course, we shouldn’t expect Azure SLAs for another year or two.

Cost Structure. Google’s “freemium” pricing is free to an impressive degree. I’ve written an application for AppEngine that I expect will be quite popular — but never popular enough to cost me even a penny. Those are powerful economics! Once you’ve passed the “freemium” threshold, however, AppEngine looks slightly more expensive than Amazon. Of course, Amazon charges the moment you start experimenting with their services; this may be cost-prohibitive for small and independent developers. Microsoft hasn’t yet announced pricing, so you develop on their infrastructure at your future economic peril.

There is one pricing issue specific to AppEngine that has yet to be resolved. The AppEngine team appears to have no firm answer on how they intend to meter CPU usage and on what, specifically, constitutes a high-CPU request. I expect more clarity on this by the time AppEngine exits beta.

No Scalability For Free. With AWS, and to some extent with Azure, the developer must manage scalability by hand. This can be a complex task and there is no silver bullet. AppEngine takes a different approach. By substantially narrowing the type of application that can be built, Google can transparently scale based on demand. Even this “scalability” isn’t free, however. Applications that perform a high percentage of writes but are not properly engineered will fall down under the weight of even ten queries per second. Application code can retry writes if they fail, but often the only correct action is to quickly return an error code to your users. In my experience, working through AppEngine datastore contention issues can be very time-consuming. Good load testing is essential.

Byzantine APIs. Amazon is especially guilty of building APIs that are far more complex than necessary. It simply isn’t possible to “hit the ground running” with Amazon’s more interesting services such as EC2 and EBS. And once you walk up the learning curve for one AWS API, you have to start all over with another. Some of Amazon’s APIs are query-based; some are REST; some are a combination thereof. The documentation is thorough but hard to make day-to-day use of.

Woeful Documentation. Microsoft has done an abysmal job explaining their new Azure platform to developers and businesses alike. After the PDC, it seems that most developers were left scratching their heads. The Azure web site lists “getting started” options for “web,” “corporate,” “ISV,” “system integration,” and “business” developers. Deciding which category you belong to is left as an exercise (in frustration) for the reader. Digging deeper, the most useful overview of Azure turns out to be a sixty-page white paper that reveals most of the Azure initiative (Live Mesh, SQL Cloud Services) to be essentially uninteresting. And once you start to use the service, you have to navigate a torturous web UI that makes a disaster out of even simple tasks like DNS configuration.

Google’s AppEngine offers worry-free scalability for a very narrow (but useful) set of problems. Your application must be driven strictly by requests to public-facing URLs. The application must render responses in a few seconds; compute-intensive tasks are not an option. AppEngine offers no work-queue and no asynchronous APIs; background processing is therefore not possible. As your application’s load increases, Google automatically allocates new CPUs to the task. This means that scalability is fully transparent to the developer.

AppEngine bottom line: If your application is a read-mostly CRUD application, Google’s AppEngine can scale you to hundreds of queries per second with little or no engineering effort on your part… and little or no financial outlay. Oh, and you’d better be happy with Python.

Amazon’s Web Services lie at the opposite end of the spectrum. Amazon has “componentized” the infrastructure that sits behind most scalable web sites. From block devices to message queues to full virtual PC instances, you can choose just the parts you need to satisfy your application’s hardware, networking, and storage requirements. Of course, all of this flexibility comes at the cost of making you the system administrator. And unlike AppEngine, you don’t get scalability for free: if you need more machines to scale your application, you’ll need to request them yourself and your code will have to determine how best to utilize the new resources.

AWS bottom line: Amazon offers maximal flexibility and is ideal if your task doesn’t hit another cloud provider’s sweet spot. If you have specialized needs, are capable of hand-rolling scalable code, or are happy acting as a system administrator, AWS is your ticket to utility computing glory.

Microsoft’s Azure, newly announced, sits somewhere between Google and Amazon. Azure allows you to deploy two distinct types of code into the data center. Web Roles are request-driven applications whose restrictions look very similar to AppEngine’s restrictions. Worker Roles are arbitrary, and potentially long-running, pieces of code. Microsoft provides several mechanisms for web and worker roles to communicate and share state, including message queues, a DHT-like datastore similar to AppEngine’s, and a quasi-relational datastore based on a variant of SQL Server. Like AppEngine, web roles can scale transparently if desired, but the developer also has the option of manual control. Worker roles are of course fully managed by the developer.

Azure bottom line: It’s too soon to tell, but if you want to build a compute-intensive application or need background processing — and you don’t want to play system administrator — Azure may be the ideal choice. If you choose to use Visual Studio for your development, Azure also delivers the best overall debug and deploy experience of the three services. At the moment, you must develop in a .NET language; C# appears to be preferred.