A REST Intervention

Responding to James Snell's article, Resource-oriented vs. activity-oriented Web services, a dissent in 4 parts and a contribution to the ongoing (never-ending?) debate on REST (Representational State Transfer), web services and SOAP (Simple Object Access Protocol)

1. A Snap Judgment
2. Changing The Frame
3. Internalizing the REST style - A case study: WebSphere Portal
4. An Argument by Analogy

1. A Snap Judgment

Sometimes you read or hear something that immediately strikes you as wrong-headed and sets you off. In the real world, if say you were in the same conference room, you might cross your arms, prepare to raise your voice a little as you ready a sharp retort. If you were on a mailing list, this is the kind of thing that will start you writing a lovely flame, complete with point-by-point rebuttals. I've learned dear lessons about the social cost and inadvisability of rushed (and rash) reactions in the online world, so I confined my dissent to bookmarking the article and tagging it with the following inchoate comment:

Completely missing the point about REST and what it means to "be on the web."

A knee-jerk response or snide and unfair commentary perhaps, but at least his piece didn't leave me indifferent like much of what I read on the web.

I've cooled somewhat in the intervening weeks, also I've re-read what Snell wrote and realized that I'd missed much of his point. It's a well-meaning and well-reasoned exposition; it's hard to argue with the form if not the substance. There are lots of caveats and "don't get me wrong"-s, cues that suggest a more nuanced picture than I obviously gave it credit. He's one of the good guys, a colleague with whose group I've often worked with, and this is very far from the typical (and occasionally disingenous) posturing of the REST/SOAP debate.

I'll start my considered dissent then by modifying my assessment: Snell does understand REST, he gets the point but is just cautious about its application and, like all of us, is struggling with the issues it raises. To him, it's just another approach in his toolchest, there's a sense of "different stokes for different folks" in his manner. Engineers are all about evaluating tradeoffs and pragmatism like his is often just sound engineering.

I think my initial reaction was coloured by having read too many journalists who cover the world of technology adoption and often cast things in the winner-takes-all mold. At ground level in the technical trenches where Snell and I live, the calculus of profits is less important; it is rather a marketplace of competing ideas and we're picky and conservative buyers on the whole.

Still in my dissent, I need to explain the second part of my statement which relates to the importance of "being on the web". There has to be a response in writing, if not in code. Ideas don't exist in a vacuum and I shouldn't take to some ivory tower, with Fielding's bible on hand, Prescod and Baker as prophets in the wilderness, and Bray as curmudgeon and loyal oppositionist-in-chief all the while pointing to Apache, Amazon, Yahoo, flickr, del.icio.us and others as favoured offspring and self-evident existence proofs. That's not a sufficient response.

2. Changing the Frame

Fair enough then, how best to articulate this lingering sense I have that Snell short-changes or side-steps the value proposition of the REST architectural style?

The following tidbit from another context seems strangely relevant here:

Oliver Hass, a 28 year-old chemist and graduate student from Oldenberg, Germany, wrote me recently about what the President's trip looked like to him. In introducing himself, Hass commented on "how necessary it can be for a chemist to forget about molecules and think about real problems."

In terms of technical advocacy, others far more experienced than I and have written eloquently and at length wonderful treatises about the virtues of REST and why wire protocols matter. Bray has a very good writeup on recent thoughts. There have been lots of attempts at articulating best practices even prominently featured in wikis. In recent months, Joe Gregorio has been embarking on the show me the code path, doing some of the most effective advocacy by plain example.

My gut feel is that the disposition of the debate will be ultimately be determined by running code and one engineer at a time, hence I won't add to the heated rhetoric. So then I'll make the advocacy argument by changing the frame, and casting the REST value proposition almost in economic terms, in terms of systemic leverage, pragmatism and ubiquity.

Positing, like Snell does, that is about a resource-oriented view rather than an activity-oriented view belies a significant question. Surely benefits would accrue if "activity-oriented" items were exposed on the web? Indeed, these days if you don't expose your offering or activity on the web, you hardly matter. If you're ultimately going to derive much of your value from web exposure, why accentuate the impedance mismatches between your technology stack and the prevailing architecture? For indeed there is an architecture to the web. This is evident when one considers the differences between HTTP 1.0 and HTTP 1.1. The REST style simply elucidates the philosophy of the underlying architecture.

For some reason, REST is much misunderstood; the handwaving description "the best of the lessons learned in developing the web" needs much elaboration, as does the elevator pitch version:

REST is defined by four interface constraints:

• identification of resources
• manipulation of resources through representations
• self-descriptive messages
• hypermedia as the engine of application state

Adopting the REST style is not a technology prescription or panacea; after all, an architectural style is not a programming toolkit. But there's a complexity and layering argument that naturally falls out of the REST viewpoint which should guide the applications that one builds.

The web is about identifying important resources and exchanging representational state. There are certain constraints that are made to enable global scope and evolution. Being on the web, is being an active participant in this scheme of things. Identifiers lead almost to having the location field as the command line. Resource modeling as your starting point sets you in direct consonance with this notion and exposes you to ease of composition. An information architecture that starts with hypermedia lends itself to the construction of simple interaction designs for both humans and machines, and there are huge amounts of tools readily available for this.

The question is then one of leverage; leverage in economic terms is all about making externalities work for you. The endpoint of this approach to system design is what I call a virtuous cycle of managed serendipity.

3. Internalizing the REST style - A case study: WebSphere Portal

At a certain point there were (at least) 8 different groups at IBM developing portal software. Incidentally, this is par-for-the-course in large companies; executives are often willing to allow this kind of competition as a kind of macabre survival of the fittest game - for a while at least. Two groups managed to ship products to customers, first Lotus K-station and then what was eventually the winner, WebSphere Portal. K-station was a product which, almost to a fault, embraced and leveraged the full range of web technologies. It demoed well, made pervasive use of CSS, DOM scripting, XML etc: web to the core in other words. Although it was a J2EE application, its fatal failing was that it shipped initially on top of Lotus Domino's servlet engine and database rather than on top of WebSphere Application Server and DB2. What became WebSphere Portal started by building on the right platform, its front end had what we euphemistically termed teething problems but it had a reasonable backend and the virtue of an architecture that looked like it could eventually evolve into the "one true portal".

As we consolidated all the efforts under the rubric of WebSphere Portal, there was of course a kind of triaging of the expertise and feature sets from the various groups. The Lotus folks emphasized collaboration and mostly owned that side of things but we also had a laundry list of technologies that we wanted to endow the platform with (from basic ideas about the way one used stylesheets for skinning, all the way up to a repartitioning of the architecture to allow more intelligence in browser clients and leverage of client side technologies that are now increasingly common). Call it youthful exuberance or perhaps in my case personal temperament, I argued each and every feature tenaciously and energetically. When it comes to technical argument, I'm not afraid to offend senior architects, product managers, or even distinguished engineers far beyond my station. In hindsight, we perhaps should have been more judicious in picking our battles because even though we managed to win many of the arguments, we lost one that we really shouldn't have. Mostly through sharp-elbowed bureaucratic maneuvering, the feature was accepted as a line-item but was marked as a nice-to-have, and then was deferred because "we didn't have the time in the schedule".

What feature was it, you might ask? There was no way to bookmark anything in WebSphere Portal.

The portal at that point bore the vestiges of an idiosyncratic framework (Jetspeed) that in my opinion actively fought against being bookmarked. No one at Lotus could fathom that a web application could ship without bookmarking. We couldn't even do the simplest use-case from K-station: when creating a place, you couldn't send an email to new members inviting them to it. An email could only direct them to the home page, once they logged in, they'd then have to navigate to the item in question and as we'd made it flexible to skin navigation, there was no easy way to describe how one would navigate to that item. The immediacy of collaboration, of "people, places and things" was irrevocably lost. I argued almost to disgust: "This is the web! Resource identification is a key tenet, almost the most important principle. It's a fundamental flaw" etc. to no avail.

It took 3 releases for this feature to be implemented and enabled in the default installation of the portal. Throughout, we continued to lobby, until the chorus grew too loud and it wasn't just "those Lotus folks" making the argument. Being able to bookmark places, pages and portlets is fundamental for many reasons beyond simple usabilty. At a first order, you were now able to paste a url in an instant message to co-workers so they could join you and see what you were looking at, at that moment. But also, almost overnight with this feature, many layers of complexity were shed in the portal framework.

Programmability in the web sense was immediately enabled, the portal became a composable platform and we were able to layer the Lotus Workplace offering on top of it. URIs give visibility to intermediaries and so things like caching (where we had cool technologies like Dynacache) were far more easily enabled. Similarly for logging and profiling the portal, we could use the same tools for processing logs as exist for regular web servers like Apache. We had new opportunities for pipelining and filter chains (to do transcoding if needed). We had more options for load balancing, we could decide to deal with remote portlets through iframe invocation rather than through immature and complex protocols like WSRP. And so on...

My biggest regret is that we hadn't been bloody-minded enough to do a stealthy check-in of the hacked prototype code for enabling a modicum of url addressability that we had developed. Instead, by being consensus-minded corporate citizens, we had allowed our platform (and users) to suffer for 3 years because it hadn't internalized this most basic aspect of being a web application framework. WebSphere Portal only acknowledged the web in its name once it embraced this part of the REST ethos and it hasn't looked back since. Indeed after internalizing resource identification, the product has taken to heart most of the other tenets of the web style. It still has some ways to go but, from that point on, it was no longer odds with the web.

As an aside: where there is friction and impedance mismatches, there inevitably will develop an ecosystem of consultants with palliating bromides and band-aids on hand. All companies claim to be "focused on the customer", my own company has the virtue (or some would say the failing) of encompassing the widest range of products and services and is apt to make money whether one buys hardware, software, or services from it. If the choice is between profitable software that internalizes the lessons of a platform and rank consultant-ware, I opt firmly for the former. Maybe I'll change my mind if I become one at some stage, but consultants, though often necessary, are mostly irksome from my standpoint.

4. An Argument by Analogy

The supposed or acknowledged (PPT) deficiencies of the REST architectural style (reliability, asynchronous event notification etc.) are to my mind good problems to have. It strikes me though that these criticisms of REST system design, while fair, are mostly concerned with boundary cases; in other words, the balance of optimization that the web style has gone for makes these mere irritants not failings. The response should be much like that typically given to those who discounted the 'best effort' internetworking or to the bellheads who harp on quality of service and see the internet as a toy. I don't see the quality of calls preventing the ubiquity of cell phones. In that case, the virtues of mobility long trumped reliability. Network standards like ethernet and TCP/IP that are the core of the Internet have succeeded and scaled because they used abstractions that embraced simplicity, transparency and existing systems. The web was able to co-opt everything in sight (from ftp to gopher to whatever newfangled scheme we'll come up with next) with only minimal constraints for the sake of scaling and composition (like the concessions to, and explicit recognition of, the interplay with intemediaries - caches and so forth). Google (pdf), Yahoo, eBay and Amazon have interesting takes on addressing these deficiencies and already operate on a global scale.

Martin Geddes, discussing a different problem, ever-quotably and far more concisely than I'll ever be, works himself to the same fundamental conclusion:

Solutions based on URLs tend to be less "efficient" than custom-designed solutions, but more resilient in the face of change. (Sound familiar?) I casually note the rapid rise of URL-friendly blogs and wikis, and the relative obscurity of vertically integrated collaboration tools like Groove.

I saw a good example of "URLization" from Joi Ito the other day. Product managers at various companies are being encouraged to use Technorati to track commentary on their products. If your product doesn’t have a clear, unique URI, then you’re in trouble...

This all brings me to my big question: on the Internet, if something doesn’t have a URL, does it really exist? Or has it just disappeared into the analog memory hole, only existing as a memory in the brains of the humans it passed through?

That's the heart of the matter, debates about "programming models" aside, this is an argument about systems design and technology adoption. If your nascent technology stack doesn't leverage, or as I think is the case in many of the WS-* protocols, actively fights against integration with the web, you're guiding yourself towards obsolesence. The REST orientation is about acknowledging the "good enough" factor implicit in the web architecture. Choosing to embrace the ethos of that layered style is arguing for leverage and simplicity. The value of ubiquity and managed serendipity that are often the outcomes of this approach should speak for itself.

Returning to what provoked this outburst, I write to James Snell: thanks for getting me thinking enough to put these thoughts together, my arms are uncrossed and there's even a smile on my face.

See also: my other REST writings. File under: REST, web, design, software, programming, architecture, Lotus, IBM, WebSphere, SOAP, technology, uri, identification, addressability, programmability, url, networks, standards, Portal, toli