The REST Elevator Pitch

• The Pitch
• The Hard Problems
• The Sequel
• The Elevator Soundtrack

The Pitch

I somewhat share Ryan Tomayko's concern for a shorter hand language for describing REST (Representational State Transfer), but I believe that the REST elevator pitch is pretty succinct already. It packs a lot of insight into the web architectural style of which dissertations have been written. The language used in technical debates is always important and rather than devolving into the fuzziness of "highs and lows", I'll suggest here that, as a matter of advocacy, one should stick to the elevator pitch.

To recap, the REST elevator pitch is:

• Identification Of Resources
• Manipulation Of Resources Through Representations
• Self-Descriptive Messages
• Hypermedia As The Engine Of Application State

In its most common application, the pitch will lead you to the HUHXtable Quartet (HTTP / URI / HTML / XML) if we handwave away the additional content types that the web style enables you to negotiate. It is quite clear that applications of all kinds have been built on these fundamentals for both human and machine interaction.

The thing is that each plank of the pitch brings its own benefits, benefits that perhaps can accrue independently. Still you get the most bang for the buck at the internet scale by the principled combination of all of them.

In my advocacy I tend to start with resource identification as the most important plank. The reason for this is that everybody can understand the value of identification. My grandmother understands the value of a bookmark to my photos. Heck we are even seeing the social benefits of ostensibly selfish bookmarking. The importance of resource identification cannot be understated and it leads us to the primacy of the URI.

The second plank, manipulation of resources through representations, is the bread and butter of HTTP, where we have an evolving set of standard formats of hypermedia exchanged using a small set of verbs.

Unpacking the HTTP proposition leads us to the four horsemen of the web: GET / POST / PUT / DELETE

Of course you can do very well if you pick up only 2 of those verbs, GET and POST. The good Reverend HTTP is a pretty tolerant fellow and won't begrudge those that don't believe in the rule of four, although in admittedly imperfect analogies, there's CRUD (Create / Retrieve / Update / Delete) in the database world and, as far as this layered internet goes there's TCP / IP / Datalink / Physical Layer (with say Ethernet or "Wi-Fi" 802.11b/a/g)

On the other hand, if you want to be a good citizen in the town of Deadwood and indeed if you want to survive in the wild west of the web, you shouldn't abuse HTTP. HTTP abuse leads to user confusion, headaches for systems designers and a tragedy of the commons. I consider HTTP a deliberately minimalist compromise, laissez-faire distributed computing if you will.

In social settings making your intentions known can help smooth interactions. We may start with small talk but we eventually manage to discern where others stand - well most of the time at least. We all operate with imperfect information but being straightforward is often a great policy. I'm sure there's some game theory or behavioural economics that can point to the value of transparency. Perhaps it is trite in light of Machiavelli and Sun-Tzu to believe that disclosure is the best policy but at least in markets transparency tends to be the font of aggregated wisdom. On the internet scale, this translates into the notion of giving visibility to intermediaries. Arguing again by analogy, the internet is an agreement, a network of networks. You can do whatever you want on your network but to interoperate with others, a small set of primitives, the TCP/IP standards in this case, have been agreed upon. I would suggest that the web style is similarly such an overlay system. REST is an architectural style which by principled design, is optimized to lower coordination costs.

Acknowledging the benefits of idempotency, of doing things safely, will lead one to making sure the use of GET is safe. Idempotency is not just about safety however; in clearly signaling your intentions, it is also about replayability. Thus idempotency is not just about the GET method. That is the wider argument for also using that oft-neglected duo, PUT and DELETE. Tunneling everything through POST (and the completely aberrant case GET) may be convenient but it adds unnecessary opacity and unpredictability about one's application. Unpredictability in the world leads to invasions and worse I might add. The use of those four core HTTP verbs helps intermediaries understand and participate more fully in the ecosystem.

The last two planks of the REST elevator pitch, self-descriptive messages, and especially that last one, hypermedia as the engine of application state, are attempts at addressing the issues of state, caching and evolving systems amongst other things.

Now of course in the preceding paragraphs, I've handwaved away entire dissertations that have been written about architectural styles and systems design, tomes written with language more precise and concepts more subtle than I've outlined. Good samples and running code are often also a determining factor in which style or application is adopted thus the view source imperative is also one of the reasons that web has caught on so quickly.

The Hard Problems

I've recently been thinking about defining the hardest problems I've encountered in software engineering, my cursory top 10 list:

• State
• Caching
• Latency
• Concurrency
• Search
• Metadata
• Persistence
• The Holy Grail Of Extensibility
• Structured data
• Character encoding

Now I'm not a database person so I handwaved away all of those data peoples' worries in one word: persistence. Of course I should probably add a few more issues and indeed you probably have your own notions. Indeed I settled just two days ago on those last two, structured data and character encoding, when I characterized them as being the gruesome twosome of computer science. Perhaps also there's the issue of discovery and bootstrapping of a system but that could be considered implementation details. There are those who mention security and reliability as other hard problems and they certainly are. Still I'll also handwave those away not as technological issues but as transaction costs; the economic benefits of ubiquity and leverage are key here. It seems that humans are quite prepared to tolerate gremlins and parasites in their systems (social, biological and technological) in moderation of course.

What is interesting is that almost all of these issues are important even if you aren't dealing with distributed computing. On the internet, which is all about distributed computing however, these are crucial problems. Now addressing these problems is a pretty tall order I must admit, but it seems to me that the web style has concrete answers to almost all of these constraints. Furthermore REST also offers a couple of those very valuable system characteristics: resilience and adaptability, almost as externalities.

Search for example seems to be addressed by valuing the importance of resource identification, the URI, that leads to PageRank and other algorithms. If you start with a viewpoint that URIs are cheap, and if you model your resources appropriately, you can deal with concurrency quite reasonably. State, caching and latency are all to do with performance which is always a bear. To that I'd argue that the commerce that is taking place daily on the internet seems to be a good indication that things are acceptable even if they could be improved. Metadata is the eternal headache of course but HTTP will give you headers and soul to help a little.

Discovery is addressed by URI as identification, the notion of hypermedia (either link headers or references in the media) and perhaps the current best practices of introspection resources. Persistence and structured data, I'll argue, are being addressed, warts and all, by the relational database, the seemingly-unstoppable spread of XML and, I believe, the current ascendancy of feeds. On this last subject, the wiring of the web, I am cautiously optimistic that the Atom store dream could be an answer; i.e. the combination of feed format (say Atom/RSS) and a wire protocol - which perhaps the Atom Publishing Protocol or similar could be. On the question of extensibility, last year I started a response that I felt was needed due to Dion Hincliffe's unbelievable assertion that "extensibility was the Achilles heel of REST". It always takes me a year to respond to minor footnotes but if there's one thing I'll do in the next few months, I'll be sure to write about REST and the Holy Grail of Extensibility.

But anyway, that is getting ahead of myself. I've been told to save my musings for "the book" or to stick to writing the running code that will "show, rather than tell" yet I keep falling for this web as conversational engine. Still there are undoubted social benefits of these exchanges and perhaps conversations like these are another virtue and externality of the web style. In any case, consider this my long winded response, Ryan: in matters of technology adoption and systems design, let's stick to the elevator pitch.

Further reading:

• REST - the web style
• A REST Intervention
• On Iframes, WSRP, REST and Portals
• The Unloved HTML Button and Other Folktales
• The Gospel of REST
• The Economics of REST

The Sequel

[Update - March 27, 2006] It seems as if everyone is getting into shaping the pitch and the language has been quite interesting.

• Danny Ayers remarked that we should go back to the source and points at Tim Berners-Lee's pitch from last year

  Web architecture 101

  • Things are denoted by URIs.
  • Use them to denote things.
  • Serve useful information at them.
  • Dereference them.

  That indeed is about as brief a pitch as some executives will be able to handle and it covers the essentials.
• Tim Bray's pitch is fairly pithy too, and for the last plank he uses

  You expect to ship a lot of URIs around in the bodies of requests and responses, and use them in operations that feel like link following.

  Now that's quite a mouthful in a presentation but it works fine in an elevator. I think "dereference them" is more terse and captures much of the same insight.
  
  As an engineer who is interested in precision in language, I tend to prefer Roy Fielding's formulation of "hypermedia as engine of application state". It's short and alludes to both the implementation and the area of focus of the web style. But your mileage may vary and I think it depends on the audience. We should be interested in a pitch that works because the advocacy struggle isn't over even if some are already thinking about the future issues. In this light, it was good to see Joe Gregorio again emphasizing the Show me the code route he embarked on last year. That View Source Imperative is the secret ingredient as far as adoption goes.

Note that everyone seems to conform to the Rule of Four as far as bullet points in the pitch go. Interestingly enough, Roy Fielding used to go with a five point pitch until he merged what were his previously separate definitions of "resource" and "URI" into the phrase "Identification Of Resources". I've proposed my own four point plan as a model for ArRESTed Development or The Low End Theory.

I concur of course with the endorsement of the formulation of REST - the web style and not just because I seem to have juice (of the google flavour) on that front having internalized that phrase long ago in my interventions, but because it gives us a tag, REST, combined with a catchphrase. As we know, every movie needs a tag line. Especially in Deadwood: "6 million ways to die, choose one". That is the B-movie of distributed computing. We want to move beyond grindhouse pulp fiction and get some of those glamourous, golden Hollywood trophies. And if that fails there's always the Gospel of REST.

The Elevator Soundtrack

• Outkast - Elevators (Me & You)
  The chorus of this laidback piece of Atlanta folklore could well be a stand-in for the inclusiveness of the web style and its architecture of participation.

  Me and You
  Your Momma and Your Cousin Too
  Rollin' Down The Strip On Vogues
  Comin' Up Slammin' Cadillac Doors

• [Update] A hat tip to the good doctor, Ernie Prabhakar, who points to a real life Huxtable Quartet. When I first started using HUHXtable, I was thinking of the Cosby Show and of comfy sweaters, thus it's great to hear that the Slide Huxtable Quartet is itself a tribute to that show and to the "Play It Again, Russell" episode in which Cliff's father, trombonist Russell "Slide" Huxtable (Earl Hyman) hosts a jam session. REST has even got a house band.
• Miles Davis - Ascenseur Pour L'Echafaud
  I've always loved Miles Davis's cool soundtrack to one of the great film noirs which saw a great print released last year. It's atmospheric and gets the job done - and with verve. I am always upset that ascenseur is translated as lift rather than elevator which is more heady to my ears. I'll conclude by suggesting that the web style will save you from heading up the Elevator to the Gallows.

File under: technology, rest, systems, software, design, architecture, advocacy, leverage, adoption, composition, tradeoffs, http, uri, xml, html, atom, web, language, strategy, toli

On Iframes, WSRP, REST and Portals

• On Prescience
• On Iframes, WSRP, REST and Portals

I'm still getting lots of good feedback to my presentation on REST - the web style (background) which I will summarize shortly. One of the more curious responses was to something that I didn't even dwell on in the presentation; indeed I think I skipped past it in my concern to get the big picture articulated. I suppose it was the proximity of yet another acronym, WSRP, to a couple of slides about complexity in the IBM technology stack, and a very public case study bemoaning the fact that Websphere Portal only added bookmarking after 3 years that raised a little concern. This was part of my subsequent response to one of those comments, slightly edited for clarity.

On Prescience

I fully agree, we should celebrate any and all who contributed to us internalizing the web style in WebSphere Portal. We should be very explicit about the pain we and our customers suffered and the evident gains that came once we learned our lesson.

The notion that it was still a struggle to get that simple feature (the ability to bookmark) included in the WebSphere Portal 5.1 release is actually a commentary on our previous attitude to the web style.

I would add to the list of those to be celebrated, those screaming Cassandras who were incredulous that we could ship a web product while ignoring url addressability and even tried to prototype some rudimentary addressability and sneak it in back when we had commit privileges to the core. If only we had had more nerve or had been able to better communicate the importance of this point... I know it normally takes 3 releases for a product to become useful, but sometimes magic can happen in the first release and with that comes thought leadership (Glory) and perhaps also some money (Gold).

On Iframes, WSRP, REST and Portals

Leading with the iframe invocation argument was always the weakest argument of those who I'll cast as the "K-stationers" in the early days of WebSphere Portal. After all, and like you note, even though iframes may provide

• easy integration of existing content
• multi-threaded download in the browser to lower the latency for the user
• caching closer the client (indeed in the client)
• offloading the portal by removing a layer of indirection in the portlet -> portal server -> client browser route
• introducting the possibility of moving aggregation to the client (leveraging Moore's law and bandwidth increases to the client)
• further recasting the "Portal Server" to a "Portlet server"(in other words there should be no difference between a "remote" and a local portlet) etc . That is the lesson of uniform interfaces in the web (e.g. URIs)

Now that's a pretty strong "weak argument" but we should acknowledge that Iframes also have downsides. Iframes mess with user expectations especially with respect to focus and some browser conventions are a little upset by them (e.g to bookmark, you need to right click on the frame in question). Browsers did solve the issues with the navigation history after a few years, but there was a little confusion in the interim with back button and refresh etc).

6 years on, I hope I have learnt to recognize the stronger aspects of an argument about systems design. What we didn't recognize or adequately evangelize was the value of the other aspect of an approach that more fully embraced the web, namely:

Basing the application semantics of the portal directly on URIs and the use of those 4 verbs that are the core of the web.

To explain, HTTP is an application-transfer protocol. It is something that works at the application layer. It has its native notion of how CRUD like operations are done (GET, POST, PUT, DELETE).

The REST proposition again:

• Identification Of Resources
• Manipulation Of Resources Through Representations
• Self-Descriptive Messages
• Hypermedia As The Engine Of Application State

I haven't kept abreast of WSRP in the past 5 years and thus my characterization of it was perhaps a little careless, but that's what you do when you make a presentation that borders on the provocative. In mitigation, I'll confess to being what I've termed a Layer Stripper.

Let me handwave a little... It strikes me that building an application protocol with CRUD like functionality that will typically (or even solely) be used over HTTP can lead to a little impedence if one is not careful.

To take my Technical Arteriosclerosis Terminator scalpel to WSRP, I would only ask:

• Do the CRUD like operations in WSRP map one to one with HTTP's native semantics? In other words, is the moral equivalent of a PUT in the WSRP world implemented as a PUT when the "transport" is HTTP?
• Is this true of at least those 3 verbs at the core of the Web (GET? POST? DELETE?). (I'll assume that WSRP adds perhaps more verbs in its applications semantics).
• Are the security primitives going to be leveraging HTTP's native security features or are we rolling our own?
• Are there any other "transports" other than HTTP? Is there a CORBA mapping for WSRP?

RFC 3205 On the use of HTTP as a Substrate should be required reading here.

I'll suggest that if we were starting from scratch we might do the following

• Start with Resource Modeling and ask: what are the core resources that need to be identified in a remotely invoked portlet?
• What representations need to be returned? For this we'd probably pick some standard Portal XML vocabulary or, for browser rendering, an XHTML format, for pervasive rendering some other Markup Language
• What are the operations that need be performed on said resources?
• What HTTP verbs are appropriate for each operation when we manipulate those resources?

Joe Gregario's Show Me The Code column has been going through this exercise with the example of a bookmarking service - well worth reading. A remote portlet may have more complex semantics but one should go through this to clarify our design and architecture. I assume that WSRP implementors went through such a process as that spec was standardized.

The result would be something like simple URI commands returning representations and manipulating them using well-worn HTTP semantics. I very much hope that is what WSRP has evolved into. We'd get:

1. Visibility to HTTP intermediaries, first the caching proxies in the web sense. (like the page caching in the portal)
2. Since we'd be using URIs, someone may think up new uses for our remote portlets and be able to easily embed them in their application through simple hyperlink and the use of the appropriate verbs etc. Third party Glue Layer people could "remix" our portlets through pipelining and filtering or whatever else those people do (I'll note that IBM recently bought Gluecode)

With this approach there is no API per se, the interface contract is explicitly manifested in the exchanged hypermedia and associated operations on the resources.

"What? No API?" "This RESTful stuff has no substance". "Where is the value add?"

That indeed is the paradox of the web style. The "API" is typically HTTP/URI/HTML/XML.

The fourth plank of the REST proposition is a difficult thing to internalize, I certainly couldn't express it adequately 6 years ago.

"Hypermedia as the engine of application state"

Ponder that for a few weeks.

[Update] James Snell adds: "HTTP is the programming model".

Some have suggested that one would additionally require some service description language, perhaps WSDL, for such a service. On that, the jury is still out. In the example I highlighted in the presentation, Google Maps didn't publish any WSDL. Craigslist didn't publish any WSDL. They simply had a clean URI api with well-defined parameters that were obvious to any developer with a week to spare and idle curiousity satisfied by the View Source impulse. It was obvious what verbs to apply: POST or GET, PUT (they don't delete their maps, but if they did, that is how one would proceed). Third party intermediaries simply figured out that url hierarchy and the associated verbs, the identified resources and what formats the representations were retuned in. Then they manufactured serendipity. Now there is no binding contract and things can get broken at any time but it wouldn't be in Google's interest in their competition with Mapquest or Terraserver to break people that compose applications on their platform and brings eyeballs and mindshare to their company.

This then is my argument for changing the frame of the debate:

There's a complexity and layering argument that naturally falls out of the REST viewpoint which should guide the applications that one builds.

The web is about identifying important resources and exchanging representional state. There are certain constraints that are made to enable global scope and evolution. Being on the web, is being an active participant in this scheme of things.

Identifiers lead almost to having the location field as the command line.

Resource modeling as your starting point sets you in direct consonance with this notion and exposes you to ease of composition.

e.g. Jon Udell's Library Lookup project

An information architecture that starts with hypemedia lends itself to the construction of simple interaction designs for both humans and machines, and there are huge amounts of tools readily available for this.

The question is then one of leverage; leverage in economic terms is all about making externalities work for you. The endpoint of this approach to system design is what I call a virtuous cycle of managed serendipity.

One response to this piece was that actually WSRP stood up quite well to those questions I had hand-waved in its direction, apparently it was not too complex, it addressed wider issues and use-cases etc. I think that is great and hopefully the leaks have been plugged in that spec so that it can be mapped reasonably to HTTP; I guess the argument is that it is not an unnecessary layer. I'm still unclear if WSRP is ever used with a "non-HTTP transport" or security, I'll dig into that at some point. In any case, the point of my talk was the same scrutiny should be brought to bear on all of IBM's technology stack and unnnecessay layers shed and/or leaky abstractions plugged to make them work better with the web style.

A later follow-up, privately-addressed, asked:

"Isn't a "portlet" fundamentally a presentation-level object? Why is the concept of "portlet remoting" not considered an oxymoron? Shouldn't we be talking instead about the simple expression of web services as data (with, perhaps, media-type detection as an enabler of polymorphism"

There was lots more and quite fulminant at that, I'd have to say that's all true and expressed more concisely than I did. It's good to know that there are like-minded people around. I didn't mention content-negotiation in my response but that is another aspect built into the web style that can be leveraged but that is constantly being reinvented at higher levels and layers. There's lots more to say but that's the lay of this technology world... In evangelism the first step is to get people sensitized, after that it's best to proceed one argument at a time.

---

P.S. I've been told there's been too much technology toli of late so I'll try to push some literary, musical or African toli to the head of the queue, it can be more fun and oftentimes more rewarding.

File under: technology, web, REST, systems, design, architecture, tradeoffs, adoption, leverage, composition, iframe, wsrp, WebSphere, ibm, portal, portlets, aggregation, software, complexity, uri, http, xml, html, xhtml, lotus, IBM, toli

A REST Intervention

Responding to James Snell's article, Resource-oriented vs. activity-oriented Web services, a dissent in 4 parts and a contribution to the ongoing (never-ending?) debate on REST (Representational State Transfer), web services and SOAP (Simple Object Access Protocol)

1. A Snap Judgment
2. Changing The Frame
3. Internalizing the REST style - A case study: WebSphere Portal
4. An Argument by Analogy

1. A Snap Judgment

Sometimes you read or hear something that immediately strikes you as wrong-headed and sets you off. In the real world, if say you were in the same conference room, you might cross your arms, prepare to raise your voice a little as you ready a sharp retort. If you were on a mailing list, this is the kind of thing that will start you writing a lovely flame, complete with point-by-point rebuttals. I've learned dear lessons about the social cost and inadvisability of rushed (and rash) reactions in the online world, so I confined my dissent to bookmarking the article and tagging it with the following inchoate comment:

Completely missing the point about REST and what it means to "be on the web."

A knee-jerk response or snide and unfair commentary perhaps, but at least his piece didn't leave me indifferent like much of what I read on the web.

I've cooled somewhat in the intervening weeks, also I've re-read what Snell wrote and realized that I'd missed much of his point. It's a well-meaning and well-reasoned exposition; it's hard to argue with the form if not the substance. There are lots of caveats and "don't get me wrong"-s, cues that suggest a more nuanced picture than I obviously gave it credit. He's one of the good guys, a colleague with whose group I've often worked with, and this is very far from the typical (and occasionally disingenous) posturing of the REST/SOAP debate.

I'll start my considered dissent then by modifying my assessment: Snell does understand REST, he gets the point but is just cautious about its application and, like all of us, is struggling with the issues it raises. To him, it's just another approach in his toolchest, there's a sense of "different stokes for different folks" in his manner. Engineers are all about evaluating tradeoffs and pragmatism like his is often just sound engineering.

I think my initial reaction was coloured by having read too many journalists who cover the world of technology adoption and often cast things in the winner-takes-all mold. At ground level in the technical trenches where Snell and I live, the calculus of profits is less important; it is rather a marketplace of competing ideas and we're picky and conservative buyers on the whole.

Still in my dissent, I need to explain the second part of my statement which relates to the importance of "being on the web". There has to be a response in writing, if not in code. Ideas don't exist in a vacuum and I shouldn't take to some ivory tower, with Fielding's bible on hand, Prescod and Baker as prophets in the wilderness, and Bray as curmudgeon and loyal oppositionist-in-chief all the while pointing to Apache, Amazon, Yahoo, flickr, del.icio.us and others as favoured offspring and self-evident existence proofs. That's not a sufficient response.

2. Changing the Frame

Fair enough then, how best to articulate this lingering sense I have that Snell short-changes or side-steps the value proposition of the REST architectural style?

The following tidbit from another context seems strangely relevant here:

Oliver Hass, a 28 year-old chemist and graduate student from Oldenberg, Germany, wrote me recently about what the President's trip looked like to him. In introducing himself, Hass commented on "how necessary it can be for a chemist to forget about molecules and think about real problems."

In terms of technical advocacy, others far more experienced than I and have written eloquently and at length wonderful treatises about the virtues of REST and why wire protocols matter. Bray has a very good writeup on recent thoughts. There have been lots of attempts at articulating best practices even prominently featured in wikis. In recent months, Joe Gregorio has been embarking on the show me the code path, doing some of the most effective advocacy by plain example.

My gut feel is that the disposition of the debate will be ultimately be determined by running code and one engineer at a time, hence I won't add to the heated rhetoric. So then I'll make the advocacy argument by changing the frame, and casting the REST value proposition almost in economic terms, in terms of systemic leverage, pragmatism and ubiquity.

Positing, like Snell does, that is about a resource-oriented view rather than an activity-oriented view belies a significant question. Surely benefits would accrue if "activity-oriented" items were exposed on the web? Indeed, these days if you don't expose your offering or activity on the web, you hardly matter. If you're ultimately going to derive much of your value from web exposure, why accentuate the impedance mismatches between your technology stack and the prevailing architecture? For indeed there is an architecture to the web. This is evident when one considers the differences between HTTP 1.0 and HTTP 1.1. The REST style simply elucidates the philosophy of the underlying architecture.

For some reason, REST is much misunderstood; the handwaving description "the best of the lessons learned in developing the web" needs much elaboration, as does the elevator pitch version:

REST is defined by four interface constraints:

• identification of resources
• manipulation of resources through representations
• self-descriptive messages
• hypermedia as the engine of application state

Adopting the REST style is not a technology prescription or panacea; after all, an architectural style is not a programming toolkit. But there's a complexity and layering argument that naturally falls out of the REST viewpoint which should guide the applications that one builds.

The web is about identifying important resources and exchanging representational state. There are certain constraints that are made to enable global scope and evolution. Being on the web, is being an active participant in this scheme of things. Identifiers lead almost to having the location field as the command line. Resource modeling as your starting point sets you in direct consonance with this notion and exposes you to ease of composition. An information architecture that starts with hypermedia lends itself to the construction of simple interaction designs for both humans and machines, and there are huge amounts of tools readily available for this.

The question is then one of leverage; leverage in economic terms is all about making externalities work for you. The endpoint of this approach to system design is what I call a virtuous cycle of managed serendipity.

3. Internalizing the REST style - A case study: WebSphere Portal

At a certain point there were (at least) 8 different groups at IBM developing portal software. Incidentally, this is par-for-the-course in large companies; executives are often willing to allow this kind of competition as a kind of macabre survival of the fittest game - for a while at least. Two groups managed to ship products to customers, first Lotus K-station and then what was eventually the winner, WebSphere Portal. K-station was a product which, almost to a fault, embraced and leveraged the full range of web technologies. It demoed well, made pervasive use of CSS, DOM scripting, XML etc: web to the core in other words. Although it was a J2EE application, its fatal failing was that it shipped initially on top of Lotus Domino's servlet engine and database rather than on top of WebSphere Application Server and DB2. What became WebSphere Portal started by building on the right platform, its front end had what we euphemistically termed teething problems but it had a reasonable backend and the virtue of an architecture that looked like it could eventually evolve into the "one true portal".

As we consolidated all the efforts under the rubric of WebSphere Portal, there was of course a kind of triaging of the expertise and feature sets from the various groups. The Lotus folks emphasized collaboration and mostly owned that side of things but we also had a laundry list of technologies that we wanted to endow the platform with (from basic ideas about the way one used stylesheets for skinning, all the way up to a repartitioning of the architecture to allow more intelligence in browser clients and leverage of client side technologies that are now increasingly common). Call it youthful exuberance or perhaps in my case personal temperament, I argued each and every feature tenaciously and energetically. When it comes to technical argument, I'm not afraid to offend senior architects, product managers, or even distinguished engineers far beyond my station. In hindsight, we perhaps should have been more judicious in picking our battles because even though we managed to win many of the arguments, we lost one that we really shouldn't have. Mostly through sharp-elbowed bureaucratic maneuvering, the feature was accepted as a line-item but was marked as a nice-to-have, and then was deferred because "we didn't have the time in the schedule".

What feature was it, you might ask? There was no way to bookmark anything in WebSphere Portal.

The portal at that point bore the vestiges of an idiosyncratic framework (Jetspeed) that in my opinion actively fought against being bookmarked. No one at Lotus could fathom that a web application could ship without bookmarking. We couldn't even do the simplest use-case from K-station: when creating a place, you couldn't send an email to new members inviting them to it. An email could only direct them to the home page, once they logged in, they'd then have to navigate to the item in question and as we'd made it flexible to skin navigation, there was no easy way to describe how one would navigate to that item. The immediacy of collaboration, of "people, places and things" was irrevocably lost. I argued almost to disgust: "This is the web! Resource identification is a key tenet, almost the most important principle. It's a fundamental flaw" etc. to no avail.

It took 3 releases for this feature to be implemented and enabled in the default installation of the portal. Throughout, we continued to lobby, until the chorus grew too loud and it wasn't just "those Lotus folks" making the argument. Being able to bookmark places, pages and portlets is fundamental for many reasons beyond simple usabilty. At a first order, you were now able to paste a url in an instant message to co-workers so they could join you and see what you were looking at, at that moment. But also, almost overnight with this feature, many layers of complexity were shed in the portal framework.

Programmability in the web sense was immediately enabled, the portal became a composable platform and we were able to layer the Lotus Workplace offering on top of it. URIs give visibility to intermediaries and so things like caching (where we had cool technologies like Dynacache) were far more easily enabled. Similarly for logging and profiling the portal, we could use the same tools for processing logs as exist for regular web servers like Apache. We had new opportunities for pipelining and filter chains (to do transcoding if needed). We had more options for load balancing, we could decide to deal with remote portlets through iframe invocation rather than through immature and complex protocols like WSRP. And so on...

My biggest regret is that we hadn't been bloody-minded enough to do a stealthy check-in of the hacked prototype code for enabling a modicum of url addressability that we had developed. Instead, by being consensus-minded corporate citizens, we had allowed our platform (and users) to suffer for 3 years because it hadn't internalized this most basic aspect of being a web application framework. WebSphere Portal only acknowledged the web in its name once it embraced this part of the REST ethos and it hasn't looked back since. Indeed after internalizing resource identification, the product has taken to heart most of the other tenets of the web style. It still has some ways to go but, from that point on, it was no longer odds with the web.

As an aside: where there is friction and impedance mismatches, there inevitably will develop an ecosystem of consultants with palliating bromides and band-aids on hand. All companies claim to be "focused on the customer", my own company has the virtue (or some would say the failing) of encompassing the widest range of products and services and is apt to make money whether one buys hardware, software, or services from it. If the choice is between profitable software that internalizes the lessons of a platform and rank consultant-ware, I opt firmly for the former. Maybe I'll change my mind if I become one at some stage, but consultants, though often necessary, are mostly irksome from my standpoint.

4. An Argument by Analogy

The supposed or acknowledged (PPT) deficiencies of the REST architectural style (reliability, asynchronous event notification etc.) are to my mind good problems to have. It strikes me though that these criticisms of REST system design, while fair, are mostly concerned with boundary cases; in other words, the balance of optimization that the web style has gone for makes these mere irritants not failings. The response should be much like that typically given to those who discounted the 'best effort' internetworking or to the bellheads who harp on quality of service and see the internet as a toy. I don't see the quality of calls preventing the ubiquity of cell phones. In that case, the virtues of mobility long trumped reliability. Network standards like ethernet and TCP/IP that are the core of the Internet have succeeded and scaled because they used abstractions that embraced simplicity, transparency and existing systems. The web was able to co-opt everything in sight (from ftp to gopher to whatever newfangled scheme we'll come up with next) with only minimal constraints for the sake of scaling and composition (like the concessions to, and explicit recognition of, the interplay with intemediaries - caches and so forth). Google (pdf), Yahoo, eBay and Amazon have interesting takes on addressing these deficiencies and already operate on a global scale.

Martin Geddes, discussing a different problem, ever-quotably and far more concisely than I'll ever be, works himself to the same fundamental conclusion:

Solutions based on URLs tend to be less "efficient" than custom-designed solutions, but more resilient in the face of change. (Sound familiar?) I casually note the rapid rise of URL-friendly blogs and wikis, and the relative obscurity of vertically integrated collaboration tools like Groove.

I saw a good example of "URLization" from Joi Ito the other day. Product managers at various companies are being encouraged to use Technorati to track commentary on their products. If your product doesn’t have a clear, unique URI, then you’re in trouble...

This all brings me to my big question: on the Internet, if something doesn’t have a URL, does it really exist? Or has it just disappeared into the analog memory hole, only existing as a memory in the brains of the humans it passed through?

That's the heart of the matter, debates about "programming models" aside, this is an argument about systems design and technology adoption. If your nascent technology stack doesn't leverage, or as I think is the case in many of the WS-* protocols, actively fights against integration with the web, you're guiding yourself towards obsolesence. The REST orientation is about acknowledging the "good enough" factor implicit in the web architecture. Choosing to embrace the ethos of that layered style is arguing for leverage and simplicity. The value of ubiquity and managed serendipity that are often the outcomes of this approach should speak for itself.

Returning to what provoked this outburst, I write to James Snell: thanks for getting me thinking enough to put these thoughts together, my arms are uncrossed and there's even a smile on my face.

See also: my other REST writings. File under: REST, web, design, software, programming, architecture, Lotus, IBM, WebSphere, SOAP, technology, uri, identification, addressability, programmability, url, networks, standards, Portal, toli