The REST Elevator Pitch

• The Pitch
• The Hard Problems
• The Sequel
• The Elevator Soundtrack

The Pitch

I somewhat share Ryan Tomayko's concern for a shorter hand language for describing REST (Representational State Transfer), but I believe that the REST elevator pitch is pretty succinct already. It packs a lot of insight into the web architectural style of which dissertations have been written. The language used in technical debates is always important and rather than devolving into the fuzziness of "highs and lows", I'll suggest here that, as a matter of advocacy, one should stick to the elevator pitch.

To recap, the REST elevator pitch is:

• Identification Of Resources
• Manipulation Of Resources Through Representations
• Self-Descriptive Messages
• Hypermedia As The Engine Of Application State

In its most common application, the pitch will lead you to the HUHXtable Quartet (HTTP / URI / HTML / XML) if we handwave away the additional content types that the web style enables you to negotiate. It is quite clear that applications of all kinds have been built on these fundamentals for both human and machine interaction.

The thing is that each plank of the pitch brings its own benefits, benefits that perhaps can accrue independently. Still you get the most bang for the buck at the internet scale by the principled combination of all of them.

In my advocacy I tend to start with resource identification as the most important plank. The reason for this is that everybody can understand the value of identification. My grandmother understands the value of a bookmark to my photos. Heck we are even seeing the social benefits of ostensibly selfish bookmarking. The importance of resource identification cannot be understated and it leads us to the primacy of the URI.

The second plank, manipulation of resources through representations, is the bread and butter of HTTP, where we have an evolving set of standard formats of hypermedia exchanged using a small set of verbs.

Unpacking the HTTP proposition leads us to the four horsemen of the web: GET / POST / PUT / DELETE

Of course you can do very well if you pick up only 2 of those verbs, GET and POST. The good Reverend HTTP is a pretty tolerant fellow and won't begrudge those that don't believe in the rule of four, although in admittedly imperfect analogies, there's CRUD (Create / Retrieve / Update / Delete) in the database world and, as far as this layered internet goes there's TCP / IP / Datalink / Physical Layer (with say Ethernet or "Wi-Fi" 802.11b/a/g)

On the other hand, if you want to be a good citizen in the town of Deadwood and indeed if you want to survive in the wild west of the web, you shouldn't abuse HTTP. HTTP abuse leads to user confusion, headaches for systems designers and a tragedy of the commons. I consider HTTP a deliberately minimalist compromise, laissez-faire distributed computing if you will.

In social settings making your intentions known can help smooth interactions. We may start with small talk but we eventually manage to discern where others stand - well most of the time at least. We all operate with imperfect information but being straightforward is often a great policy. I'm sure there's some game theory or behavioural economics that can point to the value of transparency. Perhaps it is trite in light of Machiavelli and Sun-Tzu to believe that disclosure is the best policy but at least in markets transparency tends to be the font of aggregated wisdom. On the internet scale, this translates into the notion of giving visibility to intermediaries. Arguing again by analogy, the internet is an agreement, a network of networks. You can do whatever you want on your network but to interoperate with others, a small set of primitives, the TCP/IP standards in this case, have been agreed upon. I would suggest that the web style is similarly such an overlay system. REST is an architectural style which by principled design, is optimized to lower coordination costs.

Acknowledging the benefits of idempotency, of doing things safely, will lead one to making sure the use of GET is safe. Idempotency is not just about safety however; in clearly signaling your intentions, it is also about replayability. Thus idempotency is not just about the GET method. That is the wider argument for also using that oft-neglected duo, PUT and DELETE. Tunneling everything through POST (and the completely aberrant case GET) may be convenient but it adds unnecessary opacity and unpredictability about one's application. Unpredictability in the world leads to invasions and worse I might add. The use of those four core HTTP verbs helps intermediaries understand and participate more fully in the ecosystem.

The last two planks of the REST elevator pitch, self-descriptive messages, and especially that last one, hypermedia as the engine of application state, are attempts at addressing the issues of state, caching and evolving systems amongst other things.

Now of course in the preceding paragraphs, I've handwaved away entire dissertations that have been written about architectural styles and systems design, tomes written with language more precise and concepts more subtle than I've outlined. Good samples and running code are often also a determining factor in which style or application is adopted thus the view source imperative is also one of the reasons that web has caught on so quickly.

The Hard Problems

I've recently been thinking about defining the hardest problems I've encountered in software engineering, my cursory top 10 list:

• State
• Caching
• Latency
• Concurrency
• Search
• Metadata
• Persistence
• The Holy Grail Of Extensibility
• Structured data
• Character encoding

Now I'm not a database person so I handwaved away all of those data peoples' worries in one word: persistence. Of course I should probably add a few more issues and indeed you probably have your own notions. Indeed I settled just two days ago on those last two, structured data and character encoding, when I characterized them as being the gruesome twosome of computer science. Perhaps also there's the issue of discovery and bootstrapping of a system but that could be considered implementation details. There are those who mention security and reliability as other hard problems and they certainly are. Still I'll also handwave those away not as technological issues but as transaction costs; the economic benefits of ubiquity and leverage are key here. It seems that humans are quite prepared to tolerate gremlins and parasites in their systems (social, biological and technological) in moderation of course.

What is interesting is that almost all of these issues are important even if you aren't dealing with distributed computing. On the internet, which is all about distributed computing however, these are crucial problems. Now addressing these problems is a pretty tall order I must admit, but it seems to me that the web style has concrete answers to almost all of these constraints. Furthermore REST also offers a couple of those very valuable system characteristics: resilience and adaptability, almost as externalities.

Search for example seems to be addressed by valuing the importance of resource identification, the URI, that leads to PageRank and other algorithms. If you start with a viewpoint that URIs are cheap, and if you model your resources appropriately, you can deal with concurrency quite reasonably. State, caching and latency are all to do with performance which is always a bear. To that I'd argue that the commerce that is taking place daily on the internet seems to be a good indication that things are acceptable even if they could be improved. Metadata is the eternal headache of course but HTTP will give you headers and soul to help a little.

Discovery is addressed by URI as identification, the notion of hypermedia (either link headers or references in the media) and perhaps the current best practices of introspection resources. Persistence and structured data, I'll argue, are being addressed, warts and all, by the relational database, the seemingly-unstoppable spread of XML and, I believe, the current ascendancy of feeds. On this last subject, the wiring of the web, I am cautiously optimistic that the Atom store dream could be an answer; i.e. the combination of feed format (say Atom/RSS) and a wire protocol - which perhaps the Atom Publishing Protocol or similar could be. On the question of extensibility, last year I started a response that I felt was needed due to Dion Hincliffe's unbelievable assertion that "extensibility was the Achilles heel of REST". It always takes me a year to respond to minor footnotes but if there's one thing I'll do in the next few months, I'll be sure to write about REST and the Holy Grail of Extensibility.

But anyway, that is getting ahead of myself. I've been told to save my musings for "the book" or to stick to writing the running code that will "show, rather than tell" yet I keep falling for this web as conversational engine. Still there are undoubted social benefits of these exchanges and perhaps conversations like these are another virtue and externality of the web style. In any case, consider this my long winded response, Ryan: in matters of technology adoption and systems design, let's stick to the elevator pitch.

Further reading:

• REST - the web style
• A REST Intervention
• On Iframes, WSRP, REST and Portals
• The Unloved HTML Button and Other Folktales
• The Gospel of REST
• The Economics of REST

The Sequel

[Update - March 27, 2006] It seems as if everyone is getting into shaping the pitch and the language has been quite interesting.

• Danny Ayers remarked that we should go back to the source and points at Tim Berners-Lee's pitch from last year

  Web architecture 101

  • Things are denoted by URIs.
  • Use them to denote things.
  • Serve useful information at them.
  • Dereference them.

  That indeed is about as brief a pitch as some executives will be able to handle and it covers the essentials.
• Tim Bray's pitch is fairly pithy too, and for the last plank he uses

  You expect to ship a lot of URIs around in the bodies of requests and responses, and use them in operations that feel like link following.

  Now that's quite a mouthful in a presentation but it works fine in an elevator. I think "dereference them" is more terse and captures much of the same insight.
  
  As an engineer who is interested in precision in language, I tend to prefer Roy Fielding's formulation of "hypermedia as engine of application state". It's short and alludes to both the implementation and the area of focus of the web style. But your mileage may vary and I think it depends on the audience. We should be interested in a pitch that works because the advocacy struggle isn't over even if some are already thinking about the future issues. In this light, it was good to see Joe Gregorio again emphasizing the Show me the code route he embarked on last year. That View Source Imperative is the secret ingredient as far as adoption goes.

Note that everyone seems to conform to the Rule of Four as far as bullet points in the pitch go. Interestingly enough, Roy Fielding used to go with a five point pitch until he merged what were his previously separate definitions of "resource" and "URI" into the phrase "Identification Of Resources". I've proposed my own four point plan as a model for ArRESTed Development or The Low End Theory.

I concur of course with the endorsement of the formulation of REST - the web style and not just because I seem to have juice (of the google flavour) on that front having internalized that phrase long ago in my interventions, but because it gives us a tag, REST, combined with a catchphrase. As we know, every movie needs a tag line. Especially in Deadwood: "6 million ways to die, choose one". That is the B-movie of distributed computing. We want to move beyond grindhouse pulp fiction and get some of those glamourous, golden Hollywood trophies. And if that fails there's always the Gospel of REST.

The Elevator Soundtrack

• Outkast - Elevators (Me & You)
  The chorus of this laidback piece of Atlanta folklore could well be a stand-in for the inclusiveness of the web style and its architecture of participation.

  Me and You
  Your Momma and Your Cousin Too
  Rollin' Down The Strip On Vogues
  Comin' Up Slammin' Cadillac Doors

• [Update] A hat tip to the good doctor, Ernie Prabhakar, who points to a real life Huxtable Quartet. When I first started using HUHXtable, I was thinking of the Cosby Show and of comfy sweaters, thus it's great to hear that the Slide Huxtable Quartet is itself a tribute to that show and to the "Play It Again, Russell" episode in which Cliff's father, trombonist Russell "Slide" Huxtable (Earl Hyman) hosts a jam session. REST has even got a house band.
• Miles Davis - Ascenseur Pour L'Echafaud
  I've always loved Miles Davis's cool soundtrack to one of the great film noirs which saw a great print released last year. It's atmospheric and gets the job done - and with verve. I am always upset that ascenseur is translated as lift rather than elevator which is more heady to my ears. I'll conclude by suggesting that the web style will save you from heading up the Elevator to the Gallows.

File under: technology, rest, systems, software, design, architecture, advocacy, leverage, adoption, composition, tradeoffs, http, uri, xml, html, atom, web, language, strategy, toli

Minutiae

A surprisingly large part of a software engineer's life is spent dealing with the little things. Much as I like to write about grand designs and architectural issues or people, processes and communities, all too often, the devil is in the details and I get lost chasing technological quirks. Herewith a sample of just the past week's minutiae, perhaps fodder for historians of science or anthropologists.

• Boolean Identity Crisis
• Structured Data Footnotes
• Highlights, Lowlights
• Which Side Are You On?
• The Null Hypothesis
• Camel Humps
• The East Australian Singularity

Boolean Identity Crisis

I was going over some code in my pet XForms processor implementation and wondering what was going wrong - incidentally IBM has written (at least) 6 forms processors in the past 4 years along with contributing to the Mozilla XForms effort but we won't get into that - I'll save that for a business school case study or something.

After an hour or so of head-scratching this was what I found. In XML Schema Datatypes, the lexical space of boolean values includes not only "true" and "false" but also "0" and "1". I assume the inclusion of 0 and 1 in the specification comes from the legacy of the C language and presumably that makes sense. But I ask, was that a wise decision? The designers of the specification chose a binary representation in a textual format.

The forms processor is written in Java. It turns out that in the Java language, the lexical space of boolean values is case-insensitive "true" and "false". So if you have a mapping layer that goes from schema datatype to Java you have to add special case code to deal with 0 and 1. Presumably also in the other direction, you have to make sure that you don't output "True" instead of "true". I had been linking to a library that hadn't bothered to implement this more robust logic and ran into this boundary condition. Oh well, I thought, I'll have to add an adapter around this library or rip it out and roll my own - an assignment for the weekend.

I vaguely remember Sam Ruby mentionning that this was one cause of interoperability issues between SOAP implementations and it stands to reason: we've codified an identity crisis.

Structured Data Footnotes

Jon Udell is a careful man whose

blog works differently from the rest of the InfoWorld blogs. The content is well-formed XML, and it follows certain self-imposed rules.

This discipline enables him to work miracles and regularly come up with lots of cool applications. But in the wider world, getting people to author structured data is an often intractable problem. XML is often structurally invalid; as an example the Google Reader team have reported that 15% of feeds on the web are not well formed and that is before considering the feeds' semantic validity.

Thus I was tickled by this footnote he wrote when playing with microformats.

The difference? I ran the page through HTML Tidy to get well-formed XML ²...

²   It wasn't entirely automatic, unfortunately, I had to wrestle with character encoding issues too.

Ladies and Gentlemen, I give you The Gruesome Twosome of Computer Science: Structured Data and Character Encoding. We call them footnotes.

Highlights, Lowlights

It started with an offhand comment, a report that one couldn't set the background color on text in the rich text editor in Mozilla Firefox. Every other command on the editing palette worked. You thought, "15 minutes tops, I'll take it".

So you look up the list of command identifiers in Internet Explorer and notice that the execCommand method has an attribute named BackColor which

Sets or retrieves the background color of the current selection.

You head back to the Midas spec for Mozilla and see that for the backcolor entry

This command will set the background color of the document.

Hmm... Your first thought had been that it was a case-sensitivity problem, "backcolor" rather than "BackColor", but the documentation notes that case doesn't matter. So the problem is rather what you highlighted, in Mozilla, backcolor applies to the document and not to the current selection. This means that you have to use a different identifier to achieve the same effect in Mozilla. You promptly notice one called hilitecolor

This command will set the hilite color of the selection or at the insertion point. It only works with useCSS enabled.

Okay, someone decided to not use the same command as Internet Explorer, fair enough, you've seen worse. As you change the code, you say, it's Mozilla and CSS is enabled by default so all I have to do is switch to using hilitecolor.

Of course that doesn't work. You then tell yourself that you'll just enable CSS with the useCSS attribute (you had looked more closely at the code and noticed that CSS styling had been deliberately turned off for some obscure reason).

Hmm, that didn't work...

So you go back to the spec and you notice a styleWithCSS attribute. Interesting... You try that instead of useCSS. Of course, you happen to be testing using a version of Firefox (1.04) that doesn't support that attribute so there's an exception. Presumably this attribute was introduced in Firefox 1.5 or something. Still it's a good thing that you are using an older version for testing otherwise this would have been another bug (feature doesn't work in Firefox 1.0 etc).

You bang around for a while and go back to the original Midas demo which seems to work correctly if you have the "use CSS" checkbox checked.

You view the source of the demo to figure out what could possibly be making it work and you notice that they are setting the useCSS property to false when the checkbox is checked. Huh? They set useCSS to false in order to enable CSS!

So you go back and look more closely at the documentation. First you notice that the useCSS property is deprecated. Hmm... the plot thickens. Then you read this

useCSS - value: true/false

Note: This command has been replaced with styleWithCSS. It takes the same values as styleWithCSS, but the meaning of true and false are inversed.

Up is down in other words. It's like Alice in Wonderland or something. Do note that there is no word on what version the replacement occurred. Nor indeed is there any footnote on why true and false are "inversed".

Anyway you've finally figured it out, you make the change to enable CSS styling only when applying the hilitecolor attribute, add some future-proofing to use the styleWithCSS attribute in case the deprecated useCSS attribute is removed in later browsers, submit the patches and 3 hours of your life have passed. It was frustrating but there's an object lesson somewhere. If you are building a robust cross-browser rich text editing application, you have some code that has to go through this rigmarole.

I have a confession. I recounted the above tale because I feel guilty: I could have prevented all this four years ago.

Internet Explorer was the first browser to introduce rich text editing. It was not pretty since they didn't want it to be as good as Microsoft Word, but it worked reasonably. IBM later contributed resources to Mozilla to beef up its rich text editing. Indeed four years ago, I was asked to review the resulting Midas spec by my colleagues who had been working on that functionality. The spec looked serviceable, pointing mostly to the Microsoft documentation since they wisely chose to follow the de-facto standard. I reported a few bugs with the intial implementation and we built a rich text editor around it. It's widely used on the web these days.

I obviously was a bad reviewer because I certainly didn't notice that there was a change in the semantics of the backcolor attribute from "current selection" to "document" nor indeed that a new attribute, hilitecolor, had been introduced in Mozilla. There is absolutely no reason for these discrepancies. Nor do I want to go down the forensic trail that explains why you can't have a background color on text if you don't style with CSS. That's a rathole of its own.

So what do we have here? A de-facto standard was implemented but someone took the liberty to change one aspect of it for whatever reason - semantic purity or something. I suspect I won't be the last developer to waste an afternoon on this or indeed the last user to be cursing about why I can't add a background colour to a piece of text. I hope that Opera and Safari, if and when they have their rich text editing implementation in place, will do a wholesale copying of Internet Explorer's behaviour. I don't want to be chasing these lowlights again. I often see criticism of Microsoft's inconsistent approach to standards but everyone can be as guilty as them on occasion.

Which Side Are You On?

Stefan Tilkov recently asked what's wrong with Javascript? The answer of course is nothing really, it's a fine language as evidenced by looking around current thinking on the language. Indeed Brendan Eich's biggest admitted gotcha about the language is automatic semi-colon insertion. Thus there is no reason it can't be used in environments outside the browers in which it is most widely deployed as glue - I've played with Rhino on the server-side without any problem.

From what I understand, Jotspot, which incidentally I consider a cunning plan to showcase the Dojo toolkit, uses Javascript as the scripting language on both the server and client. All power to them. Of course with Javascript from the same document executing in both environments, you can get very confused. The answer to one of Jotspot's most frequently asked questions, "Why isn't my Javascript function being called?" is that "your code is not running where you think it is" ergo, you're expecting that the current code you're looking at is server-side rather than client-side, or vice-versa. This is especially true since the object model is likely to be different, the browser DOM is a deliberately constrained environment whereas on the server side you'd want to allow your plugins to do more.

Thus there is a little impedance with using Javascript everywhere. Perhaps it's less confusing to use a different language and syntax for server side code - a tag library in jsp, or embedded java, php, asp or whatever. Or maybe clever syntax coloring in your editor or IDE would do the trick to remind you of the context. Needless to say, you have to decide what side you're on...

I've been working on a project in which the others on my team are seasoned PHP gurus and are occasionally petrified of Javascript - the reason of course being the continued brittleness of the browser platform. When I started work, this bias showed and their initial recommendation was to do as much as possible in PHP on the server-side. They recognize however that that we are living in an age of interactivity so we need that shine that comes with moving intelligence to the client. Still when you start doing data-binding and automatic JSON serialization of PHP objects, you get constructs on the browser client that you wouldn't normally use if you were a client-side person. As someone who's very comfortable with both client and server side code (perhaps more comfortable with Java than PHP), I keep running into such peculiarities all the time. Of late I find myself prototyping code in client side Javascript even if it will eventually morph into server side code. Perhaps I need to get more into Python or that Ruby bandwagon. Still you tend to develop a split personality when you develop for the web.

The Null Hypothesis

I got the note: "you can't append a column if you click on a cell in the last column of a table in Internet Explorer".

Huh? I attempted to reproduce the bug and, sure enough, that was the case. Vaguely at the back of my mind I recalled from painful experience in K-station that there were special APIs in the HTML DOM for dealing with tables. Thus I searched the codebase for insertRow and insertCell. Hmmm those functions were nowhere to be found. How were they doing the column insertion, I wondered? My guess was that this was either some innerHTML tricks or simple standard DOM manipulation. Thus I had to dig through the code and eventually encountered the Node.insertBefore conundrum.

Now insertBefore is a method on the Node interface that is part of DOM level 1 specification. Every browser claims at least DOM level 1 support.

It turns out that in certain versions of Internet Explorer the second parameter to the insertBefore call can't be null, you get an Invalid Argument exception. Mozilla handles this condition as one would expect in their Javascript binding; Internet Explorer chokes. [Obscenity]. There was too much code to change to use the HTML-specific methods so I just hacked special case code that ensures that I don't pass a null to Internet Explorer - 3 hours of my life perhaps.

Now this doesn't amaze me really, thinking back on it, this is probably the reason that somewhere in the bowels of every Javascript library that deals with dynamically adding a new option to a select control, you'll find code like the following:

function appendOptionElement(select, newoption){ if(is_ie) // test for internet explorer somehow   select.add(newoption); else   select.add(newoption, null); }

The browser is a fragile place and hopefully the frameworks that are being developed will shield you somewhat from such issues, but it is telling that you can't rely on core DOM functionality. Even if these quirks are fixed in Internet Explorer 7, this patched-up code will have to hang around for another 5 years before users will no longer use older versions of the browser. We're in a world of pain in the browser world.

Nulls however are problematic throughout computer science. The arguments around them may sound like angel and pinhead discussions but they are fair questions. What indeed is null? What is zero for that matter? For centuries and throughout the Dark Ages of the West, there was no concept of zero, it took Arabic Algebra to spread that notion. Why should one expect that programmers would have internalized the null concept? Reasonable people can and do differ on how to treat null.

Just now, reading through the latest Dr Dobbs journal, I noticed the following in an article about Consuming .NET Web Services in Oracle JDeveloper

The ATL Server SOAP handler generates an xsi:nil="1" attribute when the element's value is null, or when the array is null or zero-size. Unfortunately, Apache SOAP fails to deserialize UDTs whose fields contain the xsi:nil="1" attribute and expects zero-size arrays to be represented as XSD arrays with the dimension parameter set to zero.

The author then proceeds to outline a variety of workarounds. Now I happen to not be a fan of the SOAP style of programming (in the past I've called it Crusty Old Architecture pronounced SOA with a silent P) but as a developer, I feel the pain. The article is a catalog of kludges and likely mapping errors that have to be worked around - the word "unfortunately" is used entirely too often.

I started my career doing graphical programming thus I perked up when Raymond Chen recently outlined the consequences of invalidating the null window and his anecdote is worth quoting at length.

If however you end up passing NULL as the window handle to the InvalidateRect function, this is treated as a special case for compatibility with early versions of Windows: It invalidates all the windows on the desktop and repaints them.

Even more strangely, passing NULL as the first parameter to ValidateRect has the same behavior of invalidating all the windows. (Yes, it's the "Validate" function, yet it invalidates.) This wacko behavior exists for the same compatibility reason. Yet another example of how programs rely on bugs or undocumented behavior, in this case, the peculiar way a NULL parameter was treated by very early versions of Windows due to lax parameter validation. Changing nearly anything in the window manager raises a strong probability that there will be many programs that were relying on the old behavior, perhaps entirely by accident, and breaking those programs means an angry phone call from a major corporation because their factory control software stopped working.

The null hypothesis strikes again.

Camel Humps

Even when you get past these details, you get into matters of syntax - a longstanding pet topic of mine. Consider the separators that people use for readability. Some people like to use Hungarian notation, others prefer hyphens... well I've already written a hyphenated parable so I'll skip that aspect. Anyway, assume for some insane reason that it makes sense for your spec to have an attribute named "windowTop". There'll undoubtedly be people who will write it "window-top", "window.top" or with some other variant of case, "WindowTop"? If you're dealing with XML where case matters, things will fail. You say potato, I say pubDate anyone?

I've been playing with a product that is essentially a wiki and it turns out that by convention, camel case (or should I write it as CamelCase) is significant in the wiki world as denoting a "WikiWord". I even had to spend half an hour writing glossary entries for these concepts since in our system, user names and page names had to be in that format.

You can imagine however the kinds of issues that arise when you have an html editor that allows you to enter HTML and Javascript along with php code and a specialized wiki syntax - since for some reason, it was decided not to use angle brackets in this product. The most complicated piece of code is going to be the parser. As currently implemented, the parser is a mass (or should I say, a morass) of regular expressions and all kinds of things I'll never understand. There's even special case code in there to handle nested comments in Javascript. What was never forseen however was that you'd have to deal with user-entered script code. What would happen if camel case is used for variables inside of said script? Well it wasn't pretty when the wiki engine jumped in and treated script as wiki words, let's just say that someone had to come up with a solution.

There's this concept known as McCabe Cyclomatic Complexity which is basically an application of graph theory to software and is often used to pinpoint potential problems. The basic insight is that if there are too many decisions or paths through your code, it will be buggier, harder to maintain and test. A good guideline is that anything with complexity greater than 15 in this scheme is ripe for refactoring. Luckily I have access to some tools that can generate reports and provide some numbers to validate that nagging sense that a piece of code is getting harder to understand. What worries me is that I keep running into code that laughs at such guidelines - after the latest change the parser code just hit 41 (to give some context, values between 21 and 50 indicate "a complex, high risk program" and with the 50 barrier looming, we are verging towards that notable status of the "untestable program - very high risk"). I'm sure it didn't start out this way and that the steady accretions have been solutions to real problems. Still, technical arteriosclerosis continues its inexorable spread...

Ruby on Rails and the Zend framework for PHP are founded on favouring convention. We had this code that automatically created the schema for what amounted to a database table. We never actually told users that this was what was going on. Of course once a wider audience started to play with the product, we had to fix the bugs that arose to enforce the convention so that databases would be named as the code expected. A classic case of leaky abstractions I suppose. You can guess the complexity the additional error checking added.

Moving up a level, you get politics - the most aberrant of which have been the feed wars of the past few years perhaps best summarized hilariously by Shelley Powers. Truly, Jesus wept.

Bill de Hóra in an aside wisely noted that

Programmers would rather squabble about minutiae - it seems this transcends community or language choice.

But even when you move beyond whiplash and abrasive personalities, who would have thought that the Atom working group would have spent so much time on the concept of dates. When I saw dissertation-length essays on the various types of dates that might be used in publishing systems, I was content to continue lurking in that community. The thing however is that these details do matter and they are best dealt with up front. The aggregate waste of programmer effort in pursuit of minutiae might keep the profession in business but it surely isn't sustainable.

The East Australian Singularity

I'll conclude by noting that acts of God (or in this case, his flawed proxies: politicians) can come into the picture. Thus I read the following this week: Eastern Australia: Java applications impacted by the change to daylight savings time dates

The running of the Commonwealth Games in Australia in March and April 2006 has resulted in an extension to daylight savings time in the eastern states of Australia, which includes New South Wales, Victoria, South Australia, Australian Capital Territory and Tasmania.

Rather than the clocks being put back an hour at 03:00 on Sunday 26 March 2006, they will now be adjusted at 03:00 on Sunday 2 April 2006. The IBM Software Development Kit (SDK) or Java Runtime Environment (JRE) is not currently aware of the one-time change to the daylight savings time extension. An interim fix is required for your application to have the same time as your operating system.

This change applies to all of the Java environments, irrespective of their setup, whether they are set up to use the operating system time zone information, or the user.timezone custom property that can be set for the SDK or JRE.

I assume television schedules were behind this change to the time/space continuum since the Commonwealth games have just begun. Or perhaps the politicians were concerned about athletes missing their events or something. Still imagine if you're the Java programmer who has suffered through the badly defined date apis in Java 1.0 a decade ago, adapted to the improvements in succeeding versions and finally, finally gotten a robust application deployed. Then your manager walks in and tells you about the East Australian Singularity. You're going to have to rework everything since bank transactions might be messed up and there are likely to be blinking clocks etc. We live in a global village so you have to worry about what happens if some mission-critical application that you rely on is being run in Eastern Australia. You can't even have the special case restricted to that country, it's only a sub-region that is affected. At such times I'd prefer being the poor Eastern Australian sheep or lamb, their lot is much easier.

What a way to make a living, if it isn't one thing it's the other. And then there's version hell. Lord help me.

File under: technology, software, abstraction, details, minutiae, glitches, programming, standards, language, absurd, toli

On XForms, XPath, CSS, Brevity, Syntax And More

The always interesting Mark Birbeck of formsPlayer has a nice article up on his blog.

Ostensibly it's about "CSS, the XForms Dependency Engine, and 'Dynamic Infosets'", and he starts out by asking why there are 2 main languages (CSS and XPath) for selecting and addressing nodes within a DOM. It's a good question, and one that many have asked before. The piece is probably the definitive consideration of the question and he's a great guide, walking us through all the issues involved.

What is more interesting to me is the wider point that he goes on to make as he lurches into a very useful discussion of how we design languages and layer and model our systems. Almost in passing he addresses an issue I find most fascinating which boils down to importance of ease of authoring and syntax in technology.

The mental makeup of human beings means that brevity matters and "intuitiveness" becomes a concern. As an example, so long as the length of phone numbers was low, they were easily memorable, these days however, with 10+ digit dialing, we rely on Caller Id and programming numbers into our phones. Thus our cognitive faculties and our short-term powers of recall come into question. Being able to control the nicknames and identifiers we use in our buddy lists is a very significant factor in the spread of instant messaging and now applications like Skype. Identifiers matter significantly in this respect. The simplicity of a URI as a key, and memorable tenet, of the web architecture is a similar case in point.

From another angle on the issue, consider that not everyone can tilt their heads enough to handle the parentheses of a typical Lisp program. Most programmers can, on the whole, and some, like the Paul Grahams of the world, even wear it as a badge of honour. Of course a good computer science program should expose budding engineers to this way of thinking and many do. But these, like the Smalltalk gurus and others, are sadly outliers in the software landscape. I would hazard here that the largest impediment to the widespread adoption of the elegant programming model of Lisp is not that something like recursion is difficult to understand but rather the dissonance that the proliferation of parentheses can cause when Jane Programmer scans a listing in an editor. Vacant stares and cognitive overload ensues.

Marc Andreessen will be remembered for many things; amongst others: Mosaic, Netscape, the AOL merger, a little dotcom hubris some might say, but simply youthful exuberance I would say, evidence in the flesh of what a monopoly like Microsoft can do when provoked, and a pointer, along with Jim Clark, to the role of gravity in deflating bubbles ala Great Crash). Historians will point to all that and more.

For me though, his choice of the syntax for the image is his most lasting contribution to technology and to mankind in general. Others argued otherwise at the time and would have foisted semantic doodles on us. The "View Source" impulse that has led directly to the success of the web, that great conversational engine, would have been stymied by much head-scratching by the eveyrday people who created many a homepage circa 1995-1999. Those much mocked homepages were wonderful assertions of identity, and the lowered barriers to entry enabled many people to land their flag on this here internet where they, their friends, parents and children now live, shop and commune. If he ever receives an honourary knighthood from King Charles, his coat of arms should read

<a href="http://netscape.com">Mozilla Hyperlink Andresson</a>

It is the succint expression of the ethos of simplicity in human history.

In this vein, I was perplexed that in XPath 1.0, it is better, or rather less ambiguous, to write true() rather than true. In other words, it is recommended or even required that we treat booleans as functions and not as literals. Indeed everything is a function and as we know functions need parentheses to indicate their arguments. This always trips me up and maybe this is no longer the case in XPath 2.0. Who knows? I certainly haven't cared to look. What is true is that the cognitive impedance this caused me on my first date with the language will forever taint it in my eyes even though I have daily dealings with it.

If you had to say huh? when you did your first view source of a web page, would you have gone with that newfangled web thing or would you have written it off as one of those overly complicated buzzwords that you would look at later "when you had more time"? First impressions and snap judgments (ala Blink) count surprisingly much in these things.

One of the things that I keep thinking we need, and that I hope someone with an itch will build, is a nice XPath expression editor, a component that parses XPath and walks you through the processes of adding conditions and formulating expressions. Maybe a wizard or something, with selectors for picking the various kinds of things that are typical when building forms applications e.g. this field should be less than the value from this other field. A component that would let you add a library of custom XPath functions that could implement additional rules. Each of these libraries would be able to specify their editors but most could just be simple drop-down lists. It's not a big thing to do and you can sketch out a nice design for such a component and knock it out over a weekend. Make it open source it and be done with it.

Still, my focusing on the critical necessity of such a component is simply a recognition that hand-authoring XPath can quickly turn into a nightmare of missed parentheses, predicates and selectors. It is true that authoring in XPath doesn't require as much head scratching as say XSLT, in which context I first encountered the language, and which mere mortals like me will never understand even as I used to write in Lisp. But it is something that raises the bar quite high for the average author. In contrast there is something strangely satisfying about editing a style sheet (or maybe it's just that I've grown accustomed to that over the years). Something like this I expect is what lies behind the impulse for Web Forms 2.0, and the WHATWG, a pragmatism borne of weighing programmers' familiarity with scripting languages and a tenacious devotion to backward compatibility.

More generally though, the issue is that getting general users to author structured content is a big problem, indeed it is a nigh insoluble issue. And all the software that we produce cares very much about structure. The wonder of the spread of HTML and XML is that, ever since Berners-Lee, Bray and others unleashed their projects on us, human beings have adapted to angle brackets, < >, and now don't see them as much ado about anything. The same thing goes with CSS, the tradeoff that was made for syntax is now bearing fruits.

In the past, I've had to deal with writing a number of applications that have had things like 60,000 lines of Javascript. The messy reality is that of dealing with things like focus, issues of scope in browsers, the power and contradictory complications of late-binding scripting languages, the earlier lack of powerful debuggers and Dom inspectors, the legacy of box-model quirks as well as the powerful notion of stitching together user interfaces by leveraging the incremental rendering and multi-threaded downloading that is the basis for the hypermedia browser. All this can be done, you can have even page editors and rich spreadsheet and presentation engines in Javascript. I've written about the heroism of those who write and maintain such things. Google (Suggest, Gmail, Maps and more), Yahoo/Oddpost, IBM, and many others have competitive edges because they have developers with the skillset and more importantly the insane programming discipline required to crank out the composable browser voodoo that causes much serendipity for end users. What however about the Long Tail of Application Authoring on the web? That's what VB and other environments have catered to on desktop clients. The endpoint in all of this is when a team lead or department head can compose an application for their local concerns without much (or preferably without any) handholding from the IT departments, if indeed they have one. People just want to be able to handle their little processes and get on with things. This is what web publishing and especially blogs and wikis have done by lowering the bar for authoring with the attendant benefits in communication and global conversation.

Thus, one of the main questions that will determine the adoption (or lack thereof) of XForms or Web Forms and their ilk is the perplexing matter of whether human beings in the next decade will become as inured to writing true() in an expression as they have become with the angle brackets of html and xml. Put a different way, it could well be something completely orthoganal to the merits of the underlying technology that will determine the outcome: it will be the appearance of the kind of code you see when you do View Source on the first cool forms application you encounter. I'm suggesting then that the language acquisition cost and what I'm terming the cognitive impedance in the average human being of parentheses for functions, and forward slashes for selectors will determine the adoption rates of XForms technology.

I've been working on Forms, and XForms in particular, for the past couple of years. I happen to think that XForms has gotten the abstraction and decompostion right and that it is a great means for lowering the skillset required to model and author the kind of form-based applications that are the glue of the many custom processes of the Long Tail of Software. Indeed my only nitpick is that the specification doesn't include upfront the equivalent of the JavaScript confirm function as a concession to usablity so that you can easily put up a message for the user so that they can decide whether they "are really sure that they want to submit their form" or not. It can be done, I've been told, but it isn't emblazoned in the specification. By being too general (they'd argue that something like this needs to consider mutiple modalities and the like), they are missing out on something interaction designers would immediately point at as a shortcoming. Again, first impressions count.

I see a bright future in which that much maligned Forms "programming model" that is at the core of the Lotus Notes platform could be brought to the web platform leveraging the native primitives of the Web style (hypermedia, uris, linking etc). XForms is singularly well suited to do this. For those unfamiliar with Notes/Domino, my handwaving elevator pitch is that it is a platform essentially founded on the fundamental insight that a huge class of applications can be built based on just a few compositional building blocks: Forms, Views a standard file format, the note in Notes terms. The brouhahas made about messaging, security, directory services, and all that paraphernalia that marketing people throw about when they pitch the platform to you are all syntactic sugar around the core competency of Forms and Views and the client and server processes that can manage them. A whole cottage industry of business partners are doing very fine thank you building custom and evolvable applications for businesses, small and large, everywhere. The fact that email can be construed as a forms application is just a side benefit and detracts from the real focus of the platform. This is much misunderstood by people whose only encounter with Notes is as a Mail client. It's really just a forms and view app for people and processes. Incidentally this same platform is most likely what is funding my current work and much of the IBM Software Group, even as resources are spent on other "sanctioned" and more "strategic" approaches. C'est la vie.

One thing I've noticed is that many people seem to want to ignore the lessons learned from the Notes world over the past 15 years and and behave as if the forms space is terra incognita - a brave new world indeed. On the contrary, the Forms problem and the wider Process problem is nothing new. These are things that have been with us almost from the time that societies became organized and larger communities formed as Barry Briggs has pointed out. Whenever I plumb those depths however, I am reminded of the notion that Joel Spolksy so eloquently coined that in software it it easier to write code than to read code. In software terms, 15 years is an eternity hence we are fated to reinvent and rewrite anew old software. Just look as WS-* as opposed to Corba. Sometimes I almost despair at this notion, since it bespeaks a total lack of curiousity and historical memory even with those who are sitting in the same building who have learned comprehensive lessons about the many problems of forms: evolvable schemas, metadata, annotations and the like.

Of course I'll continue to build the tools, the processors, the renderers and the infrastructure plumbing to make the forms dream an easier reality. I'd still argue that adoption will ultimately come down to whether the View Source impulse can be leveraged and whether the average Joe will get turned off by things like true() instead of true. If I were inclined to be a research type, I'd imagine a case study or paper titled something like

"The Importance Of Syntax In Technology Adoption - Historical Insights From The Trenches 1940-2005"

A more prescient Historian of Science would note that the issue of notation in mathematics is similarly a longstanding area of concern. A linguist would add insights about how different societies adapted different writing systems and the impact on the writing system on cognition and development. Anthropologists, sociologists or psychologists would have much to say in this vein.

Technologies like XSLT and XForms which are the prime users of XPath are in still in their infancy (as are some of the other takes on this problem space from Adobe and Microsoft). Despite having many implementations at its launch, XForms is still ambling towards its inflection point and I'd hazard that the majority of XForms templates and transactions are machine-generated. Fair enough perhaps. Wearing my prediction hat however, it will be very interesting to see what happens 6 months after the default installation of Firefox includes its XForms extension. We're going to see the same thing in microcosm now that Mozilla have announced that Firefox 1.1 will include SVG which has a more limited utility for mass audiences. With very little tongue in cheek, I'd wonder what contact with a massively vaster audience of form authors will do to XForms implementors. I'd lay bets on the first XForms engine that implements a "quirks mode" for their XPath evaluation engines to dealt with common patterns of mistakes in hand-authored forms. It will be a case of omitted parentheses rather than browser tag soup that will cause much fretting in mailing lists the world over. I wonder whether Peter-Paul Koch will then have to add an XForms or XPath section on his invaluable site that documents browser quirks. The litmus test will be the teenager doing a summer job in a lawyer's office who is asked to write a little forms application to help some workflow. If parentheses make their eyes glaze over, I doubt I'll be proved wrong (although I hope to be), about whether XForms would be used for that custom application. If the typical simplified wiki syntax (whatever Jotspot or SocialText are using) is more intuitive, that will be what gets used.

The other point Birbeck raises, and here the argument is much stronger, is about the tradeoffs that designers consider when it comes to seperating the processing, addressing, eventing and styling models. He speaks to an architectural truism whatever the domain in question. This is where his clarity of thought comes to light. A clarity that stems from being one of the exalted "Invited Experts" on the XForms and HTML W3C working groups, and having an innovative product that daily explores this landscape

If, for example, you started off in the mad, slapdash world that was early browser development, you might opt instead for a very pragmatic viewpoint on these issues. That's the kind of weighing that has characterized the Mozilla folks. Håkon Lie, Hixie of Opera, fall into this category even if they appear to take it to almost militant extremes at times. Still I see where they are coming from. Your take on these things is coloured by contact with the millions of end-user authors and the daily reality of tag soup.

I've only spoken with the Opera folks a couple of times and always forgot to ask them the burning question I have. How difficult is it, by the way, to add an XPath engine to a browser? I've always assumed that the real reason (as opposed to the stated reason, "we have everything we need in scripting and css") for Opera's almost visceral objection to XForms has been a concern over footprint since the same codebase is used on desktop and pervasive clients. But with the necessity of XML engines in browsers(with the now indispensable XMLHttpRequest) and Moore's law at work on your more limited clients, at what stage do protestatations about XPath (and hence XForms which is a dependency engine over and above XPath) become simply bywords for inertia? As a very conservative software engineer personally, I am similarly not inclined to jump on bandwagons just because they are in vogue. Still I'm interested in the architectural thinking that lies behind their position.

If, on the other hand, like Birbeck, you've drunk the XForms Kool-Aid, you'd be a generalist and will be inclined to see almost everything through rose-tinted glasses in terms of seperation of model from UI and from eventing and actions. You might recite MVC chapter-and-verse as you lull yourself to sleep at night, self-satisfied at your specification. The irony is that the visual effect of mere parentheses on an average teenager could cut short your sweet dreams of empire building.

Of course it's not always so cut and dry, sometimes you're just in the middle, trying to figure out which of the 45 latest buzzwords you're expected to spout fluently tomorrow to get a raise that beats inflation - or even a promotion. Or maybe you're just trying to code for food and get some real work done by helping a doctor's assistant keep track of HMO paperwork more efficiently or something of that sort. Oh well. Food for thought in any case.

Cross-posted at the Inside Lotus weblog.

File under: technology, web, xforms, forms, xpath, css, language, design, architecture, comprehension, Lotus, IBM, notes, standards, adoption, html, xml, xform, toli,

On GMail and DHTML architecture again

So Jon Udell has been pondering GMail and its architecture on his blog. He's also trying to figure out when to augment DHTML apps with even richer client technology. Below follows what I thought would be a short email that I repeat here for blogospheric conversation's sake.

GMail's architecture is actually very generic for a DHTML app. Everyone with a clue should be trying to leverage the browser and that, in essence, is all they are doing.

I helped implement an analogous architecture in Lotus K-station back in 1999. The major difference now is that everyone is using the "v.5" versions of the browsers; no one cares any longer about Netscape 4.7x.

For me the tipping point for rich DOM/JavaScript applications came when we saw mainstream applications like Hotmail and Yahoo Mail start using DHTML menus. My Yahoo only switched to DHTML menus when it was relaunched a few weeks ago and that's my litmus test for conservatism. (Ironically GMail switched back from DHTML menus to standard HTML selects for its "More actions" and apply label bar a month or so ago - although that might just be from usability testing). A similar event happened in the past year with the way users can rate items in Amazon.com wishlists and also in Netflix without seemingly causing page reloads - you know those five stars.

I mention the spread of all these little DHTML flourishes because I remember back in 2000 spending 3 months and well over 10 heated meetings trying to convince IBM architects to allow the use of simple things like DHTML menus and losing the battle. Of course given that kind of resistance, it was unlikely that features like drag-and-drop which one got from the UI engine in K-station would see the light in WebSphere Portal.

I was arguing that judicious leverage of DHTML could improve the user experience and that it could done while addressing accessibility concerns. Ostensibly the objections were not about usability or consistency but rather about not wanting to write multiple UIs and lack of expertise in DHTML programming. I realized then that the argument was not really about "to-DHTML-or-not" but more about comfort with a repartitioning of one's architecture.

It is riskier to do more on the rich browser client because it has been a more brittle platform over this past decade. Companies that do middleware and server-side tooling take a while to move from their core competency. Architects that thrive in that environment are essentially conservative and for good reason... Four years later, I now hear mutterings about drag-and-drop and richer clients in our corridors...

This only underscores the point that Jakob Nielsen's predictions about browser adoption cycles have turned out to be pretty accurate. Even though web application developers have been quietly spreading unobtrusive javascript usage in the interim, it is only now that there's a critical mass of clients that can leverage them; when Amazon and Yahoo move, something must be happening.

The developer tools and resources have gotten (slightly) better and there's more experience with the DOM. Increased adoption of broadband also helps reduce latency for the average client so you don't have to fight the inevitable arguments about performance and can couch your advocacy in terms of user interaction. In any case if and when you do have the performance discussion you can always argue that caching as close to the client as possible is a good thing and what better cache than the browser itself. It just so happens that applications like GMail, Bloglines and Oddpost are the state of the art in terms of browser leverage.

I recently wrote about this type of architecture in my recounting of the history of the DHTML spreadsheet and presentation components that are the genetic forebears of OddPost.

The idea is to fetch an HTML skeleton, decide what content you need, fetch that (as XML), and cache it wherever you get a chance. Render incrementally.

The pattern is simple: Database <-> XML (Optional) <--> JavaScript Object Bindings <--> UI Bindings (HTML) + UI management code

This pattern works very well for page oriented applications like portals, email, aggregators. You can cache or preload your javascript objects and just manipulate the CSS display or visibility attributes for your UI Bindings.

Incremental rendering and multithreaded loading is the name of the game here. Your application is essentially architected as a hypermedia browser just like the browser and leverages the browser's built in core features, our old favorites: incremental rendering and multithreaded downloading.

The decision about using XML as the data transfer format instead of Javascript objects is a toss-up, trading off client reach and memory. XML support in Safari and Opera is not as baked as MSIE or Mozilla so having JavaScript as your interchange format with your UI engine will buy you increased reach and smaller footprint.

Ultimately though XML will be your backend data format so the temptation will increasingly be to use the various XML-on-the-wire APIs (XMLHTTPRequest). Maybe another 4 years is needed for this pattern to see more widespread usage, JavaScript on the wire suffices for now.

Perhaps K-station was too bleeding edge trying to go for XML over HTTP, DHTML and extreme leverage of the browser client 5 years ago but that experience was a great testbed for me and I learned lots of lessons about building rich REST-ful applications, the importance of URIs etc.

Again the major missing feature for this rich web application platform is offline usage and synchronization without introducing new security holes in the browser. But then that's why Bosworth is at Google as the rumour goes, right? I suspect he's got other things in mind though...

I would note that the Mozilla folks know that the offline capabilities in their platform are a bust so I'd expect some eyes on this. Look also at applications like FormsPlayer for innovation in this space or anyone doing forms in general. XForms is my current thing and offline forms are a great feature that users can understand and demos very well; it's also something that the Notes platform does very well. Our old friend, Groove, also does that kind of synchronization well but I'd want that for the web, natively in the browser.

I think that increased leverage of the browser and the DOM is a good thing. It's also a clear trend and for many applications, the browser is good enough. Good enough for Google, good enough for Yahoo, good enough for me.

Sidenote:
One note about memory consumption and pushing things onto the client. There's an end-to-end argument for this notion of pushing intelligence to the endpoints but there is a cost and in this case it's memory consumption. It's mininal but it exists in this case. One lesson I learned was that it's good to use the kind of machines that the average client would be using and consequently I always hoard older machines. I've noticed that with lots of tabbed browsing and increased use of GMail and Bloglines, my four year old Windows ME box (Athlon 900 Mhz 512MB RAM) thrashes memory more often and I've seen more sporadic crashes/freezes with Mozilla. I know this is actually a lagging indicator and that most users are now on Windows 2000 and possibly XP, but it is a pointer to the memory consumption overhead of DHTML apps. All my other machines are fine and I should note that these applications run just as fine on ME as on other platforms in terms of interactivity and all, it is just prolonged use and a piece-of-junk memory and resource subsytem in that excuse for an operating system... My next representative machine is a Win2k box and on that evidence everything is in order even with these highly leveraged applications.

File under: architecture, DHTML, programming, technology, web, javascript, xml, patterns, html, UI, udell, conversation, history, observation, adoption, DOM, scripting, xml, xmlhttp, interaction, IBM, Lotus, K-station, oddpost, yahoo, google, gmail, toli