Flickr's Godfather

The 3,000th photo I uploaded to Flickr is a howler. Well at least I think it is, let's have a look (full size image).

A little context is in order...

I, and countless others, had complained about Flickr's excessive use of Flash. First I had inveighed about rendering and accessibility concerns in Cultural Sensitivity in Technology. Then I used Flickr's Flash buttons as a prominent example in The Unloved HTML Button and other Folktales.

In any case, 2 days after the folktales were told, Flickr finally switched from Flash. They no longer use a wrapper for image display - allowing native browser image rendering, and they changed their button toolbar from Flash to DHTML. I'd like to think my snarky comments were the tipping point but I won't flatter myself. They now only use Flash where it's appropriate, for drag and drop organization of photo albums, the kind of job that Flash or applets are particularly well suited for. Now of course they didn't use unloved html buttons in their toolbar but we'll take what we get. DHTML has greater reach than than Flash and accessibility concerns are more easily addressed on that front. Also there are an evolving set of design patterns for dealing with unobtrusive DOM scripting and forms.

So there I was, pleasantly surprised by their responsiveness, and going about uploading a little comic image to punctuate some later toli essay. You'll notice from the image that there were a bunch of glitches on the first morning of the big switch. The icons for some of buttons of the toolbar weren't showing up. Oh well, we'll ignore that but simply note that if they were standard HTML buttons, there would be no images to download. Moving right along...

Then I noticed a couple of typos, I had tagged the photo as sopronos instead of Sopranos and the image's title mentioned Godfarther which tickled me somewhat.

Flickr's Godfather or Flickr Goes Further?

Well anyway, Flickr implements a Click-to-Edit feature, a little unobtrusive DOM scripting that allows you to edit in place, so I corrected the title and hit the save button that appeared. That's when this error message came up and I took the screen capture

Taking a step back for a moment, let me just say that I love glitches. They expose the interesting aspects of complex systems and, much as we aim for simplicity, software tends inexorably towards complexity. As users of software we see lots of glitches daily. As an engineer, I am always interested in the first few days of a new deployment. You can test all you want but all bets are off when you get contact with real users and the real world. As an example, Technorati's recent makeover exposed lots of unforeseen glitches and they have had to work hard to address most of them in the past month. I was chatting recently with Dale Schultz, globalization architect at IBM and noted that I have a standard set of user names when I test new pieces of software, I make sure to have hyphens (hence I use my surname), ampersands (Sun & Sun is my canonical company), and, of late, accents (Rokia Traoré) because I've been bitten by various curses in the past in the software I've written. My former team has a José López test user for the same reason. Sam Ruby uses the word Iñtërnâtiônàlizætiøn as his proving ground in the same vein. We got to discussing the tyranny of patents at IBM and I pointed him to the Prior-Art-O-Matic for a laugh. Dale is obviously many steps ahead of me and of course he tried entering a euro symbol and immediately noted that that CGI application was broken and couldn't handle euros. Glitches often tell you a lot about application internals and the things that the developers tried to foresee or, as the case may be, ignored.

But back to Flickr's Godfather, what can we say about the glitch?

• Flickr is passing xml back and forth in their API calls.
• They are likely using XMLHttpRequest to do the voodoo of incremental loading without refreshing the page.
• There's an API key, probably tied to the user's identitiy that is likely passed around in every call. Sensible enough.
• They have to implement a Javascript layer to catch API errors and display something to the user.

Now I could have determined a lot of this and more by poking around and doing the View Source investigation. At the time, I wondered if I would have done any different and concluded that my implementation would have been much the same.

I have to say that like many others I'm highly impressed with Flickr, they had defensive programming and had appropriate error messages. Most people wouldn't have bothered dealing with these boundary cases. I haven't seen similar glitches since that first day thus the teething pains were temporary and they continue to add nice features to their service.

In any case, the juxtaposition of Silvio growling and in full bloom, the Godfather typos and the error message that popped up under Silvio's hands certainly made for a little amusement then and even today and now has occasioned a short blog entry. It reminded me of an advertisement for Fosters beer I believe that goes "It touches the parts other beers fail to reach". I guess the analogue in this case is "Flickr Goes Further".

As to why I had uploaded that particular Sopranos image, well let's just say that there's a famous quote from that scene and that's for some later toli.

[Update] I tried to cross-post this to my internal IBM blog only to find that the post was chopped off at Ruby's Iñtërnâtiônàlizætiøn magic word. Thus ironically as I was pointing out glitches, I just got bitten by one. I believe BlogCentral is based on Roller software and I suppose that I'll have to figure out whether the problem is in IBM's additions or in the core framework. The interesting thing about bugs with special characters is that sometimes you can't write the issue up because the software can't handle the characters in question. Perhaps BlogCentral needs a Godfather.

File under: flickr, technology, web, dom, scripting, glitches, funny, whimsy, programming, development, javascript, DHTML, software, error handling, design, photo, error, message, collage, toli

On Bleach, Entertainments, Forms, Atom, Kiss and Sexy MFs

Some ruminations on pragmatism, Bleached Unobtrusive Dom Scripting (BUDS), Forms, Atom, XForms, Glue, Infopath, "Entertainments", Kisses and the Sexy Mom Factor in Software...

• On Bleach
• On Entertainments
• On Forms and Rich Web Apps
• Down the Infopath
• On Forms and Atom Glue
• Kiss and Sexy MFs

On Bleach

So Adam Bosworth was reconsidering a programming style that some people would have us think must be called by a snappy acronym reminiscent of bleach or legendary Dutch Football clubs. Now I won't complain about said name because I realize the value of a coinage and have been evangelizing said style for quite some time. Still I haven't used that tag because I've been mindful of Fela's injunction against Yellow Fever, the practice in many African countries for people to bleach their skin to a lighter, more palatable, shade with gruesome chemicals (most banned in the West) and the consequent horrific effects on health (read what the bleach does to the resiliency of skin to minor cuts or otherwise routine surgery). The Nigerians and Congolese are prime exponents of this practice but even Ghanaian musical icons like Daddy Lumba have fallen into this Michael Jackson-turned-Diana Ross self-hating mindset which goes to show that the distortions of race are deeply rooted and occur in every society. Looking at the evolution of Daddy Lumba and the Congolese Koffi Olomide's faces over time is much like watching Jackson 5 videos and pondering the emaciated wreck the tabloids demonize today. Like Michael, Olomide has now taken to wearing face masks. As Fela sings

You deh bleach oh You deh bleach.

But maybe we all need to suffer for our art and perhaps the musical excitement has been consequently heightened.

On Entertainments

In any case Bosworth's notes are as-ever insightful and to the point and with his now official Googlejuice there is even more scrutiny in the subsequent blogospheric discussions. Equally enlightening is his response to the feedback his note provoked.

Technology Toli readers would know that this is a pet topic of mine and I've been writing a State of the DOM address every year ever since DOM scripting became Good Enough ® (circa 1997-1998) some of which are available on the wild web and have even found an audience (e.g. Applications vs. W3C DOM, On Rich Web applications, AlphaBlox and Oddpost and On Gmail and DHTML architecture again). I've been working on this year's version but keep getting distracted by the lure of technology folktales and B-movie scripts in the town of Deadwood.

A problem with my cacophonous take on blogging is that it's sometimes easier to go with "entertainments" than to write the technical monograph for Dr Dobbs and you run the risk of being characterized as mere fodder for the titillation quotient. In mitigation, I'm of the opinion that the conversational style helps you examine various aspects of an issue and get your ideas together en route to the book (and I suspect I've got a few books in me).

You can't seperate Graham Greene's entertainments and travel journalism from his serious and high-minded literature and indeed his later works wove all these strands into a cohesive whole. In my view Monsignor Quixote is more essential to the Greene canon than A Burnt Out Case or (and here I am not seeking to be controversial, I firmly believe this) The Power and The Glory. Also the occasional parable can elucidate essential truths very effectively, plus they're more fun. And since many people are visual, a good image, as our man Edward Tufte would admit can occasionally convey your point with greater force (read camels and eyes of needles or angels and pinheads etc).

On Forms and Rich Web Apps

The leitmotif in all of those DOM scripting pieces is that the browser can be construed as a programmatic surface that can be dynamically constructed and incrementally updated. The tradeoffs in that architectural style are all about client reach and efficiency, sometimes it is more expedient to deal with Plain Old Javascript Objects (to remix the POJO acronym) as the wire format and hidden iframes as the transport of choice. XML as the data format and XMLHTTPRequest as transport are nice-to-haves but not essential as various implementations have shown.

On the supposed failure of XForms or say Infopath, that one Bosworth's commentators notes, all I would say is that technology adoption takes time. The Big Bang of the web was rapid adoption of the web primitives (HTTP, URIs, HTML, XML, the HUHXtable sweater quartet), for content authoring and service deployment. We are now seeing a second Big Bang with the irrepressible blog phenomenon, Atom/RSS, and some might add wikis although to my mind the simplified wiki syntax leaks mightily in html authoring - those quote delimeter characters. Big Bangs are said by physicists to be rare events. Three Big Bangs in a decade is a little much.

Programmatic dynamic form-like applications, which after all are the design patterns that unobtrusive DOM scripting addresses were an afterthought in the development of web browsers. There were the empire-building distractions of more flashy, if leaky, mobile code (Java, ActiveX and Flash now recasted as Flex) which are also great but which raise not only issues of plugin installation, cross-platform support and hence ubiquity, but also accessibility, matters of focus and, more importantly, those user interface standards that the browsers codified: the back button for undo, the refresh button, the ability to bookmark etc.

Down the InfoPath

The problem space of forms in browsers is slightly different than the vanilla HTML authoring issue. The dominant OS vendor doesn't really want you to write applications first for the web but rather for its latest framework-du-jour thus the incentives are perverse in the Castle of Avalon. As a case in point, I explored Microsoft InfoPath a couple of years ago, based on public articles on MSDN and a couple of Jon Udell columns and had the following comments:

Let's start with Infopath's pervasive use of XSLT. XSLT is very difficult to hand author. I've written the hairiest of code before but XSLT strikes the fear of God into me.

Microsoft tied their form rendering engine to the Internet Explorer engine. Their runtime builds what is known as a Custom DOM to augment what in in IE. A web browser implements the standard DOM, an api that all web programmers write against. In InfoPath there are 2 new non-standard object hierarchies that are implemented.

They used non-standard extensions to HTML for their rendering, (behaviors and custom html attributes) and how they do validation (through what is called expando attributes).

We can infer what is supposed to happen, and when, by looking at their file format but the semantics are not easily understood nor indeed are they documented anywhere. Indeed they can and will change at a moment's notice...

The basic thing to understand is that while in InfoPath, Microsoft has followed the same separation of Model from Controls and Instance Data and Schema as XForms or say Adobe's XFT, almost all the technology choices they made make it very difficult to interoperate with them. I'm not sure that it was intentional, their programmers might just have been more familiar with those technologies, but the net effect is to tie their forms offering very tightly to their product line. They could have added their form processing extensions for a wider audience in Internet Explorer much as the Mozilla XFoms work is doing, but as a company they had no incentive, or desire, to add such functionality to a browser that they give away for free. With Infopath they could get you to pay to get this functionality...

It is costly to implement their notion of data binding (a combination of expando attributes on elements and html behaviors), their scripting runtime has its quirks (JScript, mshtml). Their custom dom is by far the most expensive to implement, and I don't believe that the XDocument.DOM and XDocument.Solution.DOM object hierarchies, nor their semantics and lifecycle are defined anywhere.

Reverse engineering the semantics of these objects that the InfoPath runtime adds to Internet Explorer is an expensive proposition. We don't have tools to author stylesheets and are hobbled too in their use of Internet Explorer Trident rendering engine. The type of html that is generated is full of the quirks that MSIE is known for.

It is costly to implement their controls since even the simplest controls are using behaviors. As an extreme example: text input boxes are not implemented using standard html input controls rather they are implemented as spans with custom behaviors and the contenteditable style.

Infopath has been 4+ years in the making, I'd hazard a 2 year effort at the very least not to mention dealing with all the various quirks in this still evolving runtime.

On Forms and Atom Glue

2 years later I'm glad that IBM's efforts weren't dissipated into chasing rainbows or these chimeric Redmond information paths, but still it is an ongoing struggle to convince people to ignore Microsoft. The good news is that the fun will begin not with Longhorn but rather when Mozilla includes either their XForms xpi or their parallel Web Forms 2.0 work in the default install of Firefox. It also helps that the subset of XForms 1.0 Mozilla has implemented is small enough that the Mozilla XForms runtime can fit in Minomo, the slimmed down version of Mozilla targeted for smaller devices. This is sure to get some operatic attention.

In this same vein, the indication that web rendering is where the action will be is that apparently some crazy glue layer wonks (who must not have slept for the past 2 years) can now transform Infopath forms into ASP.Net.

The other outlier will be if that apparently budding Bleached Unobtrusive DOM Scripting (BUDS) community manage to build and document frameworks that are easy to use and Good Enough ®. Again a shoutout to Alex Russell, think of lay programmers when building the Dojo toolkit, the latest recasting of the NetWindows dream. Jane Programmer is looking for a VB cookbook approach to this area.

T.V. Raman, Micah Dubinko and Steven Pemberton would argue that in XForms, they have Been trying to distill the essence of those heavy scripting applications and codify the design patterns therein. This emphasis on lowering the bar for authoring is alluding to something that goes under many names:

• The obligatory Long Tail of Application Authoring
• the View Source impulse
• The Mass Amateurization imperative etc.

And yes XPath has comprehension issues but those are leaks that can be plugged with enough work as Adam points out. Also the declarative versus procedural issue is an unresolved question and I see the value in both. As with all framework approaches to technology, what one is trying to achieve is to get things Good Enough © like Visual Basic or MFC on Windows so that leverage is inevitable.

Mark Birbeck is attempting to codify design patterns and to show that XForms as opposed to heavy and supposedly unobtrusive DOM scripting is the way forward. His latest patterns being Google Desktop Search and how to handle dynamic urls and Terraserver maps. This after Google Maps and Google Suggest. Slow down Mark, leave some work for others. People will think I'm a shill linking to you every week. And presumably if you continue to dance so attractively one of the big boys will swoop in and swallow your company up into some faceless Borg entity. That may be the point of course but I'm loving the creative exploration I'm seeing and working on the competition in the hope that the best of these breeds will distill this potent xforms confection.

On matters of glue and data, PubSub appears to be on the right track with their focus on Atom. Similarly when you see Matchmaker Fielding not merely lurking, but knocking heads on atom-syntax, or Adam's theorizing that Atom, as a mere wire format and publishing protocol, "could do for Information what HTML over HTTP did for user interface", you know that Atom is worthy of attention. My long lurking on that list is perhaps not time wasted.

Kiss and Sexy MFs

Finally to pick up on Adam's harkening to pragmatism at the end of his latest comment, I've recently been having some fun with neologisms, trying to name the kind of aesthetic I detect. I had completely forgotten about the Bosworth Axis of Empathy and now have to remedy that situation.

I had always figured Adam out for a KISS after reading Keep It Simple and Sloppy which reminded me not of those obvious rock stars who killed chickens on stage but rather of Prince who was also enamoured of the sloppy and sang

U Don't Have 2 Be Beautiful
2 Turn Me On ...

U Don't Have 2 Be Cool
2 Rule My World
Ain't No Particular Sign I'm More Compatible With
I Just Want Your Extra Time And Your

Kiss

And with those Rock Star initials, I had painted him in a corner with fellow glue traveler Sam Ruby as a Radical Simplifier, Good Enougher or Technology Buzzword Demystifier (TBD). The other possible folksonomic tag was related to Godfather Bray as a Plain Old Loyal Oppositionist (Polo).

But loosely coupled with Adam's repeated emphasis on Simple, Sloppy, Standards that Scale in a web of data and the Mom Factor in technology, perhaps we should again turn to Prince who would say that were just in the business of Sexy MFs (Sexy Mom Factor Software).
But Adam, it turns out that you're a PITT (Pragmatic Inside-Out Technology Type). Glad to have you on the team, coach.

I had written about having both William Pitts as our mascots but it was pointed out to me that having Brad Pitt would be a better bet as a cheerleader not to mention that, in keeping with Kiss and Sexy MFs, he wears a mean skirt.

(ducks)

File under: technology, whimsy, bleach, entertainment, forms, Atom, glue, language, kiss, bleach, dom, scripting, javascript, xforms, infopath, writing, music, Fela, Prince, black, mom, pragmatism, sex, Adam Bosworth, Nigeria, Ghana, Congo, Koffi Olomide, literature, Graham Greene, toli

On GMail and DHTML architecture again

So Jon Udell has been pondering GMail and its architecture on his blog. He's also trying to figure out when to augment DHTML apps with even richer client technology. Below follows what I thought would be a short email that I repeat here for blogospheric conversation's sake.

GMail's architecture is actually very generic for a DHTML app. Everyone with a clue should be trying to leverage the browser and that, in essence, is all they are doing.

I helped implement an analogous architecture in Lotus K-station back in 1999. The major difference now is that everyone is using the "v.5" versions of the browsers; no one cares any longer about Netscape 4.7x.

For me the tipping point for rich DOM/JavaScript applications came when we saw mainstream applications like Hotmail and Yahoo Mail start using DHTML menus. My Yahoo only switched to DHTML menus when it was relaunched a few weeks ago and that's my litmus test for conservatism. (Ironically GMail switched back from DHTML menus to standard HTML selects for its "More actions" and apply label bar a month or so ago - although that might just be from usability testing). A similar event happened in the past year with the way users can rate items in Amazon.com wishlists and also in Netflix without seemingly causing page reloads - you know those five stars.

I mention the spread of all these little DHTML flourishes because I remember back in 2000 spending 3 months and well over 10 heated meetings trying to convince IBM architects to allow the use of simple things like DHTML menus and losing the battle. Of course given that kind of resistance, it was unlikely that features like drag-and-drop which one got from the UI engine in K-station would see the light in WebSphere Portal.

I was arguing that judicious leverage of DHTML could improve the user experience and that it could done while addressing accessibility concerns. Ostensibly the objections were not about usability or consistency but rather about not wanting to write multiple UIs and lack of expertise in DHTML programming. I realized then that the argument was not really about "to-DHTML-or-not" but more about comfort with a repartitioning of one's architecture.

It is riskier to do more on the rich browser client because it has been a more brittle platform over this past decade. Companies that do middleware and server-side tooling take a while to move from their core competency. Architects that thrive in that environment are essentially conservative and for good reason... Four years later, I now hear mutterings about drag-and-drop and richer clients in our corridors...

This only underscores the point that Jakob Nielsen's predictions about browser adoption cycles have turned out to be pretty accurate. Even though web application developers have been quietly spreading unobtrusive javascript usage in the interim, it is only now that there's a critical mass of clients that can leverage them; when Amazon and Yahoo move, something must be happening.

The developer tools and resources have gotten (slightly) better and there's more experience with the DOM. Increased adoption of broadband also helps reduce latency for the average client so you don't have to fight the inevitable arguments about performance and can couch your advocacy in terms of user interaction. In any case if and when you do have the performance discussion you can always argue that caching as close to the client as possible is a good thing and what better cache than the browser itself. It just so happens that applications like GMail, Bloglines and Oddpost are the state of the art in terms of browser leverage.

I recently wrote about this type of architecture in my recounting of the history of the DHTML spreadsheet and presentation components that are the genetic forebears of OddPost.

The idea is to fetch an HTML skeleton, decide what content you need, fetch that (as XML), and cache it wherever you get a chance. Render incrementally.

The pattern is simple: Database <-> XML (Optional) <--> JavaScript Object Bindings <--> UI Bindings (HTML) + UI management code

This pattern works very well for page oriented applications like portals, email, aggregators. You can cache or preload your javascript objects and just manipulate the CSS display or visibility attributes for your UI Bindings.

Incremental rendering and multithreaded loading is the name of the game here. Your application is essentially architected as a hypermedia browser just like the browser and leverages the browser's built in core features, our old favorites: incremental rendering and multithreaded downloading.

The decision about using XML as the data transfer format instead of Javascript objects is a toss-up, trading off client reach and memory. XML support in Safari and Opera is not as baked as MSIE or Mozilla so having JavaScript as your interchange format with your UI engine will buy you increased reach and smaller footprint.

Ultimately though XML will be your backend data format so the temptation will increasingly be to use the various XML-on-the-wire APIs (XMLHTTPRequest). Maybe another 4 years is needed for this pattern to see more widespread usage, JavaScript on the wire suffices for now.

Perhaps K-station was too bleeding edge trying to go for XML over HTTP, DHTML and extreme leverage of the browser client 5 years ago but that experience was a great testbed for me and I learned lots of lessons about building rich REST-ful applications, the importance of URIs etc.

Again the major missing feature for this rich web application platform is offline usage and synchronization without introducing new security holes in the browser. But then that's why Bosworth is at Google as the rumour goes, right? I suspect he's got other things in mind though...

I would note that the Mozilla folks know that the offline capabilities in their platform are a bust so I'd expect some eyes on this. Look also at applications like FormsPlayer for innovation in this space or anyone doing forms in general. XForms is my current thing and offline forms are a great feature that users can understand and demos very well; it's also something that the Notes platform does very well. Our old friend, Groove, also does that kind of synchronization well but I'd want that for the web, natively in the browser.

I think that increased leverage of the browser and the DOM is a good thing. It's also a clear trend and for many applications, the browser is good enough. Good enough for Google, good enough for Yahoo, good enough for me.

Sidenote:
One note about memory consumption and pushing things onto the client. There's an end-to-end argument for this notion of pushing intelligence to the endpoints but there is a cost and in this case it's memory consumption. It's mininal but it exists in this case. One lesson I learned was that it's good to use the kind of machines that the average client would be using and consequently I always hoard older machines. I've noticed that with lots of tabbed browsing and increased use of GMail and Bloglines, my four year old Windows ME box (Athlon 900 Mhz 512MB RAM) thrashes memory more often and I've seen more sporadic crashes/freezes with Mozilla. I know this is actually a lagging indicator and that most users are now on Windows 2000 and possibly XP, but it is a pointer to the memory consumption overhead of DHTML apps. All my other machines are fine and I should note that these applications run just as fine on ME as on other platforms in terms of interactivity and all, it is just prolonged use and a piece-of-junk memory and resource subsytem in that excuse for an operating system... My next representative machine is a Win2k box and on that evidence everything is in order even with these highly leveraged applications.

File under: architecture, DHTML, programming, technology, web, javascript, xml, patterns, html, UI, udell, conversation, history, observation, adoption, DOM, scripting, xml, xmlhttp, interaction, IBM, Lotus, K-station, oddpost, yahoo, google, gmail, toli

On rich web applications, AlphaBlox and Oddpost

I had been meaning to comment about the past few posts over at Loosely Coupled but for some reason the comment entry fields don't show up in Mozilla or Firefox. The idea of having to launch Internet Explorer just to enter a comment didn't sit well with me, but I guess I will do it this once. This is part of a fairly deep vein of conversation in recent months on web applications, rich clients, the browser as a platform, or the location field as the new command line. A certain critical mass seems to have been reached, maybe it's the maturity of the browsers competing with Internet Explorer (especially Mozilla), the long history of security holes in MSIE or the publicity surrounding GMail and OddPost. I might as well join in on the commentary with a view from the trenches.

I work at Lotus software, IBM and was part of the tangled Halfbrain, Alphablox, IBM, Oddpost story at least last year when I worked on the "Simple Browser Productivity Components" - the spreadsheet, presentation and rich text editors that are now bundled with WebSphere Portal, tied to a document management system out of the box. For 6 months, my job was to 'port' these components to Mozilla; IBM for obvious reasons wants everything it ships to work on all platforms and browsers. The spreadsheet and presentation were licensed from AlphaBlox.

I wrote about some of my experiences at length last year in a discussion about "Applications vs. W3C DOM" and the adequacy of standards and browser support for rich web applications. Here's some further history and commentary:

The Halfbrain folks made a quite sensible decision that browser-based productivity applications would be a good thing, after all they, like everyone else, were starting to 'live' in the browser. More to the point, there's a market for good componentry and focused applications - lots of companies have developed spreadsheets, grids, charts, and various other editors and components. You may not make as much money upfront selling these components as shrink-wrapped software, like say a copy of Microsoft Office, but you can certainly tie it to services, other value-add features or even advertising and make money in other ways.

Sidenote: IBM has tried before to crack this market with Lotus eSuite (Java Office Suite for thin clients) which I also worked on after transfering from Lotus Freelance Graphics. Smaller companies have fared better because they understand that this is a niche market that one has to nurture and they didn't have the kind of insane revenue expectations that IBM had for that product (echoes of MS Office). 5 years on I hope/pray/(think?) that the company now understands what these kinds of products/technologies can achieve as part of its Portal and Workplace product portfolio and competitive strategy. One aspect of which is Linux. (cross my fingers since that's were the salary is coming from).

In any case, the Halfbrain folks didn't know any better, forged ahead, and miraculously a year later they managed to get a DHTML spreadsheet working. Everybody who sees it goes "Wow! How did you do that? In Javascript? With the then awful state of tooling for browser development?". It simply works; the UI looks, behaves and feels essentially like Excel, selecting cells, columns, drag and drop etc. Technically I can't say enough about that achievement (especially when I think back to the effort involved in the eSuite spreadsheets and presentation - there was still far more tooling available, and of couse, a rich programming model even in the less capable early 1.0 versions of Java than in the DHTML world). Keep in mind that it's a full spreadsheet engine, complete with calculation engine, dependency-checking, macro language. They were crazy enough to even at first target IE4. The programmers were are all new to Javascript development, but they were very disciplined and persevered. They lived on the bleeding edge of DHTML, had to invent things at times and workaround countless browser bugs. They talked somewhat with the MSIE team who made a lot of fixes for them in IE 5 and especially 5.5. As a very complex DHTML application, I'm sure it was a good test bed for Internet Explorer. In much the same way the folks at Opera last year jumped at a chance to test some of their DOM support with the speadsheet and presentation..

AlphaBlox, which does business analytics, sees this demo and buys HalfBrain. It makes perfect sense for them, everything is moving to the web. They integrate it with their packages, add conversion services from Excel, beef it up and offer it as part of their products.

Having learnt from the experience of getting the spreadsheet working, and realizing that what they have is essentially a very capable, general purpose DHTML library, one of the guys (now at OddPost/Yahoo) then heroically decides: why not do a Powerpoint clone? He goes ahead and writes a presentation editor !!!! This involves not just drag and drop, selection handling etc but most importantly text editing - ie. when you add your bulleted lists you need to edit the text. He managed to get this working even without using contentEditable support - it's a hack but users are non the wiser. It worked since IE 5.5 was up to scratch as a DHTML platform. This was an even greater bravura performance. Also, with greater facility with JavaScript, the code was smaller, more object-oriented. AlphaBlox also added some level of Powerpoint import capability.

The Javascript libraries for the spreadsheet and presentation amounted to around 60,000 lines of code, much of it shared and like any good DHTML toolkits had infrastructure APIs for menuing, windowing drag and drop, selection handling etc - the kinds of things that the DOM is missing since it's document and not screen-oriented. Sure, performance on very large spreadsheets may be a little poky and the recalc engine may need to be tuned, but those making really extensive use of spreadsheets will always stick to Lotus 123, Excel or more specialized packages. Similarly the drawing primitives of the presentation editor are not what you'd have in Illustrator but I think they are good enough. These kinds of editors target a different audience, they are 'worse' by definition. Of course 'worse is better', something that Christensen knows well.

Some of the HalfBrain folks left AlphaBlox with the idea to write yet another browser version of a productivity application - email and formed Oddpost - which Yahoo bought last week.

I don't really need to comment much about IBM's goals which are self-evident. Amy Wohl had a decent article from a while back about IBM's strategy Reinventing The Office: IBM Workplace. I'm sure the story and strategy has been tweaked and revised since. IBM, with a gap in its componentry, and looking to beef up its portal story helped with the development of the Midas component in Mozilla, the rich text editor, and then licensed the AlphaBlox spreadsheet and presentation. This is where I and others come in and that was last year's palava.

As far as I can tell, given the evolution of these products, the code got better, more object oriented, and more compact. Also the state of the browser has evolved, removing the need for most of the hacks of yesteryear. If I was starting the spreadsheet from scratch with the current capable browser platforms, it wouldn't be as much of a pipe-dream as back when HalfBrain started. (Joel per contra says 'never rewrite from scratch'). Given what I know about the effort it took on the Mozilla port, if Yahoo is so inclined, it should take a 6-8 months or so to have Oddpost working well in Mozilla, maybe less since there's obviously more staff. It would be perhaps a little longer in Safari or Opera since their DOM support (especially Level 2 Range) is weaker.

I talked to the Oddpost folks a few times last year; they were happy to hear that their first born babies were still being tended to - commiserating with me about some of the hacks they had had to implement earlier. With the advent of XMLHttp in MSIE, and IE 6 being a more robust platform for them, I think things were looking good for them as history has shown. Also since their app was their regular email client, they were eating their own dogfood and have every motivation to make it work.

I suspect that IBM could (and maybe should) have acquired them and perhaps there was the kind of usual flirtation/evaluation and general testing of the waters for building business relationships. With all the various mail clients and various technologies that a large company like IBM has, and the trendy push for Eclipse-based components, 'Javascript' components were a harder sell I would guess.

If the Oddpost code is anything like the AlphaBlox code I had to modify, they could probably jump start their porting effort considerably by using a lot the code I had to write or certainly the 10 or so common design patterns I encountered in doing this work. In a sense there is definitely a missed opportunity for IBM, we've missed out on an application compares favourably to many others. Of course, there would have been a difficulty in crafting a marketing message about how it fit in with the rest of the vast IBM software portfolio (read Domino, iNotes or any of the various mail apps we have). Again, I am glad that they've found a home at Yahoo, Oddpost is good technology and a good response to GMail. IBM acquiring Alphablox right on the heels of Yahoo taking up Oddpost is just another twist to this tale and likely just a coincidence.

Back to web applications. Mozilla is now the best platform for doing such development. It has the best standards support, is cross-platform and has the best devloper tools: DOM Inspector and Venkman, and perhaps even mindshare. In fact (market share be damned), it makes sense to first write yout rich web application for Mozilla. Your code will be cleaner to start off, will be better structured and will most likely work in other browsers - but that's my opinion, others say target MSIE first, just look at marketshare! - I would rather ignore that and get the compelling app.

As of Mozilla 1.5, XMLHTTPRequest support was well baked, and people have been using it successfully. I haven't tried Safari but I assume that it is not as good as Mozilla, though fast improving. Back then Opera was missing XMLHttpRequest, ContentEditable and DOM Level 2 Range support. We got some level of Opera support for free once the Mozilla work was done for the editors. The spreadsheet was functional and one could view presentations but not edit them because of the lack of range support and scriptable textareas.

The "rich web application" strategy is a very powerful appproach to development - and one we first used when I was when working on Lotus K-station (the post-mortem on that is for another day). It entails complete leverage of the browser which, after all, is the ubiquitous client. If the browser adds features you inherit them automatically. A short description of what I think is needed:

A client side framework for managing 'widgets'; a 'widget' is construed as a parameterized blob that produces markup (either in-line html or iframe-based). The data model is pushed to the client, the page is stitched together on the client, augmented by chrome and a code layer handles drag and drop, preview mode, incremental rendering and client side caching etc.

The idea is to fetch an HTML skeleton, decide what content you need, fetch that (as XML), and cache it wherever you get a chance. Render incrementally.

The pattern is simple:

Database <-> XML (Optional) <--> Javascript Object Bindings <--> UI Bindings (HTML) + UI management code

And:

It's the Latency, stupid.

When dealing with distributed applications, it's the issue of latency that will determine which applications will rule. Users ultimately want applications that are 1. fast to load 2. capable and 3. intuitive. They want all these at the same time. This is where making increased use of the DOM should shine compared to most simple html based UIs. Of course, you have to work hard when writing DOM apps to figure out where in your architecture you can do things incrementally and where to cache. But that's what engineering is all about and how one would get paid.

In K-station the 'widget' was a portlet, your portal pages was the drawing canvas that the framework managed, and you could navigate from page to page, incrementally building your pages in memory and switch back and forth instantaneously. With client-side caching, all you're doing is toggling the css display property. In the presentation editor, the analog of the widget was a drawing object (text, image, group etc). GMail and Oddpost follow the same pattern and it is the incremental rendering and caching that distinguishes them in their performance characteristics and makes them 'feel' like desktop apps.

The major missing infrastructure piece in rich web applications is going offline and synchronizing with good security. But that's a story for another day.

See also:

• On GMail and DHTML architecture again
• Deadwood and the Web Application Leap

File under: DHTML, IBM, software, technology, toli, web, javascript, design, architecture, history, oddpost, REST, DOM, programming, best practices, scripting