On The Limitations Of Notes On The Web

• Some context
• On The Limitations Of Notes/Domino On The Web

In the week after Lotus announces lots of innovation in the forthcoming version of Lotus Notes, with lots of great screenshots and undoubted enhancements, it may seem weird to post a short article I had written a few weeks ago on the past limitations of said product in my internal IBM blog. I do this because I've been taking a historian of science's approach to my writing on technology and its effects. I am always interested in how organizations work and how they make decisions. And a certain amount of dirty laundry airing is healthy if only to pat ourselves on the back to show that now we have learnt our lessons and that there is a lot of introspection and thought in the company. Also my focus is not on the Notes client which has always sold itself, but on rather the web interfaces that product has had since the advent of Domino, its first embrace of the web. As you may know I am an unabashed believer in the web as the ubiquitous client. This piece should give some context to these three slides from my recent REST presentation. They weren't quite a twisting of the dagger but more a challenge to the development organization. I believe there's Joy in Repetition and with a hip-hop sensibility remix myself constantly. So here's the original version before the sample.

On The Limitations Of Notes/Domino On The Web

Joe Turic and others are pondering Lotus Notes vrs Gmail and coming down in favour of the latter.

The complaints are about "Why is the email program that we use internally so limited?" and some of it has to do with storage quotas and the fact that we have very low ones (even as the platform handles 49 GB mailboxes just fine as Ed Brill keeps pointing out when comparing it to Microsoft Outlook). Most of the reason for this hence is not technical but rather I suspect a hint about the direction of email retention policy.

Later David Crumpton asks "Does Notes offer a Webmail interface?"

And of course it does, and it even has two, the regular webmail interface and iNotes, the prettier dhtml version which should give a snappier performance with lower latency for the user. A third way of getting your notes mail in a browser is through a portlet in WebSphere Portal.

The question is, why did these web interfaces not get widely adopted?

Let's start with the question of investment.

1. Investment and Adoption

Plainly put, not everyone at IBM has bought into the web, html and http.

I would count Lotus/IBM's efforts over the past 7 years as half-hearted although now we are supposedly hardcore about the web, Linux and open source in general. The thing to take away is that, at the point that judicious investment could have been made, and leadership in thought, if not in software products, was possible, we blinked.

IBM does middleware and services and moves slowly due to a sense of innate conservatism. I think this is fair, there are parts of IBM that are bleeding edge and one has to get buy-in before the rest of the elephant moves. As someone who was advocating the web almost from the beginning, it has been a long road to convince others to buy-in. The fact that we are now moving (and quickly) is great but I think it is cause for rueing missed opportunities.

2. Innefficient IT structure at IBM

Jim Gray's Distributed Computing Economics (pdf) makes a devastating case for the relative innefficiency of many in the industry that haven't embraced the web style.

"Megaservices like Yahoo!, Google, and Hotmail have relatively low operations staff costs. These megaservices have discovered ways to deliver content for less that the milli-dollar that advertising will fund. For example, in 2002 Google had an operations staff of 25 who managed its two petabyte (2^15 bytes) database and 10,000 servers spread across several sites.

Hotmail and Yahoo! cite similar numbers - small staffs manage ~300 TB of storage and more than ten thousand servers."

Up until 2000, I believe the operational staff at Yahoo was 8 and included the founder Jerry Yang.

Google has embraced redundancy and written software to work around human fallibility and hardware reliability. Google File System is not too shabby, if may say so. We have efforts in the same arena whether it is the Grid and autonomic computing work and say the move to VOIP but are we eating that dogfood?

Are we prepared to change our corporate structure and have profits come more from well-designed and usable software as opposed to consultantware? Are we prepared to eat our babies and make products that have much lower operational costs?

Tim Bray had this to say $46,213,000,000.00

I looked up the answer to the question: What is IBM's consulting revenue? In 2004, IBM's gross revenue was $96B, of which $46B was Global Services, i.e. consulting. I see that basically as testimony to how our profession, the IT profession, has failed our customers. Nothing against IBM; in fact, as solution-providers go, my experience is that IBM GS is pretty good. But if you see IBM as a microcosm of the industry, it shouldn't cost $46B in consulting to deploy $50B worth of technology. It's not going to be easy to get there, and it's going to take a long time, but we just have to focus on making things simpler.

Now that Godfather Bray is at Sun, perhaps there's a little pleasure in tweaking us. But the question still remains. Have we bought into Radical Simplification?

3. Technology Adoption Takes Time

It takes time for the effects of technology to be felt and for people to get comfortable with the web. We have now had 10 years of Moore's Law in the Datacenter and so we have examples and success stories to pick from. It was a more uncertain world 5-10 years ago and perhaps as a corporation we were hedging our bets. When IBM moves it's a sign that something is ready to take off.

4. Fragility of the browser platform

One answer is that usability was a problem in webmail interfaces that are used daily. In the browser world 5-7 years ago, you had to handle that beast called Netscape 4.7 which meant that it was a case of lowest common denominator in the user interface and frequent browser round trips (read poor user interaction experience due to latency). It was controversial when iNotes was developed and proclaimed that it only supported MS Internet Explorer 5.0 and above only. The product manager who managed to get this dispensation approved deserves to be elevated to the highest levels. Many other managers didn't even bother fighting that battle. I'll hazard that even though IBM could have written something like GMail 7 years ago (and I certainly had a framework from K-station that would have made it possible with some work) it would have been a tough sell internally.

I've written about what was possible back then On Rich Web applications, AlphaBlox and Oddpost and On Gmail and DHTML architecture again

5. Transparency and Visibility

Joel Spolksy says that "it is easier to write code than to read code". This is what the Not Invented Here attitude betrays. We didn't have anything like community source where others might discover and find out about components that have solved similar problems.

(sidenote: when one asks about spending time to document one's component, one is often told that it is not mature enough, that we'd have to spend time supporting new users, we can't afford to slow down to support the one current users etc. There's a tendancy to keep everything close to our chests. Even if others will be able to help and flesh out flaws in our designs, it is thought that exposure to inchoate thinking will be a fatal flaw. This is a mindset that has to be overcome and on this, I am not so sure that it will happen, certainly I appear to me a lone voice in my parts of the wood)

See also Why Specs matter and the Law of Leaky Abstractions

Back to the subject of web interfaces for Notes... No one really invested in making great browser voodoo happen even though we had lots of great frameworks internally. Indeed I still see people grapple with problems that I have fixed before and keep digging up my old code to point them to. Ideally I would have been able to start an incubator at some site and others would have discovered it. Instead everyone is re-writing the same code all over. We haven't codified best practices or if we have, we haven't publicised them sufficiently.

6. Braindead "corporate standards"

Braindead "corporate standards" are another aspect of the problem. Back in 1999 when iNotes was just about usable and we were developing the Notes portlet for Lotus K-station (and subsequently WebSphere Portal), we were told that

1. It was against "corporate best practices" for the http port to be enabled on Domino servers (because it increased the load on the server). Now it seemed weird for an HTTP server like Domino not to turn on its HTTP port, but let's leave that aside.
2. If we got past that ridiculous plank, we were told that if the http port was to be enabled, that "corporate security standards" meant that it had to be SSL enabled, hence https had to be used. The idea is that the same HTTP basic authentication which we use in webmail and everywhere else in the web (other than when we're purchasing something) is too insecure for IBM mail (I've looked at my mail and perhaps only 3% should be confidential). Was it really so bad to just use good old HTTP?

We certainly paid a heavy price for this risk aversion.

• We found that using SSL made thing 3-10 times slower. The server load of processing SSL was insuperable not to mention that that it degraded performance on the wire. Hence iNotes which was already pushing the dtml envelope at that point was nigh unusable. The same concern for server load raised in the first objection was in fact exacerbated.
• As far as the portlet in K-station and WebSphere Portal went, where we were aggregating Domino's xml on a portal server and then marshalling it to the user, this was even worse because there was an extra level of indirection (mail server to portal server to the user's browser)
• On servers where HTTP was disabled, one had to use CORBA (DIIOP and again the SSL variant of this), this wasn't tuned since the DIIOP api was essentially the same as the local notes api, meaning that we couldn't do coarse-grained calls, and instead had to call many apis in a row to get the data that one http request would have obtained.

Now of course we worked around all of these obstacles and made the best of things, lots of people toiled to get performance fairly reasonable (the CORBA api is now (slightly) more coarse-grained, we developed Collaboration Services for Portal so that others could reuse our code and hard-learnt lessons - now the most downloaded aspect of WebSphere Portal etc), but our efforts were hobbled in many ways. We certainly could have had more things going our way.

Summing up, first impressions matter, just ask Malcolm Gladwell (Blink)...

I f the pilot deployments of the various webmail interfaces internally had been reasonable (instead of awful experiences for the executives and other audiences who tried them), we would have been able to get more buy-in, more investment and with a bit of luck might have made comparable interfaces to the snappy Gmail that not only us but also our users and customers would have loved.

It is true that many things had to go right for this vision to have transpired but let's admit that we shot ourselves in the proverbial foot and also acknowledge that it was a failure of imagination and ultimately a failure of technical leadership.

Looking at today's opportunities and the various directions in which Lotus/IBM can go in, I only hope that we don't Get on the wrong Bus this time.

File under: technology, software, web, Lotus, notes, IBM, history, strategy, organizations, adoption, decisions, iNotes, K-station, WebSphere, portal, history, toli

Cultural Sensitivity in Technology

The 'Coltrane' release of Lotus Freelance Graphics, the 1998 version bundled with Lotus SmartSuite, was famously held up for a month when it was found out that one of the clip art images had a tiny 20-pixel image of Taiwanese currency (rather than mainland Chinese currency). Or was it the other way round? An eagle-eyed QA engineer had discerned the offending image. Needless to say, this was something that would cause offense and/or affect purchasing decisions in the involved markets.

I was a little canary in that Freelance mineshaft (or was it a minefield?) when a couple of months earlier in the project I had discovered a seriously outdated map of Africa when integrating new clip art into the product. I wrote one of the sprs of which I am most proud titled something like: "Upper Volta should be Burkina Faso". The spr also made a brief mention of the fact that as far as I could tell there were similar problems in our maps with the names of the states from the former Soviet Republic which had changed earlier in the decade. I believe that my spr was deferred to a later release; it was too late in the product cycle and we'd have had to go back to the company we licensed the art from and so forth. As it turned out however, with the advent of China/Taiwan flag issue, the entire suite was delayed as the test team had to painstakingly go over all the clip art and other areas of the product vetting for similar outrages that could endanger sales.

Incidentally the acronym, spr, stands for "Software Problem Report" in Lotus's parlance. After a decade of indoctrination, I still tend to use spr rather than the more clinical, "defect", which is the favoured term at IBM, or "bug" which is the more widely-used term generally. You can still tell whether someone has the "old Lotus" DNA rather than the "new Lotus/IBM" variety by the terminology they reflexively use. For a dissection of recent linguistic trends in technology see this earlier post.

As a further aside, the Lotus spr database is actually one of the most successful and useful applications of Lotus Notes technology ever and in a similar manner, the Bugzilla project is actually one of the best things to have sprung out of the Mozilla effort. Certainly it was more useful than the Mozilla Gecko browser suite (mother to the Firefox) until the M7 milestone was reached years ago. There must be some kind of software law at work here, something analogous to Zawinski's Law on "software expanding until it can support email" (and now feed technology) and so I'll coin it as Koranteng's First Law of Software Systems:

A software platform reaches its tipping point once it can serve as a bug reporting system.

Intuitively this makes sense, once developers can view, file and retrieve bug reports using their own products, they will be more likely to use them and their confidence in their mission will improve dramatically.

Back to my topic though... I'm sure that the artists who created the clip art in that design firm had no subversive intent; they probably used outdated stock art as their source material. Still, sometimes you wonder at these things and, in this vein, I'm reminded of the Nobel Prize-winning Portuguese author, Jose Saramago's wonderful novel, The History of the Siege of Lisbon, in which a copy editor proof-reading a historical novel defiantly changes a crucial word in the text changing the outcome of a famous historical battle from a miserable defeat to victory. A masterful alternate history of Portugal then develops and its culture is fully re-imagined. I'm sure that bored developers sometimes slip similar things into their products, and not just serendipitous Easter Eggs.

More seriously though, the Freelance incident (or similar incidents relating to any new release of dictionaries or thesauruses from Microsoft) go beyond the cultural sensitivity of which this post is concerned into the rarefied realm of political sensitivity. The same thing goes when choosing product names: you don't want mere mention of your product to cause snickering or offense. A misconceived name or culturally-insensitive feature can be a black eye that will chasten even the highest-flying company. More worrying is that this can translate to a serious financial impact (well beyond a month's lost revenue for the suite) if proper attention isn't paid.

Forensic Investigations

Internationalization however is a mostly thankless, and oft-neglected, task even as I've outlined its importance. From a developer's standpoint, you first think about it mostly in terms of dealing with "resource bundles" and having to put all your resource strings into separate files for different languages. This is a pain because you probably started with a little prototype of a user interface and now you're being asked to find all those hard-coded strings in the user interface and to "do the right thing". Plus it typically becomes an issue only late in the game, when the really interesting work is over and you're ready to think about the next big thing. Instead, you're grudgingly forced to worry about handling different writing systems, dealing with bi-directional text, or that old favorite: the various "special characters" in the wide variety of programming languages, file formats, protocols or operating systems in your software. Those best at this task are akin to forensic investigators and must possess considerable patience and attention to detail.

If, for example, your software mangles names with accents, you'd have real trouble selling in France. Similarly with ampersands and apostrophes - which have significance in SGML and its derivatives (HTML, XML etc). And don't get me started about wider character sets and encoding issues... As an ironic case in point, while developing Lotus K-station, one of our best business partners was somewhat stymied in his development and extraordinary evangelism of our product because, in its earliest release, it couldn't handle the ampersand in his company's name. Luckily for us, he temporarily renamed his organization in his corporate LDAP directory; Sun & Son became Sun and Son until we worked out the quirks. To this day, I always make sure to test whatever product I work on with that organization name. It's surprising though, how often this kind of problem recurs.

The Scene at Home

In this internet era, most technology, and certainly all software development, has to have global concerns in mind. It is said that the sun doesn't set on an IBM project and I work with a very diverse set of colleagues the world over. Presumably the benefit of such a widely dispersed and diverse workforce should be to mitigate the likelihood of issues in this area. As my current project has been going through translation and localization testing of late, I've been thinking a lot about how we handle internationalization.

The old Lotus process was a more decentralized one, each product group had an internationalization team seconded to it. Members of such teams were domain experts and knew the product inside-out in addition to being localization gurus. This had many benefits, the team was involved from the very beginning in the product development cycle and could give crucial design feedback very early and iteratively. Your first prototype was immediately critiqued from their standpoint. The obvious downside of decentralization was the lack of uniform standards - chaos that our translation teams couldn't bear. Some products were very easy to localize, others were, to put it politely, far less so.

As we transitioned to the more centralized IBM globalization process, we got the benefits of uniform standards and greater resources. More languages could be supported in the initial release and you could at least point developers to documentation about the processes they should follow instead of the usual ad-hoc stumbling about. Still this was at the expense of domain expertise, sensitivity and a much slower feedback loop. Some of the test teams would be barely getting up to speed with the product by the end of the testing phase. Localization concerns would sometimes surface too late in the game and would cause much unnecessary reworking and product delays. These things are tweaked, as all processes are, and in recent years, I've seen a push towards re-enabling more domain expertise in the internationalization teams which has been very useful for our product development.

A Photography Case Study

I'll lay out a brief example then that I think is illustrative of the kinds of things we're dealing with here I hope it can tease out the technical, design and business issues that can arise...

I embarked over the past couple of weekends on a mass digitization project and scanned, retouched and uploaded 2,000 or so old photos from shoeboxes under my bed. The technology involved in this exercise was scanner hardware and image acquisition software, the bundled Adobe Photoshop Elements for color adjustment and red-eye correction and a couple of online photo-sharing services: Yahoo Photos and Flickr (coincidentally, I started the day Yahoo's acquisition of Flickr was announced).

The first thing I very quickly noticed: somehow all the photos that I uploaded to Yahoo Photos turned out darker than on Flickr. The services both resize uploaded photos; when you reduce the size of images, you have to select the pixels you are going to use and the photo-resizing algorithm used by Yahoo Photos was giving worse results. This was noticeable to me because a large number of photos featured darker-skinned people such as me. The originals looked fine on the screen and where there were lighter skin tones, everything looked good, but with darker skin tones, the resized photos were not so good. This meant that if I didn't believe in the virtues of Save Lots of Copies Everywhere (SLOCE), I would have leant towards Flickr and stopped using Yahoo Photos.

Secondly, I didn't have Flash installed on my Mozilla browsers hence my experience of the Flickr website was very different from that of my family and friends who started looking at the photos with Flash on Internet Explorer. By default Flickr used Flash to display images. There were immediate complaints about the first bunch of photos. It turns out that images that are rendered in the Flash plug-in have a slightly darker tinge than the images rendered directly by the browser itself. This is not normally noticeable but again this exacerbates contrast issues when darker skin tones are in evidence, as in this case. This was made worse when users tried the slideshow feature because the background of a Flickr screenshow (also implemented in Flash) is black worsening the contrast further.

Thirdly, when retouching photos, the Quick Fix or Auto Correct options in Photoshop seemed tailored for lighter skin tones so I was constantly having to do manual tweaking of my photos. Now this is not a big deal for a few photos and indeed it's fun to fiddle with photos but after a couple of hundred images, it gets tiresome. I found myself longing for "smarter" recognition by the software or for at least, a nice 'dark skin' option that I could set in a preferences dialog. In short I started considering using different software instead of Adobe's.

Now I mention these nitpicks on otherwise excellent and useful products because of the design issues they raise. Technology is simply a tool in aid of people and as human beings we live in different society and cultures. We find that different cultures adapt technologies in different ways to suit local preoccupations and concerns. And I certainly had my own localized concerns these past weeks.

Even when the technical "fixes" are easy, there are design issues and economic tradeoffs that also arise. True, Macromedia could implement better JPEG rendering in Flash but that comes at a certain expense - a good renderer is a hard thing to write (even if they could license the photo-rendering code from the Mozilla folks). Also what they have appears to be good enough for most people (except me obviously). When do you decide that your product is good enough and stop pandering to the Long Tail? Can you afford to do that? Or are you missing out on a vast market opportunity?

Yahoo Photos could certainly implement a better photo-resizing algorithm although presumably there's a performance penalty to be paid if you use a more color-accurate algorithm (or perhaps a larger resultant image size). At the kind of internet scale that the Yahoo service operates on with tens of millions of users and photos, this could potentially seriously affect the scalability of their platform. Conversely, if all Yahoo users switched en-masse to Flickr, which uses the more expensive algorithms, would that platform be able to handle it? Or would it generate a case of teething problems and turn users off because of poor response times?

Flickr could very easily provide a Javascript and native html browser screenshow alternative to their Flash-driven version (as all the other photo services do). I wonder how many users they are turning away by not providing a native browser screenshow. But they use Flash extensively in the rest of their product and, in the case of the screenshow, its usage serves to more easily prevent people from downloading images. Is this pseudo-DRM an essential feature in their service? An alternative they could consider is to keep Flash, but add the option to use a different coloured background for slideshows to de-emphasize the kinds of contrast issues I was encountering. Where would that option show up in the user interface?

Similarly Photoshop could implement a slight variant of their various Quick Fix and AutoCorrect features that were more attuned to my kind of skin colour (indeed I assume that photographers in Africa have written macros or filters that do such a thing). How best though, to phrase a global preference in an options dialog in Photoshop? "Adjust for darker skin tones"? Documentation writers would have a field day finding the right verbiage for such an option. Also what about usability? If you add all these preferences to your product what will your user interface look like? Try typing about:config in a Mozilla browser to get a sense of the kind of complexity that modern software developers have to cope with.

If there was a huge market for these products and services in Africa the issues I faced could be a real problem for the companies in question (although this is unlikely given the low internet penetration rates and the likely widespread software piracy). There would be demand not just for local language versions (say a Swahili language version in Kenya) but also for tweaks that would make these services more closely attuned to the prevailing culture and, in this case, ethnic backgrounds.

Thus photographers in Africa over the past 150 years have had to deal with brighter sunshine, higher contrast, as well as darker skin tones when processing their photos as photography has gone through its various evolutions and has now moved into the digital realm. The people who install photo laboratory hardware in Ghana, where I come from, always have to recalibrate their equipment to deal with the kind of skin tones that are present in the local market. The factory defaults simply won't do. I've had better results developing film in Ghana than in the US because I often forget to tell the labs here that they should "watch for skin tones".

I'd expect then that software that were truly local (and all web software should be local or adjustable to fit local concerns) might sometimes need not just run-of-the-mill language changes or even writing system changes but also, as seems needed here, algorithmic adaptations.

As software designers, we try to engineer simplicity and refrain from overwhelming users in their interaction with our services and products. Software is not the user's main focus, it's rather its use that is most important for the user, for the business and for society as a whole. Yet there are very real concerns in the application of technology in different cultures. So the next time you see a vaguely-worded so-called "Turkish option" somewhere in your application's configuration dialogs, know that someone somewhere was likely adapting their product for a local market. Join me though, in saluting the developers, testers, product managers and designers who collectively worked together to come to that solution. I'd hazard that the tweaking of the product was to fix a deal-breaker in some market.

Finally, for what it's worth, I find endlessly fascinating this notion that cultural sensitivity in technology sometimes necessitates algorithmic adaptation. Maybe though, iterative adaptation in response to local environments - evolution in short, is the name of the game. Perhaps that's simply the way things should be.

Crossposted at the Inside Lotus weblog

Postscript (a year later)

I reported my issues to Flickr and later prodded them with some other folktales about their excessive use of Flash. I know I wasn't the only one with complaints but I'd like to think that my slightly different framing of the issue helped tip the balance: a month later, they stopped using Flash to display images and moved to a much lighter weight DHTML user interface. They continue to use Flash to drive screenshows and organize albums which is fair I suppose. To their credit, they expose enough of their internals to allow third parties to step into the breach to provide alternate interfaces if needed. There is still work to do to "fix" Flash rendering but that doesn't concern me much since I don't use it. Adobe Photoshop will be a challenge however. I'll simply note that about 5 to 10 searchers daily come across this article trying to figure out "darker skin tones Adobe Photoshop" pointing to an unfulfilled need. If that number continues to rise, I'll hazard it won't be long before we see a "dark skin tone" option appearing in the Photoshop preferences.

A Footnote

An revised and expanded version of the essay appears in The Best of Technology Writing 2006 published under the digitalculturebooks imprint of the University of Michigan Press. It can be read online along with many excellent articles at the book's website. It was very exciting to be a part of this and one hopes for some more publications.

I was asked to come up with a tagline for the article, my take:

Everything is local

File under: technology, history, culture, Africa, race, Ghana, web, programming, design, i18n, localization, software, toli, Lotus, Freelance, K-station, Yahoo, Flickr, Photoshop, best practices, IBM

On GMail and DHTML architecture again

So Jon Udell has been pondering GMail and its architecture on his blog. He's also trying to figure out when to augment DHTML apps with even richer client technology. Below follows what I thought would be a short email that I repeat here for blogospheric conversation's sake.

GMail's architecture is actually very generic for a DHTML app. Everyone with a clue should be trying to leverage the browser and that, in essence, is all they are doing.

I helped implement an analogous architecture in Lotus K-station back in 1999. The major difference now is that everyone is using the "v.5" versions of the browsers; no one cares any longer about Netscape 4.7x.

For me the tipping point for rich DOM/JavaScript applications came when we saw mainstream applications like Hotmail and Yahoo Mail start using DHTML menus. My Yahoo only switched to DHTML menus when it was relaunched a few weeks ago and that's my litmus test for conservatism. (Ironically GMail switched back from DHTML menus to standard HTML selects for its "More actions" and apply label bar a month or so ago - although that might just be from usability testing). A similar event happened in the past year with the way users can rate items in Amazon.com wishlists and also in Netflix without seemingly causing page reloads - you know those five stars.

I mention the spread of all these little DHTML flourishes because I remember back in 2000 spending 3 months and well over 10 heated meetings trying to convince IBM architects to allow the use of simple things like DHTML menus and losing the battle. Of course given that kind of resistance, it was unlikely that features like drag-and-drop which one got from the UI engine in K-station would see the light in WebSphere Portal.

I was arguing that judicious leverage of DHTML could improve the user experience and that it could done while addressing accessibility concerns. Ostensibly the objections were not about usability or consistency but rather about not wanting to write multiple UIs and lack of expertise in DHTML programming. I realized then that the argument was not really about "to-DHTML-or-not" but more about comfort with a repartitioning of one's architecture.

It is riskier to do more on the rich browser client because it has been a more brittle platform over this past decade. Companies that do middleware and server-side tooling take a while to move from their core competency. Architects that thrive in that environment are essentially conservative and for good reason... Four years later, I now hear mutterings about drag-and-drop and richer clients in our corridors...

This only underscores the point that Jakob Nielsen's predictions about browser adoption cycles have turned out to be pretty accurate. Even though web application developers have been quietly spreading unobtrusive javascript usage in the interim, it is only now that there's a critical mass of clients that can leverage them; when Amazon and Yahoo move, something must be happening.

The developer tools and resources have gotten (slightly) better and there's more experience with the DOM. Increased adoption of broadband also helps reduce latency for the average client so you don't have to fight the inevitable arguments about performance and can couch your advocacy in terms of user interaction. In any case if and when you do have the performance discussion you can always argue that caching as close to the client as possible is a good thing and what better cache than the browser itself. It just so happens that applications like GMail, Bloglines and Oddpost are the state of the art in terms of browser leverage.

I recently wrote about this type of architecture in my recounting of the history of the DHTML spreadsheet and presentation components that are the genetic forebears of OddPost.

The idea is to fetch an HTML skeleton, decide what content you need, fetch that (as XML), and cache it wherever you get a chance. Render incrementally.

The pattern is simple: Database <-> XML (Optional) <--> JavaScript Object Bindings <--> UI Bindings (HTML) + UI management code

This pattern works very well for page oriented applications like portals, email, aggregators. You can cache or preload your javascript objects and just manipulate the CSS display or visibility attributes for your UI Bindings.

Incremental rendering and multithreaded loading is the name of the game here. Your application is essentially architected as a hypermedia browser just like the browser and leverages the browser's built in core features, our old favorites: incremental rendering and multithreaded downloading.

The decision about using XML as the data transfer format instead of Javascript objects is a toss-up, trading off client reach and memory. XML support in Safari and Opera is not as baked as MSIE or Mozilla so having JavaScript as your interchange format with your UI engine will buy you increased reach and smaller footprint.

Ultimately though XML will be your backend data format so the temptation will increasingly be to use the various XML-on-the-wire APIs (XMLHTTPRequest). Maybe another 4 years is needed for this pattern to see more widespread usage, JavaScript on the wire suffices for now.

Perhaps K-station was too bleeding edge trying to go for XML over HTTP, DHTML and extreme leverage of the browser client 5 years ago but that experience was a great testbed for me and I learned lots of lessons about building rich REST-ful applications, the importance of URIs etc.

Again the major missing feature for this rich web application platform is offline usage and synchronization without introducing new security holes in the browser. But then that's why Bosworth is at Google as the rumour goes, right? I suspect he's got other things in mind though...

I would note that the Mozilla folks know that the offline capabilities in their platform are a bust so I'd expect some eyes on this. Look also at applications like FormsPlayer for innovation in this space or anyone doing forms in general. XForms is my current thing and offline forms are a great feature that users can understand and demos very well; it's also something that the Notes platform does very well. Our old friend, Groove, also does that kind of synchronization well but I'd want that for the web, natively in the browser.

I think that increased leverage of the browser and the DOM is a good thing. It's also a clear trend and for many applications, the browser is good enough. Good enough for Google, good enough for Yahoo, good enough for me.

Sidenote:
One note about memory consumption and pushing things onto the client. There's an end-to-end argument for this notion of pushing intelligence to the endpoints but there is a cost and in this case it's memory consumption. It's mininal but it exists in this case. One lesson I learned was that it's good to use the kind of machines that the average client would be using and consequently I always hoard older machines. I've noticed that with lots of tabbed browsing and increased use of GMail and Bloglines, my four year old Windows ME box (Athlon 900 Mhz 512MB RAM) thrashes memory more often and I've seen more sporadic crashes/freezes with Mozilla. I know this is actually a lagging indicator and that most users are now on Windows 2000 and possibly XP, but it is a pointer to the memory consumption overhead of DHTML apps. All my other machines are fine and I should note that these applications run just as fine on ME as on other platforms in terms of interactivity and all, it is just prolonged use and a piece-of-junk memory and resource subsytem in that excuse for an operating system... My next representative machine is a Win2k box and on that evidence everything is in order even with these highly leveraged applications.

File under: architecture, DHTML, programming, technology, web, javascript, xml, patterns, html, UI, udell, conversation, history, observation, adoption, DOM, scripting, xml, xmlhttp, interaction, IBM, Lotus, K-station, oddpost, yahoo, google, gmail, toli