Solving the Web app's 'offline problem': How Zimbra did it (and others will follow)

Since first hearing about how Sun's Francois Orsini had used Apache Derby (a.k.a. JavaDB) to create a proof of concept that showed how a Web-based tax application could switch into an offline mode with little more than a browser and the Java Standard Edition plug-in, it became very clear to me that the so-called "offline problem" associated with Web apps wasn't really a problem. It was just a matter of time before Web app developers took the next logical step and married Derby to their real applications in a way that users would be able to use their Web apps with or without a connection to the Net. Not only that, when a connection was eventually restored, whatever work that someone did while offline would be synchronized with whatever data they were storing in the cloud.

Orsini eventually demonstrated his proof of concept to me, David Van Couvering, Redmonk's James Governor, and a few others at JavaOne 2006. Later that night, Governor (provoked by a beer or two) practically synthesized an entire research paper (in his head) about "the Synchronized Web." Orsini's concept now had a name.

One reason support for thick clients like Outlook makes so much sense is because of the way they solve the offline problem.

So, I'm not surprised that less than a year later (nearly on the eve of JavaOne 2007, going!), the architecture Orsini was showing is now being applied to a commercial application: the Zimbra Desktop. This past March, my colleague (and boss) Dan Farber had the scoop on Zimbra's advance on the so-called offline problem. Perhaps now, we should stop calling it a problem (although the product is still in alpha). Since then, I've been meaning to circle back to the folks at Zimbra to get more details on just exactly what they did and today, I finally caught up with John Robb, Zimbra's vice president of marketing and product management. 

Apache Derby is basically a pure Java database. In other words, whereas most database management systems (eg: Oracle, MySQL, etc.) must be compiled into different binary versions in order to run on each of the operating systems (Windows, Linux, Solaris, etc.) that they support, Derby runs on top of the Java Runtime Environment (JRE). So long as a JRE is present (which it is on a great many systems), Derby (like other Java applications) needs no recompilation. Not only that, it can run within the context of a browser since most implementations of the Standard Edition of Java are browser plug-ins. The net result is that developers of Web apps can design their applications in a way that those apps can save their information (records, documents, spreadsheets, etc.) to the local Derby-based database when no connection to the Internet is available. In other words, Derby can provide what usually isn't available to Web applications: a local persistence mechanism to which data can be saved locally (one that, thanks to its independence of the underlying operating system and filesystem, should be as portable to any JRE). 

Zimbra's unique selling proposition is that its solutions take the Web 2.0 approach to e-mail and collaboration. On the server-side is an e-mail server that's available for Mac OS X and a variety of distributions of Linux. On the client-side, Zimbra's primary approach to e-mail and group calendaring is through a rich, AJAX-driven user interface. Secondary to that are other ways to access Zimbra's collaboration server. Through a series of Web services and synchronization APIs, users that want to use a more traditional thick client e-mail interface like Outlook can not only access and synch-up with a Zimbra server via POP3 or IMAP (in the latter case, basically replacing Microsoft Exchange), they can also leverage the APIs to mashup their own collaborative apps (or Web apps). One of Zimbra's strengths is on the mobile front as well where a variety of devices (Treos, Blackberries, etc.) can work with a Zimbra server.

Unlike other API-laden solutions that might appear to be competitors to Zimbra, Zimbra itself doesn't play host to an Internet-based service. Instead, it licenses its solutions to (1) companies that in turn run the solution behind their firewalls and (2) service providers that want to provide the Zimbra solution as an offering to their customers. For the former, Zimbra's price point is almost as disruptive as its architecture: $28 per user per year (there's 25 user minimum).

One reason support for thick clients like Outlook makes so much sense is because of the way they  solve the offline problem. Under the hood, Outlook has its own way of storing and handling e-mail and calendar data and so long as a thick client can retrieve its data from a server, users can continue to work, even when a network connection isn't present. But for Web 2.0 companies like Zimbra looking to drive the cost and complexity out of thick client approaches, figuring out how to offer thin, Web-based access with the sort of persistence that's normally associated with thick clients is the Holy Grail.

With the right schemas, a database can look like anything. It can even emulate an entire filesystem. As such, technically speaking, Apache's Derby should be capable of tackling just about any persistence problem. The question for Zimbra then was how far to take it. The decision, according to Robb was relatively easy because of the architecture Zimbra uses on the server side. As it turns out, Zimbra already relies on Java and MySQL on the server side to handle metadata and user data. The message stores themselves -- the places where e-mail messages are kept -- are implemented as files in the server operating system's filesystem. To support clients like Outlook, Zimbra's Collaboration Server was already enveloped in synchronization APIs. So, according to Robb, when Zimbra first started investigating the idea of offline access for its Web client, it was simply a matter of how to come up with a miniature version of the same architecture that was on the server instead. In other words, whenever an e-mail or calendar transaction was completed, instead of saving the resulting data to a Zimbra server in the cloud, that transaction would be saved to something local that looked just like the server in the cloud, and then that local server, using pretty much the same APIs that offline thick-clients like Outlook use, would synchronize with a Zimbra server in the cloud.

"At first, we looked into the idea of having MySQL on the client, much the same way we have it on the server" Robb told me via telephone earlier today. "But we ruled that out and that's when we ended up trying it out with Derby. For clients, there's one installation file. It comes with everything they need including the Java Runtime." 

However, much like the way Zimbra's Collaboration Server keeps the messages themselves in the local filesystem, so too does the Zimbra Desktop. In other words, while Derby is used to locally persist metadata, user data, and synchronization data, the messages themselves rely on the local filesystem (eg: Windows, Mac, or Linux). So, even though the solution is available to multiple desktop operating systems, it isn't entirely self-contained within the context of a browser (at least not when the synching capabilities are being used). This in turn means it isn't truly OS independent (or relying on the OS-independence of a runtime like Java) which in turn means it's not yet truly portable the way a Web-client with offline capabilities might one day be.

That's not to say it won't happen. OS-reliant or not; By equipping it's Web-client with offline capabilities, Zimbra has put itself within reach of a Holy Grail that many said couldn't be found. Other vendors with similar interests and with the sorts of resources that dwarf those of Zimbra -- companies like Google who will want the off-line problem solved for their own Web-apps as well -- can't be far behind.