Scaling your Web 2.0 app with caching
Danny Ayers has an interesting post detailing why and how caching can be used on the Web to meet scaling requirements for Web 2.0 services. Danny wrote:
"If the Web is the platform, then as much of the data that isn’t entirely application-specific must be exposed in such a way to allow it to be devolved to other parts of the Web. Use of existing Web-based data (i.e. other people’s exposed stores) shouldn’t be the exception, it should be the rule. This mean that for the system to be robust, there will be more work on caching, and less on the construction of disconnected data silos. Caching is one of the key features that has enabled Web 1.0 to scale."
By this Danny means not just using open source technologies like Apache, PHP and MySQL for data storage - but using open XML-based data standards like RSS and Atom. Danny goes as far as to say that a startup’s software should "consist of three parts commodity to one part unique", with a lot of commodity being the data formats and storage in an application. So what will be the unique parts of an app? Danny suggested user interface and some of the data processing, as two examples.
I think Danny's main point here is that because commodity data is such an integral part of many Web 2.0 services, then caching in effect acts as a storage mechanism for data. A good proportion of an app's data is not locked up in a closed database, but moves (relatively) freely on the Web and is available at many different data points (backed up, in a sense). Scaling a Web app becomes easier, because you are utilizing the Web's distributed nature for your data storage.
Mark Nottingham also wrote on this theme in a post entitled Leveraging the Web: Caching. Mark detailed a real-life experience where he deployed a Web caching solution for a large company. He noted:
"The biggest surprise to many involved was that we were able to scale the Web site out with basically no code, using off-the-shelf components. Caching brought both scalability and reliability to the application very cheaply and easily, despite the requirement for authentication."
I'd be interested in hearing from readers about your caching solutions for the Web. Feel free to leave a note in the Talkback section.