Developing scalable and reliable .NET services
Rather than restricting its use to a few select developers, Microsoft ran what amounted to a public beta of the .NET Framework and its design tool, Visual Studio .NET. The openness of the process resulted in the availability of hundreds of books, magazine articles, newsgroups, users groups, and other resources. Although much of the information needs to be updated to reflect the changes in the new release, many of the architectural principles are still valid.
Since the beta 2 release of .NET, I’ve been developing and implementing a set of services designed to be used by organizations that want to manage their member community. I designed and implemented the system in .NET to support hundreds of thousands of daily requests from thousands of organizations. In this article series, we’ll look at the key design issues I faced while architecting an application of this scale, and we’ll discuss how I’ve used .NET to solve them.
To allow us to focus on the design principles without getting lost in the complexity of the application, we’ll look at the implementation of a specific set of functions—event notifications—in the context of the overall application. In this article, we’ll look at an overview of the system and see how its functions are divided among the different application layers—presentation, business, and data.
The WiredChurch.com system
While the design patterns used to create the system will support organizations of any type, I chose to implement it first for a specific organization type—churches. I did this for two primary reasons. First, selecting a specific target audience allowed me to engage a group of domain experts (pastors and administrators) for input on how their systems and users would interact with the WiredChurch system. Second, on the whole, churches lag far behind other organizations in their implementation of effective, Web-based community management technology, opting instead for fairly simple brochureware Web sites. So what does the WiredChurch system do for them?
WiredChurch.com doesn’t replace a church’s existing Web site; instead, it allows the church to deliver key community management services to its members without having to install and maintain additional infrastructure. These services include centralized management of users and groups, organizational calendars, discussions, surveys, media (streaming and downloadable), events, and announcements. The system allows the church to maintain its own branding and identity but to expose advanced services to its members through a .NET Web Services mechanism. Let’s drill down through the system, starting with the presentation layer.
Ladies and gentlemen, may I present…
Under the WiredChurch.com system, members will continue to access their church’s Web site through its current URL. But the types of services they can access will increase dramatically. WiredChurch implements a set of Page Generation Services (PGS). The host Web site (the host) makes Web-services calls in order to access centrally managed services.
PGS returns either raw XML or a rich GUI presentation of the data depending on the type of host requesting the data. For example, suppose a church wanted to display a calendar of all of its upcoming events. The data describing these events is stored in the WiredChurch data center. For hosts using ASP or any non-Microsoft technology, the host Web site makes a Web-services call, which returns XML that the local system can render and display. If the host site is using ASP.NET, then PGS exposes the data through ASP.NET User Controls, which encapsulate both the data access and the presentation in a control that can be dropped on a page.
Since the services interface allows very granular service calls, the church could provide an organization event calendar on the home page, a group event calendar on a group page, and a MyEvent calendar for an individual user.
The central user and group management system both authenticates users and authorizes access to their own information and to information managed on behalf of groups to which they belong. In this series, we’ll look at the implementation of the ASP.NET pages and the Web services that enable event notification and management. We’ll examine methods for increasing reliability and scalability of both the host Web sites and the PGS upon which they depend.
Sticking to the business at hand
The business layer of WiredChurch is composed of a series of “engines.” Each of these engines drives a set of functions. The Communications engine manages external e-mail, telephone, and postal mail correspondence with members. The WorkFlow engine maintains the state of current system activities and creates new processes based on a predefined action table. Both of these engines are implemented as Windows services, allowing them to work at a system level to keep these key functions active at all times. In the third article, we’ll see how the implementation allows us to improve reliability and scalability not only by developing the engines as Windows services but also by running the services on multiple machines at the same time.
The engines themselves use extensive asynchronous queuing technology. By decoupling the database from the business tier engines and the presentation layer services, WiredChurch becomes highly scalable and capable of running in both scale-up (more processors and memory in the same box) and scale-out (more boxes providing processors and memory) platforms. In the fourth article in this series, we’ll look at the theory behind using queuing technology to drive scalability and see how we’ve implemented it to support the event management functions.
Where did ADO go?
System architects with extensive ADO experience will scratch bald spots if they try to analyze ADO.NET using their understanding of ADO’s synchronous, connected nature. Microsoft designed ADO.NET as an asynchronous, disconnected replacement for ADO with scalability in mind. In the last article of this series, we’ll examine the way the WiredChurch system creates and manipulates ADO.NET datasets (the replacement for the ADO recordset) and also how it uses the .NET Framework to define data interchange based on XML messages that may or may not be persisted and manipulated using ADO.NET. We’ll show how and why WiredChurch implements almost all of its data manipulation using SQL Server 2000 stored procedures rather than implementing the data-handling code using .NET objects. Finally, we’ll discuss how to build a scalable and reliable architecture using this data-model assumption.
The case study should provide system architects with in-depth event notification and ASP.NET pages with the .NET Web services framework.