Tech
Scalability
I've been thinking about scalability alot today. The thought process actually kicked off with the Typepadoutage from last week...even thoughin the end scalabilitywasn't exactly the problem there. Yet it seems that "growing pains" is a common compliantabout many Internet social software tools -- everything from Technoratito Orkut have been criticized for straining under increased load.I've never really been a coder, butI've watched lots of different systems have to adjust -- or rearchitect-- due to scale challenges. Back in 1990, I worked for FTD when theywanted to get into an early version of e-commerce. Their approachwas to develop an in-house system which used an ASCII text colon-delimitedflat file for product records...which was fine for fifty items but notfor 5000. Needless to say, that system didn't last very long (thoughthere were hundreds of additional reasons for that, which would make aninteresting case study one of these days).In my world, cc:Mail was definitelya system which eventually burst at the seams under scale issues. Asa cc:Mail administrator back in 1992, I only had solid performance whenmy post offices were each about 200 MB. Now that's 1/3 the size ofmy own mailbox. cc:Mail had great hardcoded limits like 200 userson a mailing list, or 20 attachments per message. Obviously, Hubertand company designed an incredible system for its day, but its file-sharearchitecture eventually hit the wall for modern-day messaging requirements.Given all of that context, it's somewhatamazing that Notes has been able to evolve from 75 users per server overNETBIOS in Notes R3 to 150,000 user scalability with Domino 7, all withouta fundamental rearchitecture. It may not be sexy work, but the plumbingdone in Domino 7 to get 40-400% scalability improvements on existing hardwareis especially impressive when viewed this historical context.
I've been thinking about scalability a
lot today. The thought process actually kicked off with the Typepad
outage from last week...even though
in the end scalability
wasn't exactly the problem there.
Yet it seems that "growing pains" is a common compliant
about many Internet social software tools -- everything from Technorati
to Orkut have been criticized for straining under increased load.
I've never really been a coder, but I've watched lots of different systems have to adjust -- or rearchitect -- due to scale challenges. Back in 1990, I worked for FTD when they wanted to get into an early version of e-commerce. Their approach was to develop an in-house system which used an ASCII text colon-delimited flat file for product records...which was fine for fifty items but not for 5000. Needless to say, that system didn't last very long (though there were hundreds of additional reasons for that, which would make an interesting case study one of these days).
In my world, cc:Mail was definitely a system which eventually burst at the seams under scale issues. As a cc:Mail administrator back in 1992, I only had solid performance when my post offices were each about 200 MB. Now that's 1/3 the size of my own mailbox. cc:Mail had great hardcoded limits like 200 users on a mailing list, or 20 attachments per message. Obviously, Hubert and company designed an incredible system for its day, but its file-share architecture eventually hit the wall for modern-day messaging requirements.
Given all of that context, it's somewhat amazing that Notes has been able to evolve from 75 users per server over NETBIOS in Notes R3 to 150,000 user scalability with Domino 7, all without a fundamental rearchitecture. It may not be sexy work, but the plumbing done in Domino 7 to get 40-400% scalability improvements on existing hardware is especially impressive when viewed this historical context.
I've never really been a coder, but I've watched lots of different systems have to adjust -- or rearchitect -- due to scale challenges. Back in 1990, I worked for FTD when they wanted to get into an early version of e-commerce. Their approach was to develop an in-house system which used an ASCII text colon-delimited flat file for product records...which was fine for fifty items but not for 5000. Needless to say, that system didn't last very long (though there were hundreds of additional reasons for that, which would make an interesting case study one of these days).
In my world, cc:Mail was definitely a system which eventually burst at the seams under scale issues. As a cc:Mail administrator back in 1992, I only had solid performance when my post offices were each about 200 MB. Now that's 1/3 the size of my own mailbox. cc:Mail had great hardcoded limits like 200 users on a mailing list, or 20 attachments per message. Obviously, Hubert and company designed an incredible system for its day, but its file-share architecture eventually hit the wall for modern-day messaging requirements.
Given all of that context, it's somewhat amazing that Notes has been able to evolve from 75 users per server over NETBIOS in Notes R3 to 150,000 user scalability with Domino 7, all without a fundamental rearchitecture. It may not be sexy work, but the plumbing done in Domino 7 to get 40-400% scalability improvements on existing hardware is especially impressive when viewed this historical context.
Originally by Ed Brill from Ed Brill on December 19, 2005, 1:15pm