Supernova 2007: The Internet's black hole
KC Claffy spends her days trying to figure out what is really going on inside the Internet. She is a researcher with the Cooperative Association for Internet Data Analysis (CAIDA), which provides tools and analysis in service of making Internet infrastructure more robust and scalable.
The only problem, according to Claffy, is that the data required to study the live, operational Internet is not available. The Interent is a black hole for researchers, with the data in the hands of incumbents--telcos, ISPs, etc.--who have economic incentives to keep it out of the hands of researchers, she said. Nor does she give the software security vendors dealing in anti-virus and anti-spam product the benefit of the doubt.
"The research community has its hands tied because we don't have access to real data to study. The problem is getting access to the atoms you want to measure, so you can't figure out the real cost of infrastructure.," she said. "We need the numbers on spam, but we get them from the anti-spam vendors who have an incentive to have them look a certain ways. It's the same [problem] for viruses, the cost of broadband and file sharing."
She alleged that Hollywood, for example, uses studies that haven't been peer reviewed or properly sourced and lack a published methodology to make claims about piracy and file sharing usage. And, based on research with available data, CAIDA concluded that the incumbent players hide the fact that they lose money on non-IP revenue to subsidize other services, such as voice, and spend money on lobbying efforts rather than addressing major problems.
"There is no culture of getting and curating of Internet data, and the data is kept hidden from researchers under the rubric of privacy and data sharing," Claffy added.
In a recent published report, Claffy cited 16 problems that would be more easily solved if researchers had the data:
• security • authentication • spam • scalable configuration management • robust scalability of routing system • compromise of e2e principle • dumb network • measurement • patch management • “normal accidents” • growth trends in traffic and user expectations • time management and prioritization of tasks • stewardship vs governance • intellectual property and digital rights • interdomain qos/emergency services • inter-provider vendor/business coordination broader impact Red: persistently unsolved problems for 10+ years
"The problems on the list are not about technology--they are are rooted in economics, ownership, and trust." She recommended that the FCC offer up a part of spectrum and donate it to the commons so that researchers could see what is going on in the public network. The crowd at Supernova 2007 cheered.