Why we don't trust Devil Mountain Software (and neither should you)
Ed note: We were going to publish this investigation Monday morning after buttoning down a few more key facts. Given the fact that IDG just severed ties with Randall C. Kennedy over having an alter ego, we decided to publish our findings, which go beyond fictional sidekicks.
Devil Mountain Software has been a thorn in the side of Microsoft for years and is adept at garnering headlines. The latest effort is a report claiming that 86 percent of Windows 7 PCs were gobbling up too much memory. Can you trust these findings and the company overall? The short answer: No. Here's why.
Adrian Kingsley-Hughes recapped the technical issues with Devil Mountain Software, but frankly the concerns go well beyond mere code. Kingsley-Hughes noted that Devil Mountain Software (DMS) and its Exo Performance Network (XPNet) aren't on his "trusted list." After a discussion with many of our ZDNet experts led by Ed Bott and Jason Perlow in recent days, it's clear none of us trusted Devil Mountain Software.
Here's what we've found from our investigation of the company:
- Devil Mountain CTO Craig Barth is InfoWorld columnist Randall C. Kennedy.
- Devil Mountain's software has potential privacy issues and the company isn't afraid to show off that it can peek into your systems.
- A high-profile customer that "Barth" uses to legitimize Devil Mountain's software says there is no large implementation of the application at the company (see update on page 4).
- Numerous disclosure issues about the relationship between Devil Mountain, Kennedy and IDG, essentially the only outlet that has quoted Barth. Note: Between Saturday and Sunday, InfoWorld pulled references to Kennedy in its blog roll and said that it no longer offers the Windows Sentinel software, which is a clone of the DMS Clarity Suite.
Indeed, InfoWorld Editor in Chief Eric Knorr just confirmed the first point and IDG has severed ties with Kennedy. In a blog post, Knorr outlined Kennedy's fate. Knorr referred our questions to Kennedy, who isn't picking up the phone. ComputerWorld also said that it didn't know Barth was really Kennedy.
We saved for posterity a stray screenshot that InfoWorld forgot -- a Windows Sentinel plug :
Buckle in, because this tale goes from zero to X-files in minutes.
First, the background.
Who the hell is Devil Mountain Software? And can you trust them?
A small software company based in Florida, Devil Mountain Software regularly releases studies filled with detailed performance and market data about operating systems and browsers, with a special emphasis on Windows. When DMS publishes a report, it invariably makes headlines in some of the leading tech publications on the web. The company isn’t modest about its work, either: ComputerWorld reporter Gregg Keizer last week quoted a company source as boasting, “Outside of Microsoft, I don't think anyone knows more about Windows performance than us.”Devil Mountain Software was in the news again last week, with a report claiming that 86% of Windows 7 PCs in the network of 20,000+ machines it monitors worldwide are regularly reaching the breaking point in terms of memory usage. Our in-house Windows experts are skeptical about this data and the company in general.
We asked a team of researchers to look more carefully at Devil Mountain Software and its Exo Performance Network (XPNet) to find out more about the company and its data. What we found is alarming: Dubious claims about the company’s products and its customers; violations of privacy involving the company’s data collection software; and Web posts and interviews with a source who appears to be certainly fictitious.
In search of Craig Barth
The Devil Mountain Software goose chase began with a simple question: Who is Craig Barth? Based on the news coverage, he’s someone that only speaks to ComputerWorld directly, and doesn’t have a LinkedIn account or any other online presence.Every story we found that is based on data from Devil Mountain Software, without exception, was first reported in one of two places. ComputerWorld reporter Gregg Keizer has frequently been first on the scene with details when DMS has released a new study. We found at least a dozen stories under his by-line at ComputerWorld based on reports from XPNet, many including quotes from DMS Chief Technology Officer Barth. As we note later in this report, our reporting strongly suggests that “Craig Barth” does not exist and is in fact a pseudonym for InfoWorld contributing editor Kennedy since the late 1990s
In addition, we found multiple references to the “exo.performance.network” over at IDG’s InfoWorld, where Kennedy’s Enterprise Desktop blog regularly refers to the company in the third person. Among the samples [bold highlights added]:
Just when you think you've got it all figured out, along comes another curve ball to keep you guessing. This time around it's a set of new benchmark test results from our old friends at the exo.performance.network. (March 5, 2008)
This isn’t an estimate. It’s the raw performance delta that engineers over at the exo.performance.network (www.xpnet.com) measured while testing Windows XP (SP3) vs. Windows Vista (SP1). … The lab engineers who ran the tests did everything possible to make Vista run faster. (July 22, 2008)
After evaluating Internet Explorer 8, the folks over at the exo.performance.network… are declaring it to be one seriously bloated piece of software. (Sep 2, 2008)
In a follow-up to an earlier post documenting the RAM footprint of the second Internet Explorer 8 beta, the xpnet.com research staff has re-created its original test scenario… (August 4, 2009)
Data from the exo.performance.network seems to corroborate Microsoft’s numbers... After analyzing process metrics data collected from a network of more than 18,000 Windows IT sites, the xpnet.com research staff determined that nearly 35 percent of them are running the newer version of the Office suite. (Sep 8, 2009)
What was odd in the search for Barth is that no other outlet with the exception of one has ever quoted Barth directly. Several stories originally reported at ComputerWorld using XPNet data have been widely publicized and occasionally leap into traditional (non-technical) media, often through syndication deals with IDG. We found examples of stories mentioning Devil Mountain Software at All Things Digital (Walt Mossberg’s popular side project for the Wall Street Journal), FOXNews.com and USAToday.com and ZDNet’s international properties among others.
Barth is quoted in at least 12 articles that have appeared in ComputerWorld, beginning in November 2007. His name disappeared completely from the site for all of 2009, but he has made up for lost time in the first seven weeks of 2010, as five stories authored by Keizer and quoting Barth have appeared in ComputerWorld.
ComputerWorld also routinely attributes posts on the exo.blog to “Craig Barth,” although until today the blog itself used only a single by-line, “Research Staff,” and Barth’s name does not appear anywhere on the blog. Barth is a colorful, opinionated and attention-grabbing guy. Unfortunately, he doesn’t exist.
ZDNet editors have confirmed through multiple sources and Kennedy’s digital slip-ups that “Craig Barth” was a pseudonym used by Kennedy at multiple technical publications in the 1990s. One of those articles turned up in a Google search. As news editor for Windows NT Magazine (later renamed Windows IT Pro), Barth’s by-line appeared in an “NT News Analysis” first published in April 1998. Ironically, a brief item in that same column, also by-lined by Barth, includes several pithy quotes from Kennedy, who was identified as a “senior analyst with Giga Information Group.” As an analyst, Kennedy would have been prohibited from writing for trade publications, so using a pseudonym would have been essential in this case. That by-line appeared for the last time in the December 1998. In January 1999, the section got a new name, and Kennedy’s by-line reappeared.
The same month that he stopped contributing to Windows NT Magazine, “Craig Barth” joined Kennedy’s Competitive Systems Analysis (the CSA in csaresearch.com, which redirects to the same site as XPNet.com). The Archive.org Wayback Machine captured what appears to be the public unveiling of the CSA Online IT Research Center on December 5, 1998. The bio reads as follows:
Craig Barth is a Senior Analyst with CSA and a veteran author. His work has been a staple of Windows and Windows NT-related industry publications for the past several years, and he is a sought after speaker and consultant.
Prior to joining CSA, Mr. Barth served as the News Analysis editor for both Windows NT Magazine and SQL Server Magazine. He is also a regular contributor to Selling NT Solutions Magazine, where he employs his considerable channel marketing experience to assist others in delivering solutions based on this emerging platform.
The bio appears unchanged in future iterations of the web site, but sometime between April 10 and May 18, 2004, it was removed. The About CSA page captured on that date contains only a single “Analyst Profile,” for Kennedy.
The Devil Mountain Software website (devilmount.com), which also redirects to the content at XPNet.com, first appeared in March 2005. The About DMS page includes only a bio for Kennedy. By May 2005, all references to Kennedy had been removed from the site as well, and the About DMS page simply stated that the company had been “[f]ormed in January of 2005 by former CSA Research executives.”
The current XPNet.com design debuted in early 2007 and was captured by Archive.org on April 30, 2007. It does not now and never has contained any reference to either Kennedy or “Craig Barth.” By September 2007, Kennedy was using the page to publicize content he had posted on his new blog. Links from Xpnet.com at that time went to an early version of the Blogger-hosted exo.blog. By combing through those posts using the Wayback Machine, we were able to determine that the Research Staff profile originally belonged to Kennedy. Some posts written in 2007 on that blog contained the Randall C. Kennedy by-line. All of those posts were later changed so that the by-line was “Research Staff.”
Kennedy was more careful after 2007 not to allow his identity to leak into the exo.blog site, but on at least two occasions—in November 2007 and September 2008—he slipped up by posting graphics to Blogger using the wrong login. Those graphics were uploaded to Google’s image servers and are tagged with the ID rkennedy01ca.
We found a similar login name in one other place as well, at Symantec’s community forums. In a June 8, 2006 post under the username rkennedy01, Kennedy shared some experiences and plugged his then-new Clarity Suite software. In response to a comment, he used his full name and identified himself as CTO of Devil Mountain Software. That’s the same title that is now attributed to “Craig Barth.”
With the exception of those 1998 articles and the recent interviews in ComputerWorld, we can find no other signs that “Craig Barth” exists. There is no trace of him in any social media site. We have found only one other reporter from a non-IDG publication who has ever quoted Barth. According to that reporter, the number he called for a story in 2007 is the main number for Devil Mountain Software; he does not remember whether he actually spoke to Barth.
Kennedy’s disclosure statement on his Infoworld blog made no mention of Devil Mountain Software or XPNet.
Note: The exo.blog now has Randall Kennedy as an author. There is still no mention of Barth. The change coincided with his software's departure from InfoWorld.
Today.
And the profile on Saturday.
What does Devil Mountain’s software do? -->
What does Devil Mountain’s software do?
The next trigger that raised eyebrows about Devil Mountain came in a blog post last week. A Feb. 19, 2010 post on the exo.blog takes aim at Ars Technica author Peter Bright. Entitled Rebutting ArsTechnica, the post is an angry response to a critique (Behind the Windows 7 memory usage scaremongering) published by Bright. In the response, the blog post’s author freely admits that some person or persons at Devil Mountain Software inspected the individual data uploaded from Bright’s PC.Here’s the eyebrow raising comment:
Mr. Bright didn’t stop at simply attacking our intelligence. He took the additional step of actually contributing data from his own test PC. … [B]y connecting his PC to our network, he made his raw system metrics data available to us. And after reviewing this data, it became clear why our System Monitor widget flagged his system as being low on memory. [emphasis added]
Obviously, Devil Mountain’s Software can examine your PC, identify you, and use your uploaded data against you if XPNet/Barth/Kennedy disagree with your opinion. How does this work exactly?
Devil Mountain Software’s two flagship products—indeed the only products, as far as we can determine—are a test suite called Office Bench and a Windows monitoring tool called the DMS Clarity Suite. Both programs have been available for download from Xpnet.com for several years. In addition, InfoWorld began offering the software for download under the Windows Sentinel label. (InfoWorld pulled the software over this weekend.) At both sites, the software is free but requires registration. InfoWorld requires 13 separate pieces of user information, including name, e-mail address, company and job function, and geographic location. The XPNet registration for asks for name and e-mail address only. After registering, you are assigned a 5-digit unique user ID that must be entered during the program configuration.
We registered as users and downloaded copies of InfoWorld's Windows Sentinel tool and XPNet's DMS Clarity Tracker Agent.
Based on our tests, the InfoWorld Windows Sentinel and the DMS Clarity Tracker Agent are identical. The screens that appear during setup (including the end user license agreement) are the same. MD5 checksum hashes of the file downloaded from XPNet and the one from Windows Sentinel match perfectly. The only difference is a single letter in the file name of the executable.
As for the software itself, the installer is not digitally signed. It installs two Windows services: Cfwtracker.exe and Cfwupload.exe. The tracker program adds information at regular intervals to a database (in Microsoft Access format) stored in the user profile of the currently logged-on user. The upload module periodically sends that data to a remote server.
At its website, DMS claims that the software transmits data securely:
Running in tandem with the DMS Clarity Tracker Agent, the Tracker Upload service spools collected data to the exo.repository for later review. The service uses an SSL-secure web connection that is compatible with most enterprise firewalls and proxy server implementations.
An architectural diagram on the same page shows an HTTPS (SSL) connection:
We found this claim to be untrue. In our tests, using machines in widely separated geographic locations, the DMS software made simple (non-secure) HTTP connections on port 80, transmitting data to a server at IP address 66.115.28.220. The IP block at 66.115.28.* has DNS A records that point to devilmount.com, xpnet.com, and csaresearch.com. All of those companies are registered to Devil Mountain Software and include the name Randall C. Kennedy in the registration information.
When we attempted to use a browser to make a secure connection to https://xpnet.com, we received two certificate errors. The certificate associated with the site, originally issued by Equifax Secure Global eBusiness, had been issued to a different domain, csaresearch.com. In addition, the certificate had expired on September 7, 2009.
XPNet.com has no privacy policy on its site. Its license agreement contains no privacy information whatsoever. InfoWorld, however, made a prominent claim about privacy on its page that, until this weekend, offered the Windows Sentinel program:
Performance data is uploaded to the exo.performance.network, where your most recent one week of data is stored for viewing and analysis. Performance data will be shared in aggregate only and never identified as linked to your individual account.
The Feb. 19 exo.blog post appears to have violated that policy in a big way.
We conducted tests using the software downloaded from InfoWorld’s Windows Sentinel page and from XPNet.com and found no differences in their behavior. In both cases, the captured data was sent to the same server, which is under the control of Devil Mountain Software. As noted previously, InfoWorld pulled the software over the weekend.
Devil Mountain Software: Wall Street powerhouse? -->
Devil Mountain Software: Wall Street powerhouse?
At this point, it was prudent to question just about everything about Devil Mountain Software. The next trigger that prompted more questioning was a Feb. 19 ComputerWorld article where Barth defends the company against its doubters.Barth defended the data and his conclusions, arguing that Blue Mountain [sic] was hardly a "little-known outfit" as he ticked off his and his company's bona fides. The benchmark software, dubbed DMS Clarity Suite, was created by former Intel performance engineers, and is deployed commercially at financial firms, he said, including Morgan Stanley and Credit Suisse First Boston, as well as on Wall Street trading floors, where PC performance problems may mean millions down the drain.
That statement immediately drew my attention. Why? For starters, Wall Street firms don’t allow vendors to just randomly mention where their wares are installed. The choreography between financial titans and what tech vendors can say is tight to say the least. How could Barth yap so freely about his dealings with Wall Street firms?
Given that quote about Morgan Stanley and Credit Suisse, which by the way dropped the “First Boston” part in 2006, begged for a few follow-up calls.
A spokeswoman for Morgan Stanley through Friday was working to verify Devil Mountain Software's claims. We'll update Monday, but thought we'd push our findings forward given the fact InfoWorld pulled the Windows Sentinel software. Initially, Morgan Stanley didn't know of the company. IBM Global Services is the primary vendor for Morgan Stanley.
Separately, we’ve learned that Morgan Stanley has a tight security policy that would have found the problems with the ports for the DMS Clarity Suite. It’s highly unlikely that Morgan Stanley would allow any application to transmit data outside of its firewall given the dollars at stake. Best case scenario for Devil Mountain is that its suite is installed on a few stray PCs at Morgan Stanley in potentially violation of security policies. Worst case: The claims that Morgan Stanley is a Devil Mountain Software customer are fabricated completely.
Update 4: As of 5:30 p.m. EST Monday Feb. 21, Morgan Stanley hasn't commented on Kennedy's claims. Calls and emails still out. Will keep trying and update accordingly.
Update 5: Morgan Stanley along with other customers mentioned by Kennedy aren't commenting officially through Feb. 24 until the company finishes a full investigation. Kennedy's has claimed he can present invoices, checks and other data to prove Morgan Stanley is a customer. Kennedy adds that there's a 3,000 seat implementation at Morgan Stanley. These claims have been presented to Morgan Stanley along with Kennedy's long blog post outlining more details. We've also begun to contact other customers that Kennedy has named and will follow up in a separate post if warranted.
Update 6: While there's no official comment from Morgan Stanley yet---and there may never be---we've learned that much of what Kennedy has outlined appears to be correct regarding the Wall Street firm. The exact size of the license isn't known, but Kennedy's software, specifically the predecessor to the DMS Clarity Suite, was licensed beginning in 2000 and 2001. The software was used internally only so the security concerns we raised would not apply.
So who owns Devil Mountain Software?
We love a mystery, but tracking down who actually runs this outfit was unbelievably difficult. But after extensive research, we are convinced that Randall C. Kennedy is the owner and sole operator of this company and that it has no other employees.The company’s official website says this:
Devil Mountain Software, Inc., is an independent software development company based in FL. Our primary focus is on the financial services sector where we maintain several large installations of our commercial DMS Clarity Suite performance analysis solution. The exo.performance.network is the pet project of our engineering team and a way to give something back to the larger IT community.
That same page includes a generic e-mail address (info@xpnet.com) and a phone number in the 561 area code, which covers Palm Beach County in Florida. When we repeatedly attempted to call this number, no one answered.
The website contains no other contact information of any kind and does not have an “About Us” page or any listing of the names or biographical details of the company’s owners or operators.
In addition to the primary xpnet.com domain, at least three other domains point to this site: csaresearch.com, claritysuite.com, and devilmount.com.
The company also communicates via the exo.blog, which is prominently linked in the navigation bar at xpnet.com and leads to a site hosted at Google’s free Blogspot service.Its tagline is “The official blog for the editors and research staff and [sic] of the exo.performance.network (www.xpnet.com).”
It, too, is completely free of any names or contact information. The “About This Blog” page leads to a profile for a user named “Research Staff,” which has no additional details. That nom de blog is used, without exception, on all posts at expo.blog. As noted earlier, Kennedy's name was added to the site overnight, after this story was written but before we published.
The company is registered as a for-profit corporation with the Florida Division of Corporations. The active filing shows a principal address of 13808 10308 Trianon Place, in Wellington, Florida. The registered agent who filed the paperwork is Sobhana Retnasami, whose title is listed as CEO. An update to this document was filed with the Florida Secretary of State in April 2009, apparently to correct the ZIP code.
That address on Trianon Place is in a residential neighborhood, as confirmed by Bing Maps and Google Maps satellite images.
A South Florida real-estate tracking site confirms that the property at that address was purchased in July 2003 by Randall C. Kennedy and Sobhana Retnasami, and that the title was transferred into the name of Retnasami a year later.
As a sidelight, it’s worth noting that the predecessor of Devil Mountain Software, CSA Research, was also housed in a residence, this one in Danville, California. The CSA Research website included a street address followed by Suite 101. We were able to determine from online maps that the address was a single-family home in a residential neighborhood. According to online real-estate records, it was sold in July 2003.
(The Danville location also explains the company name. Danville is just a few miles from Mount Diablo, the largest peak in the East Bay Area. Diablo is Spanish for devil.)
Update: It appears that the InfoWorld blog noting the Kennedy developments is behind a registration wall from time to time. Update 2: This was a glitch that was fixed on InfoWorld's end. Here's the full statement from Knorr:
On Friday, Feb. 19, we discovered that one of our contributors, Randall C. Kennedy, had been misrepresenting himself to other media organizations as Craig Barth, CTO of Devil Mountain Software (aka exo.performance.network), in interviews for a number of stories regarding Windows and other Microsoft software topics. Devil Mountain Software is a business Kennedy established that specializes in the analysis of Windows performance data. There is no Craig Barth, and Kennedy has stated that this fabrication was a misguided effort to separate himself (or more accurately, his InfoWorld blogger persona) from his Devil Mountain Software business.
Integrity and honesty are core to InfoWorld's mission of service to IT professionals, and we view Kennedy's actions as a serious breach of trust. As a result, he will no longer be a contributor to InfoWorld, and we have removed his blog from this site.
Over the past 10 years, Kennedy has contributed valuable information on Windows performance and other technical issues to InfoWorld and its readers -- insight and analysis we still believe to be accurate and reliable. Based on our discovery, however, we cannot continue our relationship with Kennedy. Questions about this matter may be directed to Kennedy at rck@xpnet.com. We apologize to our readers.
Update 3: Kennedy comments on the story.