BSA figures do not add up

The BSA's latest study claims to prove that software patents are of equal importance to SMEs and large companies, a claim that political parties and some media organisations have taken at face value. But does the study really show that SMEs are of equal importance, or has the BSA presented the facts in a misleading way to lead people to the conclusions they want them to draw?

This study comes at an important time. In a few weeks time MEPs are due to vote on the software patent directive. This vote is crucial if essential changes are to be made to the directive that will restrict the degree to which software can be patented. The BSA, which is keen for the directive to be passed in its current form, published the study to persuade MEPs to vote against any amendments that would restrict patentability.

The results of the study have already been quoted by an MEP from the Alliance of Liberals and Democrats for Europe (ALDE), as an excuse for their decision to vote against many of the amendments, in apparent contradiction to their party policy. The Liberals pitch themselves as the party for SMEs, so the study was conveniently offered as proof that SMEs need this directive.

Toine Manders, the intellectual property spokesperson for ALDE, said in a press release on Tuesday: "The directive has been portrayed as benefiting only major industrial conglomerates at the expense of the small software developer, but this is simply not true. A recent study by the Business Software Alliance indicates that SMEs account for 20% of all CII (Computer Implemented Inventions) patents granted since 1998 and 81% of them rely on patent protection for their businesses."

Interesting, 81 percent of SMEs rely on patent protection for their business? Even for politicians who are used to massaging statistics that is bad maths. The study actually found that around 2000 SMEs have filed patents, which is only 0.018 percent of the 11 million SMEs that exist in Europe. So what could have caused this dyscalculia in the liberals? The BSA press release appears to have the answer.

"[These figures] show that thousands of European innovative SMEs have patents -- in fact 81 percent depend on one patent and a further 10 percent hold just two patents -- and rely on these to help develop their businesses," said Francisco Mingorance, BSA's director of public policy.

The 81 percent that Mingorance refers to is the percentage of SME patent holders that hold one patent, but you'd be forgiven for thinking it refers to all European SMEs. In fact, I'd hazard a guess that the majority of people would make this error. But then, if most people reading the release came away thinking: "81 percent of SMEs hold software patents, of course we should let this directive go through," would the BSA be upset? I doubt it.

I put this to the BSA, and they replied: "The 81 percent figure refers to those in the study". Well, surely they can be forgiven for just one example of bad English. If only it were that simple. An ALDE spokesperson told me on Wednesday that the proportion of SMEs holding patents has increased over time. In fact, the study found that the proportion of SMEs patent holders has remained constant over the past six years. Another mistake by the Liberals?

Perhaps, but this one has been repeated by the Guardian, in a story that appeared on 9 June. "The battle over the EU's software patent directive heated up yesterday when a survey showed small and medium-sized firms, the alleged victims, accounting for a growing proportion of such patents in the past few years," said the Guardian in the article.

What could cause both the Liberals and the Guardian to make this mistake? Again, the BSA's unfortunately worded press release appears to hold much of the blame. The first line of the press release states: "A new study published today by the Business Software Alliance demonstrates the growing importance of patents on Computer-Implemented Inventions (CII) for European small and medium-sized enterprises."

A few paragraphs later the BSA states that "European small- and medium-sized businesses account for more than 20 percent of all CII patents granted since 1998; And the number of SME patents granted has been rising steadily in recent years." The BSA omits to mention the important fact that the proportion of patents granted to SMEs has remained constant, so taken in conjunction with the first paragraph, it is understandable that both ALDE and the Guardian misunderstood this.

When I suggested to the BSA that it would be more accurate to change the first paragraph to say "…demonstrates the importance of patents…", the BSA replied, "Honestly, you seem to be nitpicking here."

These are not the only misleading statements in the BSA study. Mingorance states in a later quote that: "European SMEs need their patents every bit as much as big companies."

I put it to the BSA that as only 2000 of around 11 million SMEs across Europe hold patents and as SMEs are responsible for half of Europe's turnover and employ more than 53 percent of Europe's workforce, according to the European Commission, doesn't the survey actually show that disproportionately few SMEs hold CII patents? I also asked how they justify saying that the need of SMEs is equal to that of large corporations?

The BSA responded: "This seems more than obvious. If a European SME takes the time and effort to get a patent despite all the well-known difficulties that they face, it's safe to assume they have good reasons and incentives for doing so."

Ambiguous and misleading statements is one thing, but surely the BSA would shy away from factual incorrectness. Well, it appears not.

The BSA admitted two weeks ago that their claim that "European small- and medium-sized businesses account for more than 20 percent of all CII patents granted since 1998", is incorrect, as the study does not distinguish between European and non-European SMEs. Despite realising this error the BSA decided not to sent out a corrected press release because "none of the stories that appeared that day…quoted the inaccuracy you had identified." Oh well, that's ok then -- no-one quoted the inaccuracy so it's ok to leave everyone sent the press release, including MEPs involved in making a decision on the directive, with false information.

The BSA failed to update the press release on their Web site until Wednesday evening, although they apologised for this. "For reasons we don't immediately understand, the press release, as you pointed out, was not corrected on the Internet – despite our explicit instructions that it should be," said the BSA's Mingorance.

The BSA admitted in a later email that in fact only 10 percent of European SMEs hold patents -- the others being primarily Japanese and US SMEs. "The implication of the correction we made is that 20 [percent] of the CIIs are held by SMEs and about half of these, or 10 percent, are European SMEs." So, quite a big mistake then.

So, maybe it was just the BSA's interpretation of the study. Maybe they're just not that hot at maths? Yet even the study methodology is flawed, and biased towards overcounting SMEs. The study took a list of what they considered software patents, got a list of non-SMEs and deleted every patent application from the list that had been filed by a non-SME. This method results in an "error of inclusion" i.e. non-SMEs will be left in the list, which the author admits. It appears much less likely that SMEs will be deleted from the list by mistake, but despite the obvious skew that the methodology has on the results, the author has not adjusted the data suitably.

There also seems to have been a considerable lack of due diligence on the author's part. He claims in his methodology that he also eliminated academic institutions and governmental organisations from the list, but in an hour spent examining the author's list of UK "SMEs" (the SME dataset was provided by the BSA), I found five universities: Aston University, University of Oxford, Royal Holloway University of London, Imperial College, and Royal Holloway and Bedford New College. A simple search on the word "university" and "college" would have highlighted those ones.

The author defined SMEs as those employing fewer than 250 persons and with an annual turnover not exceeding 50 million Euro, although in a footnote points out that due to the algorithm used the companies may have "considerably more than 250 employees". Again, a methodology that is flawed towards overcounting SMEs. In the hour spent looking through the list, I found eight companies that can certainly not be defined as SMEs:

1) ARM - the UK chip maker that in 2004 reported revenues of £152.9m
2) BTG International - a medical company that in 2004 reported revenues of £42.7m
3) DPT Global - a company that is part of the Carlyle Group, which describes itself on its Web site as one of the world’s largest private equity firms, with "more than $29.6bn" under management, a team of "nearly 300 investment professionals". The company also claims to have "committed more than $1bn of its own capital to its funds."
4) Macrovision Europe – this is part of the digital media company Macrovision, which had reported a net revenue of $182.1m in 2004
5) Infotech Software – the Indian outsourcing company, which employs 2000 people and had operating revenues of 1919.6m Rupees (36.4m Euros) Rupees from April to December 2004
6) NTL Group – the telecommunications giant that reported an operating revenue of £2074m over the last year
7) Revolution Entertainment Systems – this is part of the Leisure Link Group, which claims to be the "UK’s largest provider of public space gaming and entertainment machines" and employs "nearly 2,500 people"
8) Energy Pool Funds Administration -- this is part of National Grid, which owns and operates the electricity system in England and Wales. The company is owned by National Grid Transco, which is one of the UK's FTSE 100 companies, with a turnover of £8.5bn for the last financial year.

So, the author of the study somehow managed to miss a FTSE 100 company, a telecommunications giant and one of the biggest outsourcing companies in India.

There are probably many more big outfits there, but the dataset makes it arbitrarily hard to find out. For some reason it misses out the majority of company names -- only 193 out of the 606 UK listed patents indicate the company name. I looked up a few of the patents that had the company name left blank on the Esp@cenet Web site, assuming they were filed by an individual inventor, but actually found there was a company name listed. If you therefore take the simplistic approach of extrapolating the 13 non-SMEs found in a quick search through 193 company names, 6.7 percent of the "SMEs" are actually non-SMEs. That's one big error.

When I pointed out the five universities and eight non-SMEs found in the study, the BSA spokesperson responded. "Congratulations! With some investment of time and effort you have discovered what the BSA and the author of the study already knew and freely and clearly disclosed – that it is extraordinarily difficult for someone to screen out non-SMEs in a systematic way for the purposes of such a study."

Yes, but then this begs the question, if the study methodology is that inaccurate, why have you publicised the study? Or, at least, why didn't you adjust the data accordingly? Why have you used figures like SMEs account for "more than 20 percent" of patents in your press release, when in fact it would be more accurate to say "fewer than 20 percent" due to the skew in the data? Why haven't you mentioned the glaring problems with the study in the press release? How can a mere journalist find so much wrong in a couple of hours, when the people commissioning, preparing and publishing the study had so much more by way of resources and expertise?

It is difficult to conclude other than the BSA is happy to allow these errors because they fit the organisation's argument. It is also hard to conclude other than that the BSA studies really should be independently corroborated if they are to be trusted. As a lobbying organisation, it is doing itself a lot of damage by its attitudes and actions, and it risks damaging the very industry it claims to support by parlaying its inaccuracies into national policies and international law.