Status Report: How we evaluate and rate technologies, and why

My fellow tech journalists and I (they more frequently) are often asked to render an opinion on the state or value of a technology, a product, or a vendor. We'll rate products or trends, perhaps with a score like 9.8 or C+; or we'll sort everything out on a rather arbitrary Top 10 list, where it's impossible to discern what made one candidate rank #3 and another one only #7. When we do this right, we make several small judgments and derive a consistent methodology for combining them together. Otherwise, we let the medium of the presentation itself obscure any substantive insights as to the source of our own, self-declared wisdom.

True insight on any topic is only achievable from multiple perspectives, some of which run in contrary directions to one another. It's why a feature article with three or five or seven guest voices tends to have greater value than a monologue, or any single op-ed page from a metropolitan newspaper tends to be more useful than single hour of a cable news channel. If an article must derive from a single voice, its author should at least make an effort to evaluate the topic from more than one vantage point.

Laudable objectives often run contrary to one another. It's hard to run a business based around intellectual property you don't own, try to create unique value for customers, and at the same time yield appreciable benefits to shareholders. Try it sometime.

We talk about disruption, we point to the presence of technology, and like amateur sleuths we attribute one to the other. But to be influential requires having demonstrated capabilities to change people's lives. Such capabilities are not often cumulatively applicable. Too many good ideas are put forth by people or companies that lack the ingenuity or willpower to build business models around them. Some capabilities run contrary to one another in varying degrees. Achieving progress — or, in the negative, enabling disruption to become destructive — requires overcoming the natural tendency of all things that counteract one another, towards inertia. Without progress, disruption is just noise.

Methodology

The goal of Status Report will be to ascertain whether a technology is genuinely influential under its own power, or whether it has succumbed to being simply a tool. 

For each technology we evaluate, we'll examine 10 categories. We'll render simple, qualitative judgments about their levels of influence in these categories, positive and negative. Each score will be the net difference: a whole number from +10 (maximum progress) to -10 (maximum regression). We're interested in the following:

• Stakeholder empowerment — A measure of how much a technology directly benefits the people who produce it or who have invested in it. This gets to the issue of whether a product or service enriches the lives of the people responsible for it.

• Competitive advantage — The extent to which a technology is capable of giving the people who produce it a business advantage over their rivals in a marketplace; or, in the negative, the extent to which it hamstrings producers from being able to compete. When a manufacturer finds itself producing a commodity that it cannot distinguish from its competitors' alternatives, this score goes negative.

• Business sustainability — An estimate of whether a business may continue to produce this technology, in whatever form it may take, into the future.

• Evolutionary incentive — The extent to which the presence of a technology in a market inspires its practitioners to improve or upgrade that technology, in a quest for greater value.

• Market enablement — An evaluation of whether the presence of a technology product or service in a market, gives rise on its own to healthy and vigorous competition among multiple participants in that market. This isn't about giving one or another player a special advantage. Rather, we evaluate whether any component in the market may benefit simply from the technology's existence. In the negative, this measures how much of a burden a technology may be to a market whose contributors must continue producing it to stay relevant.

• Customer value — A raw estimate of how likely customers of a technology will perceive beneficial value to their lives and/or work; or, in the negative, how much customers may perceive it to be a burden to them (as may be the case with set-top boxes, or on-site power generators).

• Economic contribution — The extent to which a technology may benefit the well-being of countries and/or trading blocs, typically by means of participation in their gross domestic product.

• Societal integration — The measure to which a technology product or service is becoming, or has become, an indispensable, irreplaceable part of everyday people's lives. This can happen when a product or service blends so seamlessly within its environment that it isn't much noticed. In the negative, this could be a measure of a technology's failure to be understood or embraced by its users at large.

• Cultural advancement — This gets to the heart of the question of whether the human species, and the social structures that bind it together, are made better by the technology's existence. Is humanity made better or, in the negative, imperiled?

• Ecosystemic enablement — The quality of a technology's ability to equip and inspire all of its practitioners to make genuine efforts to its own continued existence and well-being. This is very different from an industrial or an economic contribution; this refers to a technology's own innate ability to bring people together in the selfless interest of sustaining it.

Notice how these categories have been sorted, beginning with arguably the most self-serving virtue, and proceeding to the most altruistic. There have been arguments made over the years that an unwavering focus on producing stakeholder or shareholder value works against the interests of a technology's user base, preventing them from establishing a healthy community. That's why we set these categories almost, but not quite, opposite each other.

Each category's net score will be plotted as a vector, on a chart that represents what we'll call the technology's sphere of influence. Here, our metaphorical pendulum may be swung in any direction. All the progressive directions reside on the upper half of this chart, all regressive directions on the lower half. The extent to which a technology overcomes inertia is our measure of its influence.

The final score

If a technology is truly influential — capable of effectuating some degree of change, whether it's good or bad, or just "disruptive" — the combination of all the forces it exhibits needs to be capable of budging the pendulum in some direction. Progress should be something we can measure, or at least estimate.

When we give each of these ten forces of influence its own vector on a scatter chart, the result is a series of points, like holes on a dartboard. The geometric average of their locations may lie anywhere on the chart, but most likely toward the center. The extent to which the final score budges from the center, constitutes the technology's final influence score.

Without much to measure the initial scores against, it may be unwise for us at the outset to draw too sweeping a conclusion that pertains to the information technology industry, or to our society, as a whole. In all frankness, the real value of Status Report may come some months down the road, when we revisit these same spheres of influence and determine where their trend lines lay.

For now, our influence score is not a measure of disruptive force or even global interest, but of contributive momentum. It's how much a technology can change the world and the society it inhabits, for better or worse.  Because our result also includes an angle (equating to benefit on the top half of the chart, detriment on the bottom), we can actually say whether that change is positive [+] or negative [-]. And because we arranged self-serving, or "foundational," influences on the right side and selfless or "altruistic" influences on the left, we can render a general estimate of whether a technology benefits its practitioners more, or its users.

In creating this system, we experimented with evaluating the state of the World Wide Web as of September 12, 2001 — the day after one of the darkest landmark days in American history.  Our aim was to compare our estimate of the Web's influence at a critical time, with the influence of social media the day after the US Capitol insurrection.  That social media evaluation resulted in an influence score of +0.89. By comparison, our Web 2001 score yielded a +1.97 — over twice as influential for the world at large. Data points fanned out nicely in the upper left quadrant, toward altruistic progress.

On an ordinary 10-point scale, a score of 2 would sound like failure.  But ours is not a success/failure scale. It's an evaluation of how, amid all the conflicts a world-changing technology must face, it ekes out a way towards progress.  We know, from the benefit of hindsight, how the Web was influential and also how it wasn't.  As transformative as the Web was, it never established a stable business model for Web publishing, applicable to multiple enterprises and organizations of all varieties. We can see how the foundational scores didn't break through as far as +3. This is a confirmation of the history we experienced.

Science demonstrates for us quite conclusively that everything, including evolution itself, evolves. What's more, it does so in infuriatingly small increments. We want to show big steps and mighty milestones, and we end up with scores that are fractions, and explanations that are contentious and, certainly in my case, long-winded.

If we're going to do this seriously — if we're going to extrapolate the meaning behind what we're talking about, rather than making wild gestures and hyperbolic claims — then we need to do this slowly, meticulously, painstakingly. In the final analysis, the weight of our opinions will only be measurable and influential once they have emerged into the light from being challenged, scrutinized, and being held in check.