Senate IT oversight bill: Detailed analysis
Senator Thomas Carper's [D-DE] proposed IT oversight legislation (S.920) is a well-intentioned and important step toward reducing wasteful spending on mismanaged IT projects. However, the bill has substantial weaknesses that will likely compromise its effectiveness and perhaps create even more bureaucratic waste.
Following my previous blog post on S.920, a Senate staffer working on the legislation contacted me with a request to comment on the bill. Given the critical nature of this issue, potentially involving billions of dollars, I welcomed the opportunity to do so. This post summarizes my initial comments, and serves as a basis for initial discussion with that Senate staff member.
This post is unusual for several reasons:
- To my knowledge, the Senate does not often ask a blogger to serve as an expert commenter on pending legislation. In this case, my background in IT failure analysis and prevention is the defining factor.
- This post may be confusing without also studying actual full text of the bill. Usually, I attempt to make posts standalone from the original sources. Not so in this case.
- This post includes lots of detail and few overview perspectives, making it hard to read. In a sense, this post is a brain dump that I thought some readers might find interesting. My thoughts on this bill are still in formation, so this post truly remains in progress.
- This post is just not well-written. Although it pains me to publish notes rather than polished text, that's the best I can do now, given the time constraints of my own busy schedule.
Despite its shortcomings, I support this initiative and do not accept the argument that the bill does nothing except create additional bureaucratic overhead. As my comments below make clear, the bill will not accomplish its objectives without further refinement. However, even in its present state, the legislation raises the profile of important issues around IT project failure and waste.
THE PROJECT FAILURES ANALYSIS
The remainder of this post presents comment on each section of the bill.
Section 1: Short Title
This section simply states the name of the bill.
Section 2: Findings.
Presents background information and describes the rationale for this bill.
This bill addresses execution rather than planning, so the oversight measures only activate after appropriate parties identify a project as failing. Since planning takes place upstream, before any of the conditions addressed in this bill arise, a major gap exists. This shortcoming is significant.
Along the same lines, it also does not seek to validate whether the original business case of the project makes sense. This is important because some projects are well-run, from a mechanical or project management standpoint, but actually serve little useful purpose or provide business value that is commensurate with cost.
In general, the bill does not address management issues, per se. Rather, it describes a set of conditions that trigger certain intervention activities that must be performed when a project reaches defined threshold limits.
Section 3: Real-time Transparency of IT Investment Projects.
Requires the government to create a publicly accessible website showing status and progress relative to plan for government IT projects.
Although perhaps it seems like an obvious point, the bill does not specify the audience for this website. A general public audience requires simpler language and clearer presentation than an audience of professional project managers. For example, the bill requires the website to display data relate to earned value management, a measurement technique commonly used on government projects, but which is of little use to a general audience. Unless the website fully considers different audience requirements, it risks not being completely useful to either group.
It is also unclear how the website specified by this is different from an existing site, called VUE-IT, which is run by the Office of Management and Budget. VUE-IT includes a check box to highlight high risk projects and includes drill-down information. Although transparency is important, duplicating websites is wasteful and also confusing to users.
The bill demands updating the website on a quarterly basis, which is not real-time by any reasonable definition. Ideally, the website should take automatic feeds from the project management software used on a daily basis, which would create real-time insight into project status.
Section 4: IT Investment Projects.
Establishes detailed definitions and reporting requirements for projects. This section implies the CIO does not have full oversight authority today. Therefore, I wonder what are the limits of CIO oversight today.
The bill seems to primarily represent a demand for clearer reporting, which may do little aside from adding further burden to already overworked CIOs. The increased reporting has value only to the extent someone is available to read and act upon information presented in the reports.
The bill discusses specifies reporting criteria that triggers remedial activities, based on information drawn from project management systems. This value of this project data depends on ensuring that staff correctly updates those systems with accurate information throughout the life of the project. There are three potential barriers to accomplishing this goal: 1. Project participants and management may incorrectly understand or interpret project status; 2. They may not spending the time needed to correctly enter data into the system; and 3. The entire scheme is subject to manipulation if projects participants intentionally try to "game" the system with the data they enter.
In the first two categories, errors can arise from inexperience, poor source information, or lack of time to input status data. The third category involves intentional manipulation, which is substantively different from ordinary lack of information. Of course, after the fact it can sometimes be difficult to determine whether wrong information is rooted in inexperience or a deliberate attempt to manipulate. This fact alone makes it difficult to specify penalties.
This section of the bill is detailed, and here are additional points:
- The bill relies on earned-value management (EVM) as a measurement tool for determining project status. However, the importance of initial valuation presents a risk to having meaningful data downstream.
- The bill specifies specific criteria and variances that trigger project oversight steps. However, the logic behind these numbers is not made clear.
- The bill does not differentiate between projects without a useful business case and valuable projects that are constructed (or executed) poorly. The bill only covers project remediation after in-progress failures have already occurred. As a result, it is entirely likely these interventions may keep alive projects based on a completely flawed business case. This issue is a key risk for the bill.
The final bullet is a critical point, because the bill does not evaluate the upstream business case separately from project execution. If the business case is sound and sufficiently valuable, management of course should evaluate why the project has run over-budget or is late, to get project back on track.
On the other hand, if the business case is not sound, the project should be killed and not continued. Quantifying these issues into a concrete set of rules is difficult, because there are many exceptions and judgments required in making these evaluations. Complete evaluations involve going back to reexamine the project starting point, business case, and value.
Section 11319: Acquisition and Development
Acquisition and requirements analysis clearly play a significant role in downstream project success or failure. However, acquisition and project execution are so different that I wonder if they should be de-coupled from an oversight perspective. Each of these areas is substantial and I am concerned that bundling them together does not properly give each one appropriate weight or depth.
Line 23 of the bill states “(4) a process to ensure that the agency implements and adheres to established processes.” This language leads directly to bureaucratic focus on process without requiring tangible results. In addition, the people involved on the ground managing a project are often neither qualified nor have time to create new processes. The bill should offer clearer guidance on this point.
The bill describes a variety of metrics that can be used as a dashboard for reporting. This approach is excellent, but it's important to remember there are multiple audiences (including senior management, project managers, project personnel, and the general public), each of which has its own requirements, complicating creation of the dashboard. Some central group should create a common dashboard template used across all agencies. The template should specify analytic criteria, data format, and also user interface appropriate to the multiple audiences.
Section 5: Tiger Team
This section of the bill allows agencies to share top talent to fix failing projects. However, it also raises the question whether staff at these agencies has the experience to create and execute complex projects successfully. In some cases, failures may be inevitable because properly skilled resources are not available during the project definition phase and during downstream delivery phases.
Lack of accurate or complete information is one of the criteria triggering deployment of the Tiger Team. While certainly a warning flag, late reporting alone does not necessarily mean a project is experiencing problems (aside from not updating the data, of course).
As I said earlier, the bill creates additional layers of data, requiring skilled staff to analyze. Without that staff to make decisions based on the data, the entire monitoring effort becomes pointless.
Also reiterating earlier points, the Tiger team is strictly reactive to ongoing project management issues. It does not address upstream problems related to the basic business case on which the project was established in the first place. That’s fine, but the limitation must be recognized.
Section 6. Awards for Personnel for Excellence
Recognizes that project staff doing a good job should be recognized.