Persistence is often needed when collecting metrics

The dilemma
Matt’s project to upgrade all of our application databases to the newer version was winding down. Generally, the project was considered successful, having met most of its commitments in terms of deliverables, budget, and deadline. Matt’s manager was now asking for some quantitative metrics on the work the project team members did and how successful they were.

“Tom, I sat down with my team and brainstormed a list of potential metrics that we could give to my boss,” Matt said. “However, that turned out to be the easy part. Collecting the metrics is more difficult.”

“I won’t disagree,” I responded. “Are you able to collect some basic internal project measures?”

“Yes, the metrics that are within our control are not too bad,” Matt explained. “For example, we want to count the actual number of databases and tables that we migrated to the new release. We also want to report the total number of hours we spent and the total duration. The problem is when we need to get information from other people.”

I had a feeling I knew where Matt was having problems, but I let him continue.

“It is one thing to know whether we met our budget and timeline,” Matt replied. “However, my manager also wants to know how smoothly the conversion went from our clients' perspective. In our case, the clients are the application developers. We have sent surveys to the development and support teams and have received very little feedback. I think we have only six surveys returned out of around 50 that we sent out.”

“What kinds of questions are you asking?” I asked.

“The survey is simple,” Matt said with some frustration in his voice. “We’re only looking for their perception of how smoothly the migration occurred and whether they encountered any problems after the migration was completed. It should only take them a few minutes to respond.”

I looked at the short e-mail survey that Matt was sending out. It asked for answers to three questions, on a satisfaction scale of one through five:

• How satisfied were you with the time it took us to migrate your databases?
• How satisfied were you in our ability to migrate your databases with as little impact as possible to your job?
• How satisfied were you that your databases were converted correctly and accurately?

“You’re right; this survey shouldn’t require much time for a response. However, I think that it is not the time spent that is the problem,” I noted. “Our company still has a general lack of knowledge and appreciation of the value of metrics.”

Mentor advice
On the surface, I think people can easily understand the value of gathering metrics. The best way to really understand and improve your IT processes is to define metrics that indicate how the process is performed, collect them, and analyze the results. As you improve your processes, the quantitative results of your success should show up in the metrics.

Matt is discovering a problem that you may encounter in your company. Since many IT processes are service-related, you must collect metrics from outside your group to get a balanced picture of how you are performing.

Although Matt is motivated to collect the metric information, very few others are. My experience is that establishing a metrics culture is very difficult, even if you undertake an organizationwide initiative. It is more difficult when you do it on an individual project team basis.

All that being said, giving up is not the answer, either. Instead, my advice to Matt is to be diligent about gathering metrics that are within his control and then continue to try hard to gather enough feedback from the outside so that the results will be meaningful. This includes sending out the survey again to the people that didn’t respond and perhaps calling the key people in some of the areas where his team performed a large migration.

Matt doesn’t need to get a 100 percent response rate for the results to be meaningful. With most surveys, a return rate between 40 and 50 percent would be considered more than adequate.

Another option is to ask his manager for assistance. Since he asked Matt to gather feedback, perhaps his manager can work with his peer managers to make it a higher priority for the developers to respond. Matt’s manager can call or send an e-mail to the development managers letting them know that Matt is collecting this information and asking the development managers for their assistance in collecting the feedback.

Individual project managers will probably continue to struggle with collecting survey metrics until the organization becomes savvier about collecting metrics and makes this more of a core competency. If enough project managers start to collect and report metrics, senior management might see the value and push the gathering of metrics as an organizationwide initiative.