Join two executives from the IT organization at Sprint, Joyce Rainey, Program Manager of Enterprise Services, and John Felton, Director of Applications Development and Operations, for a discussion moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]
Felton: The problem that we had originally had, as any large organization has, were many applications, many of them custom built, many of them purchased applications that now are so customized that the vendor doesn’t even know what to do with it anymore.
We grew those over a long period of time. We were trying, as a way to stabilize, to get it into a centralized, single point of truth and quit the duplication or the redundancy that we built into all these applications.
The goal, as we set forth about a year-and-a-half ago, was to implement the ecosystem that HP provided, the five toolsets that followed our ITIL processes that we wanted to do. The key was that they were integrated to share information, and we'd be able to take down these customized applications and then have one ecosystem to manage our environment with. That's what we've done over the last 14 months.
[At Sprint] there are thousands of outlets, retail stores. We have our third-party customers as well, like Best Buy and RadioShack. We have about 12,000 servers, about five petabytes of storage. We serve about 39,000 customers internally at Sprint.
We host all that information to make sure that we process about a million change records a month. That information that we're capturing are configuration items (CIs). The actual content that goes in the system was, at one point, in the 24 million range. We dialed that back a little bit, because we were collecting a little too much information.
We have about 1,300 applications that were internally built. Many of those are hosted on other external vendor products that we've customized and put into Sprint. And, we have about 64,000 desktops. So, there is a lot going on in this environment. It's moving constantly and that goes back to a lot of the reasons why, if we didn’t put this in quickly, they'd pass us by.
Making it easier
Rainey: We had too many of the same. We had to make it easier for our internal support teams. We had to made it easier for our customers. We had to lessen the impacts on maintenance and cost. Simplification was the key of the entire journey.
Felton: We had to concentrate on not only making sure that the applications base wasn't duplicated, but also the data. The data is where we ended up having issues. One person's copy may not be as accurate as another person's copy, and then what we ended up spending an enormous amount of time saying whose was right.
What we did was provide one single point of truth, one copy of the truth. Instead of everybody being hidden from the data, we allowed everybody to see it all. They may not be able to manipulate it and they may not be able to change it, but everybody could have visibility to the same amount of information. We were hoping they would stop trying to have their own version of it.
Our biggest culture problem was that everybody wanted to put their arms around their little piece, their little view. At the end of the day, having one view that is customized, where you can see what you want to see, but still keeping the content within a single system, really helped us.
Having one view that is customized, where you can see what you want to see, but still keeping the content within a single system, really helped us.
It's the data that supports the application. It's the servers that host the applications. It's the third-party applications that deliver the web experience, the database experience, the back-end experience. It's the ability for us to associate fixed agents to that particular information, so that when I am calling out the fixed agent for an alarm, I'm getting the right person online first, versus having a variety of individuals coming on over time.
Rainey: The HP Excellence Award [at Discover 2011] was a very big milestone for everyone to remind us that it was well worth it, the time that was spent, the energy that was spent. I'm very glad that HP and our customers have been able to recognize that. I am very proud, very proud of Sprint. I'm very proud of the team. I'm very proud of the executive support that we received throughout this journey.
Felton: I'm also very proud of the team, as well, and we also won the CIO 100 Award. So, we’ve been able to take the same platform and the same kind of journey and show a much larger audience that it really was worth it. I think that’s pretty cool.
Importance of speed
What I might do differently is spread it out a little more, do smaller increments of implementation, versus all at one time. Don’t do the Big Bang Theory. Put in BSM, but always know that it's going to integrate with SM, and SM is going to integrate with CMS, and CMS is going to integrate with AM.
Then, build that plan, so that you integrate them. You get your customers involved in that particular application, and then when you go at the very end and put SM in, this the front door. They’re already familiar with what you’ve already done. That is something we probably didn’t do as well as we could have. It was more of a Big Bang approach. You put it in and you go.
But, at the end of the day, don’t be afraid to re-look at the processes. Don’t necessarily assume that you’re going to copy what you did today. Don’t assume that that is the best way to do it. Always ask the question, what business value does it address for your corporation? If you do that over, and over, and over, individuals will quit asking, because if you ask, these platforms are very flexible.
You can do anything. But when you get them so customized that the vendor can't even help you, then every upgrade is painful, every movement that you make is painful. What we’ve done has given us the flexibility to clean up a lot of stuff that was left over from years ago, an approach that may have not been the best solution, and given us an avenue to now extend and subtract without putting a huge investment in place.
What we’ve done has given us the flexibility to clean up a lot of stuff that was left over from years ago.
One other thing is that we had a really good idea of, "This is our business. Run it that way. You are a part of Sprint." We try to say, "We’re going to make investments that also benefit us, but don’t do them just to do them, because in this space as you look out on that floor and see all the techno wizards that are out there, shiny objects are pretty cool, but there are a lot of shiny objects."
We wanted to make sure that the shiny object we produced is something that was long lasting and gave value back to the company for a long period of time, not just a quick introduction.
Rainey: We continued to work on it. Adoption is a big key in any transformation project. One of the things that we had to definitely look at was making sure that facts can prove to people that their business requirements were either valid or invalid. That way we stop the argument of what do I want, versus what do I need?
A lot of education
We really had a lot of communication, a lot of education along the way. We continue to educate people about why we do this and why we're doing it this way. We engage them in the process by making them part of the decision-making, versus just allowing the tools to dictate whether you can do it.
With the tools, you can do whatever you want. However, you want to customize the product, but should we and for what purpose? So, we had to introduce a lot of education along the way to make sure folks understood why we were going down this path.
Felton: We implemented in 12 months. It was 14 months to get the future enhancements of the data quality and all the things we're working on right now. But as to the tipping point, I think the economy had a lot to do with it, the environment that was going on at the time.
You had a reduction in staff. You had downsizing of companies. It made it harder for individuals, to Joyce's point, to protect an application that really had no business value. It might have a lot of value to them, and in their little piece of the world it probably was very valuable, but how did it drive the overall organization?
The economy in any kind of transformational program is a key factor for investing in these kind of products. You're going to make sure that if you're introducing something it's because you're going to add value.
[Sprint CEO] Dan Hesse did a great job in coming in and putting us on a path of making sure that we're fiscally responsible. How are we improving our customer expectations and how are we moving in this direction continuously, so that our customers come to us because we're best provider there could be? And our systems on the back end needed to go that way.
So, to Joyce's point, when you brought them in, you asked "Does this help that goal?" A lot of times, no. And, they were willing to give a little bit up. We said, "You're going to have to give a little bit up because this is not a copy/paste exercise. This is an out-of-the-box solution. We want to keep it that way as much as possible, and we'll make modifications, when we need to to support the business." And, we've done that.
Rainey: It's important to recognize that data is data, but you really derive information to drive decision making. For us, the ability for executives to know how many assets they really have out there, for them to concentrate their initiatives for the future based on that information, became the reason we needed our data quality to really be good.
It's important to recognize that data is data, but you really derive information to drive decision making.
So, every time that somebody asked John why he went after this product suite, it was because of the integration. We wanted to make sure that the products can share the same information across them all. That way, we can hold truth through that single source of information.
Felton: We started with [IT] asset management. Asset management was really the key for us to understand assets and software, and how much cost was involved. Then we associated that to Universal Configuration Management Database (UCMDB). How do we discover things in our environment? How many servers are there, how many desktops are there, where they at, how do I associate them?
Then we looked at Business Service Management (BSM), which was the monitoring side. How do I monitor these critical apps and alarm them correctly? How do I look up the information and get the right fix agents out there and target it, versus calling out the soccer team, as I always say? Then, we followed that up with Release Control, which is a way for our change team to manage and see that information, as it goes through.
The final component, which was the most important, the last one we rolled out, was Service Manager (SM), which is the front door for everybody. We focus everybody on that front door, and then they can spin off of that front door by going into the other individual or underlying processes to actually do the work that they focus on.
Felton: For just BSM in itself, I'm very proud of our team. We had [another product] in 2009. We went to Business Availability Center (BAC) January 2010. HP said they had this new thing called BSM 9. Would we take it? We said sure, and we implemented it in March of that year. We took three upgrades in less than five months.
I give a lot of credit to that team. They did it on their own. There were three of them. No professional services help and no support whatsoever. They did it on their own, and I think that’s pretty interesting how they did that. We also did the same thing with UCMDB. We are on the 8x platform, about halfway deployed, and HP said they'd like us to go to 9x, and so we turned the corner and we said sure.
We did those things because of the web experience. Very few people on my team would tell you that they were satisfied with the old web experience. I know some people were, and that’s great. But, in our environment, as big as it is and as many access points as we had, we had to make sure that was rock-solid.
And, 9x for all those versions, seemed to be the best web experience we could have, and it was very similar, if I'm looking at BSM. Drop-downs and the menus, of course, are all different, but the flow and the layout is exactly the same as SM, and SM is exactly the same as CMS.
We got a nice transition between the applications that made everything smooth for the customer, and the ability for them to consume it better.
We got a nice transition between the applications that made everything smooth for the customer, and the ability for them to consume it better. I'll go so far as to say that a lot of my executive team actually log into BSM now. That would have never happened in the past. They actually go look up events that happen to our applications and see what's going on, and that’s all because we felt like that platform had the best GUI experience.
Rainey: And, if you get your CEOs and your VPs and your directors consuming and leveraging the products, you get the doers, you get the application managers, you get the fix agents, you get the helpdesk team, because they start believing that the data is good enough for decision making at that level of executive support.
Felton: We wanted reduction in our [problem resolution time] by 20 percent. Does that really mean you get a reduction? No, it means you get out there, you fix it faster, and the end-user doesn’t see it. By me focusing on that and getting individuals to go out there, and maybe more proactively understanding what's going on, we can get changes and fixes in before there was a real issue. We’re driving towards that. Do we have that exact number? Maybe not, but that’s the goal and that’s what we continue to drive for.
Additionally the costs are huge, having 35 redundant systems. We removed a lot of maintenance dollars from Sprint, a lot of overhead. A lot of project costs sometimes are not necessarily tangible, because everybody is working on multiple projects all at one time.
But, if I've got to update five systems, it's a lot different if I update one, and make it simpler on my team. My team comprised about 11 folks, and they were managing all those apps before. Now, they're managing five. It’s a lot simpler for them. It's a lot easier for them. We’re making better decisions, and we make better changes.
We’re hoping that by having it that way, all of the infrastructure stability goes up, because we’re focused. To Joyce’s point, the executive team pays attention, managers pay attention, everybody sees the value that if I just watch what this thing is doing, it might tell me before there is a customer call. That is always our goal. I don’t want a customer calling my CIO. I want the customer to call my CIO and for him to reply, "Yes, we know, and we’re going to fix that as fast as we can."
Six years ago that help desk had 400 people. As of today it has 44. The reason it does is that we bypass making calls. I don’t want you to call a fix agent to type a ticket to get you engaged. We came up with a process called "Click It." Click It is a way for you to do online self-service.
If I'm having an Exchange problem, an Outlook problem, or an issue with some application, I can go in and open a ticket, instead of it being transferred to the help desk, who then transfers it to the fix agent. We go directly to the fix agent.
We’re getting you closely engaged, hoping that we can get your fix time faster. We can actually get them talking to you quicker. By having this new GUI interface it streamlined it through a lot of wizards that we can implement. Instead of me having seven forms that are all about access, maybe now I have one. Now, there is a drop-down menu that tells me what application I want it for. That continuous improvement is what we’re after, and I think we’ve now got the tools in place to go make that easy for us.