The future of APM: more simplicity, fewer silos, and real answers

"There's an app for that."

There are few people who aren't familiar with this Apple catchphrase, which suggests that there is a solution to every problem in a self-contained application.

That is precisely the problem for businesses that are shifting more and more revenue onto applications -- whether web, mobile, streaming or cloud. When there is that much money at stake, sluggishness and crashes are not acceptable. Not if you want to keep your customers, anyway.

Enter application performance management. The practice is several years old, no doubt. But this year, it's taking off as enterprises in all industries move critical, customer-facing processes to the cloud. It's no understatement to say that I receive an e-mail about it every week.

John Van Siclen has seen the space evolve. The former chief executive of DynaTrace, he's now general manager at Compuware APM, the applications arm of the Detroit-based computing conglomerate.

To get a sense of where we are and where we're headed in APM, I gave him a call.

ZD: Let's start at the beginning. Where did APM really come from?

JVS: It was really started by Mercury [Interactive] and Wily [Technology] and Precise [Software] -- these are the pioneers of this market. You could even almost go back to Tivoli [Systems] as the grandfather, back when the network was performance. It's been around for a pretty long time.

People lost track of the companies -- they were all bought in 2005 [by companies such as] HP, IBM, Symantec. The first generation of APM got consolidated together into these big suite plays. The way the market moves, it sort of has this fragmentation when there's a disruption, and then it starts to consolidate back again. This market went through that contraction and consolidation back in the 2000s. That first generation was really built for Global 1000 companies: trading applications, credit card applications, brick and mortar e-commerce, transportation, hospitality. The focus at that time was more around an insurance policy for monitoring the applications that ran in your datacenter.

At first, the model actually mimicked the way people were running their hardware. Green, yellow, red -- they put these meters in the virtual machines to try to check the health of the componentry of these applications. It didn't tell you whether transactions were completed, it only told you whether these components were "green." On the hardware side, if there's a problem, you know which server to go to. Software's different -- it's zeros and ones. A symptom showing up in one place doesn't mean you can find the root cause. It's like the human body.

So that was the first generation, which was consolidated. What's been happening since that moment of time has been a series of things that I call "the perfect storm of complexity." On the production side, we started to virtualize our environments; on the application architecture side, we went from componentized applications to composite applications with lots of services and pieces of code. From a development standpoint, we've gone from waterfall to agile -- we're making changes faster than ever before, once a week instead of once a year. We've had this explosion, and with that, these old systems that were built for a whole different world for a different reason? Environments can't stand up.

The notion and promise of APM is more important than ever because the applications are more important than ever -- it's how we engage with customers, the supply chain, everything. Complexity is exploding. Change is exploding. And user expectations have come to the forefront -- the "Google effect," which has set the expectation that even complex queries should take less than a second. That puts a lot of pressure on everyone.

Look at mobile: it's exploding the complexity for IT. Browsers -- think about how you have to make sure your application works on all these browsers. At the center of all this disruption, which is significant, there's this opportunity for new companies. Compuware ended up buying a couple companies in Boston -- Gomez, DynaTrace -- with new approaches to APM. Compuware, although the name has been around forever, decided a few years ago to double down on APM, with its high disruption [potential].

With this change comes a whole new set of requirements, and gives rise to opportunity for new players.

ZD: One of the most interesting things about APM is how each business uses it. The applications -- and implications of them -- vary greatly by industry.

JVS: I spent a little time at a major retailer in the upper Midwest. I was wondering why they were buying our APM solution at such a large quantity and at a fast rate. I sat down with the CIO and she said, "It's because our world changed."

Now, they track every item on every palette, and some have perishable goods on them. The minute they went to perishable goods, there's a whole series of FDA requirements to meet. You can't have it sit there for more than five minutes. So all the pick and packing has to be automated. The organization of the trucks has to be perfectly timed, so the right stuff is on the right trucks to the right stores. If there's an application problem, everything stops. You can't get humans in there. The domino effect is huge.

Another one is Cars.com, for the [NFL's] Super Bowl. The ad where the little heads pop out of people and start talking? That first aired for the 2012 game, and these guys were scrambling because they knew their traffic was going to spike, but they didn't know much. In 30 days, they went through a cleanup of their application that included dozens of improvements. The traffic scaled 1,800 percent.

The new APM isn't just an insurance policy, it's actually a way of truly optimizing the performance of apps as well. The value has extended beyond monitoring for issues to extremely rapid troubleshooting as well optimization of your application.

ZD: OK, so what's driving APM forward? Any particular industries or uses?

JVS: There are several industries pushing the envelope, but it is pretty horizontal. No question, financial services. They have a lot at stake: every transaction matters to them, and there's a lot of regulation there, and they're conditioned that way -- to sit at the front edge of technology.

Interestingly, e-commerce. Not just Zappos, but Sears and Macy's and brick-and-mortar guys shifting a large part of their business to e-commerce. That's a big deal.

The other one is hospitality and travel. You might ask, isn't that e-commerce? Well, yes. But so is financial services. It's the same thing as banking: if you want to book a flight, you just go online now, you don't call. They've gone from 10 percent to 90 percent online transactions in something like 10 or 12 years. They've all done it to try to be more profitable, and engage their customers.

We're doing more and more government stuff -- there are an awful lot of online applications, and they're getting dinged if they don't work well. Plenty in manufacturing, too -- engagement with partners and supply chains. And insurance, in which a fair amount of their ability to drive volume and loyalty is ensuring their applications' [stability].

If you're not using APM, you're really working at a serious disadvantage.

ZD: One of my favorite things to ask at this point in the conversation is: then why doesn't everyone use it?

 JVS: My greatest challenge running this business is dealing with the notion of the old generation [of APM] -- it's deeply ingrained in people's heads that they've spent lots of money and gotten little value. The new generation solves many of the old issues and comes at it from a totally different point of view, and it actually works.

If you were to talk to somebody that doesn't have APM, they don't say "What is it?", they say, "Oh, it's expensive and hard." And if they're a big company, they say, "Yeah, we spent millions on it, but we didn't get a lot out of it."

I pretty much mandate that we do a proof of concept with a new customer because it takes minutes and [I want to prove the benefit]. The amount of effort to set up and maintain an application is reduced by a factor of 1,000. But you need to show it to them; they don't believe it can be done, because HP and CA and IBM said it couldn't be. So it's [a matter of] time to value.

It's also perspective. The old generation [of APM] was all about monitoring in the datacenter. You can't get access to something, and they say everything's "green" here, but how could it be, it's not working? So we're not monitoring from the datacenter, but from the browser into the datacenter. Then IT gets view of what you see. That's flipped around user perspective.

E-commerce guys know every user, every click. They keep the data for up to two weeks to address complaints. The help desk ability coming from your point of view is a big deal. One of our early customers was Zappos, which was really hell-bent on performance. They loved user perspective because it cut through so many arguments. It's some pretty cool stuff.

We're actually now beginning to connect the applications to the infrastructure, so if they find a disk failure over here, they can flip it around to find out which business transactions were impacted, which customers, which location, et cetera. It's much easier for them to prioritize things.

ZD: So where are we headed?

JVS: There are three big vectors. The first one is, single systems are going to instrument more of the componentry, so that rather than 100 tools -- mainframe, .NET, browser, hardware, operating system, hypervisor, et cetera -- there are going to be fewer systems that consolidate the facts into fewer locations so that you get a bigger, correlated view. It's a lot simpler for IT to deal with.

The second thing that I see is that we're going to continue to go down the road of [the notion that] it's not just about production. Performance-thinking need to go back upstream through test and back into development. We call it lifecycle, some people call it DevOps. It's more of a continuum, fewer silos.

The third one is analytics. You take the first two concepts -- breadth and depth; lifecycle -- and you put a brain in the middle of it. These systems have to get smarter. We know what the problem patterns are. To do better, smarter, more precise anomaly detection -- and pinpointing where the fault domain is, and isolate the root cause -- and flip the whole thing on its ear from being a system that is there for you to investigate the data to one that tells you what the problem is on your smartphone? That's the big idea. Answers, not data.

Obviously, big data models will be underneath this, but apply it to IT, and it's a big deal. The question is, who has the best raw material -- facts -- to build your intelligence off of, and the broadest reach to see the broadest set of dimensions that relate to a problem? It literally can be from bare metal to operating system to hypervisor to the business transaction itself.

ZD: And how do you get there?

JVS: I have really smart R&D people. (laughs) I have more than a hundred people working on the next generation of this. This is the next three-year horizon. It's a really exciting time. I've been in this industry for 30 years, and I've been in software infrastructure since the mid-'90s. There has never been a point in history where applications needed this class of software more than now. Where we're headed is pretty interesting.