Puppet Labs chief: Use DevOps to stop fighting fires

The founder of Puppet Labs on how IT organisations can stop fire-fighting and get productive again through DevOps.

For many the jury is still out on whether DevOps really is the force to enable fundamental change in the software business or an untried concept whose time has not yet come.

One man who has no doubts about the worth of DevOps and its likely impact on the software basis is one of its fundamental backers, the CEO of Puppet Labs, Luke Kanies.

We talked to him to find out his take on the future of DevOps and the wider implications for the software business.

luke-puppet.jpg
Kanies: "With DevOps you can make a real and meaningful difference much faster than you might think." Photo: Colin Barker

ZDNet: How did you first get started in IT?

Well I worked for a company that did IT services for call centers. And I was doing this stuff and it occurred to me that I couldn't be the only person who was doing this. There must be other people around the world who have the same problems.

Kanies: So if this was 20 years ago there would not have been much around in terms of IT for call centre management?

Yes, so in those days when you wanted help, well in the case of the company I worked for, you would just have an AT&T cell phone, you would call 411and get help from someone who was in your local city. So we had 33 call centers around the country. And we had six servers in each of these call centers so you would have 50 to 100 servers around the country and each centre would have three different server types.

But all the servers basically worked in the same way so you only had three different kinds of server [that were] duplicated across the country. That meant that the value from automation could be very high but with some level of complexity.

So within six or eight years I was working with other people who were looking at similar issues and I spent a decent amount of time working in that community. That's where I started my consulting business.

It struck me that there was no-one out there who was doing this stuff. Of all of the people I talked to, none of them was thinking about the question: "How do I make the software I am writing work for nearly everybody?" They were all focused on a much smaller target population than I was. They were looking at questions like, "How do I make the software we are using work better?"

That didn't make them right or wrong - it just meant that they had a different focus. So when you thought about what drove your decisions, what drove my decisions was: "Can everybody use my software?"

Q: So you were trying to get away from the fire-fighting mentality?

Yes. I just felt that we were never going to have software that worked for nearly everybody unless there was a company behind it. So when we looked around there were 20 different companies in that area but none of them were addressing the issue the way I wanted to address it. So in 2005, that is when I decided to tackle the problem full time. I quit my job and spent the next year or so building the product.

Q: And you did that with your own finance?

For the first year I was just doing development, making no money and living on my wife's student grant.

Q: You had no-one else to work with?

No, just myself. I tried a couple of hundred people and I failed. I couldn't find anyone else to join me, so I did what I had to. It was just me for three years. No outside financing. So I did that until I earned enough money that we could think about hiring a couple of people.

Q: So how did you feel doing that for three years? There must have been times when you wanted to give up.

It was pretty rough, but it was unconstrained, if you see what I mean. So I could work and as long as I kept my costs very, very low, I could survive on a month-by-month basis. I could survive, I could eat - that was the most important thing! And beyond that I focused on the question, what does it take to build a community? Once I started hiring people, which was about a year and a half after I should have been hiring people, it made a huge difference.

But that is what it takes to learn - you make mistakes and you figure it out.

Q: Getting the right people must have been important?

In my case, getting services help and sales help had such a huge effect on the business and in retrospect if I had just hired those people earlier...one of the things I didn't know was what kind of help I needed.

I really thought a developer was the most important help but in retrospect, hiring a services person to spend all their time at customer sites would have had a much bigger effect, and I wouldn't have had to spend more than two weeks a month at a customer site.

Next, hiring someone in sales to drive through more sales would have made a bigger difference.

Then we had our first round of funding in 2009. Since then we have raised a total of $87m and grown from three to about 385 people. We have offices in Portland Oregon, London, Belfast, Pilzen - just outside of Prague - and Australia.

Q: Why those countries in particular?

One thing was the draw of customers. We have been working with businesses for a long time - the first business I did here was in 2009. In Belfast we found a couple of really great people to help around the office and it has the great combination of time zone and no language barrier.

Q: The product has been very successful, are you at the stage where you are thinking IPO?

When you start any company has four options - you can go broke, get bought, go public or stay private indefinitely. Once you have raised as much money as we have then the stay private indefinitely option goes off the table. Of the remaining three options I far prefer going public.

If you look at the competitive landscape right now, the large companies we should be competing against, who should be selling great new software to their customers aren't doing very well.

If you look at CA, EMC, HP - they are all built pre-cloud, pre-DevOps. Really they are built for a world where companies try to protect themselves with technology and not for a world where software is the biggest asset for an organisation. That shows in the products that they build and in the sales model for the customer.

Those companies will be irrelevant in this new world and you are going to have new companies that are built for this new world.

So based on the path we are on and the amount of money we have raised we are definitely thinking about going public at some stage, but we would be a bit small if we were trying to do it next week.

Q: Shifting to DevOps, where do you think the IT world is with it in terms of getting the IT managers to think about it and how it is relevant to their organization?

Well as you know we have done a survey on DevOps and we got 5,000 respondents. Those 5,000 respondents point to the reach of the term and people's interest in it.

What I know is that three years ago, very few people knew what DevOps was outside of a small community of insiders. Today it is very, very widespread. Almost every CIO, every CEO, every sysadmin I speak to has a decent idea about DevOps. They sometimes have a wide variety of opinions, but everybody has a basic idea of what it is.

It's a connection between operations and development, it's a way of re-aligning the business and it's a way of helping companies be more agile overall.

Q: So it DevOps the sweet spot of the IT business or is it the case that it is almost there?

The amount of change we are seeing in this industry, the amount of analysis of how sysadmins work, how vendors work is huge.

If you look at the amount of chaos there is in the vendor world right now - it's insane. That chaos is a threat to a huge amount of things but it is also a huge opportunity. Whether you want to or not, if you are a big company you have to be changing how you do technology because your vendors are doing it differently, all of your creditors are doing it differently and the pressures on your business are different.

If you look back 15 years ago, if you were a bank you had a website but your website was primarily there so that your customers knew how to call you and knew where your branches were.

Today, if you are a bank, that website is the primary mechanism for your customers to interact with you. So if you think about the opportunities engendered by that change - and the threats - then if you are one of those banks who can't implement those changes, one of those banks whose website sucks, then there is a real problem.

And unlike 15 years ago you can't just call Sun or Cisco or BEA and say "Hey, here's a check - make me good at technology".

Now, the only real choice is to learn how to be good at it yourself . So today every big company has to learn how to be good at technology.

Today, DevOps is the closest thing we have that says, "Hey, here's the path to being great at technology. Here is the path to making technology a core asset as opposed to a liability."

So in the survey we found that there was a dramatically higher increase in serviceability if you implemented DevOps. There were 60x more deployments, 160x faster response to failure. So with DevOps we found that companies deployed much more quickly and had far fewer failures. And when they do have failures, they can respond to them much, much faster.

We found that the market pretty well understands that DevOps is pretty closely related to the ideas of lean working and ideas like limiting work in progress. They also understand the importance of shortening the time between starting something and finishing it. So don't have three year projects, have three month or three week projects. And ship as quickly as possible.

That means your costs are much, much lower and the whole of your operation will operate in a much more agile way.

Q: Do you feel optimistic about the industry and its ability to get these new ideas out?

I guess I want to say yes to that and no at the same time.

The only thing that can destroy the industry is the industry itself. We know there will be a significant amount of change. There are stats that show what percent of the S&P 500 won't exist anymore [at some time in the near future] and that is pretty significant.

I think you are going to see more churn and more change in that area. More instances were companies show up and by merely being better at technology are able to take over.

What it means to succeed as a grocery store is nowadays more about logistics and operations than anything else. So, what I can see is there will be an industry there and there will be companies there but whether or not it will look like the industry today is another matter. All of the big tech companies that we deal with today - and we deal with all of them - they are all under threat.

The way they are under threat is not from the competition it is from the fact that the world is starting to look very different.

It used to be that your web-site could be down for a couple of days and that would be no big deal but now, services are so important that of course you can't be out. You can't have a recovery window that involves an 18 hour restore from tape. That is just not one of the allowed things anymore.

That is because the technology is so critical and it is that criticality itself that is going to be the anchor that drives the change in our industry.

Q: But that is a change that you welcome?

Certainly the industry we had before was a horrible place. It was a place where the technology values were based on something that had nothing to do with what the company's need.

One of the things that has changed is that 10 years ago when I founded Puppet Labs we were one of the first companies to be really open about things like pricing. When I started you couldn't get a company's price sheet, you couldn't get their documentation without going through a laborious conversation with a sales person.

Fortunately, nowadays companies realise that they have to have all their stuff on the web site, they have to have transparency with the customer. I think it is good. Good for the industry, good for the customer.

One example is security. Everyone says that security is their number one issue, but of course it isn't. If it was you would just pull everything off the internet. You can't do that and why not? Well, because providing a service to your customers is number one.

So as companies get better at aligning the goals of their IT organization with the goals of their business that is where they get success.

Q: So what advice do you have for a manager who is trying to get his company behind DevOps?

It's a complicated conversation because it is a significant cultural change. Most IT organizations are in fire-fighting culture today so to do that you have to do two parallel efforts.

Various studies show that on average, services get interrupted about every 15 minutes and then different studies show that when you do get interrupted it takes you, on average, about 30 minutes to get back to productivity.

You put those two things together and you find that systems managers only respond to fires so you have to make the fires go away and then they can begin to work on the more strategic priorities of the business.

So the fire-fighting situation is resolved through automation, through consistency and the reliability that DevOps brings. You should really focus on reliability and, ironically, not on agility in the short term. Focus on reliability and consistency and that gives you the "time reclamation". You go from having 30 minutes of spare time in a week to having 25 hours spare in a week. What do you do in those 25 hours? Well that's where you go and double-down by investing in building relationships with development, building relationships with the business.

But if you start with the idea that you are going to have an 18 month implementation of DevOps in your business, you can't get there. It is a long-term, strategic cultural change that you have to drive through your business. Now that cultural change has got to be taken one step at a time.

It is the hardest part but now there are organizations that are getting better at helping people through that cultural shift by getting the right people.

I have worked at organizations with the frontline workers, the middle management; and the CIOs and in some cases they are all together and everything is working well but in others one of two of those groups have worked out that the change is necessary while the others are still in denial about it. And they are all asking, is there someone I can hire who can help get those groups across the line.

So the one thing I would caution against is that fear that you are so far behind resulting in inaction. You can get the ball rolling faster than you might think and you can make a real and meaningful difference, much faster than you might think.

Further Reading:

CA Technologies updates DevOps tools to help boost system performance

Why every DevOps team needs a contrarian thinker

DevOps: How to get started and how to make it work

What is DevOps and why does it matter?