How DevOps is proving to be a sure bet for Paddy Power

ZDNet recently spoke to Paddy Power CIO Fin Goulding about how leading-edge software development trends like DevOps, agile, and creating 'T-shaped engineers' have contributed to the company's success.

Q: What is your IT background?

I started with systems development work in large-scale banks and then I switched to Lastminute.com. I joined as the organisation had just gone through huge growth and just before the sale to Travelocity [in 2005].

Then there came an opportunity with the larger part of the organisation and I had always wanted to run an offshore team. By that time I had done everything in IT from operations through to development, and now I wanted to go offshore. Specifically I wanted to run the operation in India but they wanted someone local to do that, so I ran the Argentinian team in Buenos Aires.

Q: That must have been fascinating. Did you enjoy it?

It was something of a cultural change. I just had to do it and that is where I met my wife and so that Latin culture is part of my life now. I was there for three years and then they said, we need you to move to the Dallas head office. I didn't fancy that much and I felt it was time to get back to Ireland, so I joined Paddy Power in Dublin.

Again, Paddy Power was an organisation that had enjoyed phenomenal growth and needed a few grey hairs to get things in order. At the same time, having worked with such enthusiastic, young developers in Latin America, I could see [with Paddy Power] that the older techniques weren't going to work and we needed something that was going to fit in with the younger generation.

We needed more collaboration and younger people needed a voice at the table. I learned a lot of that in Latin America and I employ that in the way I work today.

Q; Paddy Power's website is impressive in size and scope.

Part of that is the underlying infrastructure, the VM world, removing any single points of failure, getting datacentres working in harmony, diverse routing and so on [that I am responsible for]. But all of the design and the look-and-feel I would love to take credit for but that is outside of my group.

Q: How does that work in a DevOps world? Isn't the object wih DevOps to work in separate, multi-skilled teams?

It doesn't mean that you have to move everybody into the same teams. I think you need to do that from a software engineering perspective - you have your matrix of individuals in the different teams. But sometimes it is not enough. For example, if you've got a group of people, which we call 'pods', then the pod itself is self-sufficient.

You may not have enough people to put in every one of those teams so sometimes you have to do 'one to many', which can happen with UI/UX [User Interface/User Experience]. We just make sure that in the methodology, these positions are ahead of the game in terms of assets being ready. And they are co-located with us as well because specialisms need to be matrixed.

If you separate them out, then that can lead to competition within the teams and the brand can start slipping.

Q: How do you divide your team's time between the different things they are doing?

I wish it was under my control but it's not. We have separate portfolio and product delivery functions and they are the ones working on priorities. So we respond to business drivers and business teams - that's why the teams are manoeuvrable.

Q: Looking at the matrix of your business that you use, it must be one of the more complex ones that I have seen.

That's because the actual agile team - product owner, business analyst, engineers and developers, and testers - is quite straightforward, but in a large organisation like ours, you have governance, you have security, you have production and operations, you have PMOs [Project Management Officers and Offices] and all kinds of stuff.

That is why we call it 'enterprise DevOps'. It is at another level. If I was in a small organisation with 20 people, I wouldn't really have to worry about all these other things, but in our organisation there is a lot of stuff around the outside.

Q: But in all this complexity, is there a big red button to hit when things go wrong?

That's why we have Splunk and all the other tools to warn us before they go wrong. We don't want to know that something is broken - we want to know if something is going to break.

In the DevOps world where traditionally you have a support group or a network operations team, you will see that sort of screen with all the monitors and dashboards. But if you go to our development teams, they all have the same screens because they have the same code and they build it, support it, and ship it. And they are quite clever at working out that something is going to go wrong and how it is going to go wrong, so they build-in those early-warning mechanisms.

Q: Does that mean you can devolve responsibility down?

You can. You have more ownership. If you are the product owner and I am the one that's building it, I have got this connection to you to make sure that what I am doing is actually top-quality. That's why I talk about 'T-shaped engineers' and 'Pi-shaped engineers' - it is all about giving people the ability to cover different skills. So the systems designer will write the code but will also write the test case. With this method, I get 100 percent code coverage and my quality will start to go up.

It is the same thing as if I am handing off to a production team. I am not handing off to a separate production team that doesn't understand the whole picture. There is a lot more to it, and human nature is that you don't want to be caught out. There is quite a lot of movement within the teams to make sure that what they implement is good.

One thing I have learned from management is that if you want to succeed at the most senior level, you have to have that breadth of knowledge. That's why reading theHarvard Business Review[which is where the T, Pi, and Comb-shaped management principles come from] is a good thing.

You can apply that to any job, so if you take a Java person and you give them testing skills, you get a T-shaped engineer: they write the code, they write the tests, and you get 100 percent coverage.

Q: Doesn't that hark back to an old practice which was to take people off a project when it was running late? If you had 1,000 people on a big project and it was late, you took 200 people off?

Yes, it is like that. If you think about it, it is like a motorway. If you fill it to its capacity, it doesn't move and the same thing can happen with technical teams. If you stem the flow a little, they are going to get around so much more quickly.

It's using old manufacturing techniques, applied in a technical way. We have found that some projects do speed up when you take people off them.

Q: One of your charts lists production delivery at 100 percent. Are you there yet?

I would love to say we had zero downtime and 100 percent availability - but that isn't going to happen. It's down to cost because to get there you need to have two of everything, and a- and b-type environments where you can realistically set the target at zero downtime. But we know that that is the target and we try to achieve it through everything we do.

Q: So if the aim is near-100 percent availability, is DevOps a good component of that?

For us, the bit around developers building their own test automation is important. That can increase quality beyond imagination. The other roles being embedded in the team are important too, to make sure that there are no design issues - to actually build things around what an engineer, or a systems engineer, can give you.

But at the same time, things get done faster. It's doing things very quickly and watching them failing quickly and then fixing them again and getting it right. In that atmosphere I think it is more enjoyable for people if they are working in a team that has end-to-end responsibility - from idea to production.

That's much more fun than the frustration of standing over someone's desk saying, 'Have you done the firewall?' and being told, 'I'm too busy, come back tomorrow'. Removing those constraints, and those hand-offs and blockages, actually makes the working environment much happier.

And I think once people become coaches - and fundamentally people like to teach and show people how things work - the team dynamic changes. What we are really trying to do is build startups - we are trying to build 50 startups in one, big, enterprise team.

People who join startups do so because there is less bureaucracy and it is more fun and they get to see the results of their work quite quickly. That is one of the benefits that DevOps will give you - to try to emulate that model.

Further Reading:

DevOps: How to get started and how to make it work

What is DevOps and why does it matter?

What went wrong with the UK government's £154m 'agile' online computer service?