Gambling on virtualisation: One company's mission to automate its infrastructure
"We're trying to get to the point where the infrastructure team never need to touch the infrastructure to implement change," says Betfair CIO Michael Bischoff.
He is describing his mission to automate as much of the online betting exchange's computing infrastructure as possible.
Like so many companies, Betfair uses virtualisation software to simplify the deployment, management and monitoring of its IT estate. VMware's vSphere platform virtualises servers in its datacentres and the vCloud Automation Center orchestrates and automates the deployment of these virtual machines. Monitoring the performance of this virtualised infrastructure and the applications running on top is vCenter Operations Management Suite.
Betting on the cloud
"As of last week we were running somewhere in the region of 800-900 virtual machines on the private cloud platform we're built, and that's going to be growing," he says.
But the process of automating IT operations becomes trickier as you go beyond deploying virtualised infrastructure and move up the stack to managing applications, Bischoff says.
"As we've got further up the stack towards the point where the value gets delivered, we've found it's quite hard, even with existing toolsets."
Alongside its VMware tools, the company relies on a range of third-party software and services to minimise manual intervention, such as IT automation software packages Chef and Puppet, and F5 load balancing — all of which need some level of integration to work together.
"We don't believe there will ever be a single vendor that will be able to provide a tool that does all of that stuff for you and our development colleagues are using a range of open source tools, and self-built tools as well," he says.
"We're using the orchestration platform to initiate the demand process, to do the things it needs to do, but I don't want one platform doing everything.
"Where we've invested most of our time and effort is the integration of this orchestration stack with the range of tools that need to ingest or eject into that," he says, adding that the majority of these interactions between software takes place via scripted API calls.
The orchestration platform can make API calls to Chef so it can install and set up software on machines, as well as interacting with Puppet for federated identity management.
The firm is currently working on integrating its ServiceNow platform to automatically handle some elements of change management.
"We're doing work to integrate those tools such that our change management processes can be automated," Bischoff says, giving the example of automatically logging when code is released into production.
"We use ServiceNow and we're looking for ways for the change taking place to be just an event that's recorded in ServiceNow and flagged as having taken place on that platform or that service, rather than as an interrupt that says 'Now we're going to stop all of our processes to do a something manual'. So that's what we're targeting, but it is a tool integration challenge."
Changing responsibilities
As software automation changes how the firm manages its infrastructure, so Bischoff says the responsibilities and skills of his IT team are broadening beyond a deep and narrow focus on a single infrastructure area.
"I have around 250 people in the entire IT function, and they cover a huge range of things, from desktops all the way through to data and web analytics. Our cloud team is around three to five people in total to look after the infrastructure and orchestration, and then we have a couple of hundred developers in the CTO organisation who are consumers of that service.
"Ten years ago running SANs [storage attached networks] was a data science. These days the vendors have pretty much made it easier to operate. The skill is now less in operating a storage platform than it is about how you integrate storage into the thing you're delivering.
"More technology staff are going to have broader exposure to a range of disciplines than historically was the case, where you had a very hard silo between the storage team, the network team, the database team, the software development team and the support and operations team.
"Those have started to blur, where you can't do anything in storage without being very cognisant of the network, and you can't do anything on the network without being cognisant of security and you can't do anything with security without being cognisant of what the application behaviour is like.
"Increasingly staff have experience across more than one of those disciplines and they think about automation almost as a prerequisite for what we do."
Ultimately automation is likely to be an ongoing process, Bischoff says.
"Our goal is to get continuous delivery to do both of those things, be both continuous and to deliver, without any human intervention. It's a very ambitious target and I expect very few companies have been able to achieve that, but that's the goal, we'll pick those things off one by one.
"I don't think you ever finish. I think it's a lifestyle, because tools will change and use cases will appear and disappear. Development groups and new tools and capabilities emerge all the time that they want to use, we 're going to have to be clever as to how we plug those things into our process of continuous delivery."