IT is undergoing a period of rapid change, as traditional on-premises data centres move from hosting siloed client/server applications on often-underutilised physical hardware to running cloud-native apps on highly virtualised infrastructure -- be it on-premises, in the public cloud or a hybrid combination of both. Key attributes of 'cloud-native' applications include the widespread use of microservices (small portions of independently deployable functionality), efficient packaging and deployment via containers, and communication via APIs.
A major driver of all this change is the need for speed, as businesses pursue digital transformation strategies in order to disrupt their markets, or simply remain competitive. This, in turn, has led to a new practice of combined application development and IT delivery called DevOps.
"From my perspective, the movement towards cloud, towards DevOps, is very driven by agility -- that's the foundation. Companies and organisations want to deliver functionality faster: for example, all of us, on our mobile phones, are used to apps that are updated on a daily basis, with new functionality showing up, and there's a lot of competitive pressure to continue that innovation."
"What does that mean? That means I need infrastructure that's easily available and can be provisioned on demand; it means applications and code that can be updated on a weekly or daily basis rather than once a year. Digital transformation strategies are evolving because there are so many technology changes, so you have to react fast. DevOps is critical for digital transformation, but I would say as an enabler rather than a definite requirement -- it allows you to be more agile, deliver things faster and react to market conditions quicker."
So what, specifically, does DevOps entail, and where did it come from?
The roots of DevOps
The DevOps 'movement' is an IT-centric application of ideas formulated by US statistician, engineer and management consultant W Edwards Deming, whose work on product manufacturing and business management was summarised in a set of 14 key principles, and later dubbed Total Quality Management (TQM). Deming's principles were widely credited as helping to inspire Japan's post-war economic recovery, and three of them are particularly relevant to DevOps:
- Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.
- Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
- Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and in use that may be encountered with the product or service.
Key components of DevOps include the continuous delivery of software components (such as microservices), rather than episodic releases of monolithic applications, and their continuous integration into production systems as required. Cultural factors such as the implementation of lean management practices also help teams of developers deliver improved IT performance.
You'll often hear the words 'Agile' and 'DevOps' in close proximity, which can lead to confusion. Both share the same goals of improved IT performance via the rapid, iterative development of small software components. However, compared to Agile software development, DevOps also factors in the deployment phase, with 'Ops' teams in constant communication with their 'Dev' counterparts. DevOps is, in many ways, a successor to or outgrowth from Agile.
Cloud-native applications, rapid development and deployment, plus high levels of automation and orchestration, are the norm among web-scale giants, and also startups -- 'greenfield' organisations that are minimally encumbered by legacy infrastructure and applications. But what do recent surveys tells us about the prevalence of this approach in 'brownfield' organisations such as enterprises? (See also Tech Pro Research's survey, summarised elsewhere in this special report).
What the surveys say
RightScale's annual State of the Cloud surveys have tracked DevOps adoption for the past five years, and seen it rise from 54 percent in 2013 to 78 percent in 2017. Adoption rates are consistently higher in enterprises than in SMBs:
RightScale's data also shows a big jump in enterprise-wide DevOps adoption from 21 percent in 2016 to 30 percent in 2017. At the same time, adoption rates in enterprise business units and divisions, and for projects or teams, dropped slightly (31-28% and 29-27% respectively).
The most recent State of Application Delivery survey from F5 Networks found that only 20 percent of respondents viewed DevOps as having strategic impact, up slightly from 17 percent in the 2016 and 2015 surveys. The view of DevOps' strategic importance depends on the role in the company, though: in 2017, just 17 percent of executive-level respondents saw it as important, compared to 39 percent of those with DevOps or cloud-related roles.
Given that, as A10 Networks' Kamal Anand told Tech Pro Research, "somebody, senior enough in the organisation, needs to be a champion to drive those processes," F5's data suggests that DevOps has some way to go on the strategic front. For now, F5's 2017 survey reveals a tactical view of DevOps, in which automation and orchestration frameworks are primarily used for "scalability needs and reducing OpEx" rather than achieving "the overriding mantra" of quicker time to market.
A recent survey by cloud sandbox software provider Quali identified the top ten barriers to DevOps in the enterprise. Heading the list were cultural factors, the lack of test automation facilities and integrating legacy on-premise applications into a hybrid cloud environment:
"The number-one barrier to DevOps is culture, which should not be surprising because, when all's said and done, DevOps is not a set of tools -- tools are just a means to an end. DevOps is a culture, and in calling this out as the top barrier it really brings the people part of the equation into the mix and says 'look, we can only reduce this barrier if we can bring in the ability to collaborate more seamlessly, and have a healthy mix of being able to move fast, but with the right control mechanisms in place.' These mechanisms should not necessarily become apparent when you're scaling or when something breaks, but should be put upfront in a proactive way."
"Culture is usually the hardest thing to change, and it's extremely difficult to keep everybody happy. It's also gradual change, coming about in increments, and in some ways it has to come from the top. So executive buy-in, the ability to enforce some of these things as a mandate across the organisation -- that can go a long way toward greasing the skids to make this successful."
"DevOps looks at aspects of continuous integration and continuous delivery, and test automation is usually an afterthought. And that's where it's becoming a big issue, because with DevOps the predication and the mandate is to move fast -- and when you move fast, you want to do it with reduced risk, and you don't want to compromise on quality. Without the right set of test automation principles in place, it becomes a challenge to ensure that quality requirements are met and that your risk is mitigated."
"Today we see CIOs spending budget on digital transformation, which gives the promise of productivity, efficiency, agility -- you name it. But the challenge when it comes to DevOps is dealing with the complexity of existing legacy investments and bringing that into a new scenario where the application could be cloud-based, or re-architected around microservices. We have to realise that not every application can be re-architected: some costs are way too high, and in many cases not everything can be in the public cloud or be a SaaS-based offering, due to compliance or other issues. That's where the DevOps process needs to reconcile."
Puppet has been issuing an annual State of DevOps Report since 2013. The 2016 report was released last March, and the 2017 report is eagerly awaited. Utilising the specialist research skills of DORA (DevOps Research and Assessment), Puppet's surveys aim to uncover the statistical drivers of IT and organisational performance with respect to DevOps.
Results from previous surveys have shown that the adoption of DevOps practices is correlated with IT performance (deployment frequency, lead times, failure recovery times, change failure rate), which itself correlates with organisational performance (in terms of profitability, market share, productivity). Other important factors previously identified were lean management practices, application architecture, the role of IT managers, diversity, deployment pain and employee burnout.
Key findings from the 2016 survey were that high-performing organisations deploy 200x more frequently, recover 24x faster from failures, have 2,555x shorter lead times and 3x lower change failure rates than low-performing ones. They also enjoy better employee loyalty (as measured by Net Promoter Scores), spend 22 percent less time on unplanned work and rework, spend 50 percent less time remediating security issues, benefit from an experimental approach to product development, and can achieve significant cost savings from a (DevOps-driven) technology transformation.
Puppet/DORA also found that, among respondents who had adopted DevOps, deployment automation, infrastructure automation and version control were the most commonly applied practices:
(See also Puppet's infographic summarising the 2016 report's findings.)
Tooling up for DevOps
As the graph above shows, DevOps has multiple aspects, whose key stages can be codified in a 'tool chain' diagram such as this:
There are multiple tools available -- many of them open source -- to cover one or more stages of the DevOps cycle (see Gartner's 2016 Market Trends report for an extensive, but not exhaustive, list of DevOps-Ready/Enabled/Capable tools). To address 'tool chain sprawl', CollabNet recently introduced DevOps Lifecycle Manager: this integrates with many of these disparate point solutions, providing a 'single pane of glass' view of the entire software development/deployment process.
RightScale's 2017 State of the Cloud Report found Docker (automated application deployment via containers) to be the most popular DevOps tool, followed by infrastructure configuration management tools such as Chef, Puppet and Ansible:
Also increasing in popularity are container orchestration tools such as Kubernetes, Salt, Docker Swarm and Mesosphere. Note that RightScale's survey did not ask about continuous integration tools such as Jenkins, Travis CI and others.
Quali's survey did include continuous integration tools, and placed Jenkins at the top of the list, with Docker, Puppet, Chef and Ansible also prominent:
Inspired by the world of just-in-time manufacturing and widely practised in 'greenfield' IT organisations such as web-scale businesses and startups, DevOps is now making inroads into the 'brownfield' enterprise market. The key driver for DevOps adoption in the enterprise is the increasing need for rapid and continuous development/deployment of cloud-native applications as part of digital transformation initiatives.
It's relatively early days for DevOps in the enterprise, and there are significant barriers to overcome. These include technical issues such as legacy application integration and test automation, but also factors like IT culture, budgets, skills and executive buy-in. However, surveys show that successful DevOps implementation can lead to improved organisational performance.
Read more on DevOps