Quest for the One True DevOps, Part III: Into the eye of the storm, DevOps' bravest warrior finds new hope
Video: Why DevOps is key for your business in 60 seconds
On a damp afternoon in London in early December 2015, I found myself at the base of an escalator where I would rather have been at the top. A private security sergeant hired by the ExCel Centre, trained in anti-terrorist measures, denied me access to a conference session that was part of HPE's annual Discover conference. It was the fourth such denial, from the same sergeant.
HPE's own personnel spent several minutes attempting to persuade him of my intent: that I was a member of the working press, that I had been invited by HPE, and that at least some of the folks upstairs were expecting to see me. The problem, the sergeant repeated, was that the color on my badge that designated "Press" was not the same hue as the color shown to security officials during the morning briefing. The public relations official reiterated, as emphatically as possible, that I was to be admitted.
"I will do that," he responded. "And I will treat any future instructions I may receive as rubbish."
She then reminded him that her presence and that of her company paid his salary. "Well, then, what good is my presence here?" he responded. "I mean, I'm paid to do a job. But I get four or five conflicting instructions about what my job is. And now you're telling me all those instructions are rubbish. If an organization such as yours cannot provide clear instructions as to how we do our jobs, how am I expected to be able to stop terrorists who try to blow something up?"
Then he put his hand on my shoulder where the strap of my stuffed backpack was resting, and said with a grimace that could swallow the Cheshire Cat whole, "Not to imply that you've got anything in that bag, kind sir, or that you're a terrorist, I'm not implying that at all. I would not do that. That is not what I mean. But if I don't have a job to do, and we don't have a job to do, then why are we here?"
It was actually the most prophetic question I had heard asked during the entire show. In the same way an alien bank heist is a job for Superman, this was a job for DevOps: a miscommunication between the people who programmed the HP printers to produce badges and the operators who actually oversaw the production. It's the type of problem that the implementation of DevOps principles should anticipate, and rectify before an armed team of security professionals expecting "Press" to be encoded pink, actually see something closer to orange. And in today's European security climate, the sergeant was right: If explicit security orders could be countermanded at random by anyone whose badge color could also be just as wrong as mine, what security would there be?
Read also: Agile plus DevOps is slowly but steadily reaching enterprise scale
The man at the session past the top of the escalator was Dr. Donnie Berkholz, Ph.D., then the research director for the DevOps and IT Ops channels at 451 Research.
The greatest challenge organizations face in accomplishing the goals of DevOps, Berkholz was telling his audience minutes before I was finally let in, was getting all the stakeholders in the same room together. Though a virtual room has some merits, a physical one, he argued, is far preferable. Stakeholders need to see each other, they need to be talking, and they need to be gleaning expertise from one another. Emphasizing a point I have made about tech conferences for nearly four decades now, he told attendees, when people are in the same room together, they find ways to care about one another.
But this degree of caring can't happen, he said, without clear and concerted direction at the executive level.
"You can do the tools, you can do the technologies, without having VPs and CIOs and other C-suite folks buy into this process," said Berkholz two years ago. "But you can't do the culture part. If you don't have that level of executive who knows exactly what has to happen, then you do have to bring in external expertise, absolutely."
For years, Donnie Berkholz had been that external expertise. The following year, on his own accord, he would walk down the escalator, if you will, and across the bridge to the other side of the consulting relationship, to become Vice President for IT Service Delivery at globally recognized travel firm Carlson Wagonlit. There, he would challenge himself to heed the same advice he gave his own clients.
We're on the last leg of our Quest for the One True DevOps. We began at a boat dock devoid of any computers whatsoever. We heeded the warnings that the journey's metaphorical goal (like the neon sign for the Holy Grail) may lead us in the wrong direction at any time. And now we're at the ancient valley, at the foot of the Eye of Antiquity, at the core of the realm of Corpor.
Read also: Executives overestimate DevOps maturity
Here is where the most ancient of business processes reside, and where the most benevolent aims of automation collide with the true intent of "change management." Here is where Agile initiatives, in a great many large organizations, turn themselves inside-out to become self-maintaining super silos. Here is where DevOps can become swallowed in a sea of "research projects" whose purpose eventually becomes to fully utilize its own budget to justify its existence. It is the place most feared by nearly all practitioners of change from the outside in.
Except for Donnie Berkholz, whose acceptance into the organization was greeted by a worldwide party.
The few paths toward success
Two weeks after Berkholz' hiring, Carlson Wagonlit convened its first three-day DevOps Summit. Over 1,000 employees representing the company's technology initiatives, projects, and actively maintained programs, convened together in one room. Granted, they all had great travel agents. And yes, Berkholz believed this was hard to pull off. But it happened.
"There was a bit of a movement going," he told ZDNet Scale, "and I joined because they were looking to ramp that up and accelerate it, and bring in somebody who was not only enthusiastic, but also had seen it happen in other places and had that experience of successes, pitfalls, all the different ways to fail, and the few paths toward success."
An analyst may tend to perceive the characteristics of a client organization in terms of statistics, averages, and estimates, Berkholz continued. "What I found was, there's all these little pockets of excellence out there that are masked by what you might say is an overall, relatively conservative view of technology. You go out there and find people who are pulling in all kinds of great, open source tools, and very modern SaaS and on-prem options to do observability and tracing. You've got people who are building on serverless technology. I didn't expect any of that stuff when I joined. What I expected was, by and large, your standard enterprise developers and enterprise ops folks. But there is that whole spectrum, and you find these surprises in places where you'd never expect them."
Read also: DevOps accelerates, requiring new leadership styles
In this one particular, potentially unreproducible, case the former 451 Research analyst verified a suspicion that one of the oft-stated goals of DevOps -- the elimination of barriers and the convergence of once-separate departments -- was actually a means to a different end altogether. Up until his hiring, the travel firm had not invested much time or capital in the nurturing of its technology. Once it saw the potential windfall from a complete reinvention of its IT infrastructure and services, it placed the big bet.
But some departments in the company had already begun investigating their options for going it alone -- for example, with serverless development, which enables applications to use public cloud resources on a per-function basis. There were, Berkholz admitted, "bleeding-edge adopters" of parts of the new, cloud-oriented stack, throughout the organization, in some cases driven by customer demand that was far ahead of what the travel firm was capable of delivering at the time. Carlson's "DevOps journey" might not have been a simultaneous effort.
"We've got a handful of snowflakes out there, who have done really interesting stuff in really modern ways. How do you scale that," asked the travel service VP, "to many, many more teams and products? And how do you set it up for success, once you are at that level of scale?"
In Part I, we introduced the theory that DevOps may actually be comprised of one of many infinite paths, toward a goal with certain characteristics, but which may manifest itself in many ways. Presenting that theory was Dr. Nicole Forsgren, founder and CEO of DevOps Research and Assessment (DORA), and the co-author of the annual State of DevOps Report published by Puppet. Her team's 2017 report presented research on organizations' relative state of automation, offering evidence that these firms may not actually have a firm grasp on how many of their everyday tasks are partly or fully automated. The reason: When automated processes work well, developers leave them alone and operators don't make a fuss about them. So they're never top-of-mind with executives.
The need to automate, the research concluded, may only present itself to organizations once it has reached critical mass. Theoretically, new classes of software associated with DevOps such as Jenkins for CI/CD, or Chef or Puppet for configuration management, may be perfectly capable of re-automating processes that had already been captured using an earlier generation of business process management tools, or that was otherwise already working effectively. . . albeit for a different era. Enterprises may not develop a need for these tools, or for the people skilled in using them, until they have already begun suffering the effects of obsolescence.
Perhaps their processes are indeed obsolete. But they may not be suffering yet.
Read also: 5 reasons DevOps will be a big deal in the year ahead
Change control
For whatever reasons it may have had, Carlson Wagonlit had discovered its needs had already reached critical mass. Berkholz told us his new colleagues had already begun value stream mapping -- diagramming the flow of deliverables from origin to customers. Such maps tend to be divided into service tiers, so that the flow of resources trickles down the various components like syrup over pancakes.
What the flow of syrup, if you will, ended up vividly illustrating was the lack of interdependencies between business operations. They could visibly witness the schemes of their silos. And they could spot the correlations between the exclusivity of certain business tasks, and the value -- or lack of value -- they ended up delivering to customers. Berkholz was able to spot circumstances where business units created complex processes and piled them on top of one another, essentially as a means of preserving their roles and keeping their jobs.
"If it's your job to keep this server up and running, well, it turns out that change can be antithetical if that's your only incentive," he remarked. "So you put a ton of heavyweight processes on top to make sure that you minimize change, and you put tons of oversight into every change you make, and you might tend to bulk those changes up because that's the way you think about the world.
"In many cases, it comes around to, what are your KPIs and what are your incentive structures?" Berkholz continued. "If we're reporting on uptime, and we're not reporting on delivery pace of new functionality, then we're going to end up optimizing for the things we're putting out graphs of. We have to incorporate these new KPIs into how we do our reporting; our risk/reward calculation; how we evaluate performance for individuals, for teams, for groups; how we incorporate that into the budget process."
The financing of any new process, he admitted, consumes more time and resources than he ever imagined. His company is shifting its focus to be less project-based, and more product-based. In the meantime, he's operating on a budget for a project to "do DevOps" -- specifically, to create a containerized platform with a CI/CD pipeline. It's difficult for a company that isn't really in the IT business to project an abstract pipeline of processes as a product in itself, as though it were being manufactured for customers. Instead, it's easier to project change as an extension project, over and above the real products being produced. That will change in 2018, said Berkholz, as his company adopts the goal of deliverables -- the benefits of DevOps -- as the items being funded on a set schedule.
Read also: What is DevOps? An executive guide to agile development and IT operations
"It's reminded me again of how important it is to focus on that outside-in approach," he revealed, "of what are the problems customers have, instead of what are the things vendors sell?"
Coming up, we'll take a moment to reconnoiter and take stock of what we've discovered along the long journey there and back again. Until then, hold true.
Quest for the One True DevOps
- Part one: The dream of fellowship for software developers and IT operators
- Part two: The dawn of automation and the promise of a formula for business success
- Part three: With the fullest measure of courage, DevOps holds true to its objective
Journey Further -- From the CBS Interactive Network
- "Ten ways DevOps can change how you run your IT department" by Conner Forrest, Riding the DevOps Revolution
- "10 critical skills that every DevOps engineer needs for success" by Alison DeNisco Rayome, TechRepublic
- "DevOps: Why getting the culture right is the key to success" by Colin Barker, CXO
Elsewhere
- "It's the End of Network Automation as We Know It (and I Feel Fine)" by James Kelly, Juniper Networks, writing for The New Stack
- "The Whole Bit About Performance and DX" by Scott M. Fulton, III, CMSWire
- "Automation: Not Just for DevOps" by Karen Riccio, Data Center Knowledge
Related stories
- Searching for the perimeter in cloud security: From microservices to chaos
- Micro-fortresses everywhere: The cloud security model and the software-defined perimeter
- Microservices and the invasion of the identity entities
- Machine learning and the spectre of the wrong solution
The map of Middle Ground for this series of Scale was drawn by Katerina Fulton.