Quest for the One True DevOps, Part I: The dream of fellowship
Video: Devs and Ops don't mix? New data analytics tool could fix that
"The tough question with respect to information activities is which of them belong together and which should be kept apart. There is much talk today about 'integrated total information systems.' This of course implies that all -- or at least most -- information activities should be in one component. Insofar as this means that new and different information activities, e.g., operations research or a computer system, should not be subordinated to traditional accounting, the point is well taken. But should they be coordinated? Or should they be separate?"
-Peter Drucker, Management [1973]
You're strolling along the docks at Liverpool sometime around 1949. It's lunchtime. You come across a tent with a sign identifying its owner as the Women's Temperament Association. Several ladies, probably wearing overalls with their hair tied up in buns, scarves, or nets, are serving meals to hundreds of grime-laden dock workers. In-between bites of stew and non-reprintable remarks, they infuse their discussions with messages about the virtues of sobriety.
Not surprisingly, one of these ladies struck a rapport with several of the workers. She wore classier lipstick than her colleagues -- Elizabeth Arden, specifically -- and it was starting to blow her cover. Her curiosity was intellectual, and that fact was becoming all too transparent. In the second of three docks where she served, she admitted to the men she was writing a book on the plight of dock workers.
Enid Mumford was an undercover researcher, working with the social science department at Liverpool University. Taking notes from her discussions with dock workers, she developed a theory that would become the progenitor of what today we call DevOps.
It was here that Mumford began creating a powerful idea, which would lead her through the rest of her career, and which would chart the course for seven decades of technological development. There were no computers along the Liverpool docks. But her study of the work habits of dock laborers, and the means with which management drove them towards efficiency, would bring her in direct contact with the first computers ever used in sociological research. The ideals she created, of "human machines" and "socio-technical" systems, would evolve into the founding tenets of our common philosophy of technology and society.
They also serve as our launching point for this next adventure of ZDNet Scale. Later on, you'll see it's a journey with a familiar ring to it.
Read also: Agile plus DevOps is slowly but steadily reaching enterprise scale
The machine minders
In 1947, out of concern for the many thousands of soldiers being reintegrated into life as civilian dock workers, the British Government established the National Dock Labor Board. Laborers were growing suspicious of their management, engaging in a process that was non-intuitively dubbed "soldiering." And the Labor Board set out to discover why.
Suspicion, as Dr. Frederick Winslow Taylor had written in his acclaimed 1911 essay " The Principles of Scientific Management," [PDF] was a poison among laborers that spreads like a virus and that must be extricated like snake venom. Taylor described soldiering as a type of loitering or loafing among laborers brought about by their disrespect for the work process and for the people managing them. The disassociation between laborers and management brought about by soldiering, he wrote, resulted in men working in relative states of isolation from one another. As a result, efficient work processes could not be passed down from person to person, even if science could prove those processes most sound.
"Each man should daily be taught by and receive the most friendly help from those who are over him," wrote Taylor, "instead of being, at the one extreme, driven or coerced by his bosses, and at the other left to his own unaided devices. This close, intimate, personal cooperation between the management and the men is of the essence of modern scientific or task management."
For the Labor Board, establishing this more friendly relationship at the Liverpool docks involved, for example, providing dock workers for the first time with showers and individual lockers, as well as some very friendly company at lunch.
Read also: Executives overestimate DevOps maturity
"They just thought I was some student trying to get a little bit of cash," said Mumford in a 2003 interview [PDF]. "It was quite acceptable. Students were seen as being awkward and nosy and wanted to talk to dockers and so on, but we were also seen as poor and wanting to earn a little bit of money, so that worked all right."
After a few years working among the ladies of the Temperance Association, Enid Mumford's research group received a grant from the European Productivity Association. Their mandate became, for the first time, to use computers to collect and interpret the data such as the kind Mumford was collecting. At this period of history, there were two schools of thought: One professed the ideal of computers as replacing information workers, while the other was convinced that computers could never replace human scientists in carrying out the practices of Taylor-style scientific management.
Mumford, as a researcher, had not yet made up her mind one way or the other. But as she observed perhaps the first use, or one of the first, of computers as human relations tools, she noticed right away a curious division of labor.
In her own words: "The number of jobs didn't go down, it went up, because you had this new and mysterious group of programmers, which offered a great new career to male clerks. For them it was splendid. But it brought some dreadful jobs for women, because this terrible punch-operating role appeared, where women had to punch the data into the computer. All the interesting bits were done by the computer. The women had to punch the data in and collect the output, so they were just kind of bits of machinery -- machine minders. It was a very bad period for women. Computers didn't enhance the jobs for women at all."
The machine minders were the first ops teams, the first sysadmins. And their roles were created for reasons Enid Mumford could clearly witness, but which the rest of the world would entirely miss: to keep the women busy and separated from the higher-paid men.
Read also: DevOps accelerates, requiring new leadership styles
The U.S. National Center for Women & Information Technology estimates that, in 2016, women only constituted 26 percent of the computing workforce. The artificial division of labor Enid Mumford witnessed in the early 1950s failed to corral women into one segment of the economy. Her awareness of this fact is partly what drove her to create an entire branch of systems design, based on what she called socio-technical theory.
Systems of work and the people who work with those systems, should be given the freedom to adapt to one another, she believed. In putting forth this ideal, she may have coined the phrase "open system," though she also used the phrase human systems. Along the way, she also observed an alarming side-effect of Taylorism, which back in 1911 was actually forecast as a virtue: The drive for efficiency in every step of the work process would transform general laborers into specialists who could not be replaced except by people who could perform exactly the same work, in exactly the same way. The drive for efficiency could work against her ideal.
It was a phenomenon of Taylorism that future scholars would call pipelining.
Read also: 5 reasons DevOps will be a big deal in the year ahead
An unexpected journey
Most explorations of a topic start out by defining it. With this one, we don't have that luxury. Indeed, we're on an epic quest for what the definition is.
We know this much from the outset: DevOps is an ideal, put forth by a few organizations that eliminated some or all of the barriers between the two departments, and believe they have benefitted from doing so. That benefit follows a long chain through history -- a chain that began with Enid Mumford. She discovered a separation between two factions of the work force: the creative and the practical. For most of the history of IT, there's been a cold war between these two factions. DevOps has sought to establish a middle ground between them.
For our map of the realm of DevOps, we introduce Middle Ground.
Our journey begins at a place I call the "Mumford Docks," commemorating her discovery and the triumphs that followed. The shortest path inland soon leads our expedition to the great divide: the "DevOps Mountains," separating the warm and bountiful land of "Devper" from the cool and cozy land of "Oper-tor."
The ideal of human machines is to enable technology to be responsive and responsible to all the people who make contact with it. That certainly sounds like the stuff for a techno-fable, doesn't it? Our hope is that the two peoples of these realms may eventually see their way to become one: truly, a bridging of more than just syllables.
"The first piece that organizations strive for is, they want to increase velocity," explained Robert Stroud, principal analyst with Forrester. "They actually want to deploy smaller pieces of change -- smaller new features, new ideation, net new ideas -- on a really consistent and continual basis."
Read also: What is DevOps? An executive guide to agile development and IT operations
But the distinctions between these two realms are not entirely artificial. Even before the advent of virtualization, the job of software development has essentially been about the creation of, from a virtual perspective, machines. And even after virtualization, the task of maintaining virtualized systems has been delegated to engineers, usually with expertise with physical machines. One realm is perceived as mysterious, the other somewhat pragmatic; one magical, the other strategic. Usually it's the people in one realm who bestow all the mystery and magic to everyone in the other, even when they don't feel it's been deserved.
For better or worse, there have been two clans. The DevOps ideal is expressed as a joining of the two, though there are different schools of thought as to whether this joining constitutes just collaboration, a joint partnership, or an outright merger.
We've already talked to some degree in Scale about distributed systems, and we've seen how a distributed architecture becomes pointless in a compartmentalized organization. The joint product of DevOps is the automation of processes, including software development, delivery, and maintenance, across the enterprise. Distributed architecture mandates that an automated process encompass the entire organization, without being relegated to divisions or departments. DevOps would have an organization construct a single pipeline for business processes, incorporating all stakeholders and encompassing the entire lifecycle of the product of automation (for instance, an application)
But here is where we make the boldest discovery of all: There are clearly more than two realms here. Any effective automation pipeline for a given organization must not only cross into all of these realms, but provide a two-way channel throughout.
"This is the idea of being able to connect across the business, all the way back through to the planning phase," explained Andi Mann, Chief Technology Advocate for machine data management platform provider Splunk. "In the lifecycle of software, software delivery is not a straight line. To start with, software delivery is a roller coaster... in some ways because it's up and down, round and round, and you often don't have a clear line of sight of where you'll end up. But the other part of it being a roller coaster is, it needs to loop back to the same place that it started, and needs to go again."
Every truly automated process takes place in cycles, so Mann's vision of continual iteration is not really the novel part. His vision comes from one higher level of abstraction. He believes it to be feasible that an organization can assimilate the feedback collected through platforms such as Splunk, digest that data, and re-integrate its findings into the system in order to improve the automation process, perhaps automatically.
"That is exactly what DevOps does: It re-imagines Ops, and it re-imagines Dev," explained Dr. Nicole Forsgren, CEO and Chief Scientist at DevOps Research and Assessment (DORA), in an interview with me last September for The New Stack Makers podcast. DORA produces the annual State of DevOps report, which is published by Puppet, and Dr. Forsgren is a principal author.
"Look at what Ops was 15 years ago," Forsgren continued. "Many Ops professionals -- who many times were called 'sysadmins' -- were doing very little coding, or what they would consider Dev work. . . even though it was a lot of scripts. Now, many of them are realizing that they should self-identify as developers. Now, look at what developers are doing in the DevOps world. They are necessarily needing to understand Ops, in order to build systems that scale and applications that can handle distributed systems, and many other things.
"So I don't think the reason we do DevOps is so that Dev and Ops, and everyone in-between, collaborate better. I think we do DevOps so we can deliver value to organizations."
If the final act at the end of the DevOps pipeline is indeed a customer transaction, then inevitably, that pipeline must span the organization. It is no longer essentially a software lifecycle or delivery model, but the main program with which an efficient organization communicates with its partners and customers.
Read also: Harnessing AI to make DevOps more effective
Our journey through Middle Ground brings us to a place near "The Shire," where a certain other unexpected journey began. We're now at "The Source," the beginning of the pipeline of automation. From here we can look out across a valley that appears to lead us on a clear path toward our goal, but which hides a canyon that has been the foil of many travelers: the Culture Chasm.
From there, things will get darker and spookier. DevOps seems easier to envision and perhaps to achieve from the greenfield of "The Shire." At the other end of the pipeline -- which we now know to be a loop -- is the nexus of all business processes that have defined the organization since its inception. If embracing DevOps means breaking all ties with the old world of mechanized workflows, it's here at "The Eye" where our entire perspective will shift, and our lofty goals will find themselves imperiled.
The ring goes south
In the next leg of our journey, we'll begin the long trek down the Automation Pipeline. It begins simply enough with the means to encode and repeat the processes with which software is delivered. But the closer it gets to "The Eye," as we discover in Part III, the more its aims are diverted toward behavior modification and operant conditioning.
Then for Part IV, we'll follow the journey of one prominent DevOps analyst who left his post to join an organization that might otherwise have been his client, working with it on the inside. His mission is the same, but now he's personally invested in its outcome.
Until we set forth again, hold tight.
Journey Further -- From the CBS Interactive Network
- " DevOps: The smart person's guide" by James Sanders, TechRepublic
- " DevOps is a waste of your time if you aren't fully committed" by Conner Forrest, TechRepublic
- " Time to move on from DevOps and continuous delivery, says Google advocate" by Joe McKendrick, Service Oriented
Elsewhere
- " Chef Offers Habitat, a New Kind of Application Automation" by Scott M. Fulton, III, The New Stack
- " Quickstart guide to DevOps" by Cameron Laird, Correlsense
- " The quickie guide to continuous delivery in DevOps" by Pam Baker, HPE enterprise.nxt
Related stories
- Searching for the perimeter in cloud security: From microservices to chaos
- Micro-fortresses everywhere: The cloud security model and the software-defined perimeter
- Microservices and the invasion of the identity entities
- Machine learning and the spectre of the wrong solution
The map of Middle Ground for this series of Scale was drawn by Katerina Fulton.