We are closer than ever to the long-sought vision of the "lights-out" data center, in which applications are automatically deployed and provisioned, data is securely stored in its proper places, and workloads are balanced without muss or fuss. Almost paradoxically, enterprises need more people than ever to assure increasingly sophisticated services are delivered, and align with the needs of the business.
This means both opportunities and challenges for the people in charge of today's and tomorrow's data centers. New roles are emerging, while others are seeing tasks -- such as server maintenance, scheduling, monitoring, maintenance, application delivery, and security -- being assumed by automation engines. There are even robots that can handle onsite hardware maintenance.
The Covid crisis boosted the use of automated tools and systems in data centers. This was especially important at a moment when data center staff may not have been able to come to the office, and thus began to rely more on remote access to monitor and address any issues that arise. "The pandemic has rapidly sped up many multi-year digital transformation and modernization projects," says Chris Pomeroy, VP of solutions architecture for Syntax. "Automation is necessary because manual management has been made increasingly challenging due to new data-center infrastructure and more dispersed IT teams.
While data center automation will keep accelerating, it's not clear how rapidly this is happening and the impact on existing on-premises data centers. A survey of 500 IT executives conducted by INAP finds that 85% of IT professionals anticipate their data centers will be close to full automation within the next four years. Two-thirds even go as far as to say on-premises data centers will be close to non-existent. Still, it may take time for all this to happen, and another survey from the Uptime Institute suggests that "the enterprise data center is neither dead nor dying." Workloads are moving to public cloud sites at a measured pace, with nearly two-thirds of IT workloads expected to remain in on-premises data centers through the next two years.
"The automation wave isn't likely to result in massive job displacement," says Mike Bushong, VP of data center product management at Juniper Networks. "Automation isn't about reducing staffs so much as it is about enabling profitable growth. You have to develop the new thing while still doing the old thing, so cost goes up in the short term. So while people talk about reducing staffs by 30% and data center engineers are certainly anxious about the implication, the soundbite travels faster than the reality for most companies."
Also: The main beneficiaries of AI success are IT departments themselves
What does this mean for the roles of data center managers and professionals? With the emergence of AI and machine learning to support data center operations, and increasing amounts of workloads shifting to the cloud, there will be fewer mundane tasks to worry about, thus freeing up more of time and energy for more business-level problems. "Anything repetitive will be automated," says Dr. James Stanger, chief technology evangelist for CompTIA. "This doesn't mean data center jobs are dead. Successful IT workers are morphing along with automation. This will make data centers far more efficient. Jobs will get better, too."
Data center automation "will refocus teams on a different set of tasks," says Bushong. "For example, converged technology stacks like hyper-converged infrastructure or Amazon Outposts will reduce the need to architect some parts of the infrastructure. Stronger operational capabilities that come with solutions should reduce the burden on network engineering. Abstracted and intent-based control should reduce the reliance on vendor-specific workflows. Orchestration and lifecycle management tools should simplify day-to-day operations."
Still, demand for data center managers and professionals will remain strong, at least in the short term. "The data center staffing crisis is getting worse," the Uptime Institute survey's authors report. The number of managers stating they are having difficulty finding qualified candidates has risen in recent years -- from 38% in 2018 to 50% in the most recent survey, conducted in mid-2020. Jobs "that focus on tasks that are easily scheduled would be at risk," agrees Jennifer Curry, senior VP of product and technology for INAP. "But I think automation actually creates a need for different jobs based on the addition of software into many functions that were once manual."
To be successful in data center roles today, data center managers and professionals "need to be critical thinkers, fast learners, and adept at maintaining situational awareness," says John O'Connor, manager of technology infrastructure operations at Bloomberg. "Problems occur less often, but tend to scale quickly and are more difficult to triage and remediate. They need to keep a massive amount of context in their heads and be able to leverage all the telemetry and IoT data available to them to derive the problem scope and evaluate potential solutions quickly."
Companies that have made big moves to the cloud "have typically found other roles for people working in data centers that did easier things, such as changing tapes, hard drives, or working with physical equipment," says Steve Jones, DevOps advocate at Redgate Software. "The DBAs still have to work with the databases, though their role has changed. They don't do some of the hardware-related tasks or patching, but they do need to manage security, check on performance, many of the similar tasks."
The traditional roles of a system administrator, virtualization engineers, and storage engineers "are being replaced by more DevOps roles," says Pomeroy. "Engineers today need to become familiar and proficient with automation tools such as Ansible, Puppet, Chef, or Salt. These tools will continue to be heavily utilized in the future with most of the day-to-day administrative tasks being done through automation."
Also: 81% of IT pros expect most data center and networking tasks will be automated by 2025 (TechRepublic)
Simply put, "data center roles such as tape jockeys and button pushers are rapidly disappearing, while staff who can leverage Python and query databases to form decision-making tools are rapidly becoming more valuable," says O'Connor. "Data centers do more than house data for others. They create an enormous amount of data, and the staff who manage them must adapt to use it. While there is room for specialists in particular technical or process areas, generalists make this world go around."
There will be opportunities for upward advancement in businesses as well. "People with years of experience in production data center environments -- in other words, customer-facing workers -- will become managers and strategists for hybrid environments," Stanger says. "We need people who know physical and logical security to work with programmers and AI, machine learning and robotics folks to create new solutions. Those same people will morph to become those who manage the robots and automated processes."
For what kinds of jobs should current data center professionals be preparing? Stanger points to the rise of hybrid environment management: "Today's data center workers do more than just administer servers," he points out. "Integration is critical; data center workers are asked to manage on-premises, data center, and cloud environments. This means individuals need to learn integration platforms and software, such as Eucalyptus, Bamboo, and OpenStack. They'll also need to understand ITIL-like process management."
Technology managers and professionals "should continuously improve their skills, and that is even more important as organizations move away from on-premises data centers to cloud offerings like Azure, AWS or Google," says Kathi Kellenberger, Simple Talk Editor at Redgate Software. "Database administrators, for example, need to learn about the cloud database offerings, how to migrate data, working in a hybrid environment, security differences, scripting, and differences in database management."
The shift to higher-level tasks "means allowing that person to do different things, ideally that are more closely connected to the business," Bushong says. "Application and infrastructure teams will need to coordinate to deliver applications and services. Infrastructure people are uniquely positioned to move upstack, closer to the applications. That's the opportunity."
The shift to AI-driven computing is also having a significant impact on data centers, and this is also an important career course for data center professionals. "Data center professionals can prepare for AI computing transition by increasing their knowledge of AI and machine learning as applied to application use cases to enable the organization to generate new services and products to gain competitive advantage in the market," says Rodrigo Liang, co-founder and CEO of SambaNova. "Unlike other technology trends that existed to drive down costs, AI represents a huge opportunity to generate value -- and it is being driven by all the data being generated by every computer and sensor on the planet."
The key in this emerging environment "is to understand the complex interactions of the people, processes and technologies in data centers," says O'Connor. "CIOs tend to have a top-down perspective, but data center managers often think bottom-up. To succeed, you must be able to zip up and down that continuum. Data centers are places where many divergent agendas meet physical reality. Automation, like capital spending, is neither inherently good nor bad. You need to know what matters to your business and find ways to help add value. The dynamics of leading people through constant change never goes out of style and so always leads to opportunity.
In the end, it's important to remember that every enterprise needs a highly functioning and well-aligned data center, no matter where it happens to be located, or how many people are running it. "The idea that the cloud is someone else's computers is fundamentally missing the point," says Bushong. "It's not the infrastructure that makes cloud important, it's the operations. A lot of the operational transformations that are underway -- from intent-based networking to DevOps -- are not merely about speed. In complex environments, the real problem is reliability. By developing operational practices that improve reliability, data center operators can achieve the useful benefits of speed and efficiency."