For many organizations, the last month has been spent figuring out the best way to get teams comfortable working from home. But not everyone involved with tech can carry on doing their jobs remotely, and that includes the workers who are keeping the digital world turning by keeping data centers up and running.
We are all even more reliant on online services to work and socialize while thelockdowns continue. And while it's easy to talk about all these services living in the cloud, they also have a more prosaic physical existence. Data centers typically come across as large, industrial, warehouse-like boxes; but few are aware of the fact that they are also the buildings that are keeping us connected, working and entertained.
And to keep operating, data centers need on-site staff to maintain the infrastructure. "People see data centers as large rooms of servers, but they don't realize it actually takes a certain amount of folks to maintain them," Chris Yetman, chief operating officer at Vantage Data Centers, told ZDNet. "Sure, you may need fewer people overall, but you can't have no one."
Yetman's data centers serve customers in the US, Canada and Europe, and his largest campus is in California. Early on, he explained, he started asking his staff to stay at home for a couple of weeks if they thought they'd been exposed to the virus. As the crisis escalated, Yetman switched all of his non-essential team to remote working. "But we determined pretty quickly that we could not send home engineers who work on the facilities," said Yetman. "And so we focused on making it as comfortable for them as possible."
Yetman ordered personal protective equipment (PPE) and set up sanitizing stations everywhere he could, both to minimize the risk of infection and to reassure his employees. "We've gone through and sanitized everything," he said. "You have to make people who have to work comfortable that they're going to be okay."
Fingerprint readers started coming with disinfectant or sanitizing wipes, and hand-washing stations were built near the entrances of the building; and the offices have already received two deep cleans so far. In parallel, Yetman and his team re-arranged the way shifts are worked to reduce the amount of time that employees have to work side by side, and to keep the same staff in the same buildings to lower the chance of cross-contamination. Again, risk zero proved impossible: some maintenance works require more than one person, not the least for safety. "So we still have to manage that, but we're trying to make sure we're good about how we manage it," said Yetman.
It's not an easy job trying to keep a key piece of infrastructure operational during a global pandemic. But with society at large depending on the internet, more so now than ever before, there aren't that many options, either. "You can't just stop," said Yetman. "You can't just say you're going to turn it off."
A data center is involved every time we read a post on social media or order our shopping from a supermarket; but also every time a bank processes a payment, a government delivers an online service or a teacher runs a Zoom class for students stuck at home.
"A lot of people think they are big sheds doing nothing," Emma Fryer told ZDNet. She is the associate director of the tech industry group techUK, and her focus is on data centers. "I think they get forgotten," she said.
They get so forgotten, in fact, that until last month, data centers still hadn't made it on to the UK government's list of sectors defining key workers – and very few of the buildings in the country were formally designated as Critical National Infrastructure (CNI). As a result, operators grew increasingly worried that they wouldn't be able to access their place of work, especially as lock-down became an increasingly likely prospect.
Fryer explained that the government's failure to keep up-to-date is probably simply down to the low-profile nature of data centers, and the fact that they are not really understood; and so, from early March, she started drawing attention to the issue.
"We lobbied the department for digital, culture, media and sport (DCMS) furiously," said Fryer. "We were very stern. And amazingly, the government's response was swift and decisive. They advocated internally, since this decision is made within the Cabinet Office, and within 24 hours they had got us onto the list of key workers."
In addition, DCMS set up a dedicated task force within the department to liaise with data center staff, and has partnered with Fryer's team within techUK to hold weekly calls with operators, during which they discuss policies and procedures, and share useful tips and tricks.
Fryer's feedback from those calls is that overall, and for the time being, the sector seems pretty confident that it will be able to cope with the crisis, at least from an operational perspective. Data centers are, by nature, highly risk-averse, and many operators had been keeping a pandemic response plan in the back of their business continuity planning. What's more, unlike many governments and organizations, the vast majority of data center COOs started blowing the dust off those disaster recovery plans as soon as they started hearing health experts ringing alarm bells at the emergence of a new, unknown type of coronavirus.
Operators were weeks, if not months ahead of government requirements, stressed Fryer. From segregating shifts to stockpiling sanitizers, precautions were taken earlier on to mitigate the risk of a crisis. Plus, the wider sector benefited from the feedback of multinational data center providers who were able to share insights into what was happening in their own buildings in countries worst-affected by the virus, like China or Italy. Equipped with hands-on advice on how to manage the outcomes of the crisis, data center operators in turn were able to plan ahead, and avoid being taken by surprise.
Jack Bedell-Pearce, the CEO of UK-based 4D Data Centers, explained that as an operator, early preparation is pretty much a standard response. "Companies entrust us with running their business-critical services, so we are constantly paranoid about all kinds of risks," he told ZDNet. "When you've been operating a data center for as long as I have, you can see how quickly things can go wrong. So early preparation is culturally baked into me, and into all the members of my staff."
In January, therefore, Bedell-Pearce sat down and methodically listed out all the risks associated with a pandemic, and how he would be able to mitigate them. This, combined with regular chats with fellow data center operators to discuss issues ranging from sanitizing to communicating with customers, means that he has constantly been on top of things.
"When it became apparent that the coronavirus was going to be more than a flash in the pan, we were already ordering hand sanitizers and masks," he explained. "By the second week of February, we had reviewed and updated our staff segregation policy, which is essentially a precursor to social distancing."
Like most other data center operators, Beddell-Pearce has now sent all non-essential staff back home to work remotely. For business-critical workers, he is maintaining separate shifts to avoid movement of staff, and has implemented virtual handover between shifts.
Also problematic for operators is the need to restrict customers' visits on-site, even if technical management, installation or troubleshooting of equipment is necessary. Bedell-Pearce, for his part, has specified that customers should only visit facilities by appointment; and to avoid unnecessary visits, he has implemented unlimited free daily remote hands-on requests for all clients. Server racking, replacement of components or cable patching operations can be carried out by the Bedell-Pearce's own technicians who are already on-site.
Similar efforts to encourage remote management have been deployed by other companies such as EfficiencyIT, Kao Data and Equinix. Equinix has even stated that only service-critical customers with government-designated critical infrastructure status and a special permission will be allowed on-site; all other clients will have to resort to remote access.
Bedell-Pearce explained that as far as he is concerned, it is mostly business as usual. "Operationally, in terms of the core services we offer, things haven't changed much," he said. "We continue to provide power to our customer servers, we continue to provide cooling, security and a connection. We've only put some restrictions to minimize the risk of things going wrong."
That is not to say that operators are entirely relaxed about the weeks to come. Most staff have now spent a couple of months settling in to the new routines required by health and safety measures, and no major failure has been reported as of yet; but the uncertainty, particularly surrounding the potential length of the crisis, is likely to bring about new challenges in the longer term.
High-ranking on the list of worries for operators is the risk of staff getting ill. The engineers needed to run the buildings are highly skilled, and it is likely that their skills cannot be replaced easily should they come down with the virus. An outbreak could have far-reaching consequences, especially in the case that operators don't have another team at hand to cover the work of business-critical staff.
Another issue has to do with the supply chain, which in the case of data centers often involves sourcing equipment from various different countries – including high-risk areas. Northern Italy, for instance, is a big supplier of uninterruptible power supply components, as well as of the engineers that supervise the final commissioning of the systems. Needless to say, some facilities were adversely affected as the country had to go on lockdown last month; and operators are growing nervous as governments struggle to determine when to relax confinement rules.
If the pandemic is to last longer than a few months, which is what most forecasts anticipate, some problems may also arise in the construction of new data center capacity. With users only increasing their consumption of streaming services, cloud or gaming, it is almost inevitable that more infrastructure will be needed at some point.
Vantage Data Center's Chris Yetman has identified construction as a huge challenge coming at him. With hundreds of workers "crawling around" building sites, he said, social-distancing rules are hard to implement, and even harder is to ensure the safety of staff. Yetman's sub-contractors have implemented temperature checks, and supply PPE and washing stations, but the full impact of the pandemic on building schedules is still unclear.
Still, a recent report from CBRE, a real estate services and investment firm, on the state of data center construction in Europe, showed that existing facilities should be able to cope with a surge in demand for the foreseeable future. Mitul Patel, head of EMEA at CBRE and who led the report, told ZDNet that the sector is optimistic about its ability to deliver new capacity.
"What we've found over the past year is that cloud providers have much more capacity than they usually require," said Patel. "Companies will bake into their procurement the fact that they may have a really heavy day sometimes. People around the world aren't struggling to get on Netflix or Zoom just yet, because those companies have planned for high-use levels."
Patel forecasted that European data centers are set to keep running for the next 12 to 18 months. London, in particular, has built up considerable space in the past year, which will "keep the world going" for a significant amount of time.
It seems that the data centers that power our digital lifestyles will keep running smoothly for now. But perhaps it's time to reconsider out attitude toward these facilities, too; behind the anonymous exteriors there are people working to keep the systems we rely on up and running.