Facebook's mobile device lab lets it test code on thousands of phones at once
There are tens of thousands of different mobile devices on the market, and Facebook wants its apps to run smoothly on all of them -- with every combination of operating system and network connection. Given that the social network is constantly pushing out new code, that's a daunting task.
To address this challenge, Facebook has built a mobile device lab, along with a custom-built rack system, in its Prineville, Oregon data center. With 60 racks that hold 32 phones each, the company is testing thousands of phones simultaneously, comparing new builds of an app -- including the Facebook app, Messenger, and Instagram -- to the previous version. They're specifically checking for any regression in performance or power.
At the Prineville data center on Tuesday, Facebook production engineer Antoine Reversat explained that Facebook is not just thinking about its existing 1.5 billion users, who are increasingly logging in via their phones. They're also thinking about the billions of people around the planet coming online for the first time.
"Our thinking was to get the emerging markets and get the regularly used phones here," Reversat said, explaining how the small team running the mobile lab chooses which phones to test. Along with phones that are popular in either emerging markets or the United States, the team tests high-end phones.
Each rack has a wireless access point and a camera, allowing Facebook engineers to get a visual understanding of how an updated app is impacting a phone. The racks hold 32 phones each because that's the number that will run consistently on one Wi-Fi network.
Reversat's team couldn't rely on the same racks it uses for servers in the data center because the Wi-Fi signal from one rack would interfere with the signal from the next. Consequently, they built their own racks, outfitted with insulation foam and copper tape on the inside seams so they can function as electromagnetic isolation chambers.
To test Apple devices, a rack is equipped with eight Mac Minis, connected to four iPhones each. They drive the phones to install, test, and uninstall apps. For Android testing, Facebook uses four OCP Leopard servers per rack, which are connected to eight Android devices each.
Also see: CNET: Facebook stocks data center with phones from yesteryear
Reversat's team also wanted consistency on the software side, so they built tools -- or rather, they build "recipes" using the configuration management tool Chef -- that keep phones in a consistent state through tests.
Looking ahead, Reversat said the team wants to double the density of the racks, enabling each one to hold 64 phones. That will come with some challenges, such as maintaining a four-foot air gap between the phones and the wireless access point to ensure the signal is sufficiently attenuated. Also, holding more phones will be difficult since the team wants to test larger form-factor phones next year. Meanwhile, the team also wants to see its Chef recipes better support iOS.
Eventually, Facebook wants to open source both the hardware design of its device testing rack and the Chef recipes it uses to control the phones.