Does anyone really know what time it is? Facebook does
(Image: Shutterstock)
We rarely think about keeping time. As Chicago sang in the 60s, "Does anyone really know what time it is? Does anyone really care?" But, technically, we care a lot.
Open Source
Our technology can't work worth a darn if the Network Time Protocol (NTP) isn't keeping our servers and PCs coordinated with one another. Without it, backups would fail, financial transactions would go awry, and many fundamental network services wouldn't work. To help with these issues, Facebook started making its internet clocks more accurate in 2020. Now, the social media giant is open-sourcing its technology, Time Appliances Project (TAP), and enabling anyone to turn a commodity server into a reliable NTP time appliance.
Today, Facebook uses a multi-layered time server architecture to keep time. It's made up of satellites with precise atomic clocks at the top. Facebook's own atomic clocks sync with one of these, forming the second layer.
These atomic clocks and their brethren make up the internet's primary time-keepers: Stratum-0 devices. Below those are the stratum-1 timekeepers, which bridge the gap between atomic clocks and networked computer clocks. These then connect with everything else needing accurate timekeeping using NTP.
Facebook -- instead of relying on ntpd, NTP's widely used reference implementation to synchronize clocks -- uses chrony, another NTP implementation of the NTP, and Extended NTP. To be exact, Facebook is using the IEEE 1588 Precision Timing Protocol (PTP) to create a PTP profile for data center applications and network infrastructure. The bottom line is that Facebook times are in the more accurate microseconds instead of the more commonly used milliseconds.
Now, you may think: "Who cares?" Cue that song again. But, actually, you should care.
As Oleg Obleukhov, Facebook's TAP overseer, pointed out: "For your microwave, really, it doesn't matter that much. But time precision, on a day-to-day basis, is extremely important for other services. A one-second difference doesn't make much of a difference for a human, but imagine all the transactions that can happen in a data center in the same amount of time."
Today, most companies rely on public NTP pools such as time.facebook.com to act as their stratum clocks. But, when your internet connection goes down, so does your ability to keep time with all your other remote servers and services.
To remove these dependencies, Facebook is releasing a new dedicated piece of Open Compute Project hardware: The Time Appliance. This consists of a global navigation satellite system (GNSS) receiver and a miniaturized atomic clock. With this, even without the internet or a reliable GNSS connection, your machines will keep accurate time.
Now, there's nothing new about this idea. There have been many proprietary time appliances. But they are "proprietary." They have tended to be expensive, hard to maintain, and even harder to keep secure. By open-sourcing TAP, the new Facebook board avoids these problems.
With the Time Appliance card, Facebook has found it can, even without a GNSS connection, keep accurate time within 1 microsecond per 24 hours.
So, all you need do is install its time card on any ordinary x86 computer with a free PCIe slot running Linux kernel 5.15 or newer since it includes a time card driver. You can also build it from code with the Linux kernel 5.12 or newer.
The Time Appliance Project GitHub repository includes the specs, schematics, mechanics, bill of materials (BoM), and the source code. In short, if you can deal with printing the printed circuit board (PCB) and soldering teeny-tiny components, you can build your own time card for a fraction of the cost of a regular-time appliance.
Or, if you're a bit of a klutz -- guilty -- Facebook has worked with Orolia to offer pre-made time cards, as well as NVIDIA, which is selling the precision timing-capable ConnectX-6 Dx and the precision timing-capable BlueField-2 DPU.
The Open Time Server specification gives you all the information you need on how to combine the hardware and software into your own Time Appliance. Enjoy!
Related Stories: