X

Inside CERN's datacenter helping the Large Hadron Collider probe the Big Bang: Photos

As well as acting as the hub for a grid of scientific institutions around the world, CERN's datacenter has to cope with vast amounts of raw data from its particle physics experiments.
By Toby Wolpe, Contributor
cerndsentranceangledjul2015.jpg
1 of 10 Toby Wolpe/ZDNet

Entering the datacenter

Beyond these unremarkable doors lies the computing infrastructure that deals with data from the millions of particle collisions per second generated by the Large Hadron Collider.

The collider's circular tunnel, crossing between France and Switzerland at 50m to 175m below the surface, is home to seven experiments.

The two largest, Atlas and CMS, found the Higgs boson particle in 2012. Just last month the LHCb experiment, which is designed to look at the asymmetry between matter and anti-matter, discovered a new particle called the pentaquark.

Travelling in opposite directions around the 27km ring at close to the speed of light, two beams each contain up to 476 bunches of 100 billion protons, creating collisions every 50 nanoseconds.

With the raw data per event at about one million bytes, or 1MB, produced at a rate of about 600 million events per second, filtering has to take place to avoid the experiments being swamped with irrelevant data.

That still leaves the datacenter with the task of dealing with approximately 1.05GB per second.

cerndsfisheyejul2015.jpg
2 of 10 CERN

The main data suite

Here's the CERN datacenter's cavernous main data suite. At 1,450m2, it is the largest of four. The second largest is only slightly smaller at 1,200m2, followed by two others at just over 200m2 apiece.

Together, they offer at total capacity of 3.5MW, upgraded from 2.9MW in early 2013.

With about 30 petabytes of data to sift through annually, CERN doesn't rely solely on its own on-site facilities to process the information for significant results.

cerndsgridserversjul2015.jpg
3 of 10 Toby Wolpe/ZDNet

The Worldwide LHC Computing Grid

CERN's datacenter contains about 11,000 servers with over 100,000 processor cores. But it shares the task of processing data from the Large Hadron Collider with datacenters around the world.

In 2002 CERN set up the Worldwide LHC Computing Grid, so that a wide international community of more than 8,000 physicists could exploit its data through near real-time access to the information.

This tiered distributed computing infrastructure, which connects thousands of computers and storage systems in more than 170 centres across 41 countries, builds on the technology of the World Wide Web, which was invented at CERN in 1989.

cerndsfisheye2jul2015.jpg
4 of 10 CERN

​The tier structure

CERN's datacenter is Tier 0 of the Worldwide LHC Computing Grid's (WLCG) four levels. Each tier consists of a number of datacenters and provides specific services. Between them the tiers process, store and analyse data from the Large Hadron Collider (LHC).

While CERN itself may be Tier 0 and acts as the gateway through which all Large Hadron Collider data passes, it provides less than 20 percent of the Grid's total computing capacity.

CERN's role is to store all the raw data and conduct its first reconstruction into meaningful information.

Tier 1 is made up of 13 large datacenters, which support the Grid and store a share of the raw and reconstructed data, reprocess it and distribute to the Tier 2 sites.

cerndsserversdiskjul2015.jpg
5 of 10 Toby Wolpe/ZDNet

Data storage

Here's a view down onto one of the closed aisles between the servers.

In the background, you can see the disk storage cabinets.

Although the datacenter has more 45 petabytes of data storage on disk, it also uses tape storage extensively.

Raw experimental data is held on magnetic tapes, partly because of the lower running costs but also because of what CERN sees as their greater security and longevity, and the less significant consequences of failure, compared with disks.

cerndscontrolroomjul2015.jpg
6 of 10 Toby Wolpe/ZDNet

Tech operations control

Here's the control suite, situated to one side of the main data hall, where technical operations staff can monitor events inside the datacenter.

cerndsinsideserveraislejul2015.jpg
7 of 10 CERN

The secondary site

As well as the datacenter's main site at Meyrin, near Geneva in Switzerland, CERN also shares some computer operations with the Wigner Research Centre for Physics in Budapest, Hungary.

This site functions as an extension to the datacenter and also performs the role of a Tier 0 location for the grid. Wigner also acts as a disaster-recovery facility for CERN's critical systems.

The Wigner datacenter, which is linked to CERN by two dedicated and redundant 100Gbps lines, offers more than 20,000 cores and 5.5 petabytes of disk storage.

cerndsinsideserverjul2015.jpg
8 of 10 CERN

Non-LHC infrastructure

The main data suite at the CERN datacenter is where experimental data arrives from the Large Hadron Collider, ready for access by the Worldwide LHC Computing Grid.

Along with servers and storage for data analysis for this site, which acts as the grid's Tier 0, the datacenter hosts systems used in the day-to-day running of the whole CERN complex, which runs a large number of physics experiments that are not directly related to the Large Hadron Collider.

cerndsinternetexchangejul2015.jpg
9 of 10 Toby Wolpe/ZDNet

Internet eXchange Point

One part of the CERN datacenter's main data suite is allocated to the carrier-neutral Internet eXchange Point.

It carries some historical significance as the point through which the first pan-European Internet backbone was established in 1989.

cerndsroofcoolingjul2015.jpg
10 of 10 Toby Wolpe/ZDNet

Cooling systems

Here's a view of the datacenter's roof. Beyond the rather rusty-looking ducting at the nearest edge of the roof, a platform carry the cooling plant that keeps the thousands of servers from overheating.

Related Galleries

Holiday wallpaper for your phone: Christmas, Hanukkah, New Year's, and winter scenes
Holiday lights in Central Park background

Related Galleries

Holiday wallpaper for your phone: Christmas, Hanukkah, New Year's, and winter scenes

21 Photos
Winter backgrounds for your next virtual meeting
Wooden lodge in pine forest with heavy snow reflection on Lake O'hara at Yoho national park

Related Galleries

Winter backgrounds for your next virtual meeting

21 Photos
Holiday backgrounds for Zoom: Christmas cheer, New Year's Eve, Hanukkah and winter scenes
3D Rendering Christmas interior

Related Galleries

Holiday backgrounds for Zoom: Christmas cheer, New Year's Eve, Hanukkah and winter scenes

21 Photos
Hyundai Ioniq 5 and Kia EV6: Electric vehicle extravaganza
img-8825

Related Galleries

Hyundai Ioniq 5 and Kia EV6: Electric vehicle extravaganza

26 Photos
A weekend with Google's Chrome OS Flex
img-9792-2

Related Galleries

A weekend with Google's Chrome OS Flex

22 Photos
Cybersecurity flaws, customer experiences, smartphone losses, and more: ZDNet's research roundup
shutterstock-1024665187.jpg

Related Galleries

Cybersecurity flaws, customer experiences, smartphone losses, and more: ZDNet's research roundup

8 Photos
Inside a fake $20 '16TB external M.2 SSD'
Full of promises!

Related Galleries

Inside a fake $20 '16TB external M.2 SSD'

8 Photos