Inside CERN's datacenter helping the Large Hadron Collider probe the Big Bang: Photos

1 of 10
  • Entering the datacenter

    Entering the datacenter

    Beyond these unremarkable doors lies the computing infrastructure that deals with data from the millions of particle collisions per second generated by the Large Hadron Collider.

    The collider's circular tunnel, crossing between France and Switzerland at 50m to 175m below the surface, is home to seven experiments.

    The two largest, Atlas and CMS, found the Higgs boson particle in 2012. Just last month the LHCb experiment, which is designed to look at the asymmetry between matter and anti-matter, discovered a new particle called the pentaquark.

    Travelling in opposite directions around the 27km ring at close to the speed of light, two beams each contain up to 476 bunches of 100 billion protons, creating collisions every 50 nanoseconds.

    With the raw data per event at about one million bytes, or 1MB, produced at a rate of about 600 million events per second, filtering has to take place to avoid the experiments being swamped with irrelevant data.

    That still leaves the datacenter with the task of dealing with approximately 1.05GB per second.

    Photo by: Toby Wolpe/ZDNet

    Caption by: Toby Wolpe

  • The main data suite

    The main data suite

    Here's the CERN datacenter's cavernous main data suite. At 1,450m2, it is the largest of four. The second largest is only slightly smaller at 1,200m2, followed by two others at just over 200m2 apiece.

    Together, they offer at total capacity of 3.5MW, upgraded from 2.9MW in early 2013.

    With about 30 petabytes of data to sift through annually, CERN doesn't rely solely on its own on-site facilities to process the information for significant results.

    Photo by: CERN

    Caption by: Toby Wolpe

  • The Worldwide LHC Computing Grid

    The Worldwide LHC Computing Grid

    CERN's datacenter contains about 11,000 servers with over 100,000 processor cores. But it shares the task of processing data from the Large Hadron Collider with datacenters around the world.

    In 2002 CERN set up the Worldwide LHC Computing Grid, so that a wide international community of more than 8,000 physicists could exploit its data through near real-time access to the information.

    This tiered distributed computing infrastructure, which connects thousands of computers and storage systems in more than 170 centres across 41 countries, builds on the technology of the World Wide Web, which was invented at CERN in 1989.

    Photo by: Toby Wolpe/ZDNet

    Caption by: Toby Wolpe

  • ​The tier structure

    ​The tier structure

    CERN's datacenter is Tier 0 of the Worldwide LHC Computing Grid's (WLCG) four levels. Each tier consists of a number of datacenters and provides specific services. Between them the tiers process, store and analyse data from the Large Hadron Collider (LHC).

    While CERN itself may be Tier 0 and acts as the gateway through which all Large Hadron Collider data passes, it provides less than 20 percent of the Grid's total computing capacity.

    CERN's role is to store all the raw data and conduct its first reconstruction into meaningful information.

    Tier 1 is made up of 13 large datacenters, which support the Grid and store a share of the raw and reconstructed data, reprocess it and distribute to the Tier 2 sites.

    Photo by: CERN

    Caption by: Toby Wolpe

  • Data storage

    Data storage

    Here's a view down onto one of the closed aisles between the servers.

    In the background, you can see the disk storage cabinets.

    Although the datacenter has more 45 petabytes of data storage on disk, it also uses tape storage extensively.

    Raw experimental data is held on magnetic tapes, partly because of the lower running costs but also because of what CERN sees as their greater security and longevity, and the less significant consequences of failure, compared with disks.

    Photo by: Toby Wolpe/ZDNet

    Caption by: Toby Wolpe

  • Tech operations control

    Tech operations control

    Here's the control suite, situated to one side of the main data hall, where technical operations staff can monitor events inside the datacenter.

    Photo by: Toby Wolpe/ZDNet

    Caption by: Toby Wolpe

  • The secondary site

    The secondary site

    As well as the datacenter's main site at Meyrin, near Geneva in Switzerland, CERN also shares some computer operations with the Wigner Research Centre for Physics in Budapest, Hungary.

    This site functions as an extension to the datacenter and also performs the role of a Tier 0 location for the grid. Wigner also acts as a disaster-recovery facility for CERN's critical systems.

    The Wigner datacenter, which is linked to CERN by two dedicated and redundant 100Gbps lines, offers more than 20,000 cores and 5.5 petabytes of disk storage.

    Photo by: CERN

    Caption by: Toby Wolpe

  • Non-LHC infrastructure

    Non-LHC infrastructure

    The main data suite at the CERN datacenter is where experimental data arrives from the Large Hadron Collider, ready for access by the Worldwide LHC Computing Grid.

    Along with servers and storage for data analysis for this site, which acts as the grid's Tier 0, the datacenter hosts systems used in the day-to-day running of the whole CERN complex, which runs a large number of physics experiments that are not directly related to the Large Hadron Collider.

    Photo by: CERN

    Caption by: Toby Wolpe

  • Internet eXchange Point

    Internet eXchange Point

    One part of the CERN datacenter's main data suite is allocated to the carrier-neutral Internet eXchange Point.

    It carries some historical significance as the point through which the first pan-European Internet backbone was established in 1989.

    Photo by: Toby Wolpe/ZDNet

    Caption by: Toby Wolpe

  • Cooling systems

    Cooling systems

    Here's a view of the datacenter's roof. Beyond the rather rusty-looking ducting at the nearest edge of the roof, a platform carry the cooling plant that keeps the thousands of servers from overheating.

    Photo by: Toby Wolpe/ZDNet

    Caption by: Toby Wolpe

1 of 10

As well as acting as the hub for a grid of scientific institutions around the world, CERN's datacenter has to cope with vast amounts of raw data from its particle physics experiments.

Read More Read Less

Entering the datacenter

Beyond these unremarkable doors lies the computing infrastructure that deals with data from the millions of particle collisions per second generated by the Large Hadron Collider.

The collider's circular tunnel, crossing between France and Switzerland at 50m to 175m below the surface, is home to seven experiments.

The two largest, Atlas and CMS, found the Higgs boson particle in 2012. Just last month the LHCb experiment, which is designed to look at the asymmetry between matter and anti-matter, discovered a new particle called the pentaquark.

Travelling in opposite directions around the 27km ring at close to the speed of light, two beams each contain up to 476 bunches of 100 billion protons, creating collisions every 50 nanoseconds.

With the raw data per event at about one million bytes, or 1MB, produced at a rate of about 600 million events per second, filtering has to take place to avoid the experiments being swamped with irrelevant data.

That still leaves the datacenter with the task of dealing with approximately 1.05GB per second.

Caption by: Toby Wolpe

1 of 10

Related Topics:

Big Data Analytics Data Centers Digital Transformation Robotics Internet of Things Innovation

Related Galleries

    • 1 of 3