How Fortnite approaches analytics, cloud to analyze petabytes of game data

Featured stories

Fortnite processes 92 million events a minute and sees its data grow 2 petabytes a month. With every season of Fortnite, parent Epic Games ingests more data from game clients, servers and services.

And when you have the most popular game in the world, you need an analytics architecture to match.

Chris Dyl, director at platform at Epic Games, outlined the company's analytics architecture and how it has built its system on Amazon Web Services.

Dyl, speaking at the AWS Summit in New York, outlined how Epic has moved to be all-in on AWS as well as extend usage via machine learning tools such as Amazon SageMaker, which has a bevy of built-in algorithms for developers to use. Dyl also highlighted how the company thinks about analytics for Fortnite.

The Fortnite slides from Dyl tell the tale.

Epic is an interesting case study because it has more than 125 million of Fortnite users, millions of players concurrent and telemetry data used for analytics, KPI tracking and product improvements. In addition, Epic's Unreal Engine is widely used for everything from game creation to content to enterprise applications.

"We have stretched capacity to the limits," said Dyl. He noted that Fortnite has grown 100x in recent months. Indeed, Epic has seen outages as Fortnite has grown and the company has used post mortems of those incidents to refine its architecture.

One recent event in Fortnite revolved around launching a giant rocket ship in the game. The company invited 125 million people to participate at the same time. Akamai said Fortnite set a game traffic record on its network July 12 with 37 terabytes per second delivered across its platform. See GameSpot's Fortnite coverage and learn how to play Fortnite over on Download.com.

Special feature

Special report: Sensor'd Enterprise: IoT, ML, and Big Data (free PDF)

This ebook, based on the latest ZDNet/TechRepublic special feature, explores ways IoT is improving operations and delivering business value to enterprises around the world.

Epic's analytics architecture has multiple moving parts and micro services, but generally speaking Dyl said everything is stored in AWS S3. "It's a real-time pipeline" that integrates everything from S3 to Spark to scores to telemetry data to Tableau and SQL.

"We use the data for everything from ARPU to game analysis and improvements," explained Dyl.

There are a few key themes that emerge from Epic's architecture:

S3 is used as a data lake for Epic. Dyl used the data lake term during his talk. Meanwhile, AWS CTO Werner Vogels noted the same thing. "S3 started as storage for the Internet, but since the rise of analytics in an organization S3 is growing quite dramatically," said Vogels in an interview. "We are all talking about petabyte datasets."
Architecture is critical. A company like Epic--like other gaming companies--provide good lessons for enterprises. Why? One million customers can show up at a first day product launch. Growth isn't organic in many cases and can be just one big burst. Vogels noted that AWS' architecture team gets involved with customers like Epic early. "We make sure everything is tested with best practice from day one and have the solutions architects involved. We also learn where the customers sensitivities are," said Vogels. There's also something to be said for an architecture that can serve as a base for higher level functions. "Analytics may not be something a company like Epic thinks about from day one," he said.
MapReduce and open source tools for big data are critical. Note that Spark is front and center in Epic's architecture.
Think micro services. Dyl said in his talk that Epic is looking to improve the overall management of micro services, but Fortnite is built in terms of small functions whether it's the leaderboard, stats or the analytics used to improve the experience and game.

ZDNET'S MONDAY MORNING OPENER

The Monday Morning Opener is our opening salvo for the week in tech. Since we run a global site, this editorial publishes on Monday at 8:00am AEST in Sydney, Australia, which is 6:00pm Eastern Time on Sunday in the US. It is written by a member of ZDNet's global editorial board, which is comprised of our lead editors across Asia, Australia, Europe, and the US.

The tech that changed us: 50 years of br... SEE FULL GALLERY