Fortnite processes 92 million events a minute and sees its data grow 2 petabytes a month. With every season of Fortnite, parent Epic Games ingests more data from game clients, servers and services.
And when you have the most popular game in the world, you need an analytics architecture to match.
Chris Dyl, director at platform at Epic Games, outlined the company's analytics architecture and how it has built its system on Amazon Web Services.
Read also: What serverless architecture actually means, and where servers enter the picture | Business analytics: The essentials of data-driven decision-making | What is machine learning? Everything you need to know
Dyl, speaking at the AWS Summit in New York, outlined how Epic has moved to be all-in on AWS as well as extend usage via machine learning tools such as Amazon SageMaker, which has a bevy of built-in algorithms for developers to use. Dyl also highlighted how the company thinks about analytics for Fortnite.
The Fortnite slides from Dyl tell the tale.
Epic is an interesting case study because it has more than 125 million of Fortnite users, millions of players concurrent and telemetry data used for analytics, KPI tracking and product improvements. In addition, Epic's Unreal Engine is widely used for everything from game creation to content to enterprise applications.
"We have stretched capacity to the limits," said Dyl. He noted that Fortnite has grown 100x in recent months. Indeed, Epic has seen outages as Fortnite has grown and the company has used post mortems of those incidents to refine its architecture.
One recent event in Fortnite revolved around launching a giant rocket ship in the game. The company invited 125 million people to participate at the same time. Akamai said Fortnite set a game traffic record on its network July 12 with 37 terabytes per second delivered across its platform. See GameSpot Fortnite coverage
Epic's analytics architecture has multiple moving parts and micro services, but generally speaking Dyl said everything is stored in AWS S3. "It's a real-time pipeline" that integrates everything from S3 to Spark to scores to telemetry data to Tableau and SQL.
"We use the data for everything from ARPU to game analysis and improvements," explained Dyl.
There are a few key themes that emerge from Epic's architecture:
- S3 is used as a data lake for Epic. Dyl used the data lake term during his talk. Meanwhile, AWS CTO Werner Vogels noted the same thing. "S3 started as storage for the Internet, but since the rise of analytics in an organization S3 is growing quite dramatically," said Vogels in an interview. "We are all talking about petabyte datasets."
- Architecture is critical. A company like Epic--like other gaming companies--provide good lessons for enterprises. Why? One million customers can show up at a first day product launch. Growth isn't organic in many cases and can be just one big burst. Vogels noted that AWS' architecture team gets involved with customers like Epic early. "We make sure everything is tested with best practice from day one and have the solutions architects involved. We also learn where the customers sensitivities are," said Vogels. There's also something to be said for an architecture that can serve as a base for higher level functions. "Analytics may not be something a company like Epic thinks about from day one," he said.
- MapReduce and open source tools for big data are critical. Note that Spark is front and center in Epic's architecture.
- Think micro services. Dyl said in his talk that Epic is looking to improve the overall management of micro services, but Fortnite is built in terms of small functions whether it's the leaderboard, stats or the analytics used to improve the experience and game.
ZDNET'S MONDAY MORNING OPENER
The Monday Morning Opener is our opening salvo for the week in tech. Since we run a global site, this editorial publishes on Monday at 8:00am AEST in Sydney, Australia, which is 6:00pm Eastern Time on Sunday in the US. It is written by a member of ZDNet's global editorial board, which is comprised of our lead editors across Asia, Australia, Europe, and the US.
PREVIOUSLY ON MONDAY MORNING OPENER:
- Blockchain and bluster: Why politicians always get tech wrong
- Firefox and the 4-year battle to get Google to treat it as a first-class citizen
- The future of retail: 4 trends transforming how we shop
- Cloud wars 2018: 6 things we learned in the first half
- London's tech startups are booming, but their biggest challenge is just around the corner
- Blockchain: The 2 most important things to understand
- Dissecting ZTE: What it is and what it wants to be
- Windows 10, Android and iOS: Gearing up for the next ecosystem battle
- The future of enterprise IoT: 2 factors to watch
- Digital transformation gut check: Sorry state or natural selection?
- A regulation straitjacket is no less than Facebook and everyone else deserves
- It's time to tame Big Tech: Here's how we get started
- Enterprises learning to love cloud lock-in too: Is it different this time?