Here to tell us more about how the hybrid approach to multiple, complementary cloud instances is meeting SEGA’s critical development requirements in a new way is Francis Hart, Systems Architect at SEGA Europe, in London. The case study interview is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions.
Gardner: Clearly one of the requirements in game development is the need to ramp up a lot of servers to do the builds, but then they sit there essentially unproductive between the builds. How did you flatten that out or manage the requirements around the workload support?
We work on the idea of having a central platform for a lot of these systems. Using virtualization to do that allowed us to scale off at certain times.
Hart: Typically, in the early stages of development, there is a fair amount of testing going on, and it tends to be quite small -- the number of staff involved in it and the number of build iterations.
Going on, when the game reaches to the end of its product life-cycle, we’re talking multiple game iterations a day and the game size has gotten very large at that point. The number of people involved in the testing to meet the deadlines and get the game shipped on date is into the hundreds and hundreds of staff.
Gardner: How has virtualization and moving your workloads into different locations evolved over the years?
Hart: We work on the idea of having a central platform for a lot of these systems. Using virtualization to do that allowed us to scale off at certain times. Historically, we always had an on-premise VMware platform to do this. Very recently, we’ve been looking at ways to use that resource within a cloud to cut down from some of Capex loading but also remain a little bit more agile with some of the larger titles, especially online games that are coming around.
Gardner: We’re all very familiar with the amazing video games that are being created nowadays. And SEGA of course is particularly well-known for the Sonic the Hedgehog franchise going back a number of years. What are some of the other critical requirements that you have from a systems architecture perspective when developing these games?
Hart: We have a lot of development studios across the world. We're working on multiple projects. We need to ensure that we supply them with a highly scalable and reliable solution in order to test, develop, and produce the game and the code in time. ... We’re probably looking at thousands of individual developers across the world.
... The first part was dealing with the end of the process, and that was the testing and the game release process. Now, we’re going to be working back from that. The next big area that we’re actively involved in is getting our developers to develop online games within the hybrid environment.
So they’re designing the game and the game’s back-end servers to be optimal within the VMware environment. And then, also pushing from staging to live is a very simple process using the Cloud Connector.
We're restructuring and redesigning the IT systems within SEGA to be more of a development operations team to provide a service to the developers and to the company.
Gardner: How did you start approaching that from your IT environment, to build the right infrastructure?
Hart: One of the first areas we targeted very early on was the last process in those steps, the testing, arguably one of the most time-consuming processes within the development cycle. It happens pretty much all the way through as well to ensure that the game itself behaves as it should, it’s tested, and the customer gets the end-user experience they require.
The biggest technical goal that we had for this is being able to move large amounts of data, un-compiled code, from different testing offices around the world to the staff. Historically we had some major issues in securely moving that data around, and this is what we started looking into cloud solutions for this.
For very, very large game builds, and we're talking game builds above 10 gigabytes, it ended up being couriered within the country and then overnight file transfer outside of the country. So, very old school methods.
We needed both to secure that up to make sure we understood where the game builds were, and also to understand exactly which version each of the testing offices was using. So it’s gaining control, but also providing more security.
Gardner: So we’re seeing a lot more of the role-play games (RPG) types of games, games themselves in the cloud. That must influence what you're doing in terms of thinking about your future direction.
Hart: Absolutely. We’ve been looking at things like the hybrid cloud model with VMware as a development platform for our developers. That's really what we're working on now. We've got a number of games in the pipeline that have been developed on the hybrid cloud platform. It gives the developers a platform that is exactly the same and mirrored to what it would eventually be in the online space through ISPs like Colt, which should be hosting the virtual cloud platform.
Gaining cost benefits
And one of the benefits we're seeing in the VMware offering is that regardless of what data center in the world is the standard platform, it also allows us to leverage multiple ISPs, and hopefully gain some cost benefits from that.
Very early on we were in discussions with Colt and also VMware to understand what technology stack they were bringing into the cloud. We started doing a proof of concept with VMware and a professional services company, and together we were able to come over a proof of concept to distribute our game testing code, which previously was a very old-school distribution system. So anything better would improve the process.
There wasn't too much risk to the company. So we saw the opportunity to have a hybrid cloud set up to allow us to have an internal cloud system to distribute the codes to the majority of UK game testers and to leverage high bandwidth between all of our sites.
For the game testing studios around Europe and the world, we could use a hosted version of the same service which was up on the Colt Virtual Cloud Director (VCD) platform to supply this to trusted testing studios.
Doing this allows us to manage it at one location and simply clone the same system to another cloud data center.
Gardner: When you approach this hybrid cloud model, what about managing that? What about having a view into what’s going on so that you know what aspects of the activity and requirements are being met and where?
Hart: The virtual cloud environment of vCloud Director has a web portal that allows you to manage a lot of this configuration in a central way. We’re also using VMware Cloud Connector, which is a product that allows you to move the apps between different cloud data centers. And doing this allows us to manage it at one location and simply clone the same system to another cloud data center.
In that regard, the configuration very much was in a single place for us in the way that we designed the proof of concept. It actually helped things, and the previous process wasn’t ideal anyway. So it was a dramatic improvement.
One of the immediate benefits was around the design process. It's very obvious that we were tightening up security within our build delivery to the testing studios. Nothing was with a courier on a bike anymore, but within a secured transaction between the two offices.
Risk greatly reduced
Also from a security perspective, we understood exactly what game assets and builds were in each location. So it really helped the product development teams to understand what was where and who was using what, and so from a risk point of view it’s greatly reduced.
In terms of stats and the amount of data throughput, it’s pretty large, and we’ve been moving terabytes pretty much weekly nowadays. Now we’re going completely live with the distribution network.
So it’s been a massive success. All of the UK testing studios are using the build delivery system day to day, and for the European ones we’ve got about half the testing studios on board that build delivery system now, and it’s transparent to them.
VMware was very good at allowing us to understand the technology and that's one of the benefits of working with a professional services reseller. In terms of gotchas, there weren't too many. There were a lot of good surprises that came up and allowed us to open the door to a lot of other VMware technologies.
There were a lot of good surprises that came up and allowed us to open the door to a lot of other VMware technologies.
Now, we're also looking at alternating a lot of processes within vCenter Orchestrator and other VMware products. They really gave us a good stepping stone into the VMware catalogue, rather than just vSphere, which we were using previously. That was very handy for us.
Gardner: I’d like to just pause here for a second. Your use of vSphere 4.1 must have been an important stepping stone to be able to have the dynamic ability to ramp up and down your environments, your support infrastructure, but also skills.
Hart: Absolutely. We already have a fair footprint in Amazon Web Services (AWS), and it was a massive skill jump that we needed to train members of the staff in order to use that environment. With the VMware environment, as you said, we already have a large amount of skill set using vSphere. We have a large team that supports our corporate infrastructure and we've actually got VMware in our co-located public environment as well. So it was very, very assuring that the skills were immediately transferable.
Gardner: Now that you've done this, any words of wisdom, 20/20 hindsight, that you might share with others who are considering moving more aggressively into private cloud, hybrid cloud, and ultimately perhaps the full PaaS value?
Hart: Just get some hands-on experience and play with the cloud stack from VMware. It’s inexpensive to have a go and just get to know the technology stack.