GitHub seeks to spur innovation with Kubernetes migration
GitHub on Wednesday is sharing the details of the massive technical endeavor its engineers went through to migrate the infrastructure that powers github.com and api.github.com -- some of its most critical workloads -- from a set of manually-configured physical servers to Kubernetes clusters that run application containers.
GitHub is confident the move will allow for faster innovation on the online code sharing and development platform.
"We've given our engineers a better environment in which to experiment and run the code that powers github.com," Jesse Newland, GitHub's principal site reliability engineer, said to ZDNet. "We're really excited to see the rate of innovation increase at GitHub and hopefully use that to bring better software and services to everyone around the world -- and as a result of that, have a positive impact on the software industry as a whole."
After years of running a largely static group of servers that powered the application serving web requests, the time had come for GitHub to move to a system with more flexibility. New services could take days, weeks, or even months to deploy.
"We needed to build small environments to experiment in, to allow engineers to build new stuff," Newland said. "We also we needed to more quickly respond to changes in demand. Our traffic is growing, and we always need to deal with that, and we also do see regular valleys in our traffic."
The team turned to Kubernetes for its strong open source community supporting the project, GitHub's first run experience with it, and because of the wealth of information available about the experience that motivated its design.
One of GitHub's major hurdles, however, was the lack of documentation for the deployment of Kubernetes clusters in a physical datacenter environment. In fact, much of the relevant documentation that exists on the subject is focused on home users, Newland said.
As a result, "several of us that worked on the project have a closet Kubernetes cluster in our house," Newland said, "which is great for experimentation."
While GitHub has hit the milestone of migrating the application running github.com, it's still refining how it runs Kubernetes in a physical environment. Eventually, Newland said, his team wants to share that experience "and try to improve the documentation of Kubernetes so other folks who aim to do the same thing in the future have a better experience."
The team also took a risk by choosing to start its migration with one of its most critical workloads.
"We did that very deliberately," Newland said. "We knew that that workload is extremely well understood by a large number of people around the company. You can find any corner of that application and find people with a lot of experience in that particular part of it. Any time we ran into a problem, we were able to track down engineers that were really experts in the area."
While this was the application GitHub initially targeted, it wasn't first application the team ran on Kubernetes-- they needed to run other supporting services. Still, this is what the team considered their first noteworthy success.
"I've seen, both at GitHub and other organizations, when a migration attempt is done using a toy or smaller service as its big first step," Newland said. "That can often be extremely successful -- and it can often stall there. That's what we were looking to avoid. We didn't want to get halfway there and then move to another way to build applications."
To ensure Kubernetes would work well for its needs, the GitHub team built a pre-production staging environment called "review lab."
"We were able to present something to engineers internally that allowed us to almost crowdsource, in a way, the validation of the functionality of this platform and our use of it," Newland explained. "That low-risk experimentation environment was really impactful and allowed people to move quickly with a high stakes project because they kind of had this isolated environment where the blast radius of any problem was limited."
After seeing the positive results of this migration, GitHub is working to provide its engineers to a way to deploy new applications to Kubernetes in a self-service fashion. Already, Newland said, "we've seen an increase in the rate of innovation and new ideas and software projects that not only exist and work in your laptop but exist and work in production."
Related coverage: