Virtualized web app test adds iOS and Mac to the mix

Sauce Labs, the maker of the Sauce "cloud testing service" on Tuesday announced that it added support for OS X, iOS and Android to its automated cross-platform, cross-browser testing service for web applications.

Sauce Lab's automated testing service offers virtualized testing for a number of browser and operating system combinations, including Microsoft Windows XP and Vista with versions of Firefox, Google Chrome and MS Internet Explorer; and Linux distros with flavors of Firefox, Google Chrome and Opera. With the expansion, it adds Safari, Firefox, and Google Chrome running on OS X, and Safari on iOS.

According to John Dunham, Sauce Labs co-founder, the key to offering iOS testing was through Apple's iOS Simulator, which only runs on a Macintosh with OS X. The company now has a "freight train full of Mac Pros installed in a cage in Mountain View [Calif.]" he said.

Dunham said the virtualization of OS X offers customers more security and performance. However, the effort wasn't simple. The company chose the Kernel-based Virtual Machine (KVM) project to provide the virtualization infrastructure for the OS X. However, KVM doesn't currently support OS X and Sauce engineers worked closely with several KVM committers for about two years to accomplish the port. He said Sauce will open-source the OS X virtualization work to the community.

The Mac Pros boot directly to Linux (Ubuntu 12.04), which run KVM on Linux and then run OS X as a guest VM. In addition, the OS X license requires the purchase of OS X Server for each guest VM.

On the company blog, Jason Huggins, Sauce Labs chief technology officer, wrote about the KVM development.

We started the “Apple Sauce” project with patches Alexander Graf had published for the KVM project. By 2011, however, the patches were starting to bit-rot as the underlying codebase changed. We contacted Alexander and asked if we could sponsor his further development of the patches. He brought in René Rebe, another excellent KVM contributor. With Alexander’s and Rene’s blessing, we’re pleased to now release these updated patches to the KVM community. We don’t know yet if the patches will be incorporated into the KVM project, but in keeping with our commitment to open source, we want the community to have the option.

Dunham described Sauce as the "outsourcing of the test infrastructure." The workflow is aimed at users of the Selenium Remote Control (RC) Project's automated testing tool for web application. The tool aims to provide "automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser."

While the route to iOS testing lead through the Macintosh and OS X, Sauce customers had long asked the company for OS X support, Dunham said. At the same time, he said he has yet to see a bug on OS X that can't be reproduced on Windows.

"[Developers] insist that they have this checklist that says that OS X Safari works. If they don't check it off, they are letting their constituents down. So, the customer is always right. But I've never seen this mythical bug. Now, I'm not saying these bugs don't exist but we ain't seen one. Maybe someone will send me one, I'd appreciate it," Dunham said.

He described many other interesting features of the Sauce service, including the Sauce Connect security technology, its OnDemand automated testing tool for websites, and the service's speedy provisioning of virtual machines. The company also has a project for a Sauce for Mac desktop client that launches browsers in the Sauce infrastructure.

Sauce pricing is by animated code minutes, the number of simultaneous test virtual machines. Packages range from Light ($12 for 80 minutes, 4 simultaneous test VMs) to Extra Hot ($279 for 3,200 minutes, 30 simultaneous test VMs).