Disclaimer: Scott Raymond is married to an Adobe employee who does not work in the Flash group, and to avoid any conflict of interest she did not assist in any way with this article. Also, to provide separate, corroborating tests, Jason Perlow handled his own testing independently.
There's been a substantial amount of press about web-based video performance lately, specifically aimed at Adobe Flash - and with good reason. Historically, Flash has been buggy, used a great deal of CPU, and consequently drained battery life as a result. However, Adobe has taken great pains to try to alleviate these issues. We propose to test the three most prevalent web-based video formats, comparing CPU usage and battery life.
Jason Perlow and I each picked a movie we owned on DVD over 90 minutes in length; I chose "Big Trouble in Little China" and Jason chose "The Empire Strikes Back". We used AVS Video Coverter; to generate videos of identical output in each of the three formats. The particular reason for choosing this application over others was that it was capable of generating all three video formats in one program, and that it was a stable, mature application.
For Windows Media, the codec used was the WMV9 codec for video, and the WMA2 codec for audio. For both QuickTime and Flash, we used H.264 for video, and AAC for audio.
How We Tested
Server Side: At first we assumed that it would be a simple matter of embedding the video in an HTML file on an Apache web server on our local respective networks, then playing them through a browser. This worked fine for WMV, but for Flash we needed to use a SWF player application stored on the sever, and added to the HTML code. This made things more complex.
We had been in contact with the Adobe Flash team for any technical assistance, and while they did provide us with a quite useful SWF player, we instead chose to go with Flowplayer, a stable, easy to use embeddable flash player. We also felt that by using a third-party application like this there would be no question of collusion or special code that could affect the outcome of the tests in favor of any one format.
QuickTime was another story entirely. QuickTime videos do not embed properly for standalone playing, and we ended up having to install the Darwin Streaming Server in order to stream the video to a browser. Also, the QuickTime video file had to have "hints" installed on the video and audio tracks, which was accomplished through the use of mpeg4ip-server tools. Once we had this set up, however, streaming worked properly.
Client Side: On the client side, we decided to go with Firefox as the browser for both Windows and OSX to ensure equality on both platforms. We installed QuickTime 7.6.6, Flash 10.1 RC7 and the WMP Firefox plugin on the Windows systems; on the OSX side we installed the Flip4Mac Windows Media plugin, Flash 10.1 RC7, and then later Flash Player Gala Preview 2.
Thinkpad X200, Core2Duo 2.4GHz, 4GB RAM, Intel GMA X4500HD Video Chipset, Windows 7 Ultimate 64-bit.
Apple MacBook Pro 13", Core2Duo 2.53 GHz 4GB RAM, NVIDIA GeForce 9400M Video Chipset, OSX 10.6.3.
Thinkpad T60, Core2Duo 1.83GHz, 2GB RAM, Intel GMA 950 video chipset, Windows 7 Ultimate 32-bit.
Testing was fairly straightforward. We would make sure that the battery was charged to 100%, then we started the performance monitor with logging, unplugged the AC adapter and started the browser playing the URL that would load the embedded video across the network. I used 802.11N networking on my LAN, Jason used 802.11g. The different network speeds did not significantly alter the outcome.
On our laptops, we disabled any automatic system updates, screen savers, power savers, etc. This was simply a precaution to prevent the laptops from going to sleep or dimming the screens which would greatly skew the tests.
However, we did leave our antivirus software active - Jason and I both use Microsoft Security Essentials, which is a fairly robust but low impact on CPU and memory. This was deliberate, because we were trying to emulate a real-world condition, and most people don't disable their antivirus every time they visit Youtube or the Apple QuickTime movie trailers website.
WMV: CPU Average for Firefox = 14%; 55% battery remaining
QT: CPU Average for Firefox = 24%; 57% battery remaining
FLV: CPU Average for Firefox = 18%; 62% battery remaining
As you can see, the results are fairly close. WMV wins on CPU usage, and Flash wins on battery life. Adobe has obviously improved flash a great deal on the Windows platform, rendering most of the arguments about it invalid.
WMV: CPU Average for Firefox = 29%; 12% battery remaining
QT: CPU Average for Firefox = 38%; 6% battery remaining, laptop shut down, movie not finished
FLV: CPU Average for Firefox = 19%; 18% battery remaining
This time, on older but still viable hardware, it appears that Flash wins in terms of both CPU and battery life.
It could be speculated that since both Flash and WMV have hardware acceleration with GPU offloading under Windows that Apple simply chose not to support one of the most popular laptop video chipsets in use today. However, from the accounts detailed in the forum link above, it also looks like they didn't even bother to support it on their own platform when they used the very same video chipset.
We supplied these results to Apple and Intel for comment. Apple has not responded, but we did get a response from Intel. According to the Intel spokesman:
"All Intel chipsets prior to 4 series with GMA 4500 do not support h.264 decode acceleration. Intel fully supports h.264 video acceleration in all our current gfx products. GMA 950 video HW acceleration targeted MPEG2 content given market prevalence at the time."
To confirm that this wasn't some kind of fluke concerning the GPU offloading, Jason ran the same tests on his wife's Dell desktop system with a Core2Quad 2.4GHz CPU, 4GB of RAM, a NVIDIA GeForce 8500 HD and Windows 7 Ultimate 64-bit:
WMV: CPU Average for Firefox = ~1%
QT: CPU Average for Firefox = 16%
FLV: CPU Average for Firefox = 7%
In this case, WMV was way ahead of the curve, showing nearly no CPU utilization at all. Flash came in second with very low CPU usage, and QuickTime came in last, although the CPU utilization is still fairly low. Keep in mind that this is a quad core desktop system with a more powerful GPU and is invariably going to be much faster than laptops.
Mac OSX Testing
Unfortunately the activity monitor application in OSX doesn't have logging, so I installed a copy of atmonitor. Testing on the MacBook Pro went smoothly as it did for the rest of the systems.
WMV: CPU Average for Firefox = 13%; 62% battery remaining
QT: CPU Average for Firefox = 8%; 59% battery remaining
FLV: CPU Average for Firefox = 36% = 43% battery remaining
Big difference between platforms here. It should be noted that while Flash has a very beneficial design in that it will use any available bandwidth to keep buffering until the entire video is downloaded, it was of no use whatsoever on the OSX platform.
Keep in mind, though, this is the old Flash code that did not have the benefit of the new hardware acceleration capabilities that Apple made available to Adobe recently. In fact, Adobe has made available the new Gala build of Flash for OSX, currently in early beta. Let's see how it stacked up:
FLV Gala: CPU Average for Firefox = 31% = 39% battery remaining
I was a bit disappointed since I was expecting something more spectacular. This is actually the second set of results, as I ran them again to make sure I had everything set up and installed properly with the Gala build. The CPU usage was better, but the battery drain was worse. I suspect that’s due to the GPU offloading. Video looked smoother overall.
Of particular interest is how badly HTML5 video on Chrome compared to HTML5 video on Safari. Since Apple now has a HTML5 Showcase that won't work on anything but Safari, this is highly suspicious - especially when Chrome has been reported to test higher at HTML5 compliance at HTML5Test. My install of Chrome 6.0.422.0 rates 142 out of 160.
While I am willing to cut Adobe some slack since this is a beta, I am truly disappointed that this build performed worse overall than the standard flash plugin. I contacted the Flash development team at Adobe and asked for their opinion on these results.
"The thing to keep in mind is the differences between Flash Player and desktop video players, such as Quicktime and Windows Media Player. Desktop video players play audio and video files, but don't provide a full programming interface. Flash Player, on the other hand, is a fully programmable runtime. In addition to streaming, decoding, and displaying the video, it can render other graphics elements and composite them with the video, all under the developer's control. Dynamic overlays, playlists that can fade in and out, compelling UIs (like Hulu.com, check out the previews when you 'seek' in the timeline) -- these are all types of rich features that make Flash Player more than just a video player. In order to offer developers and designers these types of rich capabilities, Flash Player requires additional resources. In addition to the progress we've made on performance to date, we will allow developers to conditionally turn off these extras in an upcoming release of Flash Player. We expect that this will result in another significant gain."
--Paul Betlem, Senior Director of Engineering for Flash Player at Adobe
Considering Apple's disdain of Flash and Adobe, I can't help but suspect that their assistance to helping Adobe improve Flash for OSX was nothing but a smokescreen. Otherwise, it means that Adobe has a long way to go in getting Flash to perform capably on the OSX platform.
One final note: There was not a single crash of Firefox during any of these tests. So at the very least, Flash stability has been greatly improved. It is actually the winner in terms of performance and battery life on the Windows platform.