Betfair: Taking no chances with technology

The US might be several steps ahead when it comes to the sophistication of its e-commerce companies but there is one business in which Europe, and more specifically the UK, is streets ahead: online gambling.

The Founding Fathers' puritanical heritage has effectively put US companies out of the game when it comes to the lucrative and growing market for Internet betting. Fortunately, their loss is the UK's gain and industrious minds on this side of the pond have not been found wanting. In June 2000, former professional gambler Andrew Black and ex-city trader Edward Wray set up online betting exchange Betfair – and won the Ernst & Young Emerging Entrepreneur of the Year award for their troubles.

Black and Wray were quick to apply the collaborative power of the Internet to the traditional bookmaking model by allowing punters to bet at odds set by other gamblers rather than a bookmaker. The model was slow to take off at first, with the fledgling site matching less than £50,000 of bets a week in its first month. A year later that figure had grown to over £1m a week and it now stands at some £50m.

Given that Betfair matches some 1.5 million transactions a day – 24 hours a day, seven days a week – it is probably one of the most advanced Internet businesses in the world in terms of the scale and robustness of its infrastructure. As the company's chief technology officer David Yu is quick to point out, if Amazon goes down for a few minutes, then at worse customers get frustrated; if Betfair goes down during a vital race then all bets are off – literally.

Betfair's vulnerability to down-time has not gone unnoticed by the criminal fraternity. Earlier this year, Betfair, along with several other online betting companies, was targeted by cybercriminals threatening distributed denial of service (DDOS) attacks unless a ransom was paid.

Earlier this month, ZDNet UK took a trip out to Betfair's high-tech offices on the banks of the Thames in London's Hammersmith to quiz David Yu, recently voted Daily Telegraph IT director of the year, on the intricacies of running a true always-on business.

This is a pretty sophisticated business model which takes a lot of tech to make it work. How did it all come about?
The company is four years old and started with Andrew Black, the founder, developing Betfair on his laptop in his spare time. He hired in a few developers and basically took his laptop program and made it into a product. The last few years have been spent refining that – we have estimated that our software is some 20 times more efficient than it was three years ago. We take around two million transactions a day now, and we serve over 80 million page views, which has more than doubled since the start of the year.

The real challenge has been understanding user behaviour, learning how users actually use the site. When you first write the site you kind of predict what you think they will do, but until you really see it in action you don't really know. So what we will do is analyse logs, we'll analyse database statistics and basically look at how trading patterns change.

We have seen that people's usage of the site has changed over time -- not just because we add new features, but because their experience increases. We have seen people who have written robots that come and screen-scrape the site; in fact, some third party has written an application that refreshes the site more often.

I think the site development is very iterative. There is no easy answer – where you just do one thing and it's right straight away. It takes constant refinement and iteration.

I guess you can't be doing development all the time - there must be a trade-off with maintaining what you already have?
As a rough rule of thumb we spend about 50 percent of our time in engineering working on new features or enhancements to existing features. We then spend the remainder of the time working on the current site. That 50 percent might mean overhead for employees, such as holidays or training, but also working on the reliability of the infrastructure. If we fall behind on that maintenance then it could well become a panic later on.

You recently re-launched the site with a newly designed user interface. What are you doing to support other browsers now there is some competition again?
The Web is predominately dominated by IE users but we are starting to see more Firefox and Mozilla usage. The new site is compatible with non-IE browsers and Mac platforms, which we didn't support before. A lot of us use Firefox, which we find to be a much faster experience than IE.

And how about other client platforms such as mobile?
We have looked at other platforms in terms of mobile, television and so on and have developed a Teletext programme - a presentation-only version where you can see odds pop up. I think the real constraint on mobile devices is simply their interface; the Betfair application is really complex and if you try to do too much over a mobile application it probably won't work that well. I have had vendors come in and try to promote SMS-based betting where you'll text in the name of a team and send that off and we would send back the odds and then you send something back… I think it is frustrating enough to send a text. At the end of the day, if it's a good phone - why not just call us?

We actually do have a light version of the site which is HTML-only, only shows prices and runs on anything -- a phone, a Web-browser, a PDA. It is just a nice to have thing but isn't a significant portion of our traffic. We are looking to offer a transactional version of that. You won't be trading on that but if you happen to be on the way somewhere in your car it might be useful.

Earlier this year you announced that you were re-engineering the site around J2EE and abandoning .Net. How has that decision worked out for you?
We use J2EE for our server platform but all of our internal tools are still written in .Net. We made that conscious decision because we found that .Net is very good at rapid development for internal tools -- it's very robust. We chose J2EE for its stability and its enterprise class and that has certainly been a very good reason to go with the J2EE platform. But we will always use the best technology for our needs -- we are pretty technology-agnostic, unlike some businesses that might dictate or mandate one technology throughout.

You also use the open-source JBoss server as opposed to proprietary technology from IBM or BEA?
In terms of JBoss, we looked at application servers at the time; it was open source which has a lot of benefits. We think open source is the strong way to go. But we always look at new technologies and constantly evaluate if we need to switch to anything else. One of the real challenges of Betfair is that line between leading edge and bleeding edge. I think we have matured as a business to the point where we will not use bleeding-edge technology -- it's just too risky for us. We will tend to use things that are innovative and leading edge, but are tried and tested.

As someone in charge of a 24/7 business which lives or dies by the quality of its software, do you have any strong views on whether open source is an inherently more robust way to develop?
One of the main complaints about open source is the lack of support behind it, but I don't actually think that is an issue. What we have found in the open-source community, particularly with JBoss, is that the support from the community and JBoss themselves is as good as what you'd get with a commercial vendor. Our decisions are going to be more around price-performance and value, and obviously open source has price advantage. We use Oracle as our database, which is very robust but certainly not free and not inexpensive but for us its quality makes for a good price performance.

How many developers do you have in-house?
About 30 developers and roughly a similar number of QAs (quality assurance) so altogether the number of people developing our software is about 50 people. The engineering team is about 140, which is made up of our infrastructure team, our 24/7 support team, business analysts, product managers and overhead like me.

We try to keep a ratio of two developers to one QA person, which is probably a better ratio than many customers but that is probably just the nature of us being a very transactional site.

What have been the most difficult skills to find?
We find them all to be very difficult. Betfair tends to be very selective – we spend more time evaluating people than technology. We are very, very picky when it comes to selecting our team so as a result we don't have very much unwanted attrition but it does take us a long time to hire. We have always had open headcount but we have never been able to fill all the seats as we can't find good enough people. I think technical QA people have been extremely hard to find for us, people who can really stress test the system.

We are finding that there is a lot more stability in terms of the service platform itself. We have also seen considerable improvements in scalability. I think some people have misunderstood the migration and see it as an IT-driven project. There are certainly IT benefits in terms of stability and capacity and so on but ultimately it is a product enhancement as it allows us to roll out a new interface, it allows us to roll out co-brands much more rapidly – in fact just before going live we rolled out a Ritz Casino co-brand. We are also able to manage multiple languages much more easily.

If it was an IT-driven decision then we would probably have made other changes first, we would probably have made some database changes. We launched the new site on the 13 December.

That was all done with in-house expertise?
We have brought in contractors in the past to augment the team we have here but we have not outsourced any work. The main reason for that is predominantly because the underlying engine that we have here, which processes hundreds of millions of pounds a week, is a financial system that quite honestly is not something that we would feel comfortable risking to a third party. Maybe over time we might feel comfortable trusting a third party with more user interface and presentation things but nothing in the core betting product. We think that is strategically important to us and proprietary.

We do have a Betfair developers programme which is for all intents and purposes an API we have developed so external third parties and other ISVs can develop applications on top of the API.