Orange: Squeezing the most out of SANs

If working in IT is on the whole an inconspicuous and mostly thankless task, then a job in storage management puts you right down there -- as far as other departments are concerned -- with the nameless individuals who come in at night to water the office plants. But if knowledge really is power, then maintaining the integrity of a company's data and intellectual property should, in theory anyway, be one of the most important jobs in the company.

One person who is quick to point out that inconspicuous doesn't mean insignificant is Steve Coombs, UK storage manager for mobile-operator Orange. He has seen the company grow to become the largest mobile provider in the UK, from an installed base of around 300,000, when it launched in 1994, to more than 13 million now. To cope with the storage issues this kind of customer base throws up, Orange created its own dedicated storage team. Coombs' group can manage around 410TB of data, using 43 EMC Symmetrix and Clariion arrays, connected into Storage Area Networks (SANs) by a number of McData director-class switches.

But Orange's approach to storage hasn't always been this sophisticated. In the glory days of the mobile industry, the company had a hard time keeping up with a snowballing customer base. From 1999 to 2000 the number of Orange subscribers exploded from 3 million to 8 million. Coombs says that over the last eight or nine years his team has had to manage sustained storage growth that in the early days severely tested a storage infrastructure initially reactive and slow to respond.

"We couldn't answer business questions quickly around our storage environment. We couldn't provision storage very quickly, we weren't really sure where our storage was, and it was pretty much maintained and supported by point solutions and point support teams. We didn't have a central support function that was able to go to the business and give them the information they needed," he explains.

But the decision to launch an ISP, Orange.net, in 2000 -- which allowed users to surf the Net over a wireless connection or receive information services such as a newsfeed over their phones -- created a real impetus within the organisation to improve its storage management and infrastructure, according to Coombs.

"In 2000, there was this big portal ISP explosion. Orange needed to provision an ISP very quickly and EMC were able to provide us with a lot of storage -- which made their account manager very happy," he says.

Already a customer of EMC since 1996, with several of the company's Symmetrix storage arrays in place, Orange took the decision in 2001 to adopt the storage giant's 'Pay As You Grow' pay-per-use billing scheme. This allowed the company to buy more storage capacity than it required on average day-to-day to allow for any spikes in demand. The company is only billed for using this extra capacity when and if it is needed. This gave Orange more flexibility and helped drive down total cost of ownership (TCO), says Coombs.

"Through this whole time we have tried to lower our TCO for storage by pay-as-you-use models and by the fact you are getting larger economies of scale from provision of the infrastructure. We have received 50 percent savings on TCO since having this infrastructure in place," he says.

At the same time, the company embarked on what Coombs terms a 'refresh' of its storage systems, which consisted of creating a more consolidated approach to storage management by improving its storage area network (SAN) infrastructure -- allowing its various business teams to share more storage devices. But rather than move all of its various business applications and units onto a common SAN, Orange created a series of distinct SAN islands that conveyed some of the scalability and flexibility benefits of a SAN without the associated disruption.

"The reason we opted for the island approach is that the cultural change within the organisation is not to be underestimated. Where you have got system administration teams very much focused on supporting the SAPs and the data warehouses, to get them to start sharing their environment has got to be a piece-by-piece approach. They are not going to do it overnight so what we had to do was put in these SAN islands to segregate our major business applications so it makes it easier to manage change in the environment," explains Coombs.

Orange is using EMC's ControlCentre 4.3 software to manage its storage infrastructure but is trialling the latest version - 5.1 - for a future migration. There have been a few troubles along the way but his team is working with EMC to correct these, says Coombs. "One of the major concerns we have at the moment is around scalability; we have effectively 2000 ports in our SAN and we have 50 disk arrays. EMC really haven't been able to test in this kind of environment, so when it comes to a live rollout these sorts of issues appear," he says. "But it still gives us the ability to manage up to 60TB per administrator; I have heard other industry figures band around figures anywhere between 1 and 30 per person."

In order to manage the new infrastructure more effectively, Orange also created its own internal storage team in 2001. Previously the company had relied heavily on EMC professional services but realised that having this expertise inside the company would be more efficient and economical.

"We were very reliant on EMC commercial services at that time because we didn't have that kind of expertise in the company," says Coombs. "The system administrators that we had were used to direct attached storage arrays and weren't really used to SAN environments. What this meant was delays in provision of new storage for projects."

A transition team from EMC helped create best practices and guidelines, while helping to recruit candidates to form the internal team. The transition from an EMC storage team to an internal team took around six months and had a significant improvement on the time it took to set up projects as well as the overall cost.

"I don't think at the time there were many other companies with a heterogeneous open-systems environment that had a dedicated storage team. We have tried to produce a centre of excellence for the storage community within the Orange group," Coombs adds.

Recruiting experienced storage staff is not easy, because a lot of the technology is so new. Coombs' advice for anyone recruiting storage administrators is to recruit skilled system administrators and train them up to become storage administrators. "Be wary of dodgy CVs," he advises. "I had a CV come across my desk last week and it said under experience 'Ten years SAN experience,' which is interesting as SANs haven't been around for ten years."

Looking to the future, Coombs believes that the job of the storage manager will never be done. "It's like painting the fourth bridge -- once you finish, it's time to start again." Having finished one storage refresh, he is already planning the next one. This will include trying to improve the utilisation levels across the SAN islands through a mixture of McData or VSAN technology from Cisco.

"SAN islands are good but they do mean that you get bad utilisation. We find that we have run out of ports in one island, whereas we have got plenty of ports in another island, so we are looking at how we can consolidate those SAN islands without having a major impact on the business," he says.

Orange is also looking at the whole IP-based SAN ICSI route, particularly for management of its distributed sites, and is actively involved with major vendors in this area. But Coombs claims that at the end of the day it's how the technology is managed, rather than the technology itself, which makes the real difference.

"It's not just about deploying technology -- it's how you manage it. You shouldn't underestimate the skills and experience required of your storage administrators."