Cambodia telco gets a touch of SAN

Mobile phone subscribers in Cambodia totaled more than 850,000 in 2004, while fixed line subscribers remained at less than 40,000, according to figures by Research and Markets. But with a population of more than 13 million, Cambodia's mobile subscriber penetration rate has far from reached its peak.

MobiTel, a US$120 million joint venture between Millicom International Cellular S.A. and Cambodia's The Royal Group, is the country's largest GSM operator. With network coverage extending to every corner of the country, MobiTel has 700,000 subscribers, with the base growing at a rate of 5 percent per month.

Cambodia is on the cusp of a mobile revolution, said Mike Johnston, MobiTel's IT manager. And its citizens are hungry for more cellular services.

As MobiTel introduced new mobile services to quell Cambodia's appetite, Johnston realized that it was time to overhaul the limiting direct-attached storage system and install a storage area network (SAN) that could handle the telco's expanding subscriber records.

At one point, the IT manager was struggling with a monstrous storage system borne out of an "ad-hoc" mentality--to throw more servers at the problem. He had 54 servers running at a storage utilization range of between 30 percent and 40 percent, and that meant he had plenty of unused capacity.

On top of that, MobiTel had to comply with the Sarbanes-Oxley Act, which mandated that the company tracked and securely retained all information pertaining to SEC financial statements. This translated into a requirement for stringent documentation of accounting and other processes, and meant that MobiTel needed better storage management to keep its house in order.

"Because of Sarbanes Oxley, we had to have a transparent, process-driven infrastructure that could retain data up to six months, and the ability to backup data on an hourly basis," said Johnston.

In April 2004, MobiTel's storage problems came to a head when its subscriber numbers ballooned after the introduction of its "killer" mobile application--the personal ring-back tone services (PRBT). The PRBT service allows a subscriber to select the ring tone that their callers listen to while waiting for their calls to be picked up.

The new service was a huge success, said Johnston. MobiTel rolled out the service, expecting to sign up 10,000 subscribers in a year. But within three days, 3,000 subscribers had signed up. And its call detail records (CDRs) ballooned, averaging about 400 million a month. Each call generated multiple records to the point that they started to bog down the billing engine, which had limited storage capacity.

Getting it right
Things finally took a turn for the better in the second half of last year, when the telco implemented an 8TB (terabyte) Clariion CX500 SAN system, together SnapView, a mirroring software; PowerPath, a software with failover functionality; and Navisphere management software.

However, MobiTel encountered a problem probably unique to emerging countries--none of its IT staff knew what a SAN was. "We had the first SAN in Cambodia, and we had to educate our staff from scratch. I have some fantastic staff, and they're very intelligent. The question was how to get their buy-in, making sure that they understood the benefits and how they could make it work," said Johnston.

Thankfully, Johnston added, EMC was more than willing to send its sales and technical team to Cambodia to assist. "Most vendors have problems delivering into Cambodia. EMC knew what they were talking about and had the patience to teach our guys who had never seen a SAN before," he noted.

EMC also suggested training MobiTel staff in Singapore as part of a more comprehensive systems implementation program.

Now that the SAN is up and running, EMC today provides continuous system monitoring, call-home notification, and advanced remote diagnostics. EMC has also tied up with a local partner in Cambodia which MobiTel staff can contact for support.

With the SAN in place, MobiTel has managed to whittle down the number of servers from 54 to 15. There is also a huge improvement in storage and server utilization which is about 80 percent today, Johnston said.

In addition, the number of staff managing different IT processes has also reduced. Now just one person is needed to oversee all storage requirements, and where there used to be 15 staff responsible for the different processes, there are now four people.

The time required to get data has also shortened. Where it used to take 45 minutes to retrieve call detail records, it now takes less than 10 minutes. With the SAN, MobiTel's backup window has reduced from 8 hours to just one.

"We're now backing up data every hour onto SCSI drives using SnapView and then onto ATA disks for staging, and we're physically backing up data every eight hours from ATA disks onto tape," said Johnston.

He also said he hopes to capitalize on the ongoing relationship relationship between EMC, Dell and Oracle to smoothen the implementation process--something that he discovered while looking at changing the company's database from SQL to Oracle.

"We've put in an EMC SAN, and replaced our old servers with new Dells. Because EMC, Dell and Oracle have a working relationship, it gives us an assurance of support," he said.

It's payback time
Overall, Johnston said he is more than satisfied with the SAN, and is planning to acquire another 10TB of storage this month, which would give MobiTel enough capacity for the next 6 - 8 months.

"Our current 8TB SAN costs about US$500,000, comes with a fully supported service level agreement, with a local partner supporting our development," he said. "It has already paid for itself, and the payback period was about nine months."

In the meantime, Johnston is looking forward to his next big project, which is to build a disaster recovery site that will replicate the original SAN 15km away, next year.

He said: "We've gone from a JBOD (just a bunch of disks) environment, and having no understanding of what a SAN is, to having a solid infrastructure, a new database. The key constraint now is to how to optimize what we have…We're putting up a second SAN, and which will mirror the [original site]. We're taking it one step further."