Secrets of starting a data warehouse from scratch

Being able to build a data warehouse right from the beginning of a company's life can eliminate some of the pitfalls typically associated with the project, but doesn't necessarily eliminate the most obvious one: uncontrolled data from multiple sources.

Being able to build a data warehouse right from the beginning of a company's life can eliminate some of the pitfalls typically associated with the project, but doesn't necessarily eliminate the most obvious one: uncontrolled data from multiple sources. And it adds a whole new challenge: do you have enough data to start planning for intelligent analysis?

I was reminded of this during a recent interview with Martin Fletcher, the chief operating officer for the Australian operations of Mortgage Guaranty Insurance Corporation (MGIC), which set up shop down under in 2006 and provides insurance for loan companies against possible credit losses.

Right from the beginning, Fletcher was determined to set up a data warehouse to allow proper tracking of the business' performance.

"I've set up companies previously and it's very clear that to get to the stage where you can analyse the data, you need it all in one place," he said.

MGIC didn't want to simply import the technology used by the American parent company. "We're a subsidiary of a very successful US business, but as we looked at systems for our home office team, they've got something that suits a business of 3,000 people and we've got 200. The scale of it was overkill for what we needed."

Effectively starting a new company meant that no existing systems had to be dismantled. "From day one, we've hired people in the team here who are very keen to design their own processes from scratch," Fletcher said.

However, that didn't mean that the familiar tangle of spreadsheets and individual documents was entirely absent. Because the company had to be in operation from the word go, individual monthly reports had to be generated until the data warehouse was up and running.

That scenario was probably unavoidable, according to Fletcher. "Realistically, we couldn't have started it any earlier. Everyone's delivered the pieces of the project on time."

Fletcher originally thought that defining the long-term requirements could be carried out in-house, but MGIC IT leader Joe Gullotta convinced him that some external advice would be useful.

"My initial thought when I talked to Joe was we would be able to define our requirements, but early on Joe was advocating using consulting experts. When you look at your business, you may not understand the best way to organise your data to analyse it." MGIC eventually engaged Altis Consulting to help plan the system.

Accessibility was a key requirement. "As we started thinking about the future capabilities of the BI tool, I wanted it to be very user interactive. I didn't want a team of reporting analysts who were the only ones who could use the system."

Even with the basic architecture in place, planning for future needs was tricky. "The biggest challenge was we didn't have a lot of raw information in our portfolio because at first, we didn't have a lot of loans insured. Often we were describing qualities about the portfolio which we didn't have any data on. That was one of the problems we had to overcome."

With the MGIC project now well-established and able to deliver daily analysis on a wide range of metrics, what advice would Fletcher offer to other companies facing similar challenges?

"Definitely spend time upfront thinking about what you're trying to achieve, not just today's requirements. If you work closely with the vendor you can get a vision for what's possible in the future. Distinguish bells and whistles from essential things to drive the business forward. We spent a lot of time thinking about what we wanted as minimum. Then once we met with the vendors we had a much more realistic discussion."

Newsletters

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
See All
See All