Intuit: Pain and pleasure in the cloud
Intuit's customers rely on the company to provide core, mission-critical financial services. For this reason, Intuit is as much an enterprise technology company as Oracle or SAP. In recent years, the company migrated its products to the cloud, emphasizing service delivery over the Internet.
Migration challenges. For many enterprise vendors, migrating products from on-premise to the cloud involves a challenging mix of changes to technology, process, and business model. In Intuit's case, with a broad product line that spans large organizations down to millions of consumers, many of these changes are particularly complex.
I spoke with Tayloe Stansbury, Intuit's Senior Vice President and Chief Technology Officer, to get a better sense of the company's efforts to migrate its products to the cloud. Stansbury explained that Intuit has invested over $300 million during the last two years to consolidate data centers, upgrade hardware, and refresh its software applications. Inuit has used much of this investment to create a robust architecture and software as a service (SaaS) technology platform.
In addition to technology investment, however, Intuit has also focused on personnel and process. To minimize human error the company instituted a peer review board, consisting of architects and others, which must approve all software and hardware changes. In addition, the company evaluated staff, including both senior management and technical personnel, to "ensure that all groups possess sufficient experience with high-availability SaaS computing."
Despite its massive upgrade and modernization program over the last 18 months, Intuit nonetheless suffered a series of outages that provoked the wrath of customers and served as a "wake-up call" inside the company, to use Stansbury's term. These outages interfered with customers' business operations and therefore harmed Intuit's reputation.
When I discussed these outages with Intuit, the company emphasized that customer satisfaction is "in our DNA." That said, the outages serve as an example of the difficulty associated with migrating a traditional software business to the Web.
OUTAGES AND DOWNTIME
Let's take a look at the four outages experienced in the 18-month period between June 2010 and the most recent, which occurred this past November 15.
In June 2010, Intuit experienced an unexpected outage that lasted several days following planned maintenance in a data center. The downtime notice, posted on an Intuit blog, raised the ire of customers, who left almost 375 unhappy comments. User comments on a subsequent update post continued the very negative theme.
One commenter explained that many users lost confidence in Intuit following that incident:
Intuit convinced me that it was a responsible fiduciary, but its handling of this situation has shown me otherwise. A 30-hour outage of mission-critical applications combined with very uninformative updates is simply not acceptable to the business owners who use your products.
After service was restored, Intuit's CIO at the time, Ginny Lee, posted several updates on the company's blog, and described what happened:
We had an accidental power failure that brought both our primary and secondary power units down. The process of restoring power caused a hardware system failure. We still don't know the root cause of that. To ensure full integrity across all our impacted applications, we immediately initiated a restoration process using our back up data. We also had to ensure proper sequencing in a way where both individual applications and our own enterprise applications that they rely on are restored in the most reliable and secure way before bringing them back on line.
Despite Lee's apologies, searchcloudcomputing.com was not satisfied with her explanation:
The notice failed to detail how a power outage of a large enough magnitude to knock out entire swaths of infrastructure was allowed to occur; modern data center operations segment electrical service delivery carefully to limit the amount of damage an outage can have.
Lee's messages did not say that an Intuit employee accidentally shut down power to the company's platform, thus causing the lengthy outage. Intuit's CEO, Brad Smith, also posted an apology on various company web sites.
Three weeks later, in July 2010, Intuit's small business accounting solution, QuickBooks Online again suffered an unplanned service outage. Searchcloudcomputing.com noted this outage lasted only a "few hours," but the notification blog post from Intuit gathered 115 comments. ZDNet reported on the negative customer comments.
Once again, in March 2011, Intuit had more unexpected downtime when problems arose in the aftermath of another scheduled system maintenance.
This time, company Executive Vice President, Kiran Patel, described what happened in an apologetic blog post:
On Monday and Tuesday, March 22-23, we had service outages triggered by separate errors made during scheduled maintenance. On Friday, March 25, we lost those services again. We're still working to identify the cause of that last disruption.
In the first two instances, blame human error. We changed our network configuration and inadvertently blocked customer access to a portion of our servers. Those initial disruptions lasted just a few minutes while we undid the change. But in both cases, restoring full service took longer than we would have liked. After fixing the problem, a surge in traffic overloaded the servers when we restored connectivity. We then decided to restore service at a purposefully measured pace to closely monitor the system and prevent another overload.
On Friday, the same services went down. We restored them much more quickly, but are still trying to determine why that happened. In all cases, we know there was no security breach or attack on our servers. And we know that your data was not lost or damaged.
Fast-forward about six months... Intuit scheduled a 4.5-hour maintenance period to take place on November 12, 2011. That planned outage again turned bad and the company faced extended downtime, not recovering until several days later. This time, 63 users left comments in response to the unexpected outage.
Although the latest outage did not draw large headlines in the press, some external observers offered scathing comments. After the outage hit, blogger David Spigelman spoke about Inuit:
This leads me to one of two conclusions:
1. They didn't do their jobs properly.
2. They don't consider these online systems as "mission-critical."
If it's the latter, that's a major problem. Their customers are small businesses, and these systems, while perhaps not entirely mission-critical to Intuit, are definitely so for their customers
Media blog, TG Daily, believes this outage was under-reported, given its severity and the impact on customers:
For some reason, the downtime wasn't widely reported, despite the fact that accountants all over the country were probably tearing their thinning hair out as they tried to process payroll without QuickBooks.
Even my fellow Enterprise Irregular, Jason Busch, a widely respected analyst on enterprise procurement, jumped in with his own comments:
Like many small businesses, we use QuickBooks Online to manage our general ledger. For a rapidly growing, self-capitalized company completely dependent on cash flow to fund payroll, healthcare costs, OPEX and new hiring, the QB URL has become more than just a general ledger, accounts receivable, accounts payable and reporting tool -- it's our lifeblood, the pulse of our business. It's a pity then that Intuit in recent weeks has had so many issues keeping the service up and running for subscribers like us. Just this week, we've even had to resort to manually writing checks because we can't access the system. And don't get us started on not being able to invoice customers!
Intuit also had an outage in November that hit credit unions, as reported by the Credit Union Times:
The lights went out for some 1,800 Intuit Financial Services customers on Wednesday at around 1 p.m. PST and it took a full six hours before services - mainly involving online banking sites - were fully restored to those credit unions and banks, acknowledged Intuit spokesperson Tobin Lee in an interview.
Lee declined to specify the cause of the outage other than to say it was a failure of the power supply coupled with failure of redundant systems that were supposed to step in with power and did not.
STRATEGIC ANALYSIS
Intuit's situation illustrates potential challenges that every on-premise software company faces when moving to the cloud. Not only is cloud technology different from on-premise software, but the transition requires operational changes, technical re-architecture, and internal culture shifts that recognize the immediacy of customer pain. This is particularly true in the case of a company like Intuit that supplies mission-critical services to a vast array of businesses. When Intuit goes down, everyone knows.
I asked Brent Leary, co-founder of CRM advisory firm CRM Essentials, and a top small- and medium-business analyst, to comment:
The challenge of moving from supporting installed applications to cloud-based solutions calls for new infrastructures, platforms, procedures, and policies. Established software vendors need time to make changes like these and get it right.
Traditional development cycles and timeframes do not meet expectations today. Companies like Intuit must get up to speed much faster than they did in years' past. They must also be much more open and communicative to their customers in order to meet the expectations of today's socially empowered customer.
Carl Brooks, a market analyst covering infrastructure at Tier1 Research, explains the responsibility that service providers like Intuit have toward customers:
Unlike many SaaS providers, an outage with Intuit is virtually guaranteed to make customers miss a payroll or face other negative effects on their business. Intuit has had to learn how to do highly available and distributed systems.
Since its Quickbooks 2007 release, Intuit has pursued a strategy that forces customers to use its online services. Although there are benefits, this approach reduces choice and customers therefore hold Intuit to a higher standard. The company needs to regain trust with unabashed transparency.
My take. Intuit's offers a cautionary tale to those migrating active products to the cloud. For developers, the journey can be difficult and fraught with technical, human, and process challenges. As Intuit discovered, even substantial financial investment may not be sufficient to ensure customer satisfaction.
Successful cloud vendors ultimately prioritize customer satisfaction above other considerations. Achieving this goal may require business model changes, such as accepting slower growth to ensure technical stability, along with creating high levels of customer outreach, communication, and transparency.
For both enterprise vendors and buyers alike, however, one thing is clear: cloud computing is far more than merely a software delivery mechanism. Success in the cloud brings requires synchronizing technology, business model, and culture.