How even agile development couldn't keep this mega-project on track

How even agile development couldn't keep this mega-project on track

Summary: The scheme sold as the world's biggest agile software project, the UK's £2.4bn Universal Credit programme, has run into cost overruns and delays. So what went wrong?

SHARE:

Touted as the world's biggest agile software project, the £2.4bn Universal Credit programme was supposed to showcase how the software methodology could be used on a grand scale.

But two years after the project — intended to create an IT system that would underpin a major reform the UK welfare system — got under way, crack began to show as it ran over time and over budget.

According to a report by the government watchdog the NAO published last week, the UK government has had to write off at least £34m on the programme and delay the national launch for the project. The department in charge of the project, the Department for Work and Pensions (DWP), can't guarantee the remainder of the £303m it has spent on the project so far will offer "good value" it said.

Agile is a flexible approach to running an IT project and an alternative to traditional 'waterfall' methods, which generally lock down specifications for a IT system at the start and then test the entire system close to the end of the project.

Agile development methodologies such as Scrum instead favour the delivery of systems as a series of smaller software modules, which can be pushed out to users at regular intervals so their feedback can inform subsequent software iterations. But it according to agile methodology specialists and the NAO which has examined the project, there have been a number of misteps in the government approach.

Not agile enough?

While agile development methodologies generally emphasise the delivery of software modules at regular intervals, in Universal Credit 's case, there was a two-year gap between the DWP starting the project design and build process, and the system going live in 2013.

That delay suggests there was not enough of an iterative feedback loop built into the programme, according to Jose Casal-Gimenez, chairman of the BCS agile methods specialist group and expert on Kanban and Lean agile methodologies. "It wasn't agile enough in this fundamental area," he said, adding that in his view there wasn't enough emphasis on "real-world testing" and learning from user feedback.

The UK's Cabinet Office — whose Government Digital Service has successfully used agile techniques in developing the gov.uk website — appears to broadly agree with Casal-Gimenez's assessment, as the NAO noted. "The Cabinet Office does not consider that the Department has at any point prior to the reset appropriately adopted an agile approach to managing the Universal Credit programme," it wrote in its report.

Contract constraints

Setting out requirements for suppliers too rigidly at the start of the programme can hinder an agile deployment, as seems to have been the case for Universal Credit.

The DWP contracted four major IT suppliers to deliver the project in 2011 but, according to the NAO "experienced problems incorporating the agile approach into existing contracts, governance and assurance structures".

"The procurement decisions appear to have been taken before agile was even thought of as a delivery approach and appear to be at odds with agile delivery," Casal-Gimenez said.

Contracts need to provide freedom for software to evolve with each new iteration — a key plank to how agile works — otherwise the nature of the methodology may see suppliers' bills go up and up.

"Hard-nosed contract managers within large suppliers will see this as a change to the requirements and will often want to charge for each iteration," said John Turner, board director at IT and business change professional services firm Xceed Group.

"Here the supplier can stand to make a lot of money and as such agile can introduce contract complexities. Agile developments need careful cost, time, quality and outcome definitions within the contract so that the power of agile will be of benefit but also impose the discipline on the need to deliver to the targets."

Too much, too fast?

Adopting agile development can't be rushed, particularly at large organisations where hundreds of people have never delivered a project using the methodology.

In hindsight it is unclear whether the DWP understood the difficulty of shifting from the traditional 'waterfall' system to an unfamiliar development methodology, while also delivering a multi-billion pound IT programme intended to deliver £74bn of benefits a year to around eight million people.

Rather than the DWP seeing the shift to agile as something that would initially add time to project delivery, the department seemingly believed it would cut development time, allowing the Universal Credit programme to be expedited and rolled out in 2013, two years earlier than it believed was possible under the waterfall system.

Underestimating how hard it is to successfully transition to agile delivery is a common mistake organisations make, according to Casal-Gimenez.

"Agile is very hard in some ways. It appears to be very simple when people talk about it, but it's very hard to implement," Casal-Gimenez said.

"Organisations that are very hierarchical will have a challenge dealing with the differing mindsets surrounding collaboration, work and being comfortable with mistakes.

"The bigger the organisation, the more complex and the more traditional an organisation is the longer it will take.

"The risk here is that many organisations have less patience than it takes to change. If you try to do too much, too quickly there is usually a day of reckoning where people lose patience."

There are also questions over whether the DWP knew if agile was suitable for projects on the scale of Universal Credit — the department "was managing a programme which grew to over 1,000 people using an approach that is often used in small collaborative teams", the NAO report said — and how the department integrated old and new metholdologies. "DWP needed to integrate Universal Credit with existing systems, which use a waterfall approach to managing changes", further adding to the complexity, the NAO said.

In January 2012, the department changed its approach to 'Agile 2.0' — an evolution of the previous agile approach which is designed to try to work better with existing waterfall approaches that the department used to make changes to old systems and, while the full national rollout has been delayed, a small scale deployment has got under way.

waterfall-agile-comp
A comparison of watefall and agile development methodologies by the National Audit Office. Image: National Audit Office

What would have been an agile approach?

Casal-Gimenez believes the Universal Credit programme may now be more on track for truly agile delivery following the DWP's decision to pilot the system at a single job centre in Greater Manchester, providing benefits to just 1,000 claimants.

"To do it the way they're doing it with the one centre in Manchester, with the controlled number of people, with the simple benefit, feels like it's much more aligned to an agile approach," he said.

"They can the start learning from that and putting in the bare bones of Universal Credit and then add more benefits, locations or claimants.

"In that way they can keep adding to the richness of what Universal Credit can do, on the basis that this is a live system and they're getting feedback and learning lessons.

"It feels like a more realistic way of tackling the big challenge of connecting and creating lots of different complex systems."

Further reading

Topics: Software Development, Enterprise Software

About

Nick Heath is chief reporter for TechRepublic UK. He writes about the technology that IT-decision makers need to know about, and the latest happenings in the European tech scene.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

19 comments
Log in or register to join the discussion
  • I don't think Agile is even wise on megaprojects

    dev shops should always switch into waterfall mode. Sometimes there just isn't a minimum viable product, and often what you're working on is too complex to be changed on the go. A waterfall based plan may be boring, but it does offer the advantage of being predictable.
    Mac_PC_FenceSitter
    • Not always true

      Some misconception regarding agile is that it is not predictable for long term projects and there will not be any plan.

      Any product differs from initial thought to the final finished product. In Agile you need to plan things everyday and every minute. else it will make you fail.

      At a release level you need to plan the stories in the order that is independent and priority ones are done first. at sprint level you plan for the current deliverable and keep the total delivery (the final thought and architecture) in mind.

      It is like, you keep a skeleton map in the picture and plan your designs and work everyday to give the skeleton the flesh and skin it required.

      if you don't have proper plan even waterfall will fail miserably but here in agile it fails fast.
      Albert Arul Prakash Rajendran
      • You are right proper planning and buy in from management are the key

        I suspect this project did not have a real management buy in for Agile. Since it was for a single client, I count the UK welfare system as part of the management. If they truly had a plan and if management was there at the end of every sprint to recieve a report, there is no way the project could have blown through that much in funding without someone putting on brakes before that.

        Unfortunately, too many people practice "sorta agile". At the developer level this includes "we don't need an architecture (or interface) documente". At the management level this is things like "why do I care about the end of every sprint, or what do you mean the developers say our time estimate are wrong and want a piece of work broken into several sprints, or we want the bugs fixed and all the planned work for this sprint done in 2 weeeks".

        It will be interesting if anyone ever breaks down where the failures were on this project.
        oldsysprog
    • you are on the dot ....

      Agile is not right where several hundred folks are involved.
      This brings back memories once again of the failed c3 project at Mercedes Benz..
      http://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System

      if was later written off as lessons learn't .....
      ug123
  • Piloting is Not Relevant

    Too often people make the mistake of thinking developing a "pilot" for one job center or equivalent will simplify the development effort. The only simplification comes from limiting the sheer number of stakeholders involved in decisionmaking. One job center or fifty, the application will still have to support the same number of business functions -- with the same complexity of code logic and data handling.

    The small number of stakeholders does offer help in getting agile to work as a development methodology, since the developers are less at risk of being swamped in a huge swirl of input from a myriad of stakeholders. Otherwise, the pilot project approach is irrelevant to this article.
    mdwalls
    • There are many benefits to a pilot

      Deployment issues are better tested at the pilot level.

      Using a pilot significantly reduces risk.

      Scaling is easier when starting small.

      Training is easier.

      There may actually be fewer "corner cases" in a caseload of only 1000.

      It may be discovered that building for all the corner cases imaginable is unnecessary in practice (manual overrides can be useful if they only occur a very small percentage of the tim).
      dimonic
      • Scaling

        Scaling up a small pilot is only practical if the overarching architecture has been designed appropriately - one of the problems I've seen with agile projects is a lack of architectural foundation - they too often think in terms of iteratively prototyping systems into production which can work OK for many small to medium projects but can produce significant rework and/or failure on large scale integration projects. Pilots are great but if the ultimate end is a very large deployment then an ongoing large scale architectural design needs to be moving in lock step - which often requires a whole different skill set than many projects bother to bring to bear. Otherwise that pilot that worked so wonderfully finds itself with unanticipated barriers and impedances to real scalability (flushing software to a hundred servers doesn't make it scale) and it gets crushed in rollout.
        archangel9999
      • Scaling 2

        Pilots are beneficial if the design is standardized across the entire user landscape. Too often local rules make this impossible (an unfortunate "normal" side effect).
        rhonin
  • No magic formula

    There is no system that can rectify a misquoted and incorrectly budgeted project from the start. While agile might be an improvement in project management, it is not a magic Band-Aid. Humans cannot predict the future. There will always be surprises.
    ozRocker
  • The blind side of Agile method for a large system

    A large system usually implies that there is a large number of users over many departments and complex business processes behind it. When a new large system is developed, it is almost always to replace an old system that has been in place for decades. The problem with a lot of smaller deliveries is not with software developers. It is the business organizations that can't handle this kind of change. Every time there is a delivery, there will be new training, new methods and procedures, new learning curve, let alone the confusion of dual systems. What you will find is that most users will simply ignore the partial new system and fall back to business as usual.
    Large business organizations are not set up for agile changes. If system replacement is necessary, they want the new system to be complete and staple when it is delivered so that they can handle the change once and for all.
    yadayada1
    • "staple"?

      I must have patronized Staples too much that I can't spell "stable."
      yadayada1
    • Golden

      Too many expect the "golden" key.
      One click does it all.
      rhonin
  • Agile? You keep using that word,

    I do not think it means what you think it means.
    Vesicant
  • Three corrections...

    1. There are 4 Pathfinder JobCentres currently provinding Universal Credit, not One. They are: Warrington, Oldham, Ashton-under-Lyne and Wigan. (Source: http://bit.ly/1eeyfx7)

    2. Para 3.10 of the NAO report says: "The Cabinet Office does not consider that the Department has at any point prior to the reset appropriately adopted an agile approach to managing the Universal Credit programme."

    3. The 1,000 existing claimants were taken on as very tightly defined 'plain vanilla' cases (out of work single people, not disabled, no children, living on their own, renting not owning etc.). Because of the 'lobster pot' principle, if their circumstances change and additional benefits become payable, they stay on Universal Credit, but their monthly statement calculations (made manually by JobCentre staff and re-keyed into backoffice systems) start to become very complicated. (Source: http://bit.ly/Universal-Credit-NAO-Blog)

    4. For up to date information please see my interview on BBC TV News last Thursday here:
    http://bit.ly/bbc-tv-news-september-5-blog

    Brian
    BrianUkulele
  • Cargo Cult Agile

    ozRocker hit the nail on the head... "No magic formula".

    This scenario is yet another classic cargo-cult. The jokers they got to try to roll out Agile clearly did not know or establish even the basics, like say frequently showing the product to "customers".

    And anyone who tries to measure the success or failure of Agile by whether it "ran over time and over budget" simply does not understand Agile. Agile is always on time and on budget, but may run _under_ scope.

    -4johnny
    4johnny
  • Article title is strawman

    The article's hyperbolic title is a strawman: "How even agile development couldn't keep this mega-project on track".

    When the Titanic is already 98% sunk, steering is not going to save you.
    4johnny
  • No "magic formula" (process)

    I agree with 4johnny and ozRocker...
    I think it is becoming apparent to those reviewing the situation.. the players involved simply didn't have a unified understanding and agreement on how "Agile" would be implemented.
    It obviously involves more than the software developers understanding of what "Agile" means.
    Everyone else involved has to understand what it means also ( management, purchasing, oversight, etc...)

    It is apparent they lacked "unity" (in vision, methods, goals).
    Without it .. it doesn't matter what methodology is put in place, it will fail (a housed divided)
    With it.. even poor methodologies can demonstrate a reasonable level of success.

    Without unity among those involved .. results will never prove one method better than another. All will fail in varying degrees. Often with the innocent given blame.
    jrlambert
  • Agile Methods are for small Agile Projects

    First of all, the article perpetuates a fallacy about the waterfall model. The waterfall model is not a development process but rather a metaphor of the classical, disciplined approach to engineering. Basically all it says is that we start with requirements, produce a design, build according to the design and verify that what was built actually satisfies the requirements. Developers well versed in this approach know that it is nearly impossible to define all of the requirements in advance. The only exceptions that I have seen personally are systems designed to replace older systems which serve as prototypes for the new system. But even in these cases, requirements can change as the client learns what is possible with the newer technology.

    Agile is just a series of little waterfalls with one important difference. The link between the system level requirements and the code is often missing. The agile method values working code over documentation (http://agilemanifesto.org/). When the documentation (aka:Design) is missing, the system design resides in the heads of the developers. It is a sad fact, but when the developers leave, management usually allows them to take their heads with them. Once the turnover reaches a certain level, the team will invariably lose sight of the initial vision and the team will get mired down with constant changes and retesting.

    Agile was "invented" by a team of developers whose project was actually canceled (http://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System). The approach is based on a failure. But like many past "silver bullets" it was adopted by a large number of organizations which never really examined the method to see if it made sense. For small projects with continually shifting requirements, it offers some advantages. For large projects where multiple software components have to work together, it typically results in a huge morass. Agile makes subcontracting out parts of a system nearly impossible.
    dilton_dalton
  • Failure or endorsement of agile?

    The executive management seems to have been somewhat unaware of the issues that come with a project of this magnitude. This was stated clearly because they had not "at any point prior to the reset appropriately adopted an agile approach to managing the Universal Credit programme." This was evident with procurement and contract constraints in place before the project had begun. Thus they were using a methodology that was not aligned with their processes -- this is akin to building a car on an ill-fitting frame.
    In addition, they were working with existing systems based on traditional systems that are more methodical, slow, and resistant to changes. Quick and flexible changes within the project could not dovetail easily with those existing systems. The organization did recognize the problem and took corrective action by implementing "Agile 2.0" to create a workable hybrid model.
    The project seems to be gaining traction now in spite of the initial problems. Before the project team was able to make headway with its project, it required some organizational changes to fit the approach they were to undertake. They showed progress after the initial setback by "getting feedback and learning lessons" that enabled them to create a pilot program. Agile was a good choice but they were not suited to it until they adapted to the environment and vice-versa.
    RobKane