Since we first published our Framework to review programmes in 2017 there has been no let-up in NAO reports on major projects and programmes, most recently on Crossrail, the Emergency Services Network and the Stonehenge by-pass road. From the need to manage the risks of untried approaches to signs warning of unrealistic cost estimates, this blog highlights some of the themes emerging from our recent work on major programmes, which have been incorporated into our updated guidance.
In 2017 we launched our Framework to review programmes. Since then we’ve published another 36 reports on major projects, gaining further insights that we’ve used to update the Framework. These are in addition to our work on government’s preparedness for exiting the EU, which illustrates some of the same challenges. We’ve also published a Survival guide to challenging costs in major projects.
Our reports cover the implementation of policy and business change as well as traditional infrastructure projects, and the findings we have brought together here are drawn from both. There are common themes across types of programme: the importance of being clear about the intended benefits and developing and delivering an approach that achieves them; the need to choose realistic and deliverable options; and the importance of ensuring there are the resources to put programmes into practice.
Signs of improvement
Over the last few years we’ve seen better project discipline in government. For instance:
- In Managing the Bank of England’s Central Services we found the Bank of England had a well thought through strategy for its reorganisation.
- In The Equipment Plan 2018 to 2028 we found that the Ministry of Defence had made more realistic forecasts of project costs.
- In Cross-government funding of research and development we found that effective leadership made a difference to the effectiveness of R&D funding.
Outputs delivered, but outcomes unclear
Our report Projects leaving the Government Major Projects Portfolio showed that most projects produced the outputs they set out to, but it was more difficult to say they had achieved the planned benefits. In some cases it was not clear what the intended benefits were, as some projects did not set out their intended outcomes in a business case.
Example from new Framework: In Rolling out smart meters we found not only that the original ambition of installing smart meters in every home by 2020 will not be met, but that gaps in monitoring are impacting on the programme’s ability to achieve the intended energy savings and pass these on to consumers.
Risky approaches being chosen
Many of our reports are about programmes where a risky way forward has been chosen. Our option appraisal examples show ambitious options being chosen without evidence that their benefits would justify the extra risk.
Example from new framework: In The new generation electronic monitoring programme we found that the Ministry of Justice (MoJ) chose to develop a new, ‘world-leading’ ankle tag and then also selected the highest-risk approach to the procurement. The process led to disputes with suppliers and two failed procurements. The MoJ finally concluded that abandoning the original plan was the least-bad option. After learning costly lessons and incurring significant delay, the MoJ changed its approach, purchasing off-the-shelf tags and bringing the integration function back in-house.
Consequences of trying to do too much
Recent examples have made it clear how departments cannot fulfil their objectives if projects cost more than they can afford. For instance the original intention of the NHS vanguard programme to integrate health and social care services was not realised because funding was reallocated to deal with NHS Trusts’ financial deficits. The Ministry of Defence’s Equipment Plan is forecast to cost £7 billion more than its budget, but the Ministry does not have much scope to reduce funds for other budgets.
Optimism in one area has consequences elsewhere. Cost increases and delays leave departments facing unpalatable options. They may have to reduce the scope, delay the project or take funding from other priorities. For example, cost increases on Network Rail’s Great Western Railway modernisation and other rail programmes meant work was deferred and ultimately electrification projects on the Great Western, Trans-Pennine and Midland mainline routes were cancelled, as we set out in our investigation. Our report on Early progress in transforming courts and tribunals also showed how the programme reduced its scope and therefore its planned benefits because its initial timetable was too ambitious.
Cost estimates need to be challenged
With increasingly complex projects, it is not easy to reach a realistic initial cost estimate, especially when under pressure to make firm commitments early on. Those responsible for programmes can be well aware of the risks, but with many factors to consider it can be hard to spot the warning signs and to challenge enthusiastic backers of projects. To help them, our Survival guide to challenging costs in major projects sets out warning signs, dos and don’ts, useful questions to ask about cost estimates and management, and a glossary of costing terminology.
Some warning signs
- It’s a “ground breaking” project”
- Alternatives are ruled out too early
- Soft benefits that take decades to deliver
- “Don’t worry, the contractor bears all the risk”
- The project is delayed to deal with cost overruns
Complex programmes are difficult to put into practice
Ambitious approaches also create challenges for later phases of programmes. Our recent blog on getting contracts right highlighted the difficulties of achieving workable contractual arrangements for complex programmes such as Crossrail and the Emergency Services Network programme. By contrast, the London 2012 programme, justifiably regarded as a successful programme, had contingency plans that allowed it to overcome last-minute difficulties.
Example from new framework: In Update on the Thameslink Programme we found that inefficiencies and service delays arose because Network Rail had not initially set up its processes to deal with the volume of design change that turned out to be needed when conditions differed from those expected. Network Rail subsequently improved its financial control of the programme, introduced measures to improve the way it managed design changes and used more sophisticated cost forecasting techniques.
Building project management capability in the civil service remains a challenge
The government’s portfolio of major projects would be huge and complex, even without departmental commitments to EU Exit work. Our March 2017 report, Capability in the civil service, showed that government had identified skill shortages in digital, commercial and project delivery skills. And in People and skills: The role of the centre of government we reported that the Infrastructure and Project Authority was helping departments to build up their project and programme management capability, but they are reliant on a limited pool of experienced people.
Even the best designed programme can founder if there aren’t the skills to manage and deliver it in practice.
We hope that our framework of questions and costing guide help programme managers to address the unquestionably huge challenges they face.
Major programmes will undoubtedly continue to feature in the NAO’s work. The NAO always welcomes your feedback on our guides and our posts and we invite you to contact us directly.
Author
Sandy Gordon
Sandy Gordon is an Audit Manager with over 25 years’ experience in conducting value-for-money studies, specialising in examining the implementation of new services and major change programmes across government. Sandy has led the NAO’s Project Delivery team, working to improve the NAO’s approach to examining programmes, including by supporting NAO teams’ use of this Framework to review programmes. He is leaving the NAO, after 33 years, at the end of July.
