© National Audit Office 2022

Modernising ageing digital services

Department for Environment, Food & Rural Affairs

Session 2022-23
6 december 2022
HC 948

The National Audit Office (NAO) scrutinises public spending for Parliament and is independent of government and the civil service. We help Parliament hold government to account and we use our insights to help people who manage and govern public bodies improve public services.

The Comptroller and Auditor General (C&AG), Gareth Davies, is an Officer of the House of Commons and leads the NAO. We audit the financial accounts of departments and other public bodies. We also examine and report on the value for money of how public money has been spent.

In 2021, the NAO’s work led to a positive financial impact through reduced costs, improved service delivery, or other benefits to citizens, of £874 million.

We are the UK’s independent public spending watchdog.

We support Parliament in holding government to account and we help improve public services through our high-quality audits.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 5 December 2022

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Gareth Davies
Comptroller and Auditor General
National Audit Office

30 November 2022

Value for money reports

Our value for money reports examine government expenditure in order to form a judgement on whether value for money has been achieved. We also make recommendations to public bodies on how to improve public services.

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.org.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

011075 12/22 NAO

This report can be found on the National Audit Office website at www.nao.org.uk

If you need a version of this report in an alternative format for accessibility reasons, or any of the figures in a different format, contact the NAO at enquiries@nao.org.uk

The National Audit Office study team consisted of:

Richard Davis, Alex Farnsworth, Sarah Hipkiss and Jonathan Pownall, under the direction of Keith Davis.

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

020 7798 7400

www.nao.org.uk

@NAOorguk

Key facts

Summary

Introduction

1 The Department for Environment, Food & Rural Affairs (Defra) provides many digital services that are critical to the country’s trade, disease prevention, flood protection, air quality monitoring and many other aspects of our daily lives. Many of these services are based on ageing IT systems and infrastructure. Landowners can use Defra’s digital services to apply for grants to plant trees or to provide environmental benefits. Defra provides a range of services for businesses. Businesses can register plant health certificates for import and export, obtain an export health certificate to export or move live animals and animal products, or report the movement of cattle, pigs, sheep, goats and deer. The Digital Assistance Scheme supports businesses moving food and animals from Great Britain to Northern Ireland.

2 Across the whole of government, ageing IT systems are a key source of inefficiency and create a major constraint to improving and modernising government services. These ageing systems are commonly referred to as ‘legacy’. Many departments have legacy systems with similar characteristics: having been built to support one specific business activity, they lack flexibility and keep data siloed for just one purpose; they are difficult and expensive to run and maintain; and there are further hidden costs arising from the need for additional business processes to overcome their limitations. As in most government departments, some of the greatest risks to the services Defra provides arise from legacy systems and technology.

3 In 2020, the Cabinet Office reviewed the legacy systems of eight departments, including Defra. It identified that government had limited visibility and understanding of the risks, compounded by a historical lack of focus on legacy systems during the budgetary process. The Cabinet Office concluded the primary barriers to improvement were that departments found it difficult to articulate the indirect benefits in business cases and had a tendency to cut funding part way through the budgetary cycle.

4 Legacy systems are becoming increasingly expensive to maintain. In July 2021, the Cabinet Office reported that nearly half of all technology expenditure across government in 2019 was dedicated to keeping outdated legacy systems running. This potentially reduces the amount of funding that could otherwise be available for modernising and enhancing such systems or contributing to business transformation. Making the transition from legacy systems to modern replacements is complex. We have found that departments typically do not have a good understanding of their IT estate and its interdependencies, and legacy systems are often poorly understood because of their age.¹

5 There are two main elements in Defra’s approach to tackling legacy: infrastructure modernisation and a Legacy Applications Programme. The Legacy Applications Programme has a four-year funding allocation of £78.5 million. It received funding of £32.2 million for the first year (2021-22) in the 2020 Spending Review.

6 In parallel with its work on legacy, Defra is embarking on a fundamental business transformation process, including consideration of potentially major structural changes to Defra Group. However, this is at an early stage and Defra has not yet developed a clear vision for how its business will operate.

Scope and purpose of this report

7 This report examines whether Defra is addressing its legacy challenge effectively. The report is based on our work specifically with Defra and Defra Group.² However, our findings and recommendations are relevant to all government departments that are seeking to address an ageing IT infrastructure and facing challenges similar to those that Defra faces.

8 Our review included assessing progress made in the Legacy Applications Programme and examining links with Defra’s business transformation programme. The report sets this out in two parts:

Part One assesses the scale of Defra’s legacy challenge and how it has arisen, and the risks to value for money and resilience. It highlights the funding challenges that Defra, in common with all government departments, faces.

Part Two assesses Defra’s plans to tackle its most pressing risks and whether plans for wider digital transformation will achieve value for money.

Key findings

The scale of Defra’s legacy IT challenge

9 Defra has one of the most significant legacy IT challenges of all government departments. In the 2021 Spending Review, which set departmental budgets for the three-year period 2022-23 to 2024-25, the government committed to spend £2.6 billion in cyber and legacy IT. In an early submission for the 2020 Spending Review, Defra estimated that it needed to spend £726 million on legacy over the four-year period from 2021-22, the second largest legacy spend requirement after the Home Office. Defra estimated that legacy accounted for 76% of its total digital, data and technology spend requirement and that it would take until 2030 to resolve all its legacy issues (paragraphs 1.12 and 1.18, and Figure 1).

10 Legacy business applications have proliferated across the Group, resulting in around 2,000 applications. Defra Group has many duplicated and overlapping applications with different versions of products that perform the same or similar functions. Many of these applications were built using software that is now outdated and 30% of them are unsupported. Unsupported applications have a higher security risk, are less reliable and are more expensive to run and maintain. The proliferation of applications is in part because, in the past, there was no centralised digital function for the Group. Digital, Data and Technology Services, Defra’s central team, was established in 2014 and now delivers the large majority of IT for the Group. As well as the large number of applications, there are many databases and spreadsheets that have been developed locally outside the mainstream technology function (these are known as ‘grey IT’). Defra does not know the full extent of these but has allocated funding to investigate sources of grey IT and begin addressing those that present the highest risk (paragraphs 1.13 to 1.15).

11 Years of low investment in Defra’s technology have resulted in a serious risk of critical service failure or cyber-attack. Major security incidents and risks to business resilience are the two top risks on Defra’s corporate risk register. Defra has been trying to deal with its legacy issues for more than a decade, but it was not until the 2021 Spending Review that it had the funding to start to tackle the problem in a strategic and planned way. Defra’s funding for digital investment in the 2021 Spending Review period, 2022-23 to 2024-25, amounted to £871 million. In addition to funding for some of Defra’s largest programmes such as the Future Farming and Countryside Programme, HM Treasury agreed funding of £366 million for general digital investment across Defra Group for the three-year period. This compares with just under £100 million for the three-year period, 2016-17 to 2018-19 (paragraphs 1.4, 1.17, 1.19 and Figure 2).

12 Defra has not assessed the full extent of the additional service costs of continuing to operate its legacy services. Like other departments, Defra does not record the business and people costs incurred as a result of continued use of legacy services, where outdated technology is adding to the costs of operating services. In its strategic outline business case for the Legacy Applications Programme, for example, Defra did not quantify the business-side operational efficiency increases that the programme would bring (paragraph 1.8).

Funding and resourcing challenges

13 Government has not yet addressed the barriers to maintaining and improving digital services created by its funding and business case approval processes. While departments need capital funding to build new digital services, they also need sufficient resource funding to maintain them. The comparative ease of getting capital funding compared with resource funding can lead to departments providing digital services without funding for maintenance costs, which they often omit from their business cases. As departments increasingly use cloud hosting, this introduces new financial and operating models involving more flexible approaches such as ‘pay as you go’ and requiring a shift away from capital to resource expenditure. One of the government’s six missions in its Digital and Data Roadmap for 2022-25 is to develop “a system that unlocks digital transformation”. Sponsored by HM Treasury, this mission aims to address barriers to digital transformation including in business case approval and other financial processes (paragraphs 1.20 and 1.21).

14 Defra has found it hard to develop and maintain long-term plans for tackling legacy because IT budgets are often cut to meet other departmental priorities. The Cabinet Office acknowledged in 2020 that new systems and services are often prioritised above fixing legacy IT. Funding provisions for legacy IT are often insufficient and, in some cases, cut during a budget cycle. Defra obtained funding from the 2015 Spending Review settlement to make efficiency savings in its supplier portfolio, to refresh some infrastructure and to transform some business applications. However, the budget for this was subsequently reassigned to work on EU Exit. Defra told us it designed some of these applications as common platforms, thereby reducing the impact of this. The Environment Agency faced similar challenges in 2016, when its work to rationalise the wide range of regulatory services it provides was curtailed because funding and resources were diverted to EU Exit preparations (paragraphs 1.21 and 1.22).

15 Defra is finding it difficult to recruit and retain people with the right digital skills. There is a digital skills shortage across UK industry and the public sector, and Defra finds it hard to recruit and retain digital talent. One reason for this is that government departments cannot match private sector pay. Over the period from October 2021 to October 2022, Defra ran recruitment campaigns for 244 digital, data and technology roles but could not fill 76 (31%) of these roles (paragraph 1.23).

Defra’s immediate plans for tackling the problem

16 Defra is making steady progress towards completing the initial phase of the Legacy Applications Programme. Defra’s focus for the initial phase of the programme is to reduce security risk and end the use of outsourced data centres as contracts expire. It is stabilising existing applications by moving them to modern, cloud-based hosting but has not been able to make other changes that would take full advantage of the benefits a cloud environment can offer, such as increased efficiency and flexibility. Defra aimed to complete this stabilisation phase by April 2022 but made a slower start than anticipated, in part due to technical problems and software licensing issues. While some improvements to existing systems will be carried out in the initial phase, Defra plans further work to enhance and transform its legacy systems and processes. It expects this to take 10 years, but has not yet developed any detailed plans (paragraphs 1.12 and 2.4 to 2.7).

17 Defra cannot deliver all its aspirations with the funding it has available. Defra received 58% of the funding it bid for in the 2021 Spending Review. According to Defra, this was sufficient to resolve some major operational and cyber risks and automation. However, it was not enough to fund a broader digital transformation of all legacy services or reduce cyber security and resilience risks to an acceptable level. Currently, only 34 of Defra’s 101 transactional services (those that allow users to exchange money for services or update official records) can be used fully online – the majority require users to phone helplines or complete offline forms. Defra Group handles about 21 million customer transactions each year and only around one-third of these are fully digital (those that do not involve paper forms). Defra employs more than 500 staff in contact centres to assist service users. As part of its business transformation programme, Defra has an ambition to fully digitise its top 20 services by 2024-25 (paragraphs 2.8 and 2.15).

Achieving transformation

18 Defra has carried out a comprehensive exercise to identify legacy spending priorities across Defra Group. Defra and its largest arm’s-length bodies (ALBs) have worked together in a formal process to set priorities for its work on legacy. In November 2021, it established a Digital Prioritisation Board, comprising Defra’s senior leaders and representatives from each of its largest ALBs. This allowed a Group-wide consensus on priorities to be reached. In March 2022, after six meetings, it selected 109 projects that it will fund. Defra received £132.3 million for 2022-23 for these projects, and it has identified 97 of the 109 to fund in this first year. By September 2022, it had spent £24 million on these projects. Spending is managed as a single portfolio and progress reported to the Digital Prioritisation Board, which can decide to reallocate funds where underspends occur (paragraphs 2.8 to 2.11).

19 Defra is not yet taking full advantage of opportunities to save money and streamline its business by creating applications that can be re-used across the Group. There are many instances where different parts of Defra Group carry out broadly similar activities and, as a consequence, there are many applications across the organisation that do a very similar job. For example, Defra runs and maintains 37 applications to give permission for something or grant a licence. To avoid duplication and to help share data, Defra has started to create applications which can be re-used across Defra Group. Defra’s principles for digital transformation include deploying common and re-useable platforms, components and functionality and organising and delivering transformation through capability groups that reflect Defra’s core functions such as payment of grants, granting of licences and incident response. However, business areas may still develop new systems and services based on their own specific requirements. Defra’s central digital team encourages collaboration between business areas but it is hard to enforce as each individual project has its own requirements, timelines and budget. Business and technology choices made in one part of Defra Group have implications elsewhere and without stronger oversight of the products and solutions chosen, it will not be straightforward to share applications for similar activities (paragraphs 2.17 to 2.19 and 2.21).

20 Many different technologies are in use across Defra Group, limiting opportunities to standardise and rationalise. The existence of a wide range of technologies is a consequence of how Defra Group has grown and changed over time. Systems were originally developed by different organisations that had their own approach to technology. Different business areas are still making different technology choices based on what they already have, and are not always taking wider corporate considerations into account. Defra has started to develop a set of principles, standards and guidelines to bring greater standardisation but has more work to do to cover all areas where these are needed (paragraphs 2.20 to 2.22).

21 Defra cannot make full use of its data because its data standards have not been applied to its legacy systems. One of Defra’s principles for digital transformation is for all datasets to be centrally catalogued and managed by established data owners. However, the data standards that Defra applies to new digital developments cannot be easily retrofitted to legacy systems. As a result, it is difficult to link databases together for greater depth of analysis. Defra recognises it is not making best use of the data it holds and has recently established a Data Exploitation Board to assess the situation (paragraph 2.23).

Conclusion on value for money

22 Across government, risks to public services posed by ageing technology have been allowed to build up over many years and Defra has been affected more than most departments. Its systems and services are out of date, creating high risks of operational failure and cyber-attack, inconvenience for service users and additional staff and maintenance costs. With the increase in funding in the 2021 Spending Review, Defra has now established a well-designed portfolio of work to deal with its most pressing legacy issues and is beginning to make progress in delivering it. Defra has been pragmatic about what it can achieve for now with a focus on stabilising its position and reducing the most immediate risks. But Defra will only get real value from its digital endeavours when it can start the process of genuine digital transformation and modernisation. Successful digital transformation extends beyond the replacement of business applications and will need engagement and contribution from the whole of Defra. It must be an integral part of the business transformation process and supported by a Group-wide data and digital strategic vision.

Recommendations

23 The following recommendations are addressed to Defra but are likely to be relevant to all departments and government bodies with legacy IT issues.

24 Defra should:

a develop an overall strategic digital vision to apply to the whole Defra Group and put in place governance and management structures to ensure that digital and data considerations are at the heart of the business transformation process. This should include providing a stronger steer on what design and architecture are needed to support business transformation to avoid opportunities for standardisation and rationalisation being missed;

b establish the senior leadership needed, for example through the appointment of a chief data officer, to develop common data models and standards that can be used to support better use of data where there are common capabilities such as grant payments;

c when new applications are being designed and built, ensure that opportunities for re-use across Defra Group are understood and built into the design;

d develop long-term priorities, timelines and funding requirements that extend beyond the stabilisation phase to maintain the momentum currently driving the Legacy Applications Programme as it progresses into the enhance and transform phase;

e carry out an analysis of the reasons recruitment is failing and work with Defra Group HR, the Central Digital and Data Office and suppliers to develop solutions; and

f develop a better understanding of the additional business and people costs of the continued use of unmodernised digital services, focusing initially on its most used digital services, to help inform investment priorities.

1 National Audit Office, The challenges in implementing digital change, July 2021. Available at:
www.nao.org.uk/insights/the-challenges-in-implementing-digital-change/

2 Defra Group is the core department plus the 20 bodies that produce joint reports and accounts with it.

Part One

The origins and nature of the legacy problem

1.1 This part sets out what legacy systems and technology are, why legacy is a problem and how it has arisen. It examines the extent and nature of legacy in the Department for Environment, Food & Rural Affairs (Defra) and how it compares with other government departments.

1.2 Government aims to deliver high-quality digital services to the public and other service users, making effective use of modern technology. Many of the services Defra provides are critical to trade, disease prevention, flood protection, air quality monitoring and many other aspects of our daily lives. A member of the general public using Defra’s digital services can buy a rod-fishing licence, check for flood risk and warnings in their area, or pay to drive through a Clean Air Zone. Farmers and landowners can use Defra’s digital services to apply for grants to plant trees or to provide environmental benefits through the Environmental Land Management scheme. Businesses use Defra’s services to register plant health certificates for import and export or move live animals and animal products, or report the movement of cattle, pigs, sheep, goats and deer. The Digital Assistance Scheme supports businesses moving food and animals from Great Britain to Northern Ireland.

1.3 Defra’s outcome delivery plan for 2021-22 states that “[its] priorities for technology investment in 2021-22 focus on reducing risks from legacy IT and technical debt and investing sufficiently in newer, emerging technologies to improve Defra Group’s most outdated IT and applications estate.³ Working in partnership across Defra Group, we will deliver digital data and platforms services that support ambition in the post-EU world and improve workforce efficiency and agility with the goal of making Defra a digital and data-driven organisation.”

What is legacy and why does it matter?

1.4 Some of the greatest risks to Defra’s services arise from ageing technology: major cyber security incidents and risks to business resilience are the two top risks on Defra’s corporate risk register. This ageing technology is commonly referred to as ‘legacy’. This term applies not only to IT infrastructure, systems and hardware, but also related business processes. Ageing technology leads to inefficiencies as additional people and processes are often needed for services to meet today’s demands. Many departments have legacy systems with similar characteristics: having been built to support one specific business activity, they lack flexibility and keep data siloed for just one purpose.

1.5 Technology becomes legacy when IT infrastructure, systems and applications are no longer supported by their suppliers, are impossible to update or no longer cost-effective to run or maintain and pose a threat to the resilience of vital services. Operational failure of many of its services would have far-reaching consequences. In the event of failure of Defra’s plant health certification service, for example, it would take a week to recover the data lost, during which time trade in plants, seeds and organic matter such as soil would come to a complete halt. The Animal & Plant Health Agency uses a major application which supports a range of business activities including livestock keeper registration and the management of bovine tuberculosis surveillance. It is old, complex, highly customised and built on products that are mostly out of support and has cost £11 million and taken several years to upgrade.

1.6 Defra’s problems with legacy services have built up over the years. They have been complicated by the additional portfolio of digital work it needed to deliver in preparation for EU Exit and to provide new ways of working for Defra staff in response to the COVID-19 pandemic. Often it has extended the functionality of old systems rather than building new systems from scratch. In February 2021, Defra described its position on ageing technology as follows: “Many business-critical applications have been in place for many years and are largely unchanged since their introduction. Their ageing architecture leads them to be less performant, reliable, secure and cost-efficient compared to modern technology. End users are presented with systems that feel outdated, lack functionality and fail to exploit the wealth of data available across Defra. [Defra’s digital team] has been trapped in a situation where any available funding has been applied to just application maintenance and remediation work to keep them functional, secure and compliant, rather than fundamentally modernising their architecture, or enhancing functionality and performance or contributing to transforming the business model.”

1.7 Legacy systems increase government’s exposure to cyber-attack because they are often incompatible with modern security and access control features. Defra has so far escaped major incident but there are many examples across government of attacks with serious consequences. In 2017, the NHS was attacked by a global ransomware called WannaCry. This attack affected at least 80 out of 236 trusts across England and a further 603 primary care and other NHS organisations. All NHS organisations infected by WannaCry had legacy systems such as unsupported Windows operating systems and so were susceptible to the ransomware. In response to this attack, the Department of Health & Social Care and NHS national bodies are working to maximise their resilience and minimise impacts by developing a response plan setting out what the NHS should do in the event of a cyber-attack and ensuring organisations implement critical care alerts and apply software patches.

1.8 Legacy services are also increasingly expensive to maintain. The Cabinet Office-commissioned report, Organising for Digital Delivery, cited government analysis which indicated that nearly half of all technology expenditure across government in 2019 was dedicated to “keeping the lights on activity on outdated legacy systems”. Continued use of unmodernised services incurs staff costs as civil servants spend time knitting together data and finding workaround solutions for inadequate legacy systems, and users are left with outdated processes that are hard to use or paper-based. Like other departments, Defra has not assessed the full extent of the additional people and process costs of continuing to operate its legacy services. In its strategic outline business case for the Legacy Applications Programme, for example, Defra did not quantify the business-side operational efficiency increases that it would bring. Our Challenges in using data across government report highlighted that departments do not measure the extra costs from managing inconsistent and poor-quality data but we found that some civil servants spend between 60% and 80% of their time on this.

1.9 The report Organising for Digital Delivery highlighted the need to address legacy IT systems across government and for government to make better use of the data it holds to improve services. It reported that there is “a universal temptation to invest in development of new features … [rather than in] ensuring security and stability in the underlying platforms”. Although the report acknowledged that the situation is particularly acute in government services, it reported that legacy issues are not unique to government. In a recent survey of private sector chief information officers (CIOs) in the US by the consultants McKinsey, CIOs reported that 10%–20% of the technology budget dedicated to new products is diverted to resolving issues related to technical debt (the future costs that will be incurred in ensuring that a system continues to remain operational and fit for purpose) and that technical debt amounts to 20%–40% of the value of their entire technology estate before depreciation.

1.10 In 2020, government established a legacy IT programme which aimed to better understand the challenges of legacy IT across government and make recommendations to address this challenge. It reported its findings, based on a review of eight departments, in November 2020. Cabinet Office identified that government had limited visibility and understanding of the risks, compounded by a historical lack of focus on legacy systems during the budgetary process. It made recommendations on how government can reduce its reliance on legacy technology, including a recommendation that suitable funding be allocated to address the legacy IT challenges. In response to the recommendations, government, as part of its digital ‘roadmap’ for 2022-25, has committed to securing efficient, secure and sustainable technology by 2025. This includes identifying ‘red-rated’ legacy systems through an agreed cross-government framework and ensuring all those systems identified have an agreed remediation plan in place. Defra is one of six departments working with the Central Digital and Data Office, which leads the digital, data and technology function for government, on the development of this framework.

1.11 Making the transition from legacy systems to modern replacements is complex. We have reported in the past that departments typically do not have a good understanding of their IT estate and its interdependencies and legacy systems are often poorly understood because of their age. In 2019, government produced guidance for departments on managing legacy technology. It identified technical blockers including:

older technology becoming unstable and it being harder to find replacement parts;

difficulty in knowing exactly what data are held and how to migrate these; and

a lack of proper documentation for legacy technology.

It also identified non-technical blockers, including:

cultural unwillingness to change current processes and ways of working;

limited IT budgets that cannot always cover the cost of a legacy migration;

difficulties coordinating with multiple suppliers; and

addressing competing priorities across the business.

Scale of the legacy problem

1.12 Defra has one of the most significant legacy IT challenges of all government departments. In an early submission for the 2020 Spending Review covering the four-year period, 2021-22 to 2024-25, Defra estimated that it needed to spend £726 million (capital and resource) on legacy over this period, the second largest legacy spend requirement after the Home Office. Legacy accounts for 76% of Defra’s total digital, data and technology spend requirement over these four years. Defra estimated that it would take until 2030 to resolve all of its legacy issues.

1.13 Defra Group has duplicated and overlapping applications with different versions of products in use that perform the same or similar functions. This is in part because, in the past, there was no centralised digital function for the Group. Until 2014, each of Defra’s arm’s-length bodies (ALBs) and executive agencies was responsible for its own IT function and for developing its own systems and applications. Digital, Data and Technology Services (DDTS), Defra’s central digital function, was established in 2014 to consolidate and harmonise its digital, data and technology functions and now delivers the large majority of IT for the Group.

1.14 As early as 2016, Defra had started reviewing its digital architecture. At the heart of its approach was to simplify and rationalise Defra’s IT estate, removing duplications and moving from onsite to offsite solutions.

1.15 In July 2022, there were 1,962 applications across Defra Group (including smaller locally developed spreadsheets and databases). Many of these applications were built using software that is now outdated. More than half (55%) of these were in extended support and 30% were out of support. Unsupported applications have a higher security risk, are less reliable and are more expensive to run and maintain. Defra also faces an as yet unquantified amount of IT which is not centrally managed (known as ‘grey IT’) and has accumulated across the Group over the years. These are applications that have been developed outside the central digital team, which is often unaware of their existence. There is a risk that these locally developed and managed applications become the sole source of data on a particular aspect of the business that may have wider use across the organisation. Defra does not know the full extent of this problem but has allocated funding to investigate sources of grey IT and begin addressing those that present the highest risk.

1.16 Livestock tracing is an example of the proliferation of applications and the problems this causes. The current landscape of applications and systems has built up over the years in response to specific disease outbreaks leading to a patchwork of inconsistent policies, processes, systems and operational delivery models. This is further complicated by livestock tracing being devolved across the UK with different administrations implementing their own systems and processes. This makes it challenging for the Animal & Plant Health Agency to respond effectively and quickly to a disease outbreak. Where it affects more than a single species, it has to bring together data from a variety of sources to get a complete picture of traceability and potential contact. It has aimed to consolidate its applications into a more joined-up multi-species database for some years, but has failed to secure investment. However, in the past few years it secured funding and the Livestock Information System Company has so far successfully delivered capability for sheep-tracing and ear-tagging, with cattle and pigs to follow in subsequent releases.⁴

Funding

1.17 Defra told us that its legacy problems had built up over the past decade in part because of a history of under-investment. HM Treasury funding for Defra digital investment, including legacy IT, increased significantly with a funding injection in 2021-22 following the 2020 Spending Review (Figure 1). There was an increase in 2020-21 which allowed Defra to begin work on addressing legacy IT. However, a significant proportion of that year’s funding was needed to cover the additional investment needed for the COVID-19 pandemic response (for example, increased provision for remote working).

1.18 In the 2020 Spending Review, HM Treasury agreed funding for 2021-22 of approximately £600 million to invest in understanding and remediating legacy IT estates across government. In the 2021 Spending Review, which set departmental budgets for the three-year period 2022-23 to 2024-25, the government committed to spend a further £2.6 billion in cyber and legacy IT. Together, the Spending Review settlements of 2020 and 2021 covered just over half of the provisional bids departments had submitted in 2020.

1.19 Although some funding for legacy was provided in the 2015 and 2020 Spending Reviews, it was not until the 2021 Spending Review that Defra had the funding to start to tackle the problem of legacy applications in a strategic and planned way. In the 2021 Spending Review, HM Treasury agreed funding for Defra’s digital investment of £871 million for the three-year period, 2022-23 to 2024-25. As well as significant funding for transformation of several of Defra’s high-priority services (including £205 million digital investment for Borders and Trade and £149 million for the Future Farming and Countryside Programme), Defra received a further £366 million (including £8 million for automation) for other digital investment requirements across the Group (Figure 2). This compares with just under £100 million for the three-year period 2016-17 to 2018-19 and was 58% of what it had bid for. According to Defra, this provided sufficient funding to resolve some major operational and cyber risks and automation, but not enough to fund a broader digital transformation of all legacy services or reduce cyber security and resilience risks to an acceptable level.

1.20 As well as the overall level of funding, having the right balance between capital and resource funding is also important. While departments need capital funding to build new digital services, they also need sufficient resource funding to maintain them. We found in our recent report The challenges in implementing digital change that the comparative ease of getting capital funding compared with resource funding can lead to departments providing digital services without the funding to maintain them. Maintaining digital services requires a continued resource budget investment in business capability. In addition, costs of recruitment and retention of digital expertise are rising in a fiercely competitive jobs market. One of the government’s six missions in its Digital and Data Roadmap for 2022-25 is to develop “a system that unlocks digital transformation”. Sponsored by HM Treasury, this mission aims to address barriers to digital transformation including in business case approval and other financial processes.

1.21 Defra’s digital team told us that its budget is often reduced to make savings, as competing priorities emerge. The Cabinet Office also acknowledged in 2020 that, across government, new systems and services are often prioritised above fixing legacy IT and that funding provisions for legacy IT are often insufficient and, in some cases, cut during a budget cycle. We reported in July 2021 that the maintenance of legacy systems is often one of the costs most likely to be cut or delayed and business cases do not always include these maintenance costs. As departments increasingly use cloud hosting, this introduces new financial and operating models involving more flexible approaches such as pay-as-you-go and requiring a shift away from capital to resource expenditure.

1.22 The need to divert resources into EU Exit preparations disrupted and delayed Defra’s plans to deal with its ageing technology. Defra obtained funding from the 2015 Spending Review settlement to make efficiency savings in its supplier portfolio, refresh some infrastructure and to transform a number of business applications. However, the budget for this was subsequently reassigned to delivering applications needed for EU Exit. Defra told us it designed some of these applications as common platforms, thereby reducing the impact of this. The Environment Agency faced similar challenges in 2016, when it began work to transform and rationalise the wide range of regulatory services it provides, including a focus on decommissioning legacy systems under its Regulatory Services Programme. However, this work was also curtailed as funding and resources were diverted to EU Exit preparations.

1.23 There is a digital skills shortage across UK industry and the public sector, and Defra finds it hard to recruit and retain digital talent. One reason for this is that government departments cannot match private sector pay. The government’s digital ‘roadmap’ for 2022-25 states that the civil service needs to compete more effectively with the private sector for skills. It aims to help departments reduce the time taken to recruit and has an objective for all departments to reduce their digital and data vacancy rate to under 10%. Over the period October 2021 to October 2022, Defra ran recruitment campaigns for 244 digital, data and technology roles but could not fill 76 (31%) of these roles.

3 Defra Group is the core department plus the 20 bodies that produce joint reports and accounts with it.

4 Livestock Information Ltd is a subsidiary of the Agriculture and Horticulture Development Board (AHDB) and is 51% owned by AHDB, with Defra owning the remaining 49%.

Part Two

Progress in tackling legacy services

2.1 This part assesses the immediate plans the Department for Food, Environment & Rural Affairs (Defra) has for addressing its legacy problems, examines the progress it has made to date and how this is supporting its wider business transformation.

2.2 Defra’s approach to modernising and transforming legacy services across Defra Group has several components including:

the Legacy Applications Programme: stabilising business applications by moving them from older data centres onto modern cloud-based platforms;

maintaining essential end-user services by modernising end-user devices, telephony and contact centre services;

establishing robust cyber security controls to mitigate against the very high risks of cyber-attacks and operational failure; and

further extending the use of common platforms to ensure that common platform services are functional and supported.

2.3 Defra has made good progress in updating its infrastructure (including laptops, mobile phones and networks): the proportion of end-user devices that run on ageing or obsolete technology fell from 95% in 2019 to 18% three years later. By July 2022, end-user devices (such as Windows laptops) were mostly using current technology and were supported by Defra’s digital services. Defra expects that all end-user devices will be supported by April 2025. Defra has made slower progress in stabilising legacy applications and legacy middleware and integration and expects 20% of its applications still to be unsupported in April 2025 (Figure 3).⁵

The Legacy Applications Programme

2.4 Defra has developed a four-year Legacy Applications Programme with a funding allocation of £78.5 million. It received £32.2 million funding for the first year (2021-22) in the 2020 Spending Review. The programme aims to address past under-investment, remove critical risks and provide a platform for future development and collaboration across Defra Group. The programme covers the 642 business applications supported by the digital team, as well as the infrastructure the applications are hosted on.

2.5 Defra aims to approach the problem in three stages:

a Stabilise: migrate existing systems to cloud platforms to reduce the risks associated with legacy platforms.

b Enhance: develop new sustainable and technical capabilities to modernise legacy systems.

c Transform: important applications will be selected for transformation by rebuilding and replacing these applications and replacing and decommissioning existing legacy IT as required.

Defra expects the full transformation of its business applications to take around 10 years.

2.6 The first phase of the Legacy Applications Programme aimed to stabilise the existing applications by moving them to modern, cloud-based hosting but making minimal other changes. This was a tactical approach to minimise risks from outdated hardware and software and because Defra needed to leave existing data centres by contractual deadlines to avoid incurring penalties. Defra originally aimed to complete the stabilisation phase in April 2022.

2.7 However, this phase made a slower start in the first year than anticipated. Defra told us it took time to establish a realistic and achievable scope and to establish the right conditions for contractors to work collaboratively. There were also delays while Defra resolved technical issues relating to software licensing and compatibility. Enhancement to applications and transformation are for the most part being addressed in the later stages of the Legacy Applications Programme but Defra has not yet developed any detailed plans for this. Defra told us it is doing small enhancements where it can as part of the stabilisation, but these are not transformational, rather improvements to the user interface.

Modernising and transforming services

2.8 In the 2021 Spending Review, Defra received £366 million for modernising and transforming its technology services over the following three years, including the Legacy Applications Programme. This provided Defra with the opportunity to move beyond the stabilisation stage and begin to plan its digital modernisation and transformation activity in a more planned and strategic way. According to Defra, this was sufficient to resolve some major operational and cyber risks and automation but not enough to fund a broader digital transformation of all legacy services or reduce cyber security and resilience risks to an acceptable level. The Spending Review settlement was 58% of Defra’s bid (£629 million) and so Defra had to prioritise its activities to determine what it could deliver with this money.

2.9 To do this, Defra established a Digital Prioritisation Board in November 2021, consisting of Defra’s senior leaders and representatives from each of its main arm’s-length bodies (ALBs). The group worked together in six meetings using a formal process to set priorities for the programme. The group assigned all activities to one of six prioritisation categories. This allowed the group to reach a consensus on programme priorities without any need, so far, to resolve conflicting views. By March 2022, Defra had identified 109 projects that it will fund out of the original 163.

2.10 The 109 projects selected for funding cover Defra and several of its ALBs. Defra has taken the important step of managing the projects as a single portfolio, overseen by the Digital Prioritisation Board. Underspend in any one project is returned to the Group for reallocation elsewhere in the portfolio. The board considers any forecast overspends and the relevant ALB or Defra’s digital services team may be asked to contribute.

2.11 For the first year of the settlement period, 2022-23, Defra received £132.3 million. Of the 109 projects, it has identified 97 projects to fund with this money in this first year. Although the forecast cost of these 97 projects is £147.6 million, 11% more than its settlement, Defra expects to end the year within budget as it anticipates that some of these projects will spend less than their budget in this first year. As of September 2022, Defra had spent £24 million on these projects. One of the projects is the Environment Agency’s Regulatory Services Programme, which aims to decommission legacy systems and move users to a single system which covers most of the regulatory activity carried out by the Environment Agency. More details of this are shown in Figure 4.

2.12 Addressing the immediate, most urgent issues is a necessary first step but prioritising contractual deadlines has so far limited Defra’s opportunities for undertaking more extensive digital transformation. Defra’s immediate focus has been to exit from arrangements with its current suppliers for running systems in dedicated data centres as contract expiry dates approach. To meet these deadlines, Defra is moving applications into more modern infrastructure provided by cloud hosting as quickly as possible and in their existing condition.

2.13 Moving to a cloud environment in this way is widely known as ‘lift and shift’. However, more extensive modifications to applications are usually needed to allow organisations to maximise the wider benefits the use of cloud services can provide, such as flexibility and efficiency. Other than in limited cases, the programme has not so far had any significant opportunity to enhance core functionality or rationalise duplication in systems.

Funding for transformation activities

2.14 The digital transformation funding Defra received in the 2021 Spending Review was focused on major projects such as the Future Farming and Countryside Programme, and Borders and Trade. Defra has not yet allocated funding for the full range of transformation activities included in its 2021 Spending Review bid. Defra is therefore not yet in a position to deliver its wider transformation aspirations, such as providing a better user experience and making better use of its data. The prioritisation exercise previously referred to identified 69 projects as falling into the transformation category.

2.15 In July 2021, Defra reported that only 34 of Defra’s 101 transactional services (those that allow users to exchange money for services or update official records) could be used fully online and the majority required users to phone helplines or complete offline forms. Defra Group handles about 21 million customer transactions each year and only around one-third of these are fully digital (those that do not involve paper forms). Defra employs more than 500 staff in contact centres to assist service users. As part of its business transformation programme, Defra aspires to ensure its top 20 services are digitised. However, none of this work is yet funded.

2.16 Defra recognises that it could do much more to join up its data. For example, it is developing a new livestock information data integration platform. Livestock information is held in multiple different systems which makes it more difficult to analyse the data across systems and different species. This could hamper the ability to manage a disease outbreak. To address this, Defra is planning to integrate livestock information data on a cloud-based integration service with a view to providing an integrated view across species and devolved administrations. Similarly, Defra operates five disparate geospatial systems and bringing these together into a single service could offer an opportunity to achieve efficiency savings.

Addressing duplication of systems

2.17 Much of the activity Defra undertakes can be classified into broad categories such as making payments, granting licences, regulation, incident response and business reporting. These are undertaken in different ways across different business areas and ALBs using a variety of systems which have been purpose-built or heavily customised according to local ways of working. For example, Defra runs and maintains 37 different applications to give permission to do something or to grant a licence.

2.18 Defra has started to recognise the potential to realise efficiency savings through building single or re-usable platforms for common activities that can be used across the core department and its ALBs with minimal local adaptation. This can avoid the expense of unnecessary duplication and also facilitate better sharing of data. Defra’s principles for digital transformation include deploying common and re-useable platforms, components and functionality and organising and delivering transformation through capability groups that reflect Defra’s core functions such as payment of grants, granting of licences and incident response. This approach does, however, require such platforms to be built in such a way that makes wider use possible in practice.

2.19 Programmes such as the Future Farming and Countryside Programme, and Borders and Trade, have their own funding outside the Legacy Applications Programme and are building platforms through Defra’s central digital function for services (such as grant management or payment systems) that have the potential for re-use elsewhere in Defra Group. However, there is no organisational ‘blueprint’ or target architecture to guide how such systems should be developed. As a result, individual programmes are developing new systems and services based on their own requirements and preferences without explicitly ascertaining that the wider needs are being fully taken into account. Defra’s central digital team encourages collaboration between business areas but it is hard to enforce as each individual project has its own requirements, timelines and budget. This is particularly the case where platforms are being developed to address an urgent need to replace an existing system.

A consistent approach to technology and data

2.20 Many different technologies are used across the Group as a consequence of how Defra Group has grown and changed over time and how systems were originally developed by different organisations within the Group that had their own approach to technology. Defra has not established an overall Group-wide standard for technology and architecture. We were told that, where work undertaken so far on the Legacy Applications Programme has resulted in greater use of common technologies, this has been to an extent coincidental rather than in accordance with an overarching design. This lack of standardisation is a problem within all government departments and, even more so, for policy areas that span multiple departments.

2.21 Different business areas are still making different technology choices based on what they already have and are not always taking wider corporate considerations into account. For example, Defra’s central digital function is building the Regulatory Services Programme for the Environment Agency (Figure 4) using one particular set of technologies.⁶ This is being done with a view to enable re-use. Defra expects Natural England to re-use this platform to some degree. However, other parts of Defra Group would find re-use harder. The Forestry Commission, for example, has expressed an interest in re-use but does not use the same technology platform. The engineering required to make services ‘portable’ across different cloud environments can be challenging. In the absence of clear architectural guidance and oversight at the Group corporate level, teams may end up solving the same problem in different ways. This will inhibit the ability to gain the full benefits of standardisation and rationalisation.

2.22 Defra has started to develop a set of principles, standards and guidelines to bring greater standardisation. These are now in place for laptop builds, cloud hosting and collaboration technologies. Further work is needed to cover all areas where these are needed and Defra aims to complete this by March 2023.

2.23 Similarly, Defra’s senior leadership has not provided direction to ensure that data needs are routinely taken into account in design. The use of disparate systems for similar purposes means that there are no clear data standards which are commonly applied across Defra Group. As a result, Defra is not making best use of the data it holds and it is difficult to link databases together to achieve greater depth of analysis. Defra has recognised that bringing together data held in different systems, for example those relating to animal disease tracking, is a difficult and manually intensive undertaking. One of its principles for digital transformation is for all datasets to be centrally catalogued and managed by established data owners and available and interoperable to service multiple processes and applications. However, the data standards that Defra applies to new digital developments cannot be easily retrofitted to legacy systems. It has recently established a Data Exploitation Board to assess the situation.

2.24 A 2019 report on the Environment Agency’s monitoring and assessment estate, which had grown organically over an extended period of time, identified a number of core data weaknesses including:

multiple independent data repositories;

a lack of consistent naming and data standards and of common data structures or overarching data model;

complex integration of IT systems involved in the overall business workflow;

difficulties and inconsistency in data extraction; and

a wide variety of competing tools used in the presentation and analysis of results.

These characteristics, coupled with the lapse in support for an ageing and technically complex infrastructure, constitutes a real and present operational risk to the business.

Digital and technology influence on wider business transformation

2.25 Defra is considering proposals for reforms to structures and ways of working across Defra Group, including how to reduce duplication between ALBs. Defra is still at the stage of developing its proposals and terms of reference for transformation. There is a risk that decisions taken now in relation to Defra’s digital infrastructure will not be able to accommodate whatever changes are introduced. Business transformation and digital transformation are closely linked but continued effort is needed to consistently embed digital and data considerations into the heart of the organisation. Without sufficient clarity on a vision for what the transformed Defra will look like, it is difficult for Defra’s digital specialists to prepare. Decisions are having to be taken now, for example on data architecture, that may need to be reversed when the new Group structure starts to be implemented and this could result in systems having to be rebuilt.

2.26 The transformation programme is at an early stage of maturity and numerous officials we interviewed expressed the view that digital specialists do not yet have a strong enough voice in helping to shape transformation options. Digital is one of four ‘legs’ that support transformation (the others being policy and delivery, outcomes focus, and skills and capability) rather than being at the forefront of developing proposals. There have been recent improvements, including work with the business transformation team to develop an approach to building a digital service transformation programme that was approved by Defra’s Executive Committee in September 2022. These have led to digital specialists playing a more central role in discussions and there is a growing understanding among policy officials that digital, data and technology must play a central role in the business transformation process. However, there is further work to do to align the different perspectives of policy, future business needs and digital, data and technology so there is a clear and consistent vision and ‘blueprint’ for Defra’s transformation and to create a digital and data-driven organisation. Digital transformation and service transformation are key components of a future Defra operating model. This will require increased digital capability across all areas of Defra’s business.

5 Middleware is software that helps programmes and databases work together including when they are on different computers.

6 Microsoft Power Apps in the Microsoft Azure cloud.

Appendix One

Our evidence base

1 This report evaluates whether the Department for Environment, Food & Rural Affairs (Defra) is managing its spending on legacy technology effectively. Our review includes Defra’s progress on the Legacy Applications Programme, whose current focus is on mitigating Defra’s most pressing risks. It also examines how Defra is preparing for potential structural changes to Defra Group arising from its business transformation process.

Interviews

2 We carried out 27 interviews, four with the Environment Agency and 23 with Defra officials. These interviews took place between June and August 2022 and were carried out online. They covered a variety of topics including how Defra is tackling the legacy issue, the transformation process and how Defra is liaising with its arm’s-length bodies (ALBs) to create common platforms.

Document review

3 The purpose of the document review was to gain understanding and insight into the approach taken by Defra in its Legacy Applications Programme, how it prioritised the projects and the background to the legacy situation. We reviewed the documents between June and August 2022 and they included, but were not limited to, documents such as Spending Review settlements, the legacy IT strategic narrative, prioritisation documents and Defra Legacy Applications Programme board papers.

4 We used the information from these documents to provide background information and to inform the study team of the approach and management structures used by Defra and its ALBs. It also provided insight into how Defra was prioritising the funding it received and how comprehensive its approaches were.

Limitations

5 Defra was unable to provide any documents related to its funding for the 2010 and 2015 Spending Reviews. While there were some documents for the 2020 and 2021 Spending Reviews, these provided little detail on spending specifically on legacy.

Case studies

6 We selected case studies to highlight the issues raised in interviews and document reviews. As these were illustrative, we did not explore these in detail. The case studies took place through initial interviews with people leading the programmes and followed up with document requests to provide us with more information and evidence of progress. We selected case study topics through discussions with Defra to identify programmes that provided insight into some of the challenges Defra faces. The case studies were the Regulatory Services Programme, the Future Farming and Countryside Programme, geo-spatial and business information, the livestock data information programme and the animal health customer account management system.

Design and Production by NAO Communications Team
DP Ref: 011075-001