To successfully develop an AI programme, you’ll need strong governance processes because of the risks related to lawfulness, security, bias and data. Regardless of whether these processes are already built into your existing governance frameworks or implemented as a new governance framework, they should focus on:
continuous improvement through the inclusion of new knowledge, methods and technologies
identifying and working with important stakeholders representing different organisations and interests, including Civil Society Organisations (CSOs) and sector experts. This will help create a balanced view throughout the life cycle of any AI project or initiatives
planning for the long-term sustainability of AI initiatives, considering scalability, long-term support, maintenance, ongoing stakeholder involvement and future developments
You should manage governance of AI through an AI governance board or AI expert representation on an existing governance board. You can include an ethics committee as part of your governance framework depending on your operating context. Each has different roles and responsibilities.
AI governance board or AI representation on an existing board
The role of an AI governance board or representation on a board is to provide oversight, accountability and strategic guidance to help the organisation or team make informed decisions about AI adoption and use. It covers aspects such as risk management, compliance, assurance, resource allocation, stakeholder engagement, and aligning with business objectives and ethical principles.
An AI governance board helps you make sure your project is on track and that strategic, legal, ethical and operational risks are managed.
Ethics committee
The primary focus of an ethics committee is to assess the ethical implications of various actions, projects and decisions about AI within an organisation or programme. The committee will evaluate AI from an ethical standpoint, focusing on values such as fairness, transparency and privacy, and is more specialised than that of an AI governance board.
An ethics committee usually includes legal experts, representatives from other relevant organisations related to the service you’re delivering and community members and stakeholders. All of these members can provide a specialised perspective on ethical matters such as health and security issues.
Before creating an ethics committee, you should consider the ethical, strategic and operational context of your organisation or programme. For example, the department may be too small or the programme too low risk to have a committee like this. It might be sufficient to have an AI governance board or an AI expert representative on a programme board to help you manage ethical considerations. An AI governance board should be able to guide you on whether you need an ethics committee. Refer to the ethics section for more information.
Creating an AI systems inventory
To provide a comprehensive view of all deployed AI systems within an organisation or programme, organisations should set up an AI and ML systems inventory. This is in addition to the Algorithmic Transparency Recording Standard Hub that all government departments and certain arm’s length bodies must use to ensure public transparency around the algorithmic tools used in their decision-making processes.
A live inventory will help your team, your organisation and your stakeholders understand the scope and scale of AI usage. It does this by providing better oversight and awareness of any potential risks such as data quality, model accuracy, bias, security vulnerabilities and regulatory compliance. An inventory will also be helpful for audit purposes.
The inventory should be regularly kept up to date with:
a description of each system’s purpose, usage and associated risks
details like data elements, ownership, development and key dates
use protocols, structures and tools for maintaining an accurate, detailed inventory
You should consider sharing your inventory with the AI community of practice. This will enable the community to support your work and connect you with the teams that have developed similar projects across government so that you can share expertise and best practices and possibly reuse existing solutions.
Governance structures for teams
For all programmes or services that use AI systems, teams should:
set out how the AI model will be maintained and managed over time
develop a comprehensive plan for knowledge transfer, and for training new and existing staff to ensure the model’s sustainable management
establish clear roles and responsibilities to ensure accountability within teams, including who has the authority to change and modify the code of the AI model
ensure diversity within the project team by incorporating a range of subject matter expertise, skills and lived experiences
establish pathways for escalation and identify key points of contact for specific AI-related issues
adopt a risk prioritisation plan with specific project controls throughout the delivery and post-delivery cycle, such as how you will evaluate data sets for bias
establish a data reporting mechanism that captures how data flows are managed and maintained throughout the delivery and post-delivery cycle
set out how the programme or project team will work with and report to their programme boards and the ethics committee, if one has been set up
Managing risk
Risk management is part of governance. It helps you to strategically plan and manage your AI project to achieve objectives and respond to challenges in an agile way.
A risk assessment is critical in ensuring that AI projects are only undertaken if the potential benefits outweigh the risks. You should base this threshold on an objective assessment of the project's potential risks and benefits, and by defining acceptable levels of risk and ensuring that any possible risks are identified and addressed early in the project life cycle. Relevant laws, regulations and ethical considerations should inform the assessment.
If you’re managing AI programmes as part of a portfolio of work, the Orange Book and its Portfolio Risk Management Guidance provide a complete overview of risks. You can use the Central Government’s Assurance Directory (CGAD) guide to navigate the numerous standards, codes and guidance documents currently issued by the centre of government which complement the Orange book. You should also refer to the Cabinet Office guidance on a human-centred approach to scaling and de-risking AI tools to consider the hidden risk of AI in light of the cultural, organisational, and human factors that can influence AI adoption and design.
As part of the design of your project, you should conduct a risk assessment to understand the risks and their potential to cause harm to individuals or groups, as well as the likelihood of the AI service being misused or exploited. The impact should be calculated based on the complexity of the AI system, the quality of the data used to train the system, and the potential for human error or malicious intent.
When conducting your assessment, you should consider a number of risks around AI including security, managing bias, legal and operational risks. Consider also that the scale of autonomy of an AI service can increase operational risks. For example, in the case of autonomous vehicles, the Society of Automotive Engineers’ Levels of Driving Automation ranks autonomy on a scale from 0 (no autonomy) to 5 (full automation for all features under all conditions). This scale correlates to the level of risk.
Alongside the risk assessment, you need to create a robust risk management framework that sets out defined roles and responsibilities and includes clear escalation routes to help mitigate risks.
Mitigating risks
You can mitigate some risks related to how the AI service performs by building or establishing programme and technical guardrails, or best practices. These will guide the design, implementation and operation of an AI service or application, and are an essential element of delivering great services.
In the case of autonomous AI services that make decisions in areas such as social care or healthcare, the impact of autonomy of an AI service can be mitigated by including human intervention. These decisions need to be made in a controlled environment so as to not reintroduce bias into the AI service.
Whether your AI service is autonomous or includes elements of human intervention, it should be evaluated throughout the all stages of the project life cycle, including design, development and operation. Your risks and mitigation strategy should also cover how your team will manage continuous performance monitoring to prevent biased or inaccurate outputs.
There are also security and data protection risks which are covered in detail in the AI security risks data protection and privacy sections.
AI quality assurance
AI quality assurance ensures that the AI service meets the service level requirements and provides evidence that the service is fit for purpose. It helps you to check that robust techniques have been used to build, test, measure and evaluate AI systems. It also helps organisations communicate that their systems are trustworthy and aligned with relevant regulatory principles. It should be used throughout the AI service life cycle, including during testing and validation in the development phase and monitoring once the AI service is being used.
To meet quality assurance requirements, AI systems must be trustworthy, accountable, transparent and robust. They must ensure safety, respect privacy, mitigate bias, ensure fairness, and be secure and resilient. Given the complexity of AI systems, you may require a toolbox of different products, services and standards to ensure their effectiveness. For example, DSIT’s introduction to AI assurance identifies the key elements of an assurance process which includes risk assessment, impact assessment, bias audit, compliance audit, conformity assessment and formal verification. To ensure your AI tools are properly tested and assured, you can also use the AI Testing and Assurance Framework for Public Sector designed by the Cross-Government Testing Community.
The Magenta Book and associated guidance on the impact evaluation of AI interventions offer a structured, evidence-based approach to assessing the effectiveness and value of AI systems. Integrating evaluation activities throughout the AI lifecycle supports the development of trustworthy systems by ensuring that your services are built using robust methods and are continuously assessed for performance, unintended consequences and real-world impact.
Validation of AI
Being able to assert the quality of an AI service is critical in ensuring both the safety of the system and the reliable accuracy of the service.
You must be sure that any updates to the AI system have a quantitative testing and validation process as part of the change control process. Validation is part of the testing of an AI system. ISO9000:2015 - Quality management systems specifies that it is ‘confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled’.
The deployment of AI systems that are inaccurate, unreliable, or poorly generalised to data beyond their training creates and increases negative AI risks and reduces trustworthiness. You should consider the complexity of your AI systems and identify the different products, services and standards necessary to ensure their effectiveness. To do so, you must also make sure that these products and services comply with the required standards as defined by standards development organisations (SDOs), such as the International Standards Organisation (ISO).
Operational monitoring
Once you’ve released your AI system for use and it’s operational, you should have ongoing performance monitoring in place. This will ensure your system is operating as expected, and you should be able to provide evidence of this. It will also help you to identify and manage any changes to the model.
Any updates to the model need to go through a managed release process. This will help you to mitigate the impact of any process change and to clearly document changes made for future reference. You should ensure that the release can be withdrawn and the system reverted to an earlier version if required.
As systems and environments evolve, the current process may diverge sufficiently from the training period of the AI system. This is known as model drift and may require retraining or implementation of a new model within the AI system. Close monitoring is essential so that you can catch this as early as possible and reduce possible disruption to the AI system.
Practical recommendations
Connect with your organisation’s assurance team and review the portfolio of AI assurance techniques.
Set up an AI governance board or include AI experts on existing governance boards.
Consider setting up an ethics committee made up of internal stakeholders, cross-government stakeholders, sector experts and external stakeholders like Civil Society Organisations.
Set up an AI/ML systems inventory to provide a comprehensive view of all deployed AI systems within your department.
Make sure your programme or project teams have clear governance structures in place.
Conduct an assessment of your AI product to identify potential risks and implement a robust mitigation strategy.
Use quality assurance techniques to make sure your AI product is trustworthy, accountable, transparent, robust, secure and resilient, and that it respects privacy, mitigates bias and ensures fairness.
Conduct evaluation activities in accordance with the Magenta Book.
Make full use of the training resources available, including the courses on the business value of AI and understanding AI ethics on Civil Service Learning.
AI governance
This information complements and updates the AI Playbook for the UK Government . It reflects the latest guidance and best practice. This will be updated as needed.