Responsibly buying AI

This guide is structured around the specific roles involved in the commissioning and procurement of AI within councils in England. To effectively utilise this resource, users should navigate using the sidebar to locate questions and prompts tailored to their particular role in the process.

In an AI context, data protection law applies to personal data issues e.g. when organisations use personal data to train, fine-tune, or deploy an AI-based technology, and when they use AI-based technologies to process personal data or make decisions about people on that basis. As a starting point, data protection law – specifically the UK GDPR, which needs to be read alongside relevant data protection legislation including, the Data Protection Act (2018) – requires:

• A valid lawful basis to process personal data, which must be determined before you begin processing personal data, based on your purpose and relationship with the individual whose personal data you will process.
• Compliance with the seven key principles of the UK GDPR – these lie at the heart of the data protection law and fundamentally inform everything that follows.
• A DPIA when the personal data processing is likely to result in a high risk to the rights and freedoms of individuals. Examples of such processing include the use of innovative technologies, such as AI.
• You must have appropriate measures and records in place to be able to demonstrate your compliance. You must be able to demonstrate that you have appropriately assessed and mitigated risk following data protection principles.

The ICO has published extensive guidance to explain the requirements of the UK GDPR and Data Protection Act (2018) including the data protection audit framework and specific AI-guidance to support this. A DPIA is a necessary way to demonstrate your compliance. In the context of AI-based technologies, a DPIA should begin by including:

• a systematic description of the processing activities, including data flows and the stages when AI processes personal data, including any solely automated decisions that may produce significant effects on individuals
• an explanation of any relevant variation or margins of error in the performance of the system which may affect the data protection law fairness principle
• a description of the scope and context of the processing, including the data processed, the number of data subjects involved, the source of the data, and the extent to which individuals are likely to expect the processing. 

A DPIA is a foundational document for data protection compliance. It is also a powerful tool for understanding the benefits and risks of deploying an AI-based technology, identifying who you need to consult, and setting out roles and responsibilities. You should involve your Data Protection Officer (DPO) at the earliest possible stage of procuring an AI-based technology and in developing a DPIA.

Full information on how to conduct a robust DPIA in the context of AI-based technologies is available in the ICO’s AI guidance and DPIA advice.

The Procurement Act 2023

Councils will need to ensure that the considerations noted in this guidance are made at the appropriate point of the procurement process and full consideration is given to the legal requirements laid out in the new Procurement Act. Officers should engage with procurement and legal professionals in their organisation to ensure this. 

Human Rights considerations

The Human Rights Act 1998 requires that everyone who works in public authorities must act in a way that is compatible with the Act. The close relationship between equality and human rights means it could be efficient to think about them together when developing policies. Therefore, you may also want to consider and gather evidence on whether your policy can have a potential impact on: 

• groups of people sharing a characteristic that is not specifically protected under the Equality Act 2010, for example, people living in different geographical areas, children on free school meals, asylum seekers, refugees, or single parents
• human rights as set out under the Human Rights Act 1998, for example, the right to family life or to be free from torture and degrading treatments.

Ethical practices

While this guide does not specifically address ethics, ethics as a system of moral principles has overlap with both PSED and data protection’s statutory principles, as you assess equity of access, equity of treatment and protection of people's rights and freedoms as outlined in the Human Rights Act 1998. Some organisations have an ethics board or processes to support ethical decision making. For delivery of health and social care services, you may employ a Caldicott Guardian. Any existing ethical processes within your organisation should be brought into the processes for commissioning, procuring, and delivering services using AI-based technologies.

Algorithmic Impact Assessment

While not required by law in the UK, you may benefit from undertaking an Algorithmic Impact Assessment (AIA) that promotes investigation into AI-specific and automated decision making risks or otherwise integrating elements of AIAs into organisational DPIA and EqIA templates. If you work with international organisations, especially from the EU member states, you may need to consider an AIA.

Exemptions for sensitive data

In addition to special category data, a council should consider what extra protections it should put in place for other sensitive data such as social care provision data. These extra protections should be determined depending on the risk of the data and service area where the technology could be deployed. For example, a council which is processing data, such as that related to abuse monitoring, should consider legal conditions for these activities. The ICO has specific guidance on the legal conditions for processing sensitive data, as required by law, including data related to health and social care.

Under the PSED, public authorities must assess the potential impact of a new policy on people with protected characteristics under the Equality Act 2010. They must do so before making the decision to implement the new policy and continue monitoring its equality impact after implementation. 

While you are not legally obliged to consider other characteristics not listed in the Equality Act, your council may want you to consider the impact on other characteristics such as people who are care experienced, or socio-economically deprived. You should speak to a subject matter expert in your organisation such as an EDI officer if you are unsure of additional considerations that may need to be made.

If you are considering contracting out an AI-based technology, you will need to collect robust and sufficient evidence to support the questions and prompts laid out here. The evidence you collect to answer these can be quantitative or qualitative and will help you establish whether your proposal to commission an AI-based technology:

• could have either a positive or negative impact on people with protected characteristics
• is relevant to one or more of the three aims of the PSED.

Considering the below questions may lead you to decide that AI-based technology is not suitable for your needs at this time, or you may need to reconsider your objectives and approach.

• Could the use of the AI-based technology contravene actions prohibited by the Public Sector Equality Duty? Could using this AI-based technology lead to direct or indirect discrimination, harassment, victimisation, or any other conduct prohibited by the Equality Act 2010?
• How will service users and employees be affected? Could using AI-based technology affect how service users or employees access services or participate in activities relevant to your policy area?
• Could the use of the AI-based technology disproportionately impact people with particular protected characteristics? Such as those who have a low level of access to services, participation in public life or other activities.
• How will it impact your communities? Could it create or worsen disadvantages and inequalities in your community? Could it remove or minimise disadvantages and inequalities in your community? Could it lead to prejudice, community tensions, conflicts, isolation, or segregation? Could it help tackle prejudice and promote understanding between people with different protected characteristics? Could using AI-based technology affect how people perceive or interact with others?
• How does the AI-based technology impact on people with protected characteristics? Does using AI-based technology consider the needs of people who share a relevant protected characteristic? Could the AI system have a greater negative impact for people who belong to more than one marginalised or disadvantaged group? For example, individuals such as Black women, disabled children, or elderly Muslim women may face compounding harms due to the intersection of characteristics such as age, race and religion, especially where those characteristics are already linked to social or structural disadvantages.

If your evidence shows that your proposal to commission an AI-based technology may have an impact on people with a particular protected characteristic, we strongly recommend you seek their views, either directly or by consulting with organisations that represent their interests. A public body cannot avoid complying with the PSED by claiming that it does not have enough time or resources to do so. The methods and degree of engagement should be proportionate to the size and resources of the body and the significance of the issue. This should be recorded.

Examples of engagement may include:

• meeting with a stakeholder group that represents people with a particular protected characteristic
• a focus group with relevant voluntary or community organisations
• a full public consultation or face-to-face consultation event.

For more information on carrying out engagement in councils see the LGAs guide to engagement: New Conversations.

The amount of time and resources a public body puts into considering the equality impact of a particular policy must be proportionate to its potential impact on people with different protected characteristics and its relevance to the three aims of the PSED.

For example, decisions an organisation makes about procuring AI-based technologies to manage traffic will have less relevance to the PSED and people with protected characteristics than decisions to procure technologies to identify potential criminal behaviours or vulnerable citizens (e.g. homelessness, poverty, Not in Education, Employment or Training). More time and effort should therefore go into considering the potential impact of the latter.

When you do not anticipate that using the AI-based technology you seek to procure can lead to unlawful discrimination but is still likely to have a negative impact on advancing equality of opportunities and/or good relations between people with different protected characteristics under the Equality Act 2010, you must consider the actions your council could take to:

• mitigate any negative impact using the AI-based technology may lead to for people with protected characteristics
• meet the needs of disabled people specifically - these actions are called reasonable adjustments and are covered under a separate specific duty in the Act.

Finally, you may want to consider positive action measures your council could take to alleviate disadvantage or under-representation or meet the particular needs of those who share a protected characteristic and ensure that they have the same opportunity to use/benefit from the technology you seek to procure.

In doing so, we would encourage you to think about the actions and adjustments you should make and which ones you could reasonably expect AI providers to:

• have made in developing the technology they are selling to you, or
• make as part of their contracting arrangements with you.

Deciding which actions and adjustments you expect AI providers to have made/make will help you build your contract specification, tender, interview questions and thereafter your contract arrangements.

For more information on conducting robust EqIAs, the EHRC has produced a 10-step guide for public authorities in England.

Depending on the context, the use of AI-based technology is highly likely to involve a type of personal data processing that can result in a high risk to individuals’ rights and freedoms and will therefore trigger the legal requirement for you to undertake a DPIA. You should assess this on a case-by-case basis, and the earliest stage of procurement is an ideal time to do this. 

Considering these questions may lead you to decide that AI-based technology is not suitable for your needs at this time, or you may need to reconsider your objectives and approach. If you proceed with procurement and identify a high risk that you cannot sufficiently mitigate then you are legally required to consult with the ICO before any processing takes place. If your DPIA identified a high risk, but you have taken measures to mitigate this risk, you do not need to consult the ICO.

• Why and how are you going to use an AI-based technology to process personal data? You should begin by clearly understanding your aims and why you intend to procure an AI-based technology. This could incorporate existing documentation, such as project proposals. This is an ideal time to engage your DPO. To help understand the risks, some questions you should consider include:
  • Does the AI-based technology involve processing of special categories of personal data?
  • Is there any systematic monitoring of publicly accessible areas on a large scale?
  • Is there solely automated processing, including profiling, on which decisions are made that produce legal or similarly significant effects?
  • Is this a new technology or novel application of an existing AI-based technology?
  • What are the intended outcomes for individuals or wider society, as well as for your organisation?
     
• What is the nature, scope, context, and purpose of any processing of personal data? This can be a challenging task due to the complex nature of both AI-based technologies and councils. Data flow diagrams are an important tool here. You need to clearly describe the data processed, the number of data subjects involved, the source of the data in this way, and the extent to which individuals are likely to reasonably expect the processing of their data in this way. You can reproduce technical information from an external provider to help with this process but remember that the proposed supplier may not be the original developer of the AI-based solution. Specific questions you may want to ask include:
  • Do your agreements and contacts leave space for the expectation of the supplier to use any personal data for its own purposes, including potentially training their AI-based technology?
  • Is personal data leaving or entering your organisation, and if so, what is the destination or source? Will the data be subject to any international transfers?
  • Is solely automated decision-making taking place that produces any effects on individuals? Is human review meaningful? Being documented? Are human reviewers adequately trained, and do they have the authority to override decisions?
     
• Have you thoroughly consulted everyone? It is good practice for your DPO to be involved in your procurement process from the start. Under the accountability principle of the UK GDPR, as a controller you are responsible for understanding how the AI-based technology works to comply with the data protection law and should be able to explain this to consultees. Your DPO should work collaboratively with data scientists or AI engineering teams. Your DPO should be mindful of obligations of the law enforcement and intelligence service processing regimes. You should also seek and document the views of individuals whose personal data you process, or their representatives, on the intended processing of your AI-based technology. For example, you should consider if the processing involving AI would require seeking appropriate consent from the users. You may not engage with them if it would be disproportionate or impractical, but you still must be transparent about it to them. Questions to ask include:
  • Is the DPO engaged from the start of the procurement process and the development of the DPIA?
  • Have independent domain experts who have a deep understanding of the context in which the AI-based technology deployed been consulted?
  • To what extent have local residents or their representatives been consulted about the impact of the AI-based technology?
  • Is there a diverse, well-resourced team supporting this procurement process, including data scientists, engineering teams, senior management, the DPO and commissioners?
     
• Have you assessed the necessity and proportionality of the processing? You should consider if processing personal data using AI-based technology is necessary and proportionate to the purpose for which you require it. You should consider, and document, whether you could achieve the same goals without the processing of personal data involved in the proposed use of the AI-based technology. Of course, this will be case specific. This may also lead you to conclude that procuring AI-based technology is not suitable for your intended purpose, nor poses a sensible solution to a problem. Some key questions to ask yourself include:
  • Do individuals expect their data to be processed in this way, and have you done any engagement to check, where it is proportionate and practical?
  • Are the relevant rights and freedoms of product users fairly balanced and does this AI-based technology complement or replace human decision-making?
     
• Have you identified and assessed the risks to individuals? You should document and score each risk you identify. Beyond rights and freedoms, you should carefully consider the potential impact of other material and non-material damage or harm on individuals. You should also consider legal frameworks in this context, such as the Equality Act 2010 and PSED. If the data special category or a high volume, this will increase the overall risk for whether the use of AI is safe and proportionate. Question you should consider include:
  • Does the processing involve special category or criminal offence data, or include the data of children?
  • Are there concerns over this type of processing or any security flaws in the AI-based technology, such as model inversion attacks?
  • Has the supplier clearly communicated and addressed the potential fairness, accuracy (including bias issues) in their AI-based technology? Do they continue to monitor the risks?
  • Does the training data, including any fine-tuning data controlled by the council, reproduce or introduce any biases?
  • What is your relationship with the individual data subjects? How will you tell individuals what is happening with their data?
     
• Have you identified measures to reduce risk? For every risk you identify, you should also identify measures to reduce or eliminate the level of assessed risk. You should ensure that your DPO is involved in this process You must consult the ICO if you cannot sufficiently reduce the risks identified. Questions you may want to consider include:
  • Are the individuals or teams responsible for the development, testing, validation, deployment, and monitoring of AI-based technology adequately trained and understand the data protection (and other legal) implications of the processing?
  • What technical and organisational measures are in place to support the deployment of this AI-based technology?
  • How will you ensure data quality and data minimisation?
  • How will you respond to data subject access requests?
  • What protections do you need to establish to ensure that the data is kept securely and with an appropriate amount of control by you?
  • How will you prevent function creep, or a situation where success in one context or use case of the AI-based technology may encourage people to think that it can be used in other circumstances without a proper review of how proportionate and legally compliant that new use may be?

Before you make the decision to commission AI-based technology, you must consider the rationale for doing so, including whether such technology is suitable and proportionate to the outcomes you wish to achieve.

You will need to assess the equality impact and data protection risks of commissioning or procuring AI-based technology and we recommend you use a EqIA and DPIA. There is crossover between these assessments so that you will not need to start from scratch with each. To streamline assessments, we propose that local authorities merge their DPIA and EqIA and voluntarily undertake an AIA.

You should review and revise these assessments throughout the procurement process, and early use of them will help you to decide on some of the specifics that should be included in the tender. These specifics might be a particular standard of cyber security, expected accreditations of bidders, or facilities for the council to review the AI-based technology during the assessment of bidders.

You should consider the following questions and prompts and integrate your answers into any EqIA or DPIA you complete. Considering the following questions may lead you to decide that an AI-based technology is not suitable for your needs at this time, or you may need to reconsider your objectives and approach. 

• What is the problem you are trying to address and what outcomes do you need to achieve? Is there non-algorithmic or non-AI options which may be used to achieve the same goal? If you choose AI-based technology, you will need to explain why that option is favoured.
• What is the rationale behind choosing this specific AI-based technology? Do you have evidence of its success in similar circumstances? If your proposed use is novel, you should factor this into your risk assessments (e.g. DPIA and EqIA), consider safeguards such as undertaking reviews of its use more often, engagement with impacted individuals and shortened contract terms or break clauses
• What personal data do you expect the AI to process? Does it include special category or criminal offence data? If the data is special category or a high volume, this will increase the overall risk for the use of AI-based technology. High-risk in your processing may be mitigated by shorter review periods or pilot periods with more limited data, your DPO should be consulted on this.
• How will you ensure data quality and data minimisation? Can you reduce the volume or improve the quality of data that you control and introduce to an AI system?
• Do you understand the personal data you will need to process as part of the deployment? Is it proportionate to the purpose of the AI-based technology? How will you respond to rights such as to restrict, deletion or access of data? Data flow diagrams may help.
• Do the individuals impacted by the AI-based technology share particular protected characteristics defined by the Equality Act 2010? If so, you are expected to have greater safeguards to protect those individuals from bias and unfair outcomes. You may need to involve more professionals, such as an equality or ethics specialist, or undertake more regular reviews of the activities.
• How much of a choice will people have in terms of having their personal data processed by or to train the AI-based technology? Can users opt out of personal data processing? Can users challenge, correct, or reverse the AI-driven output? If so, how and to what extent? If you decide to procure and use the AI-based technology, the above expectations need to be described in the tender and built into processes for when the contract is live.
• Would individuals expect you to use their data in this way? What information do you already provide to individuals in privacy notices and similar policies? Have you done any engagement to check or learnt from other authorities that have tried this approach? If not, you may need to plan more engagement.
• Who will be using this AI-based technology within your organisation or service users? What is the level of competency for both service users and staff? For example, if service users will be using the technology, you may anticipate that the majority will not need any particular competency to use it. However, users with little IT competency may need training on how to use the technology and users with specific disabilities are likely to need reasonable adjustments. What training will you need to provide to each user group, internal and external? Do you need to consult internal stakeholders, legal experts, or technical experts?

• How will using the AI-based technology affect individuals? Who is likely to benefit from this technology and who may experience negative outcomes or be denied opportunities? What characteristics protected under the Equality Act 2010 are at stake? Are there any other characteristics you should be considering here? Consider the perspective of individuals (staff and/or service users), the organisation, and society. This information must form part of your EqIA and can also be recorded in your DPIA.
• How will personal data be collected, used, stored, shared, deleted, anonymised and who is responsible for those actions? What responsibilities for the AI-based technology supplier do you need to establish in the contract to ensure that the data is kept securely and with an appropriate control by you? Control will include whether the data can be used for training the AI-based technology during the contract or in the future, and whether this is solely for the benefit of the council or if the newly trained AI will be purchased/used by others.
• Is there an expectation for the supplier to use the data for purposes beyond what a user is expecting, including future training the AI-based technology? This must be decided before award and documented in the contract. Successful decommissioning of an AI-based technology without people losing control of their personal data should be planned in advance (for example if a developer has trained an AI-based technology on their personal data in a way that encodes it but does not delete the model following decommissioning). You might find it useful to refer to a flow diagram or another way of describing data flows, including documenting whether any other party is expected to access and use the data. Your proposed supplier may not be the original author of the algorithm. You must understand and clearly state in the contract who has access to the personal data and for what use.
• Will the data be stored outside the UK? Speak to your DPO early, as it may be appropriate to specify the locations of data (data sovereignty) in the contract, which may impact the cost.
• Are there concerns over this type of processing or security flaws? If you decide to continue with the tender, what expectations will you set with bidders and how will you draft contracts to help reduce risk and clearly describe the responsibilities of the supplier?
• What evidence do you want from the prospective suppliers to allow you to consider whether their standards are appropriate from a regulatory point of view? Do you expect bidders to comply with an approved code of conduct or certification scheme? This should be written into the tender and evaluation documents.
• How will you measure success? What information will you need to identify and evidence that using the AI-based technology is delivering the intended outcomes/benefits?
• What information will you need to ensure personal data is handled in ways that people would reasonably expect and not used in ways that could lead to negative impacts, undermining of their rights and freedoms including unlawful discrimination, inequalities, community tensions, isolation, or segregation? You should consider these needs when drafting the tender, awarding the contract, and drafting the contract. People should not lose control of their personal data.
• It is good practice to aim for a diverse project team, especially in terms of – but not limited to – sex, gender, age, race, colour, language, religion and belief, national origin, ethnic origin, social origin, disability, and sexual orientation. Consider how does team diversity reflect the complexity and diversity of the population groups expected to be affected by the AI-based technology or using it. How could this introduce biases?

• How will you assign responsibility for answering these questions and the council’s decision-making during pre-tender, tender, implementation, and contract monitoring stages? Consider the diversity of the decision-making team and the measures you expect to meet or the activities or actions that you will not tolerate.
• How will you ensure effective business continuity practices and processes exist and are implemented if the specific AI-based technologies fail or shut down? There should be relevant questions within the tender. You also will need to ensure that suitable contract clauses reflect the standards you expect. In this context it makes sense to consider the security principle of data protection and any risks to the confidentiality and integrity of the data, or whether any groups of users are more likely to be negatively affected during an incident.
• Are there widely accepted evaluations, such as functional tests, to assess potential risk areas for the AI-based technology? Is the testing methodology robust? Could a supplier game the results of this test? How will you document risk and justify your decisions based on those risks?
• You will be using AI-based technology to achieve specific outcomes. Even if that use is initially successful, as the time passes, the AI-based technology may no longer be performing well in new circumstances or when it interacts with new data. Therefore, the continued use may necessitate the adaptation of the AI-based technology or stopping its use. You need to consider this before tender and build suitable checks into the contract and processes, such as regular retraining of the AI-based technology, or break clauses.
• How will you prevent function creep, or a situation where success in using the AI-based technology in one context may encourage people to think that it can be used in other circumstances without a proper review of how proportionate and successful that new use may be? If you consider using the personal data for a compatible purpose, a compatibility test should be carried out and results recorded in your DPIA. It is also something to consider as part of your EqIA and you may want to introduce a specific decision-making process for any potential further use. Successful decommissioning of an AI-based technology without people losing control of their personal data should be planned in advance (for example if a developer has trained an AI-based technology on their personal data in a way that encodes it but does not delete the model following decommissioning). For further information on this see ICO guidance.

Councils will differ on how they have devolved their procurement functions. We use the term ‘procurement officers’ to include any staff involved in (1) producing the contract specification and invitation to tender and (2) asking bidders questions relevant to equality and data protection to shortlist and select the successful supplier.

The questions and prompts below will help procurement officers to build equality and data protection considerations throughout the tasks listed above in (1) and (2). 

Considering these questions may lead you to decide that AI-based technology is not suitable for your needs at this time, or you may need to reconsider your objectives and approach.

• Has a DPIA and an EqIA been undertaken or at least started? Have they raised issues that need to be managed in the contract specification and tender? Has any new evidence emerged on the potential equality impact and data protection risks of using the AI-based technology your council is considering? Evaluating and mitigating equality and data protection risks is a continuing process. As such, if new relevant evidence emerges including from tenderers, staff involved in procuring AI-based technology must review and update the Council’s EqIA and DPIA.
• What is the relevance of equality to the subject matter of the contract? The relevance of equality to the subject matter of the contract will help determine whether and the extent to which it should form part of the contract specification. 

• How will you set out what the successful bidder will have to do in relation to equality in your contract specification?
• Does the potential supplier have the relevant capability within the organisation to assess equality and data protection risks for this type of AI-based technology?
• Can you partially complete a contract data protection schedule that will identify for the bidders the expected processing of personal data? Speak to a Data Protection Officer for advice.
• What type of due diligence do you need to undertake on potential bidders? You will either need, or direct the commissioner or project manager to liaise, with officers responsible to ensure compliance with equality and data protection laws as there are likely to be questions that should be asked at the bidding stage. This is particularly important when procuring AI-based technologies as receiving and assessing information on how the AI-based technology works may be a long process.
• How will data be collected, used, shared, stored, and deleted and who is responsible for those actions? Will the data be subject to international transfers? What responsibilities for the supplier do you need to list in the tender and the contract to ensure the processing of personal data in accordance with data protection?
• Is there an expectation for the supplier to use the data for its own purposes, including potentially training the AI-based technology? This must be decided before award and documented in the contract. Your proposed supplier may not be the original developer of the AI-based technology. You must understand and clearly state in the contract who has access to the personal data and for what use.
• The AI will be used to achieve specific outcomes. If that use is successful, the AI may no longer be reflective of the new circumstances. Continued use may not be suitable without adapting or stopping its use. You need to consider this before tender and build suitable checks into the contract and processes, such as regular retraining of the AI-based technology, or break clauses. Successful decommissioning of an AI-based technology without people losing control of their personal data should be planned in advance (for example consider seeking legal advice if a developer has trained an AI-based technology on their personal data but does not delete the model following decommissioning).
• The tender should ask how the AI-based technology will be trained and whether the supplier expects to access and use the data from the council, for example, to further train the AI. You may wish to state what is acceptable to you, or you may wish to see the state of the market offering before deciding. Your consideration of this will come through the DPIA and EqIA as you assess whether using the data for training the AI or for the suppliers’ own purposes is likely to be appropriate.
• How will you ensure effective business continuity practices and processes exist and are implemented if the specific AI-based technologies fail or shut down? There should be questions within the tender, and you will need to ensure that suitable contract clauses cover the standards you expect. This is an area where you are considering the data protection risk of the availability or integrity of the data, and whether any protected groups are more likely to be negatively affected during an incident.
• Consider asking bidders to provide you with their own EqIA and DPIA as part of the tender process. Doing so will help you assess their content and quality when shortlisting bidders and awarding the contract. Incorporating these into the award process will evidence the council’s commitment to equality and compliance with data protection laws.
• Consider publishing your contract specification and invitation to tender alongside the council’s EqIA and DPIA. Doing so should help the council build trust with its communities through demonstrating that it considers the data protection, and equality impacts of the AI-based technologies it wishes to buy in a transparent and accountable way. You should make this clear when publishing the tender and ensure any confidentiality and transparency clauses in the contract reflect this.

You can build these questions into your tender and bid assessment process. The questions below are guidance for the council officers assessing bidder answers. 

Accreditation and documentation

Bidders may have produced their own assessments in the form of DPIAs, EqIAs, AIAs for their AI-based technology. Not all providers will have carried out these assessments, but you can reasonably request them to compare to your own risk assessments. Questions you can include:

• Can you provide a copy of the Data Protection Impact Assessment (DPIA) you completed for development of the AI-based technology? If not, why not? Review the DPIA completed for AI development and check it against the one that has been carried out for this tender. Consider whether the bidder has covered all the areas. Seek advice from your DPO, as well as any relevant AI specialists your council employs or has access to. If you identify a high risk in your DPIA that you feel cannot be mitigated, contact the ICO Innovation Service, DPIA team for support.
• Can you provide a copy of the equality considerations/Equality Impact Assessment (EqIA) you have completed for the development of the AI-based technology? If not, why not? Review the information provided and check it against the documents such as an EqIA that has been carried out for this tender. Consider whether the bidder has covered all the areas. Seek advice from your EDI officer as well as any relevant AI specialists your council employs.
• Can you provide a copy of an Algorithmic Impact Assessment (AIA) you completed for development of the AI-based technology? If not, why not? AIAs are optional but you could review and check any AIA as you would with a DPIA or EqIA noted above.

Training data, bias, and data use

AI-based technologies can introduce, preserve and perpetuate pre-existing biases, by learning to replicate bias found in the data on which they are trained. The ICO has produced extensive guidance on this in its Fairness in the AI lifecycle guidance. Discrepancies in sample sizes, with lack of accurate representation of individuals, can raise the risk of bias. Additionally, the speed at which an AI-system can complete a large number of tasks may then result the impact of that bias being amplified. You must take care when evaluating AI-based technology, and when considering evaluations conducted by suppliers, to ensure that the testing methodology is robust, and that the results are reflective of the real-world performance of the AI-based technology. Questions you can ask include:

• How is personal data used to train the AI-based technology? How often is the AI-based technology trained and what is the source of this data?
• Is any personal data processed under the contract expected to be used to train the AI-based technology? Will the newly trained AI-based technology be available only to the council or as part of the product for other organisations?
• Does the use of the AI-based technology require sharing data with third parties? And if so, what is the destination or source? Is such sharing documented and communicated to data subjects?
• Can the supplier confirm that the AI-based technology is trained on lawfully obtained data?
• Can the supplier provide a list of controllers, joint-controllers, processors, sub-processors that will be involved in the processing, and the roles and responsibilities each one has?
• What access to corporate information does an AI supplier need? Is this access necessary and proportionate to what the council wants to do with the AI-based technology?
• What consideration has been given to the diversity of the AI development and testing team, especially in terms of – but not limited to – sex, gender, age, race, colour, language, religion and belief, national origin, ethnic origin, disability, and sexual orientation, including how this reflects the complexity and diversity of expected user population, and how this could introduce biases?
• How is the data that is used to train the system well-balanced, and does it reflect the diversity of the targeted or end-user or impacted population? Does this data include information on people with protected characteristics under the Equality Act 2010 either explicitly or by proxy e.g. data of birth as a proxy for age?

The ICO advises that organisations should conduct a ‘proxy analysis’ of AI models. This should detect whether any features of the model are proxies of protected characteristics or special category data. For example, if a model detects correlations between welfare benefit fraud risk scores and carers’ days off work, it may unjustifiably lead to women disproportionately being targeted for fraud investigation. This is because women tend to have more caring obligations, so carers’ days off work is operating as a proxy for sex. Having detected a proxy, you should ascertain if you need to remove or adjust the feature to avoid any false correlations. Read more about ‘proxy analysis’ on the ICO website .

• What processes did you use to test for biases? What processes are in place to continue testing the model for biases during implementation? Will data/results of these tests be made available to the council during the bidding process? Can the supplier test the AI-based technology using scenarios representative of the council use case, to assess the likelihood of bias?
• What differences can you foresee between the data used for training and the data processed by the AI-based technology during deployment? How do you manage and reduce the possibility of the AI-based technology producing discriminatory outcomes or discrepancies in its performance for different groups?
• How has the AI-based technology been tested with people with relevant protected characteristics under the Equality Act 2010? What differences were identified in the statistical accuracy rate (or any other performance metric used)? Please describe any difference of this kind. E.g. false positive/ false negative?
• How did you assess whether the AI-based technology is accessible by those with disabilities e.g. accessible to screen readers, including alternative text for images, colour-blind friendly palettes?
• If you have identified that using your technology could have a negative impact on people with specific protected characteristics, how have you identified actions or reasonable adjustments to mitigate against this? What actions/reasonable adjustments have you already used and were they effective? If not, which actions are you proposing to put in place during implementation and how will you monitor their effectiveness? What would you suggest if we find that these actions/reasonable adjustments are not effective during implementation? 

Integrity and confidentiality

Councils have a duty to protect the data of service users and staff, when assessing bidders, you want to ensure the AI-based technology is trustworthy and that data is protected. Questions you could use include:

• What is your process to document how data quality issues have been resolved during the design process? Show evidence.
• What processes will you implement to ensure that unforeseen risks can be flagged, or spurious decisions contested throughout the lifecycle of the AI-based technology?
• What fallback systems will be in place to ensure an adequate continuity of essential services if the AI-based technology needs to be temporarily or permanently suspended?
• Describe any relevant variations or margins of error in the performance of the system which may affect the fairness of the personal data processing and its outcomes.
• What controls are in place to protect the AI-based technology from adversarial attacks that seek to extract personal data?
• What is the response process if there is a serious security incident involving the AI-based technology?

Training and usability

It is good to understand from bidders the skill level service users or staff may need to be able to use the AI-based technology and review its outputs. Bidders may be able to offer training to upskill users of the AI-based technology, to support its roll out. Questions you could use include:

• Is there any training available to help staff or service users to understand the benefits and risks when using this AI-based technology?
• How much training will officers need to be able to effectively use the AI-based technology and challenge the AI outputs/decisions?
• Will staff be made aware of the potential for their own human bias towards accepting an AI-based technology output without scrutiny?
• Can council officers challenge the decision-making of the AI-based technology? If so, how had at what points? How much training will officers need to be able to effectively use and challenge the AI outputs/decisions?

If you were not involved in the EqIA and DPIA processes, familiarise yourself with them and identify anything that needs to be included in the contract, such as cyber security standards or clauses to ensure you are informed before a supplier begins or amends use of AI. 

You should include relevant clauses, schedules, and specifications into your contract to help your council manage equality and data protection risks associated with using the AI-based technology. Any clauses within the contract should be:

• proportionate and quantifiable.
• referred to in the contract notice or tender documentation; and
• clear and unambiguous, and understood by tenderers and suppliers.

You can also specify performance targets in your contract. When you do, you should be explicit about how you expect the supplier to monitor their performance against these targets. 

The questions below aim to help you think about the equality clauses and performance measures to include in contract arrangements with your AI provider:

• How will data be collected, used, shared, stored, and deleted and who is responsible for those actions? Will the data be subject to international transfers? What responsibilities for the supplier do you need to list in the tender and the contract to ensure the privacy and safety of personal data?
• The contract must be clear on what control the supplier has to access and use the data, for example, to further train the AI-based technology.
• What equality data will the provider gather and provide you with to ensure that using the AI-based tech is:
  • delivering the intended benefits, including for people with specific protected characteristics?
  • is not leading to unlawful discrimination?
  • Is not leading to bias that need to be addressed through additional mitigation actions or reasonable adjustments?

• How and how often will your AI-based technology provider gather such data? e.g. data will be collected via user surveys and provided to the council on a monthly basis.
• What actions will the provider be expected to take to deal promptly and sensitively with complaints about discrimination?
• What actions will the provider be expected to take to mitigate any negative impact you have identified or that may become apparent when the AI-based technology is in use?

• Whether and how personal data under the council’s control will be used to train the AI-based technology?
• Whether you need to put in place any exemptions for sensitive data which is inputted into the AI-based technology?

• How often the supplier is expected to retrain the AI-based technology?
• Is there an expectation for the supplier to use the data for its own purposes, including potentially training the AI-based technology? This must be decided before award and documented in the contract. Successful decommissioning of an AI-based technology  without people losing control of their personal data should be planned in advance (for example if a developer has trained an AI-based technology on their personal data in a way that encodes it but does not delete the model following decommissioning). You might find it useful to refer to a flow diagram or another way of describing data flows, including documenting whether any other party is expected to access and use the data. Your proposed supplier may not be the original author of the algorithm. You must understand and clearly state in the contract who has access to the personal data and for what use.
• Within the lifecycle of your AI-based technology deployment and use, it is possible that you may need to stop or pause using the AI due to a change in circumstances. You should consider this before tender, and build suitable checks into the contract, such as requirements for regular retraining of the AI, or break clauses.
• Suitable contract clauses should cover the standards you expect for effective disaster recovery and business continuity practices and how they will be implemented if the specific AI-based technologies fail or shuts down. This is an area where you are considering the data protection risk of the availability or integrity of the data, and whether any protected groups are more likely to be negatively affected during an incident.
• Consider carefully what you expect to happen during a potential incident, such as a cyber-attack. A cyber-attack can cause significant issues in a very short period of time and can take months to fully fix. Your contract clauses need to be clear that you expect immediate and regular notifications from your supplier, whilst recognising that some aspects of the incident may have a limited distribution where the police or the National Cyber Security Centre (NCSC) are involved. You may want to consult the UK government’s Code of Practice for the Cybersecurity of AI.

Ongoing contract management

You must undertake contract monitoring; this can include asking the supplier for updated answers to your due diligence questions and reviewing these against the due diligence answers you received previously (either when the contract was created or during the last annual review). If negative outcomes are occurring or standards have changed in a way that is unacceptable to you, what action will you take with or against the supplier to reduce or change these circumstances?

• You will need to review and, if necessary, amend your EqIA and your DPIA throughout the life of the contract. This may be when:
  • you reach Minimum Viable Product (MVP).
  • the pilot stage is complete.
  • annual contract review.
  • a change in activity or legislation; and
  • a change to the product or upgrade, for example, to include an AI-based technology (sometimes called ‘slipstreaming’).

You also need to review:

• whether the expected benefits are being achieved.
• whether the processing of personal data continues to be fair, lawful, and secure and if not, what action you will take to rectify this.
• whether there have been any incidents or complaints relating to data protection, failure to provide reasonable adjustments, discrimination, failure to provide users with particular protected characteristics with the same opportunity to access or use your services compared to those who do not share these characteristics. If so, what has been done to rectify these issues and reduce the likelihood of them occurring in the future? This may be an area where you consider penalties against a supplier or ending the contract.
• whether there is evidence of discrimination occurring through use of the AI-based technology. If so, what has been done to rectify this for the future and to eliminate discriminatory outcomes?
• Whether the AI-based technology need to change. This may be because of a change in society, or that successful use has changed the situation, and the original technology is no longer statistically accurate for the population it is interacting with.