If you are using almost any modern office tools you are likely using some form of AI, whether it is machine learning, automated processing or agentic agents doing automated tasks using a large language model. It doesn’t really matter what tool you are using, what service or how much you need to consider the risks and the data protection guidance and regulations.
As of 2026(1) the UKGDPR now differs from the European GDPR in some significant ways, one of those concerned a seemingly relaxed allowance in regards to automated decision making.(2) The important feature we cover here is that the UK has altered the manner in which you can safely use automated tools to allow them to make decisions on data processing.(3)
Allowing automated decision making and processing carries significant risk which can breach data protection legislation, you are still liable even if an automated system negatively impacts individuals, their fundamental rights or freedoms. A recent case in Germany has brought this to immediate attention as it has been decided by a court that Google is directly liable for its AI summaries.(4)
In this article we are going to break down some of the risks so that you can consider if you are taking steps to address them.
One: Function Creep and Incompatible Processing (5)
Automated tools are being developed at a staggering pace and they are being integrated into systems almost as fast as they are created. Even the most dedicated organisation doesn’t always know exactly how its staff are using these tools.
There is a real concern, and proven examples, that the algorithms may learn may develop in unpredicted ways. It is also probable that you might use the automated system for purposes beyond what you originally intended.
This is “function creep” and “incompatible processing”. You can easily lose control over personal information and you have a duty to inform individuals how you use their data and what automated processing is performed. Get this wrong and you will find yourself performing unlawful data processing and the erosion of trust between you and the data subjects is the least of the issues this causes.(6)
Two: Bias and Discrimination
Automated systems are subject to occasional hallucinations, and they are often plagued by bias dependent on the training data or on data pollution. They might produce outcome that are discriminatory, unfair, inaccurate or not diverse. This result might see an AI rely on characteristics that can lead to decisions that breach regulations.(6)
As an example: something as innocuous as an automated system for profiling CVs can be subject to an access request from those people it rejected. Do you know why it rejected them? You better as if it was in any way biased you could find yourself in trouble.
Three: Security Vulnerabilities and Purposeful Data Poisoning
How are your security practices? What are the security practices of the tools your staff are using? What are the data retention policies? Out of date, old, or even un-tested systems might leave the software stack open to unauthorized access and vulnerabilities. Does the model you are using have a security audit? Did the training data?
You could be using exposed training data. you might have deployed code or systems that have been the subjects of data poisoning attacks, often referred to as a poisoned well attack. Any system that has the result of an accidental or purposeful loss, destruction, or removal of personal data can be an issue. You don’t want to be responsible for exposing people to a financial loss or a fraudulent use of their data.
Four Lack of Transparency
Many AI systems utilize highly complex learning algorithms and high dimensional data, which can make their logic and outputs inherently difficult to explain. If you cannot clearly explain how the AI agent is making decisions, individuals are disempowered, unable to understand how their data is being used, and hindered from exercising their information rights.(7)
Five: Excessive Data Collection
Don’t take the approach of collecting as much data as you possibly can to train or test an automated process or system without a good reason. If you do need to collect a large amount of data to help train a system or process then take steps to apply de-identification techniques and remove unnecessary, or irrelevant, features.
Always try to follow the core data protection principle of data minimization.(8)
Six: Unlawful Automated Decision-Making
If you rely on an automated system or agent to make sole decisions that have and legal, or any significant effects, without any human review this could violate data protection laws. When we talk about software we at Shadowcat mention trust, governance and ownership and the same is true of data.
Current UK reform to data protection laws mean that strict safeguards must be used for automated decisions. The worst thing you can do is a casual nod to data protection rights and procedures, or even worse do nothing at all.
And a review by a human, where they lack training, knowledge or the authority to challenge or override any automated output will be insufficient. So it is wise not just to be aware of the whole processes used in the automation, but to have a human-led overview and input into them.(9)
Seven: Model Drift
The last area is model drift. This is important to understand if you use a subscribed automated system and vital if you have a version of your own, such as a local LLM and agentic system.
As mentioned above, an AIs accuracy can degrade over time if deployed without a regular maintenance and testing. If the original training data is no longer relevant or has become biased or corrupted. All of this can potentially lead to model drift which can lead to inaccurate or unfair processing of data.(5)
Know the Impact of the Data
So what’s the solution to knowing all of this? We suggest that you, or you and a partner organisation conduct a Data Protection Impact Assessment (DPIA) prior to deploying an automated system; and to review and reconduct it when changing, evolving or just over time to map any changes.
You need to actively map out where you data flows, what systems it touches, what automation is done and how. You need to implement and review security measures, and ensure human reviewers have “real” authority and “complete” ownership over the automated system’s outputs.
Want to talk about this more then use the contact form on our website (https://www.shadowcat.co.uk/#contact) to get in touch.
(1) The Data Use and Access Act
(2) https://ico.org.uk/about-the-ico/what-we-do/legislation-we-cover/data-use-and-access-act-2025/the-data-use-and-access-act-2025-duaa-summary-of-the-changes/data-protection/#automated
(3) There are very pertinent exemptions mostly around restrictions to special category data already specified in the UKGDPR.
(4) https://www.dw.com/en/german-court-holds-google-liable-for-fake-ai-answers/a-77527661
(5) https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/principles/purpose-limitation/
(6) https://ico.org.uk/for-organisations/guide-to-data-protection/key-dp-themes/guidance-on-ai-and-data-protection/what-do-we-need-to-do-to-ensure-lawfulness-fairness-and-transparency-in-ai-systems/#howshouldweaddress
(7) https://ico.org.uk/for-organisations/guide-to-data-protection/key-dp-themes/explaining-decisions-made-with-ai/
(8) https://ico.org.uk/for-organisations/guide-to-data-protection/key-dp-themes/guidance-on-ai-and-data-protection/how-should-we-assess-security-and-data-minimisation-in-ai/
(9) https://ico.org.uk/for-organisations/guide-to-data-protection/key-dp-themes/guidance-on-ai-and-data-protection/how-do-we-ensure-individual-rights-in-our-ai-systems/
No responses yet