By Anna Antimiichuk
for the 2024 Risky Women Writing Competition, presented in partnership with Bovill Newgate
This article discusses the rising threat of data poisoning in AI, where attackers corrupt datasets to manipulate outcomes, risking personal privacy and ethical standards. It emphasizes the need for robust Governance, Risk, and Compliance (GRC) frameworks to protect data integrity and ensure ethical data use.
In the digital age, data has become not only a critical asset, but, regretfully, also a growing vulnerability. With the rapid advancement of artificial intelligence and machine learning, the amount of data collected, processed, analysed is increasing exponentially. It is often used to train AI models that power everything from personalised recommendations to predictive analytics in healthcare and finance. However, as data becomes more central to decision-making processes, it becomes an attractive target for malicious actors.
One such threat that has gained significant attention in recent years is data poisoning, where attackers intentionally corrupt datasets to manipulate outcomes. To undermine machine learning models, false data is introduced into training datasets. It can take many different forms, such as inserting malicious code that modifies the decisions made by AI models in particular ways or adding errors that distort algorithmic outputs. The intention is to undermine the AI system’s integrity and dependability, regardless of the approach used. A data poisoning attack could introduce biases, create vulnerabilities, take control of AI-powered manufacturing equipment, or even train AI-powered threat detection solutions to identify attack traffic as benign. Only their skill and imagination limit a threat actor.
Unlike other cyberattacks, which may focus on stealing or encrypting data, data poisoning seeks to subtly distort, degrade the performance of AI systems. By contaminating the data that is used to train models, attackers can cause AI systems to make incorrect predictions, inaccurate decisions, or even behave erratically having the potential to seriously disrupt vital machine learning-based applications like cybersecurity, fraud detection, and medical diagnostics.
As more businesses rely on AI models to process sensitive data, there is an increasing risk to personal privacy from poisoned data. Numerous actual cases have already shown how it can jeopardise security and privacy. In one case, University of Maryland researchers ran an experiment to demonstrate how adversarial data could be used to contaminate face recognition systems. They were able to fool the AI into misidentifying people by making subtle, undetectable adjustments to the images, which raises questions about the possibility of abuse in security systems where personal identity verification is essential.
In another case, a cybersecurity firm discovered that attackers had poisoned data used by a predictive policing algorithm. The poisoned data led to false predictions about criminal behavior, disproportionately targeting certain demographics. This not only raised ethical
concerns but also exposed vulnerable populations to unfair surveillance, violating their privacy and human rights.
The rise of data poisoning brings ethical questions to the forefront: how data is collected, stored, shared. If datasets are poisoned, not only is the AI system compromised, but the very privacy of individuals whose data is part of the training set is also put at risk. For example, malicious data could be injected into a dataset that organisations use to develop consumer-facing applications. This could cause AI systems to reveal sensitive personal details, directly or indirectly, without users realising it, and leaking information about an individual’s preferences, health conditions, or even personal identifiers like social security numbers.
Firms must ensure that the data they collect is not only used ethically but is also protected from manipulation. However, the reality is that many firms lack robust governance, risk, and compliance (GRC) frameworks to safeguard against these emerging threats. From a GRC perspective, data poisoning creates new challenges for ensuring compliance with privacy regulations such as the General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA) and others. If a dataset is poisoned and causes an AI system to expose personal data, organisations could face hefty fines for failing to protect it.
The opaqueness of AI systems, or “black boxes” that make it hard to see how decisions are made, raises ethical questions. When poisoned data is involved, this lack of transparency becomes particularly problematic because it can be nearly impossible to identify and lessen the effects of the attack. It’s possible that those impacted by data poisoning caused by an AI system won’t even be aware that their privacy has been breached.
Another issue revolves around the sharing of data when datasets are shared with third parties for research, development, and training. Without proper safeguards, shared datasets can be compromised, putting both the original data owners and the third parties at risk. Data could flow through multiple systems, magnifying the harm and complicating efforts to trace the source of the attack.
To address the risks posed by data poisoning, a proactive GRC approach should be adopted strengthening governance frameworks to ensure that data is collected, stored, and processed in a secure, ethical manner. Regular audits of training datasets should be a part of risk management strategies in order to spot any indications of poisoning or tampering. Strict controls should be put in place by businesses to protect sensitive data from both internal and external threats, and compliance with privacy regulations should also be a primary priority. Furthermore, firms should invest in advanced AI security measures, such as anomaly detection systems that can flag potential poisoning attacks. By integrating security into the AI development lifecycle, it is possible to mitigate these risks. To combat this emerging threat, organisations must not only strengthen their GRC frameworks but also embrace a more ethical approach to data. This will safeguard both their AI systems and the privacy of the individuals relying on them.