By Laura Hawkins
for the 2024 Risky Women Writing Competition, presented in partnership with Bovill Newgate
Learn how organizations can prepare for AI-related risks, including employee misuse, algorithmic biases, and external threats, to maintain trust and ensure responsible AI use.
AI systems are a hot topic in the world of regulation, where legislators and regulators across the globe are reacting to the more accessible and scaled use of AI systems, such as Generative AI, and the challenges they pose. Many existing pieces of regulation and good practice risk management principles require organizations to test and exercise their crisis management response, both technically but also strategically. Understanding where your business is using AI, how a disruption can impact your business services (both internal and external facing services) and the types of scenarios you may need to respond to is paramount to be able to maintain trust and confidence with your consumers and partners.
The AI crises of tomorrow are nuanced with new and evolving challenges that mean organizations need to review how they approach using AI. This is best achieved through dynamic wargaming, where updated scenarios are used to test procedures, systems, and teams, as well as testing specific use cases before deployment.
In the last few years, scenarios have focused on threats such as ransomware, third party outages and technical updates leading to disruption. However, scenario libraries need to evolve to meet this technology, but what kind of scenarios should organizations look to test, and what remediations should be considered to respond?
Employee Gone Rogue
With certain types of AI more accessible than ever, everyday users now have AI systems in the palm of their hand. While this can lead to greater knowledge, creativity, and curiosity, it can also lead to negligence, misuse, and in some cases, illegal activity.
Scenarios could cover an employee uploading personal data or confidential business information to an insecure or public AI system, employees manipulating productivity or hiring AI systems to amplify biases and the use of AI to excessively monitor employees or automate tasks that require human oversight.
An organization will need to look at internal use cases of AI and test where misuse could occur. Then work to improve internal processes, such as a review of acceptable use policies and training materials to cover how employees should use AI systems across the business, or implementation of ongoing technical bias testing to monitor for model drift and misuse.
Algorithm-Fuelled Falsehoods
AI systems in themselves can present challenges for organizations. Hallucinations (where an AI generates information or answers that seem believable but are false or made-up), data bias and uncertainty are all potential pitfalls an algorithm can fall in to, but at the end of the day the algorithm works for you and needs to represent the brand and values of the business.
Scenarios could cover a chatbot that isn’t presenting accurate information to a customer, algorithms offering biased decisions to one demographic over another, or algorithms developing deepfake or false marketing material that is released to the public.
Here, organizations need to consider reputational harms, as well as potential legal harms. There are legal and regulatory requirements around transparency and explainability, that means decisions made by or with the help of an algorithm, need to be fully documented and logically understood. Ensuring your technical teams have a firm and documented understanding of how an AI system works and the data it uses, with a clear human in the loop to provide oversight, is key.
AI Under Siege
AI presents a new attack surface for malicious third parties to take advantage of, while the aim of disrupting the confidentiality, integrity or availability of data might be the same, the ways of doing so have shifted.
Scenarios could cover attack types such as data poisoning (where a third-party injects false data into a data set to change AI learnt patterns) or malware attacks that lead to potential biases or inaccurate decisions. Traditional data impacts, such as loss or unavailability, can also occur due to AI vulnerabilities and exploitation of how this could occur should also be considered.
Organizations must review their technical response plans and conduct technical testing that incorporates the AI systems. Understand where technical impacts also necessitate a regulatory response, for example the EU AI Act or proposed legislation across multiple jurisdictions, will require organizations to report material breaches of high-risk AI systems, as well as consideration of existing requirements for personal data breaches.
Safe, responsible, and secure AI
AI system use needs to be incorporated into wargaming to prepare leaders to respond to potential challenges across legal, reputational, ethical, and operational risks to the business. AI presents an opportunity to innovate, but doing so responsibly is not only good for society but also an organization.
However, this can only be achieved if an organization has laid the groundwork first. Similar to traditional approaches to operational resilience, understand what you have (by creating an inventory of data and model use cases across the organization), understand how data and AI supports your business (align AI use across strategy, services and processes), understand the risks the AI system can impact (review and update your risk registers and tolerances) and then test, test, test.
By building your business to embrace AI in a safe, responsible, and secure way, you will build your business able to withstand the scenarios of tomorrow and build products and services with safety by design.