You only have to spend a few moments asking ChatGPT some questions to understand the value AI can bring in solving problems at pace, with companies such as Gartner listing generative AI as a pivotal AI technology going into 2024. (What’s New in Artificial Intelligence From the 2023 Gartner Hype Cycle™)
As a major global defence supplier, Leonardo is transforming its business through the digitisation of its processes, manufacturing and services. At the same time, the company’s cyber and security business is proactively developing an innovative range of AI-based capabilities to assist in solving the AI challenges defined by the Defence Artificial Intelligence Centre (DAIC) in the Defence AI playbook.
Ethics aside, these problem spaces have large technological challenges. In addition to achieving the desired functionality, they must also achieve the required level of safety, security, legal and specific regulatory requirements such as airworthiness, in order to be a viable solution. Current assurance frameworks and methodologies cannot deliver this. A parliamentary enquiry into Autonomous Weapon Systems in Defence, published in February 2024, called out these challenges. Overcoming them using current assurance methodologies and frameworks will not be enough.
A Step Into the Unknown
Current guidance published by the UK NCSC (Guidelines for secure AI system development - NCSC.GOV.UK) in collaboration with other countries is a great start to outlining the lifecycle phases of AI solutions and nuances between them from a security sense. However, experience and tailoring are required to apply this guidance successfully in safety- and security-critical environments. Despite this progress in holistic guidance, low-level frameworks to enable threat modelling do not exist for AI, which means threat modelling cannot be done consistently. The Open Source Foundation for Application Security (OWASP) Top 10 for large language models is a framework that bucks this trend, though it is focused on a particular AI type in an enterprise implementation scenario.
Additionally, recent failures of even simple automation software in safety-critical environments have resulted in greater mistrust of AI solutions in those environments. This demonstrates the need for greater threat-led assurance of non-traditional digital systems that directly and indirectly control physical devices in the real world.
The question therefore, is how AI-based solutions can be robustly assured from a security perspective throughout their deployment lifecycle in safety-critical environments.
When looking at the assurance of AI in safety- and security-critical environments across the defence sector, the following areas must be considered:
-
Emulate Adversary Behaviour – Be threat-led in your approach; model and emulate adversaries’ techniques against your protection and focus on how the AI capability might be attacked in the context of its use case. It is too easy to take modular COTS solutions, bolt them together and run a traditional penetration test. For AI, the focus must be on ‘context-aware’ testing approaches that seek to exploit the decision logic of an AI, not just its traditional hosting environment. In doing so, you can gain a better understanding of which part of your AI tool is exploitable and where to better defend.
-
Validate, don’t just Verify – In security, never has V&V (Validate and Verify) been more relevant. AI is an emerging technology, and document and configuration reviews only tell half the story. The way in which modern neural networks reach a decision is not yet fully known (Why humans will never understand AI - BBC Future). Therefore, testing and validating operations in the given use case, is key. This means you cannot just use a checklist of items against a design to assure it; you have to logically test against a range of attacks and do this at regular frequency to account for step changes in adversarial capability.
-
Manage Ambiguity, Don’t Ignore It – Adversaries have always exploited adaptability in a given system, which makes AI solutions attractive targets for attackers. Do not just accept an AI tool can be used ambiguously; by focusing on constraining its inputs, you will constrain its outputs. ChatGPT, for example, uses prompt engineering (The Essential Guide to Prompt Engineering in ChatGPT - Unite.AI) to constrain user queries. This is a major roadblock in safety-critical environments that requires a system to behave in a probabilistic way, so controlling inputs in this way can enable greater consistency in behaviour.
-
Attackers are not always people – AI has brought about a potential new wave of cyber weapons that could operate almost entirely unilaterally. Generative AI has the ability to produce and use bespoke code based on its trained mission. This means a cyber-kill chain against your organisation might be realised in seconds rather than minutes. The most effective tool against an AI solution is a counter AI solution. Therefore, in the future, counter AI solutions may offer a feasible way to emulate adversarial behaviour at pace against a given AI capability.
Invest in the Bigger Picture
As ever in security, financial investment can be a barrier and none of these considerations can be properly executed without appropriate investment. If the massive benefits of AI within an organisation are universally agreed, then the business case for greater investment around assurance across the lifecycle of the capability should also be recognised.
To fully realise this investment, the assurance of AI within safety- and security-critical environments cannot be siloed within one department. It must also be highly contextuliased to a given use case. A scientific and threat-led approach to assurance will enable greater focus on achieving system resilience and go some way to providing confidence in its consistent operation and functionality, even in the face of adverse cyber events.