We have all been startled by the power of OpenAI’s ChatGPT and its ilk of Generative AI tools, like Google’s Gemini or Meta’s Llama. One crucial question is can these be weaponized to make our cyberspace more insecure? I will reflect on three primary aspects of this question.
First, there is some evidence that these can be used for generating phishing emails [1], or more generally, misinformation campaigns [2]. This is an area where detection techniques will quickly improve to ensure that this stays like an arms race, but we do not fundamentally become insecure. The ultimate objectives of phishing are few and not changing (such as, to extract money or credentials) and detection techniques will focus on catching at that point, irrespective of how the phishing mails were generated. Misinformation is more challenging as the volume and sophistication will increase significantly, and detection techniques, and human training, will have to also improve substantively. These are not trivial to achieve, but also not fundamentally impossible.
Second, there is some early concerning evidence that Large Language Models (LLMs) can be used to generate malware (such as, viruses or ransomware) [3], [4]. However, what LLMs generate are rarely usable because they rely on large data sets and security vulnerabilities, by definition, do not have large existing trails of data. There will be significant, and expert, manual effort involved in crafting malware and then the defense techniques will be able to keep up with the volume of malware created through such means.
Third, the ownership structure of LLMs provides us with some confidence that they cannot be used for rampant security attacks. These models are sophisticated, and expensive, to create and to continuously update. Consequently, only few organizations can afford to own them. Such organizations operate under legal norms and they will put guardrails such that bad actors cannot weaponize them. Such guardrails have been shown to be imperfect, but it will be an arms race where we will learn to erect better defenses with new attacks. For example, as the vendors observe attacks to create illegal content (such as creation of bio weapons), they will deploy techniques to prevent such generation. There is some evidence already of this happening [5], [6]. Interestingly, these guardrails are initially rule-based but they will themselves evolve to employ gen AI’s automation.
So, to sum, GenAI will change our cyber space, and dare I say, our lives. But it will not fundamentally alter the balance of security in cyber space in the medium term (i.e., next 3 years).
1. CNBC Technology Executive Council. "AI tools such as ChatGPT are generating a mammoth increase in malicious phishing emails". November 28, 2023. At: https://www.cnbc.com/2023/11/28/ai-like-chatgpt-is-creating-huge-increase-in-malicious-phishing-email.html
2. Freedom House. "The Repressive Power of Artificial Intelligence". November 2023. At: https://freedomhouse.org/report/freedom-net/2023/repressive-power-artificial-intelligence
3. Bank Info Security. "Malware Taps Generative AI to Rewrite Code, Avoid Detection". May 5, 2023. At: https://www.bankinfosecurity.com/malware-taps-generative-ai-to-rewrite-code-avoid-detection-a-21972
4. Secure Frame. "Generative AI in Cybersecurity: How It’s Being Used + 8 Examples". October 24, 2023. At: https://secureframe.com/blog/generative-ai-cybersecurity
5. Google. "Our responsible approach to building guardrails for generative AI". October 12, 2023. At: https://blog.google/technology/ai/our-responsible-approach-to-building-guardrails-for-generative-ai/
6. Ernst and Young. "Effective guardrails transform gen AI fascination into solid foundations". November 9, 2023. At: https://www.ey.com/en_ca/technology/effective-guardrails-transform-gen-ai-fascination
Distant Whispers from an Academic Engineer's World 