2026.06.01
AIガバナンス
It was not very long ago that we first saw the dawn of this new AI age. OpenAI’s ChatGPT started out as a fun talking robot. Since that day, AI has become a cornerstone not only of the technical world but of our lives and our future in general. This article explores the imminent risks that AI brings with it, walking through the major classes of attack and the strategies used to defend against them.
Today, AI systems often have direct and complete access to a company’s knowledge base. Many companies and executives fail to anticipate the creative ways in which something as simple as a RAG bot can begin spilling out company secrets through nothing more than careful prompt engineering.
A startup called Vanna AI, which used generative AI to produce SQL queries and visualizations and ran them on the backend against company data, was found to be vulnerable through prompting alone. Vanna AI relied on an internal function that generated a Python script to render visualizations. By influencing the AI to generate harmful code, attackers were able to reach deep company secrets, because the AI had direct access to company servers.
Companies are actively encouraging employees to use AI for tasks such as development and data aggregation. A major risk that surfaces from these practices is that an employee may unknowingly feed sensitive company code or data into tools like ChatGPT or Claude. Once ingested, these tools have the potential to either display or otherwise use that information.
It is worth noting that a breach like this does not stem from an attack. The employees themselves share the information. The most notable example is Samsung, where executives realized that employees were using external AI tools and that their data was leaking. They promptly issued a company-wide policy restricting AI usage. Amazon did the same, restricting AI usage among its employees.
The Samsung and Amazon incidents demonstrated that AI-related data exposure does not always originate from external attackers. In both cases, employees interacting with generative AI systems became the entry point through which proprietary information left the organization’s security perimeter.
For decades, cybersecurity strategies have focused on protecting networks, applications, databases, and user accounts from unauthorized access. Firewalls, intrusion detection systems, antivirus software, and access control mechanisms have formed the foundation of enterprise security programs. The widespread adoption of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, and AI agents introduces an entirely new attack surface. Unlike traditional systems, AI models can be influenced through natural language, creating risks that conventional controls were never designed to address.
The deeper issue is that the natural-language capability of almost every AI tool has made malicious intent far more accessible, even to those who are not technically sophisticated. This widens the pool of potential attackers and the range of attack domains. Current security standards are still catching up to the behemoth of AI.
Traditional cybersecurity benefits from decades of accumulated knowledge, testing methodologies, compliance frameworks, and industry standards. Organizations can rely on established practices such as penetration testing, vulnerability scanning, secure development lifecycles, and frameworks like NIST, ISO 27001, CIS Controls, and the OWASP Top 10. For AI systems, the situation is far less mature. While organizations rapidly deploy LLMs, RAG systems, and autonomous agents into production, there is currently no universally accepted methodology for comprehensively evaluating AI-specific security risks. As a result, many organizations are deploying AI faster than they can assess its security posture.
A strong example is the Microsoft Copilot EchoLeak vulnerability. Attackers sent emails containing malicious prompts and commands that looked harmless to traditional email defenses. When Microsoft Copilot indexed and read those emails for summarization, it was compromised purely through the prompt. Every conventional security system was doing its job correctly, yet the leak still happened because there was no AI-specific guardrail. Similarly, Chevrolet’s AI chatbot was talked into agreeing to sell a car for one dollar.
As organizations increasingly deploy LLMs, RAG systems, and AI-powered assistants, a new class of vulnerabilities has emerged that differs significantly from traditional cyber threats. Rather than exploiting software bugs or network weaknesses, these attacks target the AI model’s interpretation of information. Among them, prompt injection has become one of the most significant and widely discussed risks in enterprise AI security.
Prompt injection is an attack in which an adversary embeds carefully written instructions into the input an AI will read, in order to make the AI do things it should not. The goal is to manipulate the system and extract sensitive details or unethical outputs. Unlike traditional attacks that exploit software bugs or network vulnerabilities, prompt injection exploits the model’s interpretation of language. A sample injected prompt might look like this:
This is alarming because the space of possible injected prompts is effectively infinite, given the non-deterministic nature of natural language. For any system designed to filter out malicious intent, an attacker only needs to craft a better prompt that slips past the flags. The Microsoft Copilot EchoLeak incident was itself an example of prompt injection.
Where prompt injection manipulates through input, data poisoning begins the corruption at the source. The attacker deliberately introduces malicious or misleading data into the training dataset or retrieval source, effectively planting a backdoor to exploit later.
Data poisoning can occur at multiple stages of an AI system’s lifecycle. In machine learning systems, attackers may inject manipulated samples into training datasets to alter the model’s behavior. In RAG systems, attackers may poison knowledge bases, documentation repositories, or external information sources the model relies on. An indirect example is Microsoft Tay, which was launched to learn from Twitter interactions. Attackers coordinated to feed Tay racist statements and hate speech, corrupting its behavior.
As AI becomes increasingly valuable, attackers are no longer interested only in manipulating systems. A newer goal is to replicate the entire model. Unlike traditional cyberattacks that seek unauthorized access to servers or databases, model extraction focuses on learning the decision-making patterns of the AI itself. This is especially concerning because modern models often represent years of research, massive training datasets, and millions of dollars in computational investment.
Consider a customer support chatbot. Suppose an organization has built a highly specialized model trained on years of proprietary customer interactions. An attacker can repeatedly interact with the chatbot, submitting thousands or even millions of carefully designed prompts while recording the responses. Over time, they accumulate enough information to train a separate model that behaves similarly. Although the attacker never gains direct access to the source code or training data, they effectively reproduce a significant portion of its capabilities. A strong real-world example is the Meta LLaMA leak in 2023. The model was given to select researchers for development, but within days the weights were leaked online. Thousands of copies soon circulated, and even competitors built derivatives.
Now that the dangers are clear, it is just as important to understand the precautions and mitigation strategies that protect these systems.
Prompt injection presents a unique challenge because LLMs process both instructions and data as natural language. Unlike traditional software, where commands and content are clearly separated, models often struggle to distinguish trusted instructions from untrusted information. No single control eliminates the risk. Mitigation requires multiple layers of defense operating throughout the AI pipeline.
The first step is to sanitize inputs of all kinds, both direct and indirect. Filtering out keywords is not enough, since an attacker may prompt in different languages, human-readable or not. A watchdog that looks for intent rather than just words goes a long way. AI tools should also not hold high privileges within an organization. Even when an injection succeeds, hard-enforced restrictions prevent things from getting out of hand. Separating contexts for a RAG bot is one of the keystones of protection: a strong system prompt that is isolated from user input and retrieved context is a powerful defense, because most injection attacks succeed when retrieved documents are treated as instructions. Finally, a firm audit and logging system is needed to close off any open ends.
Data poisoning is particularly challenging because the attack often occurs long before the system produces an output. Unlike prompt injection, where malicious instructions are visible during inference, poisoned data may remain hidden within training datasets, knowledge bases, or retrieval systems for extended periods. By the time incorrect outputs appear, identifying the original source of corruption can be extremely difficult. Ensuring data integrity becomes just as important as securing the models themselves.
It is important to verify sources and put strong checks in place before training begins. Each source should have a clear point of origin and a modification history. Organizations that invest in detailed version-history records stand a better chance against these threats. In many cases, the earliest indicator of poisoning is a change in the system’s outputs, so teams should monitor generated responses for unusual recommendations, policy inconsistencies, or factual inaccuracies. Behavioral monitoring serves as an early warning system for poisoning attempts that evade traditional controls, and it is important to recognize that a single source of trusted knowledge will not always remain trustworthy.
Model extraction is unique because the attack does not necessarily require unauthorized access to the target system. Instead, attackers interact with the AI through legitimate interfaces, collecting inputs and outputs over time to learn how the model behaves. Since modern models represent substantial investments in data, infrastructure, and research, the model itself becomes a valuable target for theft and replication.
Rate limiting is a strong countermeasure, since attackers rely on consistent input-output patterns to understand a model’s intricacies. Query quotas and usage restrictions significantly increase the cost and difficulty of reconstructing model behavior. Strong access and API authentication controls help ensure that only authorized users can reach AI services; they may not eliminate the risk entirely, but they provide accountability and deterrence. A non-technical precaution is to make sure the legal framework around the application is sound, using licensing agreements, policies, and enforcement mechanisms to discourage unauthorized replication.
| Attack | Defenses |
|---|---|
| Prompt injection Attackers embed malicious instructions to manipulate model behavior. |
Input sanitization · context isolation · system prompt hardening · output filtering · red teaming & adversarial testing · monitoring & logging |
| Data poisoning Malicious or misleading data is introduced to corrupt knowledge or training data. |
Data provenance & validation · access controls · version control & audit logs · content moderation · continuous data-integrity monitoring · periodic accuracy reviews |
| Model / agent abuse Attackers trick AI agents or extract model capabilities to misuse or replicate. |
Least-privilege access · action confirmations & approvals · tool & API restrictions · rate limiting · anomaly detection · kill switches & incident response |