LLM Security Basics: The Full Threat Model

Almost all LLM vulnerabilities originate from the architectural reality that models receive instructions and external data in a single token sequence with no structural boundary between them. While traditional software relies on parameterization to strictly separate executable commands from user data (such as in SQL queries), no such mechanism exists for natural language where instructions and data are both plain text. This allows prompt injection to succeed either directly via chat inputs or indirectly through retrieved external content like emails or documents. Relying solely on filtering mechanisms is insufficient, as evidenced by attacks that bypass dedicated injection classifiers.

The OWASP Top 10 for Large Language Model Applications maps critical AI security risks directly onto the stages of an LLM data pipeline, spanning input, retrieval, model processing, tools, and output. Each pipeline stage exposes distinct vulnerabilities, including prompt injection, knowledge base corruption, and excessive agency. Unlike localized pipeline stages, supply chain vulnerabilities underpin the entire architecture and affect all downstream components simultaneously. Mapping attacks to these stages clarifies where untrusted data enters and which permissions are abused, guiding effective defense strategies.

Attacks targeting a model's interior—such as model theft, training-data extraction, and training-time poisoning—pose real technical risks but are currently low priority for most standard application developers. Demonstrations show these attacks are bounded, expensive, or promptly mitigated by commercial model providers following disclosure. However, responsibility and risk rise significantly for teams that host open weights, fine-tune on proprietary data, or run their own training infrastructure. For downstream developers, common operational vulnerabilities like over-privileged agents present a far greater threat than interior model attacks.

LLM attacks that cause material damage exploit a structural vulnerability known as the lethal trifecta, which occurs when an agent possesses access to private data, exposure to untrusted content, and a channel to act externally. Model alignment cannot prevent these exploits because LLMs are inherently designed to follow instruction-like inputs. Real-world incidents have already compromised systems like GitHub's and Anthropic's MCP implementations, GitLab Duo, and autonomous trading agents. Mitigating this risk is typically achieved most affordably by narrowing access permissions or severing outbound communication channels rather than relying on stronger filtering.

Supply chain vulnerabilities in AI stacks pose severe risks by bypassing runtime defenses prior to input validation. Many distributed models use serialization formats like Python pickle that allow arbitrary code execution upon loading, exemplified by the nullifAI technique evading Hugging Face's Picklescan. The widespread scope of this threat is demonstrated by Protect AI flagging roughly 352,000 issues across over 50,000 models. Mitigating this risk requires strict provenance management, adoption of safe serialization formats, and model signing.

A single security guardrail cannot reliably defend AI models against adaptive attacks, necessitating a defense-in-depth architecture where multiple independent layers prevent single-point failures. Empirical research from late 2025 demonstrates that adaptive attacks can defeat multiple existing prompt injection and jailbreak defenses. Rather than attempting to make the model itself impenetrable, effective security architectures isolate and constrain the model's environment, treating its outputs as untrusted. Practical implementations of this philosophy include frameworks like Google DeepMind's CaMeL and operational heuristics like Meta's Agents Rule of Two.

The foundational threat model for language models arises because they process instructions and data as an indistinguishable sequence of tokens. Attacks like model theft and training-data extraction are largely bounded, but catastrophic risk concentrates when an AI agent simultaneously accesses private data, untrusted content, and external communication channels. Mitigating these risks demands a defense-in-depth approach across the supply chain, though safeguards such as human review inevitably trade off autonomy, latency, and cost.