Security & Robustness

The Problem

As AI systems are deployed into critical infrastructure, enterprise workflows, and consumer products, they become a new and poorly understood attack surface. They are also brittle: small, deliberate manipulations of an input can make a state-of-the-art model fail. Security and robustness are two sides of the same problem - keeping systems reliable in the presence of an adversary.

Key challenges include:

• Prompt injection — Models process instructions and data through the same channel, so any text a model reads (from a web page, document, or tool) can hijack its behavior.
• Adversarial attacks — Small, often imperceptible perturbations to inputs that cause misclassification or unsafe outputs.
• Red-teaming at scale — Surfacing failure modes before adversaries do requires systematic, automated probing rather than ad-hoc testing.
• Deployment risk — Large-scale AI deployment carries security implications reaching far beyond the model itself, into the systems that host it.

What We're Working On

• Red-teaming & threat detection — Systematic methods to surface vulnerabilities, jailbreaks, and unsafe behaviors in deployed models.
• Prompt-injection attacks & defenses — Finding new direct and indirect injection techniques and developing defenses that separate trusted instructions from untrusted content.
• Adversarial robustness — Training and evaluation methods that keep models reliable under adversarial and out-of-distribution conditions.
• Security of AI deployment — Analyzing the security implications of large-scale AI, including in critical and maritime infrastructure contexts.