\ AIS
Research
We study the security and behavior of AI models and agents through three research clusters: AI Red Teaming, AI Alignment and AI Assurance.
AI Red Teaming
Developing scalable, algorithmic frameworks for AI red teaming that leverage optimization techniques and language models to autonomously generate diverse, complex edge-cases and jailbreaks.
We evaluate AI agents against the OWASP Top 10 for LLM Applications to identify critical vulnerabilities in logic, memory, and external tool integration.
Injection Attack Vectors
- Direct Injection
A user directly commands the AI agent to override its safety protocols.
- Indirect Injection
The AI agent is manipulated by processing external content (e.g., a website) that contains hidden malicious commands.
- Memory Injection
Poisoning the AI agent’s long-term memory or session history to influence future behavior.
- Tool Injection
Manipulating the tools and APIs the AI agent uses to interact with the outside world (e.g., databases).
Automating AI Red Teaming
Automated red teaming makes vulnerability discovery scalable, systematic, and interpretable. We move beyond ad‑hoc manual testing toward repeatable procedures that (i) uncover novel failure modes, (ii) remain effective as models and defenses evolve, and (iii) separate true safety behavior from evaluation artifacts such as brittle refusals.
We develop an automatic red-teaming framework that improves attack discovery through iterative feedback. Rather than relying on a fixed set of hand-written attacks, the framework repeatedly proposes, tests, and refines attack attempts to produce diverse and effective ways to expose model weaknesses over time. In parallel, we treat refusal behavior as a core part of the evaluation signal. We characterize the prompts and interaction patterns that trigger refusals and design controlled evaluation procedures that reduce refusal-related artifacts. This helps distinguish genuine safety behavior from broad or brittle refusals that can mask underlying failure modes, enabling clearer diagnosis of vulnerabilities and more faithful measurement of safety performance.
Operational Security Outcomes
Reduced Unknown Jailbreak Paths
Systematically exposing and closing safety gaps to minimize the surface area available for zero-day exploits.
Hardening of Tool Invocation
Reinforcing the security boundary between the model and external APIs to prevent unauthorized execution.
Lower Mean-Time-to-Fix via Reproducible Traces
Generating deterministic logs of attack chains to allow engineers to instantly replay and debug exploits.
CI Fails on Reintroduced Vulnerabilities
Integrating automated regression testing to block deployment if a fixed vulnerability resurfaces.
AI Alignment
AI alignment investigates how AI models and agents follow intended goals, instructions and constraints. Our research examines the gap between intended and observed behavior, including goal misgeneralization, reward gaming and deceptive behavior. We study how evaluation and oversight can reveal these failures as systems become more capable and autonomous.
Deceptive behavior and oversight
Within this cluster, we investigate behaviors in which an AI system misrepresents its knowledge, intent or actions. We examine how such behavior emerges, how it can be distinguished from ordinary errors, and how to evaluate it through controlled experiments.
We connect critical statements and tool-mediated actions to verifiable evidence and traceable execution, studying checks that identify discrepancies between reported intentions, uncertainty and observed behavior.
AI Assurance
Our policy-driven enforcement isolates untrusted content, validates and constrains actions at execution time, and applies output and data-handling checks to reduce injection-driven behavior changes and prevent sensitive information leakage.
AI Firewall
Agent Control Layer
Operational Security Outcomes
Fewer unauthorized or unintended actions
Policies gate tool calls, data access, and side-effecting operations so the agent executes only what is explicitly permitted and in scope.
Lower prompt injection and instruction smuggling success
Trust boundary controls treat untrusted inputs as data-only, reducing the chance that content from web pages, documents, users, or tool outputs can hijack control flow.
Reduced data exposure and safer information handling
Controls limit what can be retrieved, stored, or transmitted based on sensitivity and purpose, decreasing leakage risks and accidental disclosure.
Smaller blast radius and more predictable execution
Runtime enforcement, least-privilege permissions, and parameter validation constrain actions at the moment of execution, limiting impact from errors, compromised inputs, or misconfiguration.
Faster incident response and stronger audit readiness
Decision records and action traces link each operation to the governing policy and checks applied, improving triage, accountability, and compliance evidence.