Tuesday starts with a pager alert and ends with a call from the CISO. A B2B support agent with access to a CRM and an email connector has read a customer ticket containing poisoned instructions. It follows those instructions, retrieves an internal pricing sheet, and forwards it to an external address. The customer sees a normal reply. The support team sees a normal audit trail, until someone notices the unusual outbound message.
That isn't a model-quality issue. It's an operational incident involving data exposure, unauthorized access, and a system that acted beyond the user's intent. The uncomfortable question on the escalation call isn't whether the agent was “intelligent” enough. It's who allowed it to read that content, call that tool, access that data, and send that message without a control stopping it.
This is the operator's view of AI agent security. The model matters, but the trust boundary around every tool, connector, credential, and data source matters more. The practical objective isn't to make an agent perfectly obedient. It's to make bad behavior constrained, visible, reversible, and expensive to continue.
Table of Contents
- The Moment Your Agent Becomes a Liability
- What an AI Agent Actually Is and Where It Breaks
- The Threat Models That Hit Agents in Production
- Secure-by-Design Patterns You Can Ship This Quarter
- Runtime Controls That Keep Agents Honest
- Governance, Red-Teaming, and the SDLC Loop
- Your 90-Day Playbook and Operator FAQs
The Moment Your Agent Becomes a Liability
The first Slack message usually sounds harmless: “Can someone check why the support agent sent this?” The second includes a screenshot of an email that nobody approved. The third asks whether the agent had access to customer records, internal documents, or shared mailboxes.
By then, the system has already crossed the line from assistant to liability. A read-only summarizer can produce a wrong answer. A read-write agent can send mail, update a CRM record, alter a ticket, expose a document, or trigger a downstream workflow. The difference is not philosophical. It determines whether the incident stays inside a review queue or reaches customers, regulators, counterparties, and legal counsel.
The poisoned-ticket scenario is easy to underestimate because every individual step can look legitimate. The agent reads a ticket, searches a connected system, selects an email tool, attaches a document, and sends a message. If nobody records the full chain, investigators may see only a series of ordinary API calls rather than the sequence that turned untrusted content into an unauthorized side effect.
Practical rule: Treat every agent action as a production operation with an owner, an authorization decision, and a replayable record.
Why the incident starts before the alert
Teams often ask whether the model was “jailbroken.” That question is too narrow. The incident began when the organization gave the agent a broad capability set and allowed external content to influence tool selection without an enforceable boundary.
The OWASP Top 10 for Large Language Model Applications places prompt injection at the number-one risk, describing how attackers can manipulate LLMs into unauthorized access, data breaches, and compromised decision-making. The risk becomes operationally serious when an agent can use tools, connectors, and external data sources, because the manipulated instruction can produce a real side effect.
Integrity verification also belongs in the incident conversation. Teams need a clear answer to who controls integrity verification for prompts, tool metadata, model artifacts, and connected data. If ownership is ambiguous, nobody can explain which trusted component changed or who approved it.
The operator's response
The correct response is not to shut down every experiment indefinitely. It's to inventory the agent's permissions, revoke unnecessary access, preserve traces, inspect tool parameters, and identify the first untrusted input that influenced the run. Then fix the boundary that allowed the agent to convert content into authority.
Your Tuesday-morning checklist should be blunt:
- Contain the capability: Disable high-impact tools or move them behind human approval.
- Preserve the trail: Retain prompts, retrieved context, tool calls, policy decisions, outputs, and side effects.
- Inspect the identity: Determine which service identity, user delegation, or connector credential performed each action.
- Rebuild the path: Reproduce the run with the same inputs and confirm where policy should have stopped it.
That is what useful AI agent security looks like. It starts with incident response discipline, not a debate about whether the model is aligned.
What an AI Agent Actually Is and Where It Breaks
An AI agent is a loop. It receives a goal, sends context to an LLM, interprets the model's decision, calls a tool, observes the result, and repeats until it reaches a stop condition or a human intervenes. The tools may include APIs, files, browsers, databases, email systems, CRM actions, or Model Context Protocol servers.
The AI agent workflow is easier to secure when operators draw the full loop rather than treating the model as a standalone component. The model generates a decision, but the surrounding system determines what that decision can do.
Three trust boundaries
Think of the agent as a microservice with three boundaries that require different controls.
The input boundary contains user messages, support tickets, web pages, uploaded files, retrieved passages, and tool responses. These sources may contain instructions disguised as ordinary content. A browser page can tell the agent to ignore its original task. A document can request secrets. A tool response can embed text that looks like a system directive.
The model boundary includes the system prompt, model behavior, retrieved context, adapters, and orchestration logic. Prompt injection targets the relationship between these elements. The model may not reliably distinguish an authorized instruction from text that merely appears imperative, especially when the orchestration layer places both into one context window.
The tool boundary includes every connector, API, MCP server, service identity, and credential. The blast radius becomes concrete here. A model can propose an email, but the email connector sends it. A model can request a database query, but the identity and query policy determine what comes back.
Why the tool boundary gets priority
Most operators spend too much time trying to make the model refuse malicious instructions and too little time limiting what happens after a bad decision. Model behavior is probabilistic. Tool permissions, schemas, network routes, approval gates, and audit records can be deterministic.
That doesn't make the model boundary irrelevant. It means the security program should assume the model can be confused and design the tool boundary so confusion doesn't become compromise. The next threats all exploit that gap, through content, credentials, connectors, or sequences of actions.
The Threat Models That Hit Agents in Production
Agent attacks rarely arrive as a single dramatic exploit. They appear as ordinary actions assembled into an unauthorized chain. The table below maps the main threat models to what operators can observe and what their existing telemetry often misses.
AI Agent Threat Models at a Glance
| Threat | Typical Vector | Primary Asset at Risk | Common Detection Blind Spot |
|---|---|---|---|
| Data exfiltration | A poisoned ticket or document persuades the agent to include sensitive data in a normal outbound tool call | Customer records, pricing, internal documents | Teams log the destination but not the data selected for the payload |
| Prompt injection | Malicious instructions appear in a web page, file, retrieval result, or tool response | Agent intent and downstream actions | Security teams inspect user prompts but ignore indirect content |
| Model and prompt theft | Repeated endpoint queries extract behavior, system instructions, or proprietary adaptations | Model logic, prompts, retrieval strategy | API monitoring tracks availability and cost, not extraction patterns |
| Privilege escalation | A read-only workflow chains a weak connector or delegated identity into a write operation | Business records, accounts, workflow state | Permissions are reviewed per tool, not across the entire action chain |
| Supply-chain compromise | A modified MCP server, tool description, dependency, or retrieval index steers agent behavior | Agent integrity and every connected system | Approved integrations remain trusted after metadata or artifact changes |
Data exfiltration
An agent can leak data without invoking an obviously malicious function. A routine enrichment request may contain an unexpected customer list. An email call may include an internal document because the model interpreted an embedded instruction as part of the task.
Failure indicator: outbound payloads contain fields, attachments, or volumes unrelated to the user's request.
Detection blind spot: teams record that an approved tool was called but don't inspect the parameters, data classification, destination, and authorization context together.
Prompt injection and indirect injection
Direct prompt injection targets the user-facing interaction. Indirect injection hides the instruction in content the agent retrieves or browses. StakeBench research reported indirect prompt injection attack success rates from 41.67% to 68.16%, while direct prompt injection exceeded 79% across tested configurations, and it observed no attack scenario that leading GPT-5 and Gemini-based systems consistently blocked. Those findings are documented in the OWASP resource covering agentic application risks.
Failure indicator: the agent changes task scope after reading external content or invokes a tool the user never requested.
Detection blind spot: prompt logs capture the visible user message but omit retrieved passages, web content, tool metadata, and context ordering.
Model and prompt theft
An attacker can probe an exposed endpoint to infer system instructions, retrieval behavior, or proprietary fine-tuning. The objective isn't always to copy weights. Extracting the control logic may reveal sensitive business rules or help an attacker craft more effective injections.
Failure indicator: repeated, adaptive queries produce progressively similar outputs designed to expose hidden instructions or behavior.
Detection blind spot: rate limits are tuned for availability rather than behavioral patterns, so extraction looks like ordinary usage.
Privilege escalation
A read-only agent may gain write capability through a connector that accepts delegated credentials, a workflow that trusts agent-generated parameters, or a sequence where one tool creates authorization for another.
Failure indicator: a run crosses from retrieval into mutation without a distinct policy decision or approval.
Detection blind spot: each tool appears correctly configured when reviewed alone, while the combined path creates unintended authority.
Supply-chain risk
An MCP server or retrieval index is part of the agent's operating environment, not a harmless plugin. A changed tool description, poisoned document, or compromised dependency can redirect behavior before the user notices anything unusual.
Failure indicator: tool metadata, schemas, publishers, endpoints, or retrieval content changes outside the expected release path.
Detection blind spot: allowlisting happens at onboarding, then the organization stops verifying provenance and integrity during operation.
The benchmark evidence matters because it rejects the comforting idea that one filter will solve the problem. Recent RAG-agent testing used 847 adversarial test cases across five attack categories and found that a layered defense stack reduced successful attacks from 73.2% to 8.7% while preserving 94.3% of baseline task performance. The results appear in the RAG-enabled agent security benchmark, and the operational lesson is clear: combine anomaly detection, prompt guardrails, and response verification instead of betting the workflow on a single refusal mechanism.
Secure-by-Design Patterns You Can Ship This Quarter
Secure-by-design work should create engineering tickets, not architecture theater. The right question is simple: if the agent is manipulated during one run, what access disappears, what action gets blocked, and what evidence remains?
Start with least agency, not merely least privilege. An agent may have limited permissions and still cause harm if it can act autonomously across too many systems. Separate read, draft, approve, and execute capabilities. Put irreversible actions behind an explicit approval step.
Four tickets for the next sprint
Issue per-task credentials. Give each run a narrowly scoped identity with a short lifetime. When the task ends, the capability expires. A stolen or misused credential should not remain useful after the workflow completes.
Broker tool calls. Route actions through a policy service rather than placing a broad secret in environment variables. The agent requests an operation, the broker evaluates identity, tool, parameters, destination, and approval state, then issues only the capability required for that call.
Validate at the boundary. Enforce JSON schemas, field types, allowlists, content-type restrictions, destination rules, and maximum payload sizes before a connector receives model-generated arguments. Do this outside the prompt. Instructions are not a substitute for validation.
Block dangerous outputs. Use structured tool contracts and policy guards to reject destructive actions regardless of model confidence. A model saying “approved” shouldn't authorize deletion, external sharing, or financial changes.
Protect the supply chain
Pin model artifacts, adapters, prompts, tool schemas, and connector versions. Verify signatures and hashes through a registry that fails closed when an artifact drifts. Review tool descriptions as carefully as system prompts, because the agent may treat metadata as behavioral instruction.
Sandbox code interpreters and browsers with isolated filesystems, restricted egress, and disposable environments. Store important logs and datasets in append-only or write-once systems when modification would destroy forensic evidence.
These patterns don't require a new research program. They require named owners, acceptance criteria, and tests that fail when a permission expands or a checksum changes.
The implementation sequence should be practical:
- First ticket: Inventory tools and identities.
- Second ticket: Replace shared credentials with brokered, per-task access.
- Third ticket: Add schemas and policy checks to every high-impact tool.
- Fourth ticket: Capture immutable traces and test replay with representative attacks.
Runtime Controls That Keep Agents Honest
Design controls limit what an agent should do. Runtime controls stop what it tries to do. You need both, because a correctly designed workflow can still encounter poisoned content, a compromised connector, a model regression, or an unexpected loop.
Enforce decisions on every call
Checking access at session start isn't enough. The policy engine should evaluate every tool invocation using the agent identity, user context, requested operation, parameters, destination, data sensitivity, and current approval state. Log the decision with timestamp, reason, policy version, and outcome so an investigator can replay the run.
The agent should request capabilities, not see raw credentials. Keep secrets in a short-TTL vault, inject them only into the broker or connector, and prevent model output from containing tokens. If a tool needs a secret, the orchestration layer should handle that exchange outside the model context.
The controls described in AI data security guidance for operational teams fit into this same boundary-first model. Protect data by controlling which operation can access it, where the result can go, and which human can approve the exception.
Slow down runaway behavior
Rate-limit per agent, per tool, and per downstream identity. A runaway loop shouldn't be able to drain a payment workflow, scrape a partner service, or generate an uncontrolled stream of outbound messages. Set separate ceilings for reads, writes, retries, and external destinations, then make exceptions explicit.
Sandbox browsers and code interpreters with deny-by-default egress, restricted filesystem access, process limits, and disposable state. A browser agent that can reach every internal hostname is not an assistant. It's a network pivot with a language interface.
Observe side effects, not just text
Store the complete execution trace:
- Context lineage: user request, retrieved documents, web pages, tool responses, and system instructions.
- Action lineage: every selected tool, argument, identity, destination, approval, and returned value.
- Outcome lineage: records changed, messages sent, files created, and downstream workflows triggered.
Alert on new tools, new endpoints, unusual data volumes, unexpected parameter fields, repeated denials, and outputs that violate policy. A clean final response doesn't prove a clean run. The agent may have taken a harmful action before producing a reassuring sentence.
The useful alert isn't “the model looked strange.” It's “this identity sent sensitive data to a destination it has never used, after reading an untrusted document.”
Governance, Red-Teaming, and the SDLC Loop
Security can't be a launch gate that disappears after the demo. Agent behavior changes when prompts, models, retrieval indexes, tool metadata, permissions, and connectors change. The release process has to treat those components as one system.
Put security into each release stage
During design, threat-model the workflow. Draw the data flow from user input to retrieval, model, tool broker, connector, and side effect. List the untrusted sources, identities, approval points, and failure modes before the first production integration.
During staging, red-team the actual agent. Use the actual tool manifests and representative data. Test direct prompt injection, indirect instructions in retrieved documents, browser content, tool abuse, privilege escalation, and cross-context contamination. A generic penetration test won't expose a flaw in a connector chain the tester never saw.
During release, audit the evidence. Require signed evidence packets containing model and prompt versions, tool schemas, permission scopes, test results, policy changes, and unresolved findings. Screenshots are not release evidence. They lack provenance, context, and reproducibility.
In production, enforce policy continuously. Monitor decisions, connector changes, tool metadata, anomalous sequences, and approval bypass attempts. Feed incidents back into threat models and regression tests.
Make CI test the dangerous paths
Tool calls need schema tests. Guardrails need policy unit tests. Known jailbreaks and injection strings belong in regression suites. Canary prompts should block deployment when a new model or orchestration change causes a protected action to proceed.
Governance frameworks can provide vocabulary without replacing engineering. Map controls to SOC 2, ISO 27001, NIST AI RMF, and the EU AI Act, but keep the mapping tied to concrete evidence: identities, logs, approvals, testing, change review, and incident handling. A compliance matrix that doesn't point to an enforceable control is paperwork.
The AI governance and compliance guidance is useful when translating operational controls into leadership and audit language. Keep the ownership model equally concrete. Product owns the workflow, engineering owns implementation, security owns control requirements and testing, and a named business owner accepts residual risk.
Your 90-Day Playbook and Operator FAQs
A defensible rollout doesn't require waiting for a perfect platform. It requires sequencing the work so every phase reduces uncertainty and blast radius.
Days 1 through 30
Inventory every agent, tool, connector, model, prompt, retrieval source, identity, and downstream side effect. Map the data flows and name an owner for each workflow. Mark which actions only read, which draft, which mutate records, and which communicate externally.
Capture the current baseline before changing it. If you can't show what an agent can access or which tools it calls, you don't have a security posture. You have an assumption.
Days 31 through 60
Replace broad credentials with scoped identities and brokered calls. Add input and output validation at tool boundaries, move secrets into a vault, and record prompts, retrieved context, tool calls, decisions, approvals, and side effects.
Prioritize high-impact actions first. External email, customer data export, financial changes, account administration, and record deletion deserve enforcement before low-risk drafting or internal search.
Days 61 through 90
Run a scoped red-team against the most consequential workflow. Close the top three findings, repeat the tests, and establish an audit cadence tied to releases. Require re-approval when model artifacts, tool descriptions, permissions, retrieval sources, or destinations change.
Akto's 2025 report describes the adoption-control mismatch directly: 69% of enterprises were piloting or running early production AI agents, while only 21% had full visibility into agent actions, tool invocations, or data access. The same report says 79% lacked a formal governance policy for AI agents or MCP connections, and 60% hadn't conducted an AI or agentic risk assessment in the previous 12 months. Those figures come from Akto's 2025 State of Agentic AI Security Report. Use them to frame the control gap, then measure your own inventory and evidence rather than repeating industry averages as a substitute for facts.
Operator FAQs
How should I frame the budget?
Fund controls around the workflows that can expose data, change records, spend money, or contact outsiders. Start with identity, policy enforcement, secrets, telemetry, and testing. Don't lead with a vague “AI security platform” request.
Who owns agent security?
The business owner owns the risk of the workflow. Engineering owns implementation. Security defines controls, testing, and evidence. Infrastructure owns identity, network, and secrets. If nobody is named, the agent is unmanaged.
How do I explain this to the board?
Say that an agent is a software system with delegated authority and an untrusted-content attack surface. Report which agents exist, what they can do, which actions require approval, how activity is monitored, and whether releases produce audit evidence.
When should I gate a release?
Gate it when a new tool, connector, identity, model, prompt, retrieval source, or high-impact action lacks an approved threat model and passing tests. Also gate it when traces are incomplete or policy decisions can't be reconstructed.
How is this different from classic AppSec?
Classic AppSec focuses heavily on code, dependencies, identities, and network paths. AI agent security adds probabilistic instruction following, retrieved context, tool metadata, dynamic planning, and model-generated parameters. Keep the classic controls, then add context lineage, action-level policy, adversarial testing, and runtime behavior monitoring.
Cyndra helps teams turn real workflows into production-grade AI agents with scoped credentials, human approval for high-stakes actions, and audit logging for tool calls and agent activity. Visit Cyndra to discuss an agent security design that fits your systems, owners, and release process.