Prompt Injection and Data Leakage in Financial AI Assistants
Treating Everything Your Assistant Reads as Hostile Input
Financial institutions have decades of practice treating user input as untrusted. Every form field is validated, every parameter is bound, every upload is scanned. Language model applications quietly break that discipline, because the content a model reads and the instructions it follows travel in the same channel, so a document your assistant retrieves can tell it what to do.
That is the whole of prompt injection, and it is structural rather than a bug awaiting a patch. The OWASP top ten for large language model applications, whose 2026 list was published in August 2026, ranks prompt injection first, describing manipulation via crafted inputs that can lead to unauthorised access, data breaches, and compromised decision-making. Defending against it in a bank means designing for containment rather than hoping for detection.
What makes prompt injection structural?
Instructions and data share one channel, so no wording reliably separates them.
| Type | How it arrives | Why it is hard |
|---|---|---|
| Direct injection | A user types instructions attempting to override system behaviour | Detectable in part, but paraphrase and encoding defeat filters |
| Indirect injection | Instructions hidden in retrieved content such as a document, email, or web page | The user is an unwitting carrier and never sees the payload |
| Multi-stage injection | Content instructs the model to plant instructions elsewhere for later | Persists across sessions and users |
| Tool-mediated injection | A tool's response contains instructions the model then follows | Trust is misplaced in your own integrations |
Why can't prompt engineering solve it?
Because separation of instructions from data is not enforceable in a shared channel.
System prompts telling the model to ignore instructions in documents reduce success rates and do not eliminate them, and each model update changes the picture. Treat prompt hardening as one layer among several, useful and insufficient, and put your engineering effort into limiting what a successful injection can accomplish. That is the same reasoning that made least privilege the foundation of modern security architecture, as set out in this guide to zero-trust security architecture.
Which attack paths matter in a financial institution?
Anything the assistant reads that someone outside your control can influence.
| Path | Realistic scenario |
|---|---|
| Client-supplied documents | A borrower's financial pack contains hidden instructions, ingested during preparation |
| Inbound email and attachments | An assistant summarising a mailbox reads an attacker's message |
| Web and vendor content | Market commentary or a supplier portal page retrieved for context |
| Third-party data feeds | A field in an external record carries instructions |
| Internal documents authored externally | A contract draft returned by a counterparty |
| Tool and API responses | A compromised or manipulated integration returns instruction text |
| Agent-to-agent messages | One agent's output becomes another's untrusted input |
The first row is the highest-frequency risk in banking, because financial institutions receive documents from outside constantly and process them enthusiastically. Any assistant that reads client documents is exposed by design, which is why the copilot patterns in relationship manager copilot design put read-only scope first.
What is the attacker actually trying to achieve?
Exfiltration, unauthorised action, or influencing a decision, in roughly that order of likelihood.
Exfiltration means getting data out, either directly in output the attacker can see or indirectly through a request to a destination they control. Unauthorised action means using the assistant's tools to do something: create a payee, alter a record, send a message. Influencing a decision is subtler and specific to finance: content crafted so a credit summary omits a weakness or a risk assessment reads more favourably, which harms the institution without any data leaving. That third objective is rarely modelled and deserves to be, because it exploits exactly the trust a copilot is built to earn.
Have you modelled an attacker whose goal is a more favourable credit summary rather than stolen data?
How does data actually leak?
Mostly through outbound calls whose parameters carry the payload.
| Channel | Mechanism | Control |
|---|---|---|
| Outbound tool call parameters | Data placed into a query, URL, or message body | Egress allowlist, parameter validation |
| External fetch or link rendering | Model induced to request an attacker-controlled URL | Block outbound fetches, disable auto-rendering |
| Output to the user | Restricted content surfaced to someone unauthorised | Entitlement-aware retrieval, output checks |
| Logs and traces | Sensitive content copied into log stores with wider access | Redaction, access control on logs |
| Model provider retention | Prompt content retained or used beyond your terms | Contractual and technical verification |
| Error messages | Stack traces and diagnostics carrying data | Sanitised errors |
Why are outbound calls the main channel?
Because they leave the boundary programmatically and often look like normal operation.
If the assistant can fetch a URL, send an email, write to an external system, or call a search API, then a successful injection has a route out that no output filter inspects. Close it structurally: no arbitrary outbound HTTP, an allowlist of destinations enforced at the network layer, validated parameters per tool, and no automatic rendering of model-supplied links or images. Those controls cost little and remove most of the exfiltration surface, which makes them the highest-return work in this whole area.
Which defences actually work?
Privilege limits first, then containment, then human gates, then detection.
| Defence | Effectiveness | Notes |
|---|---|---|
| Least privilege on tools | High | An assistant that cannot act cannot be made to act |
| Egress allowlisting | High | Removes the primary exfiltration route |
| Entitlement-aware retrieval | High | Limits what an injection can reach in the first place |
| Human confirmation for sensitive actions | High | Breaks the automated chain at the consequential step |
| Content isolation and structured prompting | Moderate | Clear delineation helps and is not a guarantee |
| Injection classifiers on inputs | Moderate | Catches known patterns, evaded by novel phrasing |
| Output filtering for known sensitive patterns | Moderate | Useful backstop, blind to paraphrase |
| A second model checking the first | Moderate | Adds cost and can itself be injected |
| Prompt hardening | Low to moderate | Necessary hygiene, never the primary control |
The ordering matters. Institutions that invest first in detection classifiers and last in privilege limits end up with an assistant that can do a great deal of damage and a filter that catches yesterday's phrasing.
What does containment architecture look like?
An untrusted content boundary, with no tool access in the same context as untrusted material.
The practical pattern is to separate the phase that reads untrusted content from the phase that takes action. Retrieve and analyse documents in a context with no tools and no egress, produce a structured summary, then let a separate step act on that structured output with tools available and deterministic validation applied. An injection in the document can corrupt the summary, which is bad, and cannot directly invoke a tool, which is much worse. Add deterministic gates on anything material, so a proposed action is validated against business rules rather than trusted because the model produced it, which is the boundary discipline described in agentic AI for back-office operations.
Can your assistant read an untrusted document and call a tool in the same context?
Talk to Digiqt about containment architecture for AI assistants
How do you test for injection?
With a maintained injection corpus, embedded in realistic documents, run as a regression suite.
Build a corpus of injection payloads covering instruction override, data exfiltration attempts, tool invocation attempts, encoded and obfuscated variants, multi-language payloads, and payloads split across document sections. Embed them in realistic financial documents, since a payload in a bare text file tests less than one hidden in a formatted statement or a contract appendix. Run the suite on every model version, prompt change, and tool addition, because all three change behaviour. Then add adversarial testing by people who did not build the system, and scope AI assistants into your wider threat-led programme, as described in threat-led penetration testing.
What should the suite measure?
Whether the injection succeeded, what it achieved, and whether anything detected it.
Record per payload whether behaviour changed, whether data left, whether a tool was invoked, and whether monitoring alerted. That third column is the one people forget: an injection that succeeds and is detected immediately is a different risk from one that succeeds silently. Track the suite's pass rate over time as a release gate, and treat a regression as a blocking defect rather than a known issue, since the alternative is shipping a capability whose safety properties you have stopped verifying.
What should production monitoring watch?
Unusual tool patterns, new egress destinations, refusal rate changes, and canary tokens.
Alert on tool call sequences that deviate from normal patterns, on any egress attempt to a destination outside the allowlist, and on sudden changes in refusal or escalation rates, which frequently indicate either a model change or an attack pattern. Plant canary tokens, meaning distinctive fake identifiers, in sensitive records so any appearance of them outside the boundary reveals disclosure. Log prompts, retrieved sources, tool calls, and outputs under one trace so an incident can be reconstructed rather than inferred. And define an incident response path specifically for AI assistants, including who can disable the assistant, because during the first real incident nobody will want to debate that.
What governance applies?
Inventory, an AI framework, security testing as a release gate, and a defined incident path.
Register assistants in the model inventory, govern them under your own policy plus a recognised framework such as the NIST AI Risk Management Framework 1.0 and its 2024 Generative AI Profile, and use the NIST Cybersecurity Framework, now at version 2.0 released in February 2024, to place the controls inside your existing security programme rather than creating a parallel one. OWASP's list gives you a shared vocabulary with your security function, including sensitive information disclosure at LLM06 and excessive agency at LLM08, which is the entry that most directly warns against giving an assistant unchecked authority to act. Note that OWASP has moved active development to its GenAI security project, so treat the top ten as a widely understood taxonomy rather than a fixed specification. The wider governance framing sits in this guide to bias and governance in AI systems and the responsible-deployment patterns in building responsible generative AI systems.
How should defence be sequenced?
Privilege and egress first, then containment, then testing, then monitoring, then detection layers.
| Phase | Duration | Deliverable |
|---|---|---|
| Tool and egress lockdown | 2 to 4 weeks | Allowlisted destinations, minimal read-only tools, no auto-fetch |
| Entitlement-aware retrieval | 1 to 2 months | Permission enforced at selection, not display |
| Containment split | 1 to 2 months | Untrusted analysis context separated from action context |
| Injection test suite | 1 month | Corpus, realistic embedding, release gate |
| Monitoring and canaries | 1 month | Tool anomaly alerts, egress alerts, canary tokens, tracing |
| Detection layers | Ongoing | Input classifiers, output filters, second-model checks |
| Incident readiness | 2 weeks | Runbook, kill switch, named owners, rehearsal |
The first row can usually be done in weeks and removes most of the realistic damage, which makes it the right place to start even if the rest of the programme takes months.
Which metrics matter?
Injection suite pass rate, blocked egress attempts, tool scope, detection rate, and time to disable.
Report the injection regression suite pass rate per release as a gate. Count blocked egress attempts, since a nonzero number confirms the control is active. Track tool scope per assistant, meaning how many actions it can take and how many are irreversible, and drive it down deliberately. Measure detection rate on known payloads, distinguishing prevented from detected. And rehearse and measure time to disable an assistant, because that is your containment capability during an incident and it is invariably slower than anyone expects the first time.
Prompt injection is not a temporary weakness that better models will remove. It follows from mixing instructions and data in one channel, which is what makes these systems useful. The institutions handling it well stopped trying to make the model trustworthy and started making the surrounding system safe when the model is not.
Frequently Asked Questions
What is prompt injection?
Manipulating a language model through crafted input so it follows an attacker's instructions instead of yours, which can lead to unauthorised access, data disclosure, or unintended actions.
What makes indirect injection worse than direct?
The attacker never touches your interface. Instructions hidden in a document, email, or web page reach the model when your system retrieves that content, so the user is an unwitting carrier.
Can better prompt engineering solve it?
No. Instructions and data share one channel, so no wording reliably separates them. Prompt hardening reduces frequency and cannot be the primary control.
Where does data actually leak?
Mostly through outbound tool calls and fetches whose parameters carry the data, and secondarily through output to the user, logs, and error messages.
Which defence gives the most protection?
Least privilege on tools and egress. If the assistant cannot reach an external destination or perform a sensitive action, a successful injection has far less to work with.
Does output filtering help?
As a secondary layer. It catches known patterns such as card numbers or identifiers leaving, and it cannot catch paraphrased or encoded disclosure, so it must not be the main control.
How do you test for injection?
With an injection corpus embedded in realistic documents, run as a regression suite on every change, plus adversarial testing by people who did not build the system.
What should production monitoring watch?
Unusual tool call patterns, egress to new destinations, sudden changes in refusal rates, and canary tokens planted in sensitive records to detect disclosure.
