Your support agent passed its accuracy evals. Then a customer from one tenant asked about pricing, and the agent answered fluently, confidently, and with another tenant's numbers. Nothing in the model failed. The retrieval layer did exactly what it was built to do, which was the problem. Protection by design is how you make that class of failure structurally impossible, not just unlikely.
The Cross-Tenant Leak That Started With a Prompt
A marketplace support agent receives a routine question about negotiated pricing. The system prompt says to only use the current customer's documents. The retriever, however, searches a shared index and ranks by similarity alone. A chunk from another tenant scores highest, enters context, and the model paraphrases it into a helpful answer.
No alert fires. The answer is well written, properly cited in tone, and wrong in the only way that matters. The customer now holds a competitor's terms, and your logs show a successful response.
Why the prompt was never a control
A prompt instruction is a request to a probabilistic system. It isn't an authorization check, and it can't remove content that's already in context. Once unauthorized evidence reaches the model, you're relying on the model to notice a boundary it was never given the tools to verify.
The same shape recurs elsewhere:
- Stale memory reuse: An old conversation summary outlives the policy it described and gets retrieved as current fact.
- Prompt injection: Retrieved text instructs the agent to call a tool, and nothing downstream asks who authorized that call.
- Unauthorized escalation: The agent issues a refund because the conversation sounded convincing, not because a policy check passed.
If a control only works when the model behaves, it isn't a control. It's a hope with formatting.
In each case, the fix isn't better wording. It's moving the decision to a layer that can say no deterministically, before generation begins. That means tenant scoping enforced in the retrieval query, tool calls gated by authenticated policy, and missing evidence producing an abstention instead of a plausible answer.
Teams often discover this ordering backwards. They add a review queue, a warning banner, or stricter prompt language after an incident. Those measures catch some outputs, but they leave the underlying path open. The next leak just needs a slightly different phrasing.
What Protection by Design Actually Means for AI
Protection by design means the safe behavior is a property of the architecture at runtime, not a statement in a launch review. If a guarantee disappears when a prompt changes, a model version rolls, or an integration is rewired, it was never architectural.
The idea has a measurable lineage outside software. A European study published in 1991 concluded that 60% of fatal accidents resulted partly from decisions made before construction work began, and a United States analysis of 224 fatalities between 1990 and 2003 found 42% linked to design factors, according to NIOSH prevention through design research presented at Purdue. The implication transfers directly. Layouts and constraints decided early determine whether later operators can fail safely at all.
From secure by design to AI systems
In April 2023, CISA and international partners urged manufacturers to take ownership of customer security outcomes rather than shifting the burden to customers, and CISA later described more than 200 software manufacturers as having joined its Secure by Design pledge, per CISA's Secure by Design guidance and pledge resources. Applied to AI, that ownership means the system that retrieves and generates carries the burden of proof, not the operator reading the output.
Two invariants follow from that shift:
- No action without authenticated authority: Retrieval, tool use, and state changes require a verified identity, tenant, and scope on every call.
- No output across a boundary without evidence: A claim that can't be tied to authorized, current evidence doesn't ship as an assertion.
This discipline mirrors what good product design principles demand in interface work, where constraints are built into the flow instead of explained in a tooltip after the user errs. In AI systems, the equivalent is removing the unsafe path entirely. The model proposes, deterministic policy disposes, and abstention counts as a legitimate outcome rather than a failure to be papered over with fluent text.
Five Principles That Anchor Protection in Architecture
Principles only matter if each one kills a named failure mode. These five do, and each one can be stated as a testable invariant in a design review.
The diagram below maps the principles as a hierarchy rather than a checklist of independent items.
The five invariants
Evidence-bounded claims require every substantive assertion to carry its supporting evidence identifiers, the protocol that validated it, and the denominator behind any reported rate. This prevents the quiet upgrade of a guess into a supported fact. If you can't point to the evidence unit, the claim doesn't render as a claim.
Authority-before-autonomy means the model never authorizes its own actions. It proposes a retrieval, a tool call, or a state change, and an authenticated policy layer checks identity, tenant, resource, and purpose against canonical state. A prompt injection can still produce malicious text, but text alone can't execute anything.
Fail-closed grounding covers the missing-data cases. Expired tokens, unresolved tenant identity, unavailable policy data, or unverifiable provenance produce denial, not best-effort continuation. Most production incidents I've seen trace back to a fallback path that treated unknown as allowed.
Composable isolation keeps credentials, billing, data authority, and operational responsibility separate, so one product or integration can be adopted or revoked without widening another's access behind the scenes. Abstention as a valid state completes the set. Refusing or deferring is an explicit, logged transition, not an error the system routes around.
The secure by design literature supports this front-loading. An analysis spanning 600 enterprise customers reported that organizations with more than 7,000 trained developers reduced vulnerabilities introduced into software by 47 to 53%, according to reporting on secure by design return on investment. Scope matters here. That figure covers conventional software vulnerabilities, not AI retrieval failures, so treat it as evidence for early invariant-setting generally, not as a promised AI outcome.
The video below walks through how these principles interact under a realistic agent workflow.
Authorization at the Retrieval Boundary
Retrieval is where multi-tenant AI systems leak, so it's where authorization belongs. Checking permissions at login and then trusting every downstream query leaves the widest door unguarded.
OWASP guidance on vector and embedding weaknesses identifies cross-tenant data leakage, poisoned index writes, and context bleeding as core risks, and recommends scoping indexes per tenant and task while enforcing authorization in every retrieval query, per OWASP guidance on generative AI advertising and vector security controls. The model should receive only evidence that was already authorized before similarity search ran.
The safe request path
A defensible path looks like this. First, bind the authenticated tenant and user identity to the retrieval call itself, not to the session that started minutes earlier. Second, apply tenant and task scoping as a hard filter before ranking, so unauthorized chunks never compete on similarity. Third, reject any result missing valid ownership metadata, even if it scored highest. Only then does content enter the prompt.
A shared physical index is fine under this model. Convenience isn't the enemy. The enemy is treating tenant identity as instructional text instead of a query constraint, a distinction explored further in patterns for AI access control in production systems.
Prove it with adversarial tests, not happy-path demos:
- Cross-tenant search: Query with tenant A's credentials for content that only exists in tenant B and confirm exclusion.
- Stale token reuse: Replay an expired credential and confirm denial at retrieval time.
- Unauthorized memory writes: Attempt to persist memory outside the caller's scope and confirm rejection.
- Poisoned document retrieval: Ingest a document with embedded instructions and confirm it can't expand its own audience.
Practical rule: If your isolation test asks the model whether it leaked, the test is wrong. Check the retrieved set, not the final prose.
Provenance as a First-Class Property of Every Evidence Unit
A citation string appended after generation tells you almost nothing. Provenance is structured metadata attached to each evidence unit before it's ever retrieved, describing origin, version, and the conditions under which it may be used.
NIST generative AI material defines data provenance as information about where data came from, how it was generated, its ownership and license, and how it was distributed, with recommended fields covering source, legal rights, generation date, method, intended use, and restrictions, according to NIST generative AI risk management material on provenance metadata. That definition sets the floor for production RAG.
Minimum provenance fields for an evidence unit
| Field | Purpose | Example |
|---|---|---|
| Source identifier | Stable reference to origin system | policy-db/refunds |
| Document version | Pins the exact text used | v14, supersedes v13 |
| Ingestion timestamp | Supports freshness checks | Recorded at ingest |
| Authorization scope | Defines who may retrieve it | tenant-id, support role |
| Transformation history | Shows chunking or redaction steps | split, PII redacted |
| Usage restrictions | Captures retention and license limits | internal use only |
A worked example
A support agent answers a refund question. The response record preserves the claim made, the evidence identifiers supporting it, the retrieval policy decision, the model version, and the mapping between each sentence and its source. When the refund policy is revised next quarter, you can find every answer built on the old version without re-reading transcripts.
Provenance doesn't prove a claim is true. It establishes what evidence was used and under which conditions, so separate checks for authority, freshness, consistency, and entailment have something concrete to operate on. Without it, claim validation runs against untracked text assembled from multiple systems, and every incident review becomes archaeology.
The trade-off is ingestion discipline. Someone has to populate and maintain these fields, and legacy corpora rarely arrive with clean lineage. Start with new and high-impact sources, and treat unprovenanced legacy content as lower-trust evidence with narrower permitted uses.
Fail-Closed Retrieval, Tools, and Policy Decisions
Fail-open systems continue when a check can't complete. Fail-closed systems deny. For AI control points that touch customer data or state changes, only one of those defaults is defensible.
Zero trust architecture guidance describes a policy enforcement point as the component that approves or denies each connection and logs the transaction using default-deny construction, per zero trust architecture guidance on policy enforcement points. Transplant that pattern into an agent stack and every retrieval, tool invocation, and state mutation gets an explicit allow or deny decision.
Two defaults compared
Under fail-open, a policy service timeout means the tool call proceeds and someone reviews logs later. The refund is issued, the record is mutated, and rollback becomes a business process. Latency looks great in the dashboard. Under fail-closed, the same timeout produces a denial or abstention, the user gets an honest deferral, and the state stays intact. You trade a small amount of availability for a large amount of containment, and for high-impact actions that trade isn't close.
The model sits upstream of this gate as a proposer. It can suggest issuing a refund, but the policy layer evaluates identity, tenant, resource, operation, purpose, and current canonical state before anything executes. A compromised or manipulated model can therefore suggest unauthorized actions without possessing the authority to carry them out, which is also why disciplined AI data pipeline design matters upstream of the gate itself.
What the decision log must preserve
- Decision inputs: Identity, tenant, resource, operation, and evidence identifiers presented.
- Decision outcome: Allow, deny, or abstain, with the specific reason code.
- State reference: The canonical state version the decision was evaluated against.
That shape lets operators distinguish a legitimate refusal from a retrieval failure or a policy defect. Without reason codes, every denial looks identical and teams respond by loosening the gate, which in turn converts fail-closed back into fail-open.
Making Protection Operational and Measurable Across the Lifecycle
Design reviews end. Production doesn't. Protection that isn't re-verified after model updates, integration changes, and permission edits decays over time while the documentation stays pristine.
ISACA's 2025 global privacy survey found that 67% of respondents said their organizations practiced privacy by design when building applications and services, yet only 27% reported always doing so, according to ISACA's State of Privacy 2025 survey results.pdf). That gap between claimed and consistent practice is exactly what runtime measurement exists to close.
Metrics with honest denominators
Report failure categories separately instead of collapsing them into one accuracy score. NIST's generative AI risk profile separates confabulation from unauthorized exposure of sensitive information, per NIST's Generative AI Profile in AI 600-1, and your metrics should honor that split:
- Unsupported-claim rate: Assertions lacking valid evidence, over total assertions evaluated.
- Sensitive-data leakage rate: Responses exposing unauthorized data, over total responses tested.
- Correct-abstention rate: Cases where evidence was insufficient and the system properly declined.
A system can improve factuality while still leaking, or reduce leakage by refusing everything. Separate denominators keep those outcomes visible, and frozen test cases with versioned results keep comparisons honest across releases. This is the working territory of agentic AI observability practices, where each run links input, retrieved evidence, policy decision, output, and any human intervention into one reproducible record.
Operational tooling reinforces the habit. Just as a tool for creating tutoring invoices turns ad hoc billing into consistent, reviewable records, an evidence protocol turns ad hoc agent behavior into consistent, auditable transactions. AletheionAGI offers one implementation of this approach as a grounding and evidence-control layer that works with existing LLMs, RAG pipelines, and vector databases rather than replacing them.
Documented intent is a hypothesis. Demonstrated control, measured against frozen cases after every material change, is a result. Keep those two categories labeled separately in every report you publish internally.
From Checklist to Architectural Property
Checklists record that someone considered a risk. They don't establish that the system resists it on a Tuesday afternoon after three integrations changed. Governance thinking makes the same point for content operations, as the Franchise Foundry content governance guide shows when it separates documented rules from enforced workflows.
Open questions remain, and pretending otherwise helps nobody. The minimum audit trail for an agent workflow isn't standardized. Measuring unsafe escalation is harder than measuring factual accuracy. The threshold between answering with caveats and abstaining outright is a product judgment, not a purely technical one.
The trade-offs are real too. Fail-closed behavior costs some availability. Provenance costs ingestion effort. Per-call authorization costs latency. What those costs buy is containment, reproducibility, and claims you can defend with records instead of recollections.
Treat protection by design as an evidence protocol with explicit denominators, versioning, and failure outcomes. That's a position you can carry into architecture reviews and procurement conversations alike, because it survives the only test that matters. It still holds at runtime.
If you're shipping agents or RAG over multi-tenant data, AletheionAGI provides grounding and evidence-control infrastructure with authorized retrieval, provenance-aware delivery, and fail-closed behavior that layers onto your existing models and pipelines. Bring your hardest isolation or unsupported-claim failure case and evaluate it against frozen tests before your next release.
