Prompt Injection and Untrusted AI Context

Bounded lesson outcome

Natural-language systems can blur the visual difference between instructions and data. That does not mean every item placed in model context receives authority. The bounded outcome of this lesson is to model Untrusted AI Context as a trust-boundary problem, keep data separate from authority, constrain tool and action scope, and require independent validation or human approval before a model-influenced proposal crosses a privileged boundary.

The evidence destination is Untrusted AI Context Control Record v1. It does not claim that prompt injection has been completely solved, that a guardrail is infallible, or that one local test proves production security. It records, for a bounded synthetic fixture, each Context Item, Trust Level, whether it May Inform Content, whether it May Grant Authority, the independent Validation applied, the Allowed Tool Scope, the Approval Boundary, the Observed Decision, and the Residual Risk.

The lab identifier is SEC008-LAB1. The entire lab is local and synthetic. It does not contact an external LLM, a real tool, a production account, a real credential, or an external target. The “model” is represented by fixed test outputs and every action is a harmless local stub. The exercise is about defensive boundaries, not bypass techniques or offensive prompt construction.

1. Context is not authority

A user request, a retrieved document, an external note, a tool result, and a model output can all appear in one workflow, but they do not automatically have the same security role. The first question in the control record is therefore not “what does this text say?” but “what is this item allowed to influence?”

In the fixture, trusted_policy is the only policy authority. user_intent expresses the requested harmless local task. external_note is untrusted data: it can inform content but cannot rewrite policy or expand action scope. tool_result remains data until independently validated. model_proposal is a proposal, not an authorization decision.

This creates a deliberate distinction between May Inform Content? and May Grant Authority?. An external note can be useful input to a summary, so the first answer can be yes while the second remains no. Collapsing those two columns is a trust-boundary error.

2. Prompt formatting is not a hard security boundary

Structured prompts, delimiters, role labels, and instruction hierarchies can make workflows clearer. They are useful engineering tools, but the lesson does not treat them as a hard authorization boundary. If policy says only a small set of actions is allowed, enforcement must remain visible in a deterministic policy check or an explicit approval gate.

In SEC008-LAB1, fixed model output cannot rewrite the local policy. Even when a model proposal looks reasonable, the evaluator checks the action name, requested scope, argument schema, user intent, and approval requirement again. The workflow therefore follows a simple rule: the model may propose, but policy decides.

A well-written system prompt can still reduce ambiguity. The evidence requirement is stronger: if the model proposes something outside scope, an independently inspectable gate must deny it. “The model usually follows the prompt” is not sufficient evidence.

3. Classify untrusted context before model influence

The learner begins with a small set of synthetic context items. For every item, the record identifies the source, trust level, allowed influence, and authority restriction. Classification is not a complete prompt-injection defense; its purpose is to make assumptions explicit so later controls can enforce them.

The external note is classified as untrusted_data. A synthetic tool result is also data until verified. The fixed model output is classified as untrusted_proposal. Even when a proposal aligns with the user request, it still passes schema validation and policy validation before any local action stub can run.

Negative Evidence is created by deliberately changing the external note metadata to trusted_authority. The validator must reject that classification. The probe uses no adversarial language and no live model; it mutates trust metadata so the boundary is observable.

4. Apply least privilege to tool and action scope

Every action in the lab is harmless and local, but the miniature environment is enough to demonstrate least privilege. The policy defines only three action stubs: show_preview, save_local_draft, and create_local_reminder. A model proposal cannot invent a fourth action and cannot expand the arguments or scope of an allowed action.

The record must contain an explicit Allowed Tool Scope, not a vague phrase such as “necessary tools.” When a proposal names an action outside the allowlist, the evaluator returns DENY_OUT_OF_SCOPE_ACTION. That denied path is evidence and must remain visible.

Least privilege also applies to fields. The reminder stub accepts only local fields defined by policy. There is no network destination and no external side effect. The resulting evidence proves only the local fixture boundary; it is not a production-security claim.

5. Validate actions independently of model output

Before any stub runs, the proposal must satisfy a fixed schema: action, arguments, source context, and declared intent. A deterministic evaluator then checks that the action is allowed, the argument names are permitted, the proposal does not conflict with user intent, and any approval requirement has been met.

Malformed model output returns REJECT_OUTPUT_SCHEMA. A well-formed but out-of-scope action returns DENY_OUT_OF_SCOPE_ACTION. An approval-gated action without approval returns DENY_APPROVAL_REQUIRED. These are implementation observations. The security interpretation is written separately in the evidence record.

This separation prevents an unsafe shortcut: a fluent or confident model answer does not become authorization evidence merely because its language is orderly.

6. Guardrails are defense in depth, not authority

A classifier or guardrail can contribute a risk signal. In this lesson the guardrail output is advisory and recorded separately from the final decision. If the guardrail says PASS while deterministic action policy says deny, the final decision remains deny. A guardrail warning likewise does not create new authority.

The design intentionally rejects the claim that a second LLM or guardrail guarantees prompt-injection prevention. Such components can improve detection or monitoring, but deterministic access control, least-privilege scope, validation, and approval boundaries remain independent.

A useful Negative Evidence probe leaves the advisory guardrail at PASS while changing the model proposal to an out-of-scope action. The correct result is still a denial, demonstrating that the guardrail did not replace policy enforcement.

7. Treat model output as untrusted before render, parse, store, or action

Model output is also untrusted. If it will be rendered, the rendering boundary must be appropriate. If it will be parsed as a structured action proposal, schema validation comes before the action. If a value is stored in the local evidence record, type, length, and allowed-field checks still apply.

SEC008-LAB1 uses only fixed JSON-like proposals. There is no dynamic code evaluation and no executable content. Unknown fields are rejected and only policy-defined actions are recognized. This demonstrates that output validation is a distinct control from input or context classification.

Output validation does not prove semantic correctness. A schema-valid proposal can still be wrong about the world. The Residual Risk field must preserve that limitation and explain why sensitive decisions remain dependent on policy or human review.

8. Make the human approval boundary visible

Some action classes can require approval even in a synthetic fixture. The lesson uses create_local_reminder as an approval-gated local action for demonstration. If the approval flag is false, the action does not run and the control record preserves the denial.

The lesson does not claim that human-in-the-loop is always sufficient. Humans can make mistakes. The instructional goal is to keep accountability visible: the learner should be able to identify what was deterministic, what was model-influenced, and where a human decision entered the flow.

The Approval Boundary field uses explicit states such as none, required, or satisfied. Owner acceptance is not prefilled, and the final changed-context decision is not supplied by AI.

9. Positive Evidence and Negative Evidence

Positive Evidence shows that the baseline works under declared assumptions. For example, the user requests show_preview, the fixed model output proposes that action, the schema is valid, and the local policy permits it. The observable result is ALLOW_LOCAL_STUB.

A bounded security claim also needs denied paths. The required probes are: reject a false authority classification for external context; deny an out-of-scope action; mark the workflow incomplete if the required approval gate is removed; and mark the evidence record incomplete if a required field is missing. Each probe separates prediction, observation, and bounded inference.

observed_status=DENY_OUT_OF_SCOPE_ACTION is an observation. “The local fixture enforces its action allowlist” is a bounded inference. “The production application is secure against every prompt-injection technique” is not supported by this evidence.

10. Untrusted AI Context Control Record v1

The record contains nine columns:

1. context_item
2. trust_level
3. may_inform_content
4. may_grant_authority
5. validation
6. allowed_tool_scope
7. approval_boundary
8. observed_decision
9. residual_risk

Every row must be traceable to a fixture, a policy rule, or an observed local output. Final decision fields and owner acceptance are not prefilled. A record without Negative Evidence or without a residual-risk statement cannot support the final bounded decision.

The table is an inspection surface, not decoration. If a reviewer cannot trace a row back to the local fixture and observed decision, the evidence quality is insufficient.

11. Transfer

The changed context contains a synthetic document summarizer and a scheduling helper with different authority. The summarizer may use an external document for content, but it has no action authority. The scheduling helper may produce an action proposal, but its scope and approval requirements must be rebuilt independently.

The learner should not copy the baseline policy mechanically. They must decide again which context items are data, which local actions are permitted, what validation is needed, and whether human approval belongs in the workflow. The transfer decision remains bounded and must preserve Residual Risk.

This final task intentionally removes the decisive scaffold. AI may help organize observed evidence, but it may not supply the learner’s final authority, scope, or residual-risk conclusion.

12. Non-goals

• A system prompt does not create an infallible hard security boundary around untrusted content.
• A guardrail or second LLM does not guarantee prompt-injection prevention.
• Model output does not receive privileged authority merely because it comes from an internal model.
• Prompt injection is not identical to SQL injection and is not solved by one shared escaping technique.
• The local lab does not establish production security.
• No offensive payload, bypass recipe, or external target is required.

13. Verification and evidence discipline

Completion requires the learner to run the baseline, preserve denied paths, fill the evidence template without a prefilled final decision, and solve the transfer case independently. static_preflight.py checks local-only boundaries. public-contract.test.py checks the observable fixture behavior. Hidden oracles independently verify trust, action-scope, approval, and evidence invariants.

A green test proves only the declared fixture behavior. If the policy, action set, model interface, tool integration, or threat model changes, decisive evidence must be rerun. This is especially important in AI-enabled systems because context, model behavior, and connected tools can evolve.

14. Sources and freshness

Current claims stay inside the frozen authority OWASP-LLM-PROMPT-INJECTION. The Bible snapshot was retrieved on 2026-09-02 and the next review date is 2026-10-02. During this authoring run the official primary source was checked again without adding a new authority.

The source is used for current prompt-injection threat patterns and defense-in-depth guidance including structured separation, output monitoring and validation, human-in-the-loop controls, least privilege, and action validation. The evidence artifact, status tokens, and local schema are MAKNOW instructional designs, not standards-defined record formats.

15. Completion boundary

The lesson is complete when the learner can explain, for the baseline and at least one negative probe, what was observed, which policy gate made the decision, which context item did not have authority, and what residual risk remains. Completion does not mean proving production-wide security or model/guardrail infallibility.