What Can AI Do in Programming—and What Should You Not Delegate to It?

AI can quickly propose code, tests, explanations, alternatives, and debugging hypotheses. That speed creates a specific engineering risk: a plausible proposal can be mistaken for a justified decision. A clean diff or confident explanation still does not tell you whether the result matches the actual requirement, allowed data, system constraints, or acceptance standard.

This lesson does not build a permanent list of tasks that AI “can” or “cannot” do. Tool capabilities change. Instead, you will build a more durable decision boundary: for one bounded task, define what AI may propose, what must be verified independently, and who retains the final acceptance decision and accountability. Your output is an AI Delegation Boundary.

The key question is not whether AI can produce an answer; it is what you are justified in accepting

Suppose you ask AI to fix a bug. It produces a patch, gives a convincing explanation, and even writes a few tests. Is the change now acceptable? Not yet. Three different things are involved:

• Candidate: something AI proposes or produces;
• Evidence: something that actually tests the claim you care about;
• Acceptance decision: a judgment, based on evidence and constraints, to accept, modify, defer, or route the change for further review.

For a bounded programming task, AI may help generate candidates, surface alternatives, summarize provided evidence, or propose tests. But its own output should not automatically count as sufficient evidence for the correctness of that same output. If a patch and an AI-generated test share the same mistaken assumption, “two AI outputs” are not necessarily two independent checks.

Your Delegation Boundary must make five things explicit:

1. What bounded outcome do you want?
2. What may AI do, and what context or tool access may it use?
3. What output is still only a candidate?
4. What independent evidence or oracle supports acceptance?
5. Who makes the final decision and owns the consequence?

This lesson does not claim that a particular model is always good or bad, and it does not establish technical mastery or job readiness.

Replace “is AI good at this?” with a bounded decision

Statements such as “AI is good at programming” or “never trust AI” are too broad for engineering decisions. A tool can be useful for generating options while still being insufficient as the final authority. The access it receives can also change the risk of the same task.

Shrink the unit of decision. Instead of “coding,” write something like:

“For this bounded function, AI may propose a skeleton. Expected behavior comes from the accepted specification. The change is accepted only when checks tied to that behavior pass and the responsible reviewer approves it.”

Or:

“For this sensitive change, AI may inspect only the minimum approved context and has no direct write authority. Its output remains a proposal.”

The central issue is not the model's potential capability. It is the authority boundary and the evidence required for this task.

The model: Task → AI Role → Access → Candidate → Evidence → Human Decision

Build the AI Delegation Boundary with six questions.

1. Task: what bounded outcome are you trying to achieve?

Write the result precisely. “Improve this code” is vague. “Remove duplication from this function without changing its observable behavior” is more inspectable.

If the outcome is ambiguous, no test or review can fully rescue the decision because the acceptance target itself is unclear.

2. AI Role: what role may AI play?

The role might be candidate generation, option generation, explanation, summarization of supplied evidence, counterexample search, or test suggestion. Keep role separate from authority. “May propose” and “may decide” are not the same permission.

3. Access: what may the tool see or change?

Three simple modes are useful for reasoning:

• No tool access: AI has no access to the real environment and works only from supplied context.
• Read-only: it may inspect approved information but cannot mutate the environment.
• Write-capable: it may create or apply changes or actions.

As the environment becomes more changeable, context scope, permission boundaries, reversibility, and acceptance evidence need to be more explicit. This does not mean write access is always forbidden; it means access should match the task and its risk.

4. Candidate: what is still only a proposal?

Code, tests, explanations, migration plans, queries, refactors, and reports remain candidates until they are checked against an independent acceptance basis. Clean presentation, confidence, or a large number of generated tests does not automatically change that status.

5. Evidence and oracle: how do you know the candidate is acceptable?

Evidence is something that tests a specific claim: observed behavior, relevant tests, an interface contract, an accepted specification, an invariant, trustworthy logs, compiler/type-checker results, or human review, depending on the claim.

In this lesson, an oracle is a source or rule that tells you what should count as correct for a particular claim. It might be a specification, agreed expected output, an invariant, or another authoritative reference. AI must not become its own oracle merely by repeating or endorsing the result it generated.

Evidence must fit the claim. Passing a unit test does not prove whole-system security; compiling does not prove product behavior is correct.

6. Human Decision: who approves accept, modify, defer, or route?

Human accountability does not mean humans must perform every check manually. Automated tests, static analysis, CI, and review workflows can provide strong evidence. But the acceptance standard, interpretation of evidence, and final acceptance decision must not simply be delegated back to the same generator that produced the candidate.

For higher-risk work, ownership may be shared across roles such as developer, reviewer, or product owner. What matters is that the ownership is explicit and traceable.

Build an inspectable AI Delegation Boundary

For each task, record one row with these fields:

Two rules matter.

Rule one: constrain the tool to the task, rather than expanding the task to match advertised capability. If read-only access is enough for the outcome, write access is not automatically required.

Rule two: define acceptance evidence before becoming attached to the candidate. If you invent the standard only after seeing an attractive result, you risk unconsciously reshaping the standard around the output.

Why “AI wrote the patch and the test” is not automatically independent verification

Suppose the requirement says empty input must return a specific error. AI writes a patch that converts empty input into an empty list. It then generates a test that expects exactly that behavior. The patch and test agree with each other, but they share the same mistaken interpretation of the requirement.

The problem is not that the test was written by AI. The problem is dependency of reasoning. If the candidate and its check originate from the same bad assumption, adding more checks from that assumption does not create independent confidence.

Break the loop by grounding at least part of the oracle or acceptance standard outside the same generator: an accepted specification, pre-defined cases, a clear invariant, reference behavior, or independent review that re-derives the claim.

Human accountability is ownership, not manual execution

Thousands of automated tests may run without a person reading each one. They can still be useful evidence if you understand what claims they test and what they do not test. The human role is to own the claim boundary, evidence threshold, and response to failure.

Context is part of the decision

A model may be technically capable of processing more information than you are justified in providing. Sensitive data, secrets, customer information, or irrelevant repository areas should be included only when required and permitted for the task. Technical capability does not grant permission to use every data source or access level.

Three examples: separate candidate generation from authority

Example 1: code skeleton, behavior from the specification

A small data transformation function must satisfy defined input, output, and two error cases. AI may propose a skeleton, naming options, and candidate tests.

A defensible boundary is:

• AI Role: generate candidates;
• Oracle: the accepted specification and expected behavior;
• Evidence: checks tied to those requirements plus observed behavior;
• Decision Owner: the responsible developer or reviewer;
• Non-claim: passing those checks does not establish overall architecture quality or whole-system security.

Example 2: patch and test repeat the same mistake

AI writes a bug fix and a regression test. Both pass. Later you discover that the test copied the same incorrect assumption as the patch and omitted an edge condition from the requirement.

The conclusion is not “AI-generated tests are worthless.” The conclusion is that evidence independence must match the claim. A generated test may be useful, but acceptance still needs an expected behavior grounded somewhere other than the candidate's own reasoning.

Example 3: sensitive context and a write-capable tool

You are diagnosing a production defect. Real logs contain information that is not all necessary for the diagnosis, while the tool can both read files and apply changes.

A reasonable boundary might:

• minimize context to the necessary, permitted subset;
• begin with read-only analysis if that is enough;
• make any write-capable change explicit, logged, and reversible;
• accept the change only from system evidence and a named decision owner.

The fact that a tool can do more does not mean the task requires or permits more authority.

Four common delegation failures

Failure 1: “the answer sounds convincing, so it is correct”

Tone, length, confidence, and professional-looking code are not evidence.

Repair: name the claim and identify evidence that actually tests it.

Failure 2: “AI wrote tests too, so it verified itself”

The candidate and tests may share the same assumption.

Repair: define an independent oracle and connect at least one important check to a requirement, invariant, or other external evidence.

Failure 3: “more context is always better”

Extra context may be irrelevant, sensitive, or misleading.

Repair: provide only context needed for the task and constrain access to the minimum required for the bounded outcome.

Failure 4: “humans must manually check everything”

That is also a bad interpretation. The goal is not to remove automation.

Repair: use automated evidence, while keeping decision ownership and the claim boundary explicit.

For experienced engineers, compress the boundary around risk and reversibility

For a low-risk, reversible change, a reasonable flow may be: AI proposes the change, pre-existing tests run, the diff is reviewed, and the small change is accepted.

For a higher-risk change, the same model may require narrower access, stronger evidence, an independent reviewer, or an explicit rollback step. The difference is not simply whether AI is “smarter.” It comes from consequence of failure, reversibility, and the evidence threshold needed for the task.

An experienced engineer should be able to record that tradeoff briefly but explicitly rather than relying on intuition or tool confidence alone.

Three tool modes, one invariant principle

Compare the same task in three settings.

No tool access

AI has no tool or real-environment access, but it may still generate a candidate or alternatives from supplied context. Evidence must come from a claim-matched basis such as an accepted specification, an independent check, observed behavior, or human review. Human ownership of the final decision remains explicit.

Read-only tool

AI may inspect approved context, suggest likely causes, or summarize available evidence. Because it cannot mutate the environment, mutation risk is lower, but its recommendation is still not the authority.

Write-capable tool

AI may apply a candidate directly. That can reduce friction, but the acceptance gate must not disappear. Write scope, logging, reversibility, and evidence should remain explicit.

The invariant across all three modes is this: candidate generation, evidence collection, and acceptance are different roles. A tool may assist with more than one, but the authority boundary must remain visible.

Practice: build a Delegation Boundary for one real task

Round 1: bound the outcome

Choose one small real task. “Write the entire parser” is too broad; “support one defined format with three specified error cases” is more useful.

Round 2: choose AI role and access

State whether AI only proposes, has read-only access, or is write-capable. Exclude context that is not needed.

Round 3: define evidence before the candidate

State what must be true for acceptance. A requirement, independent test, invariant, compile/type check, runtime behavior, or review may contribute, as long as you match it to the claim.

Round 4: record the decision and non-claim

After inspecting evidence, choose accept, modify, defer, or route for review. Then state what this acceptance does not prove.

Round 5: transfer across three contexts

Redesign the same task for:

1. no tool access;
2. read-only access;
3. write-capable access.

For each mode, explain what changes and what remains invariant. If the answer is only “AI does more or less work,” the authority boundary is incomplete. You must also account for access, evidence, and ownership.

Five questions before acceptance

1. Is the task and outcome bounded and inspectable?
2. Is AI's role separated from decision authority?
3. Is context and access limited to what the task requires?
4. Does acceptance evidence actually test the claim instead of merely echoing the candidate?
5. Are the decision owner, non-claim, and update condition explicit?

If any answer is unclear, the Delegation Boundary is not yet strong enough to support an acceptance decision.

Main assessment: defend the boundary in a changed context

You will receive a new task for which AI can generate a candidate. You must define:

• the bounded outcome;
• AI's role;
• permitted context and access;
• what counts only as a candidate;
• the independent oracle or evidence;
• the decision owner;
• what triggers accept, modify, defer, or escalation;
• what the decision does not prove.

Then transfer the same task across no-tool, read-only, and write-capable modes. Passing this assessment only shows that you can construct and defend a bounded AI Delegation Boundary. It does not establish broad mastery of security, architecture, or all programming work.

Why this lesson does not maintain a list of current model capabilities

Tool capabilities change, while the core question here is more stable: what is the candidate, what evidence is required, and where does final authority remain? If a future version adds a current claim about a specific model, version, or provider capability, that claim must be checked separately against an official source with a review date. This version intentionally makes no provider-specific claims about current capabilities.