Human Approval for Risky Actions

Scenario

Example company: an internal ops agent helps engineers clean up cloud resources. Its tools include list_resources, stop_instance and delete_volume. All tool calls run automatically.

What you see in production

An engineer asks the agent to "clean up the old test volumes". The agent lists volumes, decides that anything with "test" in its name and no recent writes is old, and deletes eleven volumes, two of which held a staging database's backups. The agent did exactly what it understood; nobody saw the plan before it ran.

The naive code

def run_tool(block):
    return TOOLS[block.name](**block.input)          # every tool, every argument, no questions asked

Reproduce it

A policy gate that classifies each call and holds risky ones for approval:

RISK = {"list_resources": "read", "stop_instance": "reversible", "delete_volume": "destructive"}

def gate(call, approver):
    level = RISK.get(call["tool"], "destructive")          # unknown tools are treated as dangerous
    if level == "read":
        return "run"
    if level == "reversible" and call.get("count", 1) <= 3:
        return "run"
    summary = f"{call['tool']} on {call.get('count', 1)} item(s): {', '.join(call['targets'][:3])}..."
    return "run" if approver(summary) else "blocked: approval denied"

def cautious_engineer(summary):
    print("  approval requested ->", summary)
    return "backup" not in summary                         # refuses when backups are in the list

calls = [{"tool": "list_resources", "targets": ["vol-*"]},
         {"tool": "stop_instance", "targets": ["i-test-1"], "count": 1},
         {"tool": "delete_volume", "targets": ["vol-test-backup-db", "vol-test-old-2", "vol-test-old-3"], "count": 11}]
for call in calls:
    print(call["tool"], "->", gate(call, cautious_engineer))

list_resources -> run
stop_instance -> run
  approval requested -> delete_volume on 11 item(s): vol-test-backup-db, vol-test-old-2, vol-test-old-3...
delete_volume -> blocked: approval denied

Root cause

The agent had the authority of its tools with none of the checks a human operator would apply: no plan review, no confirmation for irreversible actions, no limit on blast radius. Ambiguous instructions ("old test volumes") are normal; without a checkpoint, the agent's interpretation goes straight to production.

The fix

• Classify every tool by risk: read-only, reversible, destructive or external (sends money, email, messages). Unknown tools default to the highest risk.
• Gate in code, not in the prompt. The host enforces approval; the model cannot talk its way past it.
• Show the full action to the approver: tool, exact arguments, number of items affected.
• Plan first, then act. For multi-step changes, ask the agent for a plan (list of actions) and approve the plan; execute only what was approved.
• Offer dry runs (dry_run=True) and limit blast radius (at most N items per call).
• Prefer reversible operations: snapshot before delete, soft delete with a retention period.

Trade-offs

• Too many approval prompts train people to click "approve" without reading; auto-approve read-only and small reversible actions.
• Approval adds latency; for background agents, queue actions for review instead of blocking a conversation.
• Approval does not fix bad tools: a destructive tool with a vague scope is still dangerous even when approved.

Detect it in production

• Log of every risky call with the approver, the arguments and the outcome.
• Approval rate and time-to-approve; a near-100 % approval rate with seconds of review suggests rubber-stamping.
• Alerts on destructive actions above a size threshold, even when approved.

Key points

• Classify tools by risk and gate destructive or external actions in code.
• Show approvers the exact action and blast radius; approve plans before execution.
• Prefer dry runs, limits and reversible operations.

Further reading: Building effective agents · OWASP LLM: Excessive Agency