How do you evaluate an agent?

Summary

On outcomes, not exact paths: run tasks in resettable sandbox environments, check the final state in code (record created, ticket closed), check constraints (no forbidden actions, approvals obtained), measure efficiency against budgets, and run each task several times to report success rates.

Longer Answer

Agents reach goals through different valid paths, so matching a golden tool sequence fails good runs and passes runs that look right but produce wrong results. Grade the final message with a rubric where it matters.

Key Points

• Grade final state and constraints.
• Efficiency against budgets.
• Repeated runs for success rates.
• Sandboxed, resettable environments.

Deep dive: Evaluating Agents