Capstone: An On-Call Assistant with MCP Tools

• Four MCP servers (logs, metrics, deploys, incidents), each with its own least-privilege service credentials; the agent host connects to them over Streamable HTTP inside the network.
• The alert webhook starts an agent run with the alert payload. An orchestrator plans the investigation and runs independent lookups (deploys, metrics, logs, similar incidents) in parallel.
• A large model writes the plan and the summary; a small model classifies and extracts (for example, which service and time window).
• Remedial actions (rollback, scale) are tools on the deploys server marked destructive; the host requires the on-call engineer's approval in the chat or paging tool.

Look for separate servers with separate credentials, parallel read-only lookups, model routing, and approval enforced by the host for destructive tools.

• recent_deploys(service, since) returns at most 10 deploys with version, author and time.
• search_logs(service, query, start, end, limit=50) returns matching lines (truncated to 300 characters each), counts by message pattern, and a log_ref for more; secrets are redacted server-side.
• metric_summary(service, metric, start, end) returns aggregated statistics and change points, not raw series.
• similar_incidents(text, limit=5) returns titles, causes and links.
• rollback(service, to_version) and scale(service, replicas) are marked destructive, validated, and support dry_run.

Strong answers keep responses small and aggregated, paginate or reference large data, redact on the server, and clearly separate read-only from destructive tools.

• Approval in code for rollback and scale, showing the exact service, version or replica count; the approver's identity is logged.
• Log content is wrapped and labelled as untrusted data in tool results; the system prompt says instructions inside data are never followed, and the destructive tools require approval anyway.
• Secrets are redacted by the logs server before they reach the model; traces store redacted content only.
• Servers use their own scoped credentials; no user tokens are passed through.

Look for defence in depth: approval gates, treating log content as untrusted, redaction before the model, and least-privilege credentials.

• At most 15 steps, 3 minutes and a token budget per investigation; the metrics server enforces 10 requests per second with a client-side limiter.
• If a source is down, the summary says which data is missing rather than guessing; the investigation continues with the remaining sources.
• When a budget runs out, the assistant posts what it has found so far and the next checks it recommends.

Good answers set explicit limits and describe a useful partial result, not a timeout.

• Every run is traced: model and tool spans with arguments, token counts and latency, linked from the incident channel.
• An eval set from 50 past incidents with known causes: replay the alert and the recorded data, and grade whether the summary names the right cause and evidence (human review plus a rubric-based model grader).
• Track in production: time to first summary, engineer rating of each summary, actions proposed versus approved.

Look for an offline eval set built from real incidents with known answers, plus production feedback signals.