AI Engineering Tutorial · Quality and Production · Lesson 24 of 25
Logging and Tracing LLM Calls
Log what you need to debug and to control cost: model, prompt version, tokens, latency, stop reason, tool calls.
What to record for every call
- Model ID and prompt version, so behaviour changes can be traced to a change.
- Input and output tokens, cost, latency and time to first token.
- stop_reason, errors and retries.
- For agents, every tool call with arguments and a result summary, as spans in one trace.
- Content of prompts and answers only with redaction, sampling and limited retention, because it may contain personal data.
import json, re, time
def redact(text):
return re.sub(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+", "[EMAIL]", text)
def log_call(model, prompt_version, question, answer, usage, started, stop_reason):
record = {"model": model, "prompt_version": prompt_version, "input_tokens": usage[0],
"output_tokens": usage[1], "latency_ms": round((time.perf_counter() - started) * 1000),
"stop_reason": stop_reason, "question": redact(question)[:80]}
print(json.dumps({k: v for k, v in record.items() if k != "latency_ms"}))
start = time.perf_counter()
log_call("claude-sonnet-5-5", "support-v12", "Email me at [email protected] about order 10042",
"Sure...", (812, 64), start, "end_turn")
Output
{"model": "claude-sonnet-5-5", "prompt_version": "support-v12", "input_tokens": 812, "output_tokens": 64, "stop_reason": "end_turn", "question": "Email me at [EMAIL] about order 10042"}OpenTelemetry has semantic conventions for generative AI so traces work with standard tools.
Test yourself
Why log the prompt version and model ID on every call?
The API requires itTo attribute quality, cost or behaviour changes to a specific changeTo reduce latencyTo encrypt logs
Show answer
Answer: To attribute quality, cost or behaviour changes to a specific change
Without versions in logs you can't tell which change caused a difference.