When do you use code graders, model graders or human graders?

Summary

Code graders for anything checkable (valid JSON, labels, tool calls, numbers, required or forbidden phrases), model graders with a specific rubric for qualities code can't check (correctness against a reference, tone, completeness), and humans to calibrate model graders and judge the hardest cases.

Longer Answer

Measure a model grader's agreement with human labels (accuracy, Cohen's kappa) before trusting it, and re-check periodically. One criterion per grader prompt with a fixed output format.

Key Points

• Code graders first.
• Rubric-based model graders.
• Humans calibrate.
• Measure grader agreement.

Deep dive: Code Graders, Model Graders and Humans