Reviewing AI-Written Code

The problem

A teammate opens a pull request: "Generated with AI, tests pass." The tests do pass, because the assistant also wrote them, and they only check the happy path. The function silently drops the last record when the input length is a multiple of the batch size.

Real-world analogy

Proofreading your own essay is hard because you read what you meant to write. Reviewing AI code has a different trap: it looks so tidy that you skim. In both cases, a checklist makes you read what is actually there.

The idea

Review generated code at least as carefully as a colleague's:

1. Run it yourself, including the tests, and try a few inputs of your own.
2. Read every line; if you can't explain a line, ask the assistant or remove it.
3. Edge cases: empty input, one item, exact multiples, very large values, unicode, time zones.
4. Tests: do they test behaviour you care about, or just echo the implementation?
5. Dependencies: does every imported package exist, and is it the one you expect?
6. Security: user input in SQL, shell commands or HTML; secrets; permissions.
7. Fit: naming, structure and error handling match the project.

The code

A batching function with an off-by-one, the AI-written test that misses it, and an edge-case test that catches it:

def batches(items, size):
    out = []
    for start in range(0, len(items) - size, size):        # bug: stops one batch early
        out.append(items[start:start + size])
    if len(items) % size:
        out.append(items[len(items) - len(items) % size:])
    return out

def flat(groups):
    return [x for g in groups for x in g]

# test written with the code: one convenient input
print("happy path:", flat(batches(list(range(7)), 3)) == list(range(7)))
# reviewer's edge cases: exact multiple, smaller than one batch, empty
for n in (6, 2, 0):
    data = list(range(n))
    print(f"n={n}:", "ok" if flat(batches(data, 3)) == data else f"lost items -> {batches(data, 3)}")

happy path: True
n=6: lost items -> [[0, 1, 2]]
n=2: ok
n=0: ok

Common mistakes

• Trusting tests written by the same assistant in the same step as the code.
• Skimming because the code is well formatted and commented.
• Approving imports of packages nobody checked.
• Reviewing only the diff, not how it interacts with surrounding code.

Key points

• Run, read every line, and test edge cases yourself.
• Be sceptical of tests generated together with the code.
• Check dependencies, security and fit with the project.

Further reading: Google engineering practices: what to look for in a code review · OWASP Top 10