Why can AI-generated code look correct but still be wrong?

The model predicts plausible code from patterns; a subtle bug (such as forgetting leap years) reads just as fluently as correct code, and the model has not run it against your cases.

What does an assistant know about your project?

Only what is in its context (your request, files it can read, earlier messages, project instruction files) plus general knowledge from training, which may be outdated.

Which tasks are assistants typically strongest at?

Boilerplate, tests, data conversions, explaining unfamiliar code and first drafts, all of which you then review and run.

How AI Coding Assistants Work, and Why They're Confidently Wrong — Coding with AI Assistants

The problem

You ask an assistant for a function that returns the last day of a month. It answers instantly with clean, well-commented code. It works for the three months you try. A week later, a report breaks on 29 February.

Real-world analogy

An assistant is like a very well-read colleague who has seen millions of code examples but has never run your program. They can write something that looks exactly like what usually works. Whether it works here, with your data and your edge cases, is still your job to check.

The idea

A language model writes code by predicting, token by token, what is likely to come next given everything in its context: your request, open files, earlier messages. That has consequences:

Fluent is not the same as correct. Plausible code with a subtle bug reads just as confidently as correct code.
It only knows what is in its context (plus general knowledge from training). It does not know your conventions, your library versions or last week's decision unless you tell it or it can read them.
Knowledge can be out of date or mixed up: APIs from older versions, or functions that don't exist.
It is fast at the boring parts: boilerplate, tests, conversions, explaining unfamiliar code.

The code

The "last day of the month" function, written the plausible way, and checked:

A plausible function, a hidden edge case, and the test that finds itTry it yourself

import calendar
from datetime import date

def last_day_plausible(year, month):
    days = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]      # looks right, forgets leap years
    return date(year, month, days[month - 1])

def last_day(year, month):
    return date(year, month, calendar.monthrange(year, month)[1])

for year, month in [(2026, 1), (2026, 2), (2028, 2), (2026, 12)]:
    a, b = last_day_plausible(year, month), last_day(year, month)
    print(year, month, a, b, "" if a == b else "<- bug")

Output

2026 1 2026-01-31 2026-01-31 
2026 2 2026-02-28 2026-02-28 
2028 2 2028-02-28 2028-02-29 <- bug
2026 12 2026-12-31 2026-12-31

Common mistakes

Accepting code because it looks right, without running it or reading it closely.
Assuming the assistant knows your codebase, library versions or business rules.
Asking for large chunks at once, which makes subtle errors harder to spot.
Not testing edge cases (empty input, leap years, time zones, very large values).

Key points

Assistants predict plausible code; plausibility is not correctness.
They know what is in their context, so give them the context that matters.
Run and test what they write, especially edge cases.

Further reading: Claude Code overview · GitHub Copilot documentation · Python calendar.monthrange