What is package hallucination and why is it a security risk?

A model suggesting a package name that does not exist; attackers can register that name with malicious code, so blindly installing suggestions can compromise your system.

How do you make a SQL query built from user input safe?

Use parameterised queries (placeholders with separate values) instead of string formatting, so input is never interpreted as SQL.

Name three insecure defaults to watch for in generated code.

Disabled TLS certificate verification, overly permissive CORS, debug mode enabled in production (also weak password hashing and hard-coded secrets).

Security Pitfalls in AI-Generated Code — Coding with AI Assistants

The problem

An assistant writes a search endpoint using an f-string to build SQL, and suggests installing a helper package with a plausible name. The SQL is injectable. The package name did not exist when the code was written, and someone has since registered it with malware inside.

Real-world analogy

A recipe from the internet may be delicious and still call for an ingredient that is unsafe for you. You check the ingredients list before cooking, especially the ones you have never heard of.

The idea

Generated code inherits common patterns, including insecure ones, and models sometimes invent package names ("package hallucination"); attackers register such names (sometimes called slopsquatting). Watch for:

Injection: user input in SQL strings, shell commands, file paths, HTML.
Unknown dependencies: verify every new package exists, is the intended one, and is maintained.
Weak defaults: disabled TLS verification, permissive CORS, debug mode, weak hashing for passwords.
Secrets in code: hard-coded keys copied from examples.
Missing authorisation checks on new endpoints.

The code

Two quick checks: string-built SQL versus a parameterised query, and new dependencies against an approved list:

Injection with string SQL, safety with parameters, and a dependency checkTry it yourself

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE users (name TEXT, is_admin INTEGER)")
db.executemany("INSERT INTO users VALUES (?, ?)", [("ana", 0), ("root", 1)])

user_input = "x' OR '1'='1"
unsafe = db.execute(f"SELECT name FROM users WHERE name = '{user_input}'").fetchall()
safe = db.execute("SELECT name FROM users WHERE name = ?", (user_input,)).fetchall()
print("string-built SQL returns:", unsafe)
print("parameterised SQL returns:", safe)

approved = {"requests", "pydantic", "sqlalchemy", "httpx"}
suggested = ["httpx", "fastjson-utils-pro", "pydantic"]
print("needs review before install:", [p for p in suggested if p not in approved])

Output

string-built SQL returns: [('ana',), ('root',)]
parameterised SQL returns: []
needs review before install: ['fastjson-utils-pro']

Common mistakes

Installing packages suggested by an assistant without checking they exist and are reputable.
Accepting string-built SQL or shell commands because the example input was harmless.
Copying example code with placeholder secrets or disabled security settings.
Skipping security review because "the AI wrote it".

Key points

Check generated code for injection, weak defaults, secrets and missing authorisation.
Verify every new dependency before installing it.
Use parameterised queries and safe APIs by default, and run security scanners in CI.

Further reading: We Have a Package for You! Package hallucinations by code LLMs · OWASP SQL injection prevention · OWASP Top 10