Testing in Practice · Testing APIs, Contracts and User Journeys
Finding and Fixing Flaky Tests
Diagnose tests that pass and fail without code changes, caused by time, ordering, shared state and randomness, and fix them for good.
What you'll build
Three flaky tests and their fixes: one depends on the current time, one on test order through shared global state, and one on unseeded randomness. Running the suite in both orders exposes the order dependence.
The idea
- A flaky test passes and fails on the same code. Flaky suites train teams to ignore red builds.
- Common causes: time (
now(), time zones, end of month), order and shared state (globals, databases, caches), randomness, concurrency and waiting, and external services. - Fix the cause: inject clocks and random generators, isolate state per test, wait for conditions instead of sleeping, fake external services.
- Quarantine a flaky test only with a ticket and a deadline; never just add retries.
The code
# file: promo.py
from datetime import datetime, timezone
def is_happy_hour(now=None):
now = now or datetime.now(timezone.utc)
return 17 <= now.hour < 19
_cache = {}
def cached_price(sku, lookup):
if sku not in _cache:
_cache[sku] = lookup(sku)
return _cache[sku]
def clear_cache():
_cache.clear()
# file: test_flaky.py
from datetime import datetime, timezone
import pytest
import promo
# Flaky: depends on the wall clock (passes only between 17:00 and 19:00 UTC)
def test_happy_hour_flaky():
assert promo.is_happy_hour() in (True, False) # rewritten below with an injected clock
# Fixed: inject the time
@pytest.mark.parametrize(("hour", "expected"), [(16, False), (17, True), (18, True), (19, False)])
def test_happy_hour(hour, expected):
assert promo.is_happy_hour(datetime(2026, 1, 15, hour, tzinfo=timezone.utc)) is expected
# Flaky: these two depend on run order through the module-level cache
def test_price_from_old_catalogue():
assert promo.cached_price("mug", lambda sku: 8.5) == 8.5
def test_price_from_new_catalogue():
assert promo.cached_price("mug", lambda sku: 9.0) == 9.0
# file: main.py
import subprocess
import sys
def run(label, *tests): # a fresh process per run, like separate CI jobs
print(label)
out = subprocess.run([sys.executable, "-m", "pytest", "-q", "--tb=no", "-rf", "-p", "no:cacheprovider", *tests],
capture_output=True, text=True).stdout
print("\n".join(line for line in out.splitlines() if line.startswith(("FAILED", "1 ", "2 ", "7 ", "6 "))))
run("normal order:", "test_flaky.py")
run("reverse order:", "test_flaky.py::test_price_from_new_catalogue", "test_flaky.py::test_price_from_old_catalogue")
Output
normal order: FAILED test_flaky.py::test_price_from_new_catalogue - AssertionError: assert ... 1 failed, 6 passed in <time> reverse order: FAILED test_flaky.py::test_price_from_old_catalogue - AssertionError: assert ... 1 failed, 1 passed in <time>
Run this on your computer — see “Get Started” for setup.
The two catalogue tests each pass alone, but whichever runs second fails: they share the module-level cache. (The clock-dependent test is written to always pass here; its real version would fail outside happy hour.) The fix is an autouse fixture that isolates state:
# file: promo.py
_cache = {}
def cached_price(sku, lookup):
if sku not in _cache:
_cache[sku] = lookup(sku)
return _cache[sku]
def clear_cache():
_cache.clear()
# file: conftest.py
import pytest
import promo
@pytest.fixture(autouse=True)
def fresh_cache():
promo.clear_cache()
yield
promo.clear_cache()
# file: test_prices.py
import promo
def test_price_from_old_catalogue():
assert promo.cached_price("mug", lambda sku: 8.5) == 8.5
def test_price_from_new_catalogue():
assert promo.cached_price("mug", lambda sku: 9.0) == 9.0
# file: main.py
import subprocess
import sys
for order in (["test_prices.py"], ["test_prices.py::test_price_from_new_catalogue", "test_prices.py::test_price_from_old_catalogue"]):
out = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", *order], capture_output=True, text=True).stdout
print(out.strip().splitlines()[-1])
Output
2 passed in <time> 2 passed in <time>
Run this on your computer — see “Get Started” for setup.
Common mistakes
- Adding automatic retries to hide flakiness instead of fixing the cause.
- Tests that read the real clock, environment variables or global caches without controlling them.
- Unseeded random data in tests (
random.choice); seed it or use Hypothesis, which reproduces failures. - Fixed sleeps when waiting for asynchronous work.
Key points
- Flaky tests come from time, order, shared state, randomness and waiting.
- Inject clocks and randomness; isolate state per test.
- Run tests in random or reverse order in CI to expose dependencies (e.g. pytest-randomly).
Further reading: pytest: flaky tests · Google Testing Blog: flaky tests
Practice questions
What are the most common causes of flaky tests?
Answer
Dependence on time, test order and shared state, randomness, timing of asynchronous work, and external services.
How do you make time-dependent code testable?
Answer
Inject the current time (a parameter or clock object) instead of calling now() inside the logic, and pass fixed times in tests.
Why is retrying flaky tests a poor fix?
Answer
It hides real bugs (including race conditions in the product) and slows CI; the root cause stays and failures become normal.
Open in the app to mark lessons complete, take notes and track your progress.