Properties, Coverage & Honest Green
In short: an example test says this input gives this output; a property says for any input, this relationship holds — a round trip, an invariant, idempotence — and you let generated examples hunt for a counterexample instead of writing them down by hand. A found failure is usually noisy; shrinking repeatedly deletes or simplifies it while it keeps failing, until what's left is the essence of the bug. Coverage shows which lines ran, which is not the same as which inputs were tried — 100% coverage with a crash on an untested input is normal, not a contradiction. And a green suite can lie in several specific, recognisable ways.
Quarry's vault stores each record as one line — level, stage and message
joined by commas — and reads it back when the report runs. It had tests.
They wrote WARN,yard,disk full and read it back, and they passed. Then a
real message arrived — disk 91% full, retrying — and the vault split it
into four fields instead of three. Every test was green. None of them had
ever tried a comma.
State what must hold for every message, not just the ones you thought of
A few property shapes cover most of what you'll write: round trip
(decode(encode(r)) == r), invariant (something about the output is
always true — a sort's result is in order and has the same items as the
input), idempotence (normalise(normalise(s)) == normalise(s)), and
oracle (a fast, clever version agrees with a slow, obviously-correct
one). Properties don't name the answer, which is why they can be checked on
inputs nobody wrote down:
Neither property alone is enough — "ordered" is satisfied by always
returning []; "same items" is satisfied by returning the input untouched.
Together they pin sorting down completely. Before generating anything, see
what one awkward message does to the vault's encoder:
record = ["WARN", "yard", "disk 91% full, retrying"]
text = ",".join(record)
print(len(text.split(",")))You found that because you already knew to try a comma. A property test finds it without knowing — generate many random records from a seeded source (so a failure replays exactly) and stop at the first one that breaks the round trip:
Change the seed and the counterexample changes, but one always turns up within a handful of tries — the alphabet is deliberately small and nasty (spaces, colons, commas), because what you generate decides what you can find. That cuts both ways: a generator that can't produce the dangerous input makes every property pass:
dedupe_sorted is wrong for any list with a repeated number, but the property check passes all 200 examples. The bug is in the generator, not the check. Fix it so the check can see the bug.
On your own machine the standard tool is Hypothesis
(pip install hypothesis): you describe the shape of inputs with
strategies and it generates, runs as an ordinary pytest test, deliberately
favours edge cases, remembers failing examples between runs, and
shrinks every failure before showing it to you.
from hypothesis import given, strategies as st
records = st.fixed_dictionaries({
"level": st.sampled_from(["DEBUG", "INFO", "WARN", "ERROR"]),
"stage": st.text(min_size=1),
"msg": st.text(),
})
@given(records)
def test_vault_round_trip(r):
assert decode(encode(r)) == rHypothesis doesn't run in this browser, so we build the shrinking step ourselves — it's the part that turns a baffling failure into an obvious one.
Shrink a 15-character failure down to one comma
A random counterexample is usually noisy:
{"msg": "ab: ,b a:ba ,,a"}. Which character matters? Keep deleting one
character at a time; keep the deletion only if the result still fails.
When no smaller version fails, what's left is the essence of the bug.
Fifteen characters became one — the comma is the bug. Step through a shorter case to watch each deletion either stick or get rejected:
If a bug fires on a fraction q of all possible inputs, n random examples
miss it with probability (1 − q)ⁿ. A 1-in-100 bug needs about 300 examples
to be found 95% of the time — Hypothesis runs 100 by default, which is why
good generators bias towards edge cases instead of relying on luck.
Measure which lines ran — and see why that isn't enough
Coverage measures which lines executed while the tests ran. On a real
project it's pytest --cov. The mechanism is a trace hook — Python calls a
function you register on every line executed — and the same hook works
here:
Line 5 — return "fail" — never ran. That's what coverage finds: code no
test has ever executed. The reverse is not true — every line running
doesn't mean every line is right:
def pct(part, whole):
return round(100 * part / whole)
def test_pct():
assert pct(1, 4) == 25
test_pct() # passes, and covers 100% of pct's lines
print(pct(1, 0))Catch a suite that can't tell right from wrong
A test suite is only worth its ability to turn red when the code is wrong. Mutation testing makes a small deliberate change — a mutant — and sees whether any test notices; a mutant that survives marks a behaviour nobody checks:
The first suite looks thorough and lets two mutants live, because it never
tests the boundary. A suite also goes green dishonestly in other ways worth
recognising on sight: no assertion (the test calls the code and checks
nothing), swallowed failure (a broad try/except around the check — the
trap from lesson 1), always-true assert
(assert (got == want, "msg") asserts a non-empty tuple — the message goes
after the comma, no brackets), testing the double (the mock returns 3,
the test asserts 3), and a generator that can't reach the bug, as
above. Each one is caught by the same habit: make the test fail once on
purpose.
Quick practice
mutants = {
"original": lambda n: n % 2 == 0,
"odd check": lambda n: n % 2 == 1,
"always True": lambda n: True,
}
def survivors(suite):
return [name for name in mutants if name != "original"
and all(mutants[name](n) == want for n, want in suite)]
print(survivors([(4, True)]))def is_sorted(xs):
return all(a <= b for a, b in zip(xs, xs[1:]))
def test_is_sorted():
assert is_sorted([1, 2, 3])
test_is_sorted()
print(is_sorted([]), is_sorted([5]))🎯 Your turn
Build round_trip(record) — the property that guards Quarry's vault. Write
encode(record), which joins level, stage and msg with commas but
first escapes each field (a backslash becomes \\, a comma becomes
\,), and decode(text), which walks the text character by character: a
backslash means "take the next character literally", an unescaped comma
ends a field. round_trip returns decode(encode(record)) — and it must
equal record for every record:
round_trip({"level": "WARN", "stage": "yard", "msg": "disk full, retrying"})
# → {"level": "WARN", "stage": "yard", "msg": "disk full, retrying"}The starter has the naive version that breaks on a comma. Escape the backslash before the comma, or the backslashes you add will be escaped a second time.