Capstone — The Test Rig
In short: every feature here is something you've already met — the
capstone is wiring them into one loop. Collect with an explicit
registry, not a guess over globals(). Expand a parametrised test into
one job per row, built immediately rather than deferred into a closure.
Inject fixtures fresh, after the row's own values. Run in isolation
so one crash never stops the rest. Classify into passed, failed, error
or skipped — with the specific exceptions caught before the general ones.
Report, grouped by outcome, with an exit code CI can read. That is what
pytest is on your machine; here you build a small, honest version of it,
then point it at Quarry's real suite and read what it finds.
Five lessons, five pieces: a table of parser cases, a fixture factory, a fake source, a round-trip property, a failure narrower. Each works on its own. What Quarry doesn't have yet is the thing that holds them together — one command that finds every test, gives each what it needs, runs it in isolation, and tells you in a few lines whether Quarry still works.
Register tests explicitly instead of scanning and guessing
Lesson 1 collected by scanning globals() for names starting test — that
works, but it guesses. A registry is explicit: a decorator adds each
test to a list the moment it's defined, and the rig runs exactly that list,
in definition order.
Both decorators hand back the same function they were given — they only record it. That matters when decorators stack:
REGISTRY = []
def test(fn):
REGISTRY.append(fn)
return fn
def parametrize(*cases):
def attach(fn):
fn.cases = cases
return fn
return attach
@parametrize(("WARN",), ("INFO",))
@test
def test_level(level):
pass
print(len(getattr(REGISTRY[0], "cases", ())))Classify an outcome — and get the except order right
pytest's pytest.skip("reason") and unittest's SkipTest are both an
exception the runner recognises, which puts a trap in the classifier:
import unittest
def outcome(fn):
try:
fn()
except AssertionError:
return "failed"
except Exception:
return "error"
except unittest.SkipTest:
return "skipped"
return "passed"
def test_later():
raise unittest.SkipTest("the vault encoder isn't merged yet")
print(outcome(test_later))Put the specific classes first — SkipTest, then AssertionError, then
Exception — and each lands in its own bucket.
Expand a parametrised test without the late-binding trap
A parametrised test becomes several runnable jobs, one per row. The obvious way to build them has a famous bug:
Each job should check its own row, giving [True, True, False]. Instead every job checks the last row. Fix how the jobs are built.
A function body looks up outside names when it runs. All three lambdas
share one case variable, and by the time they run the loop is over. A
default argument is evaluated when the function is defined, so
case=case freezes each row into its own job. The rig avoids the problem
entirely by running each row immediately as it loops, rather than
collecting jobs for later.
Here's the heart of the rig for one test ID: split the parameters between row and fixtures, build fresh fixture values, run, classify.
zip(names, row) fills the first parameters from the row; every name left
over must be a fixture. If one isn't registered, looking it up raises
KeyError — before the test runs — and that is an error, not a
failure: the test never got to check anything.
Report flakiness honestly, and let the exit code be the verdict
A flaky test passes and fails on the same code — usually timing,
ordering or the network. The tempting fix is to rerun failures until they
pass — look what that does: with r tries, a test that fails half the time
shows green with chance 1 − 0.5ʳ, 94% at four tries. The suite looks
healthy while the bug behind the flakiness ships. Rerun to detect
flakiness, then report it; never rerun to hide it — a test that both
passed and failed across runs of the same code is flagged as flaky, and
that list is worth more than the green tick.
Your numbers will differ run to run — timing is the one measurement in a test report that's never exactly repeatable, which is why the rig reports it but never asserts on it. When pytest finishes it exits with a number CI reads: 0 if everything collected passed (skips are fine), 1 if anything failed or errored, 5 if nothing was collected at all — a suite that runs nothing proves nothing.
Quick practice
def exit_code(outcomes):
if not outcomes:
return 5
if any(o in ("failed", "error") for o in outcomes):
return 1
return 0
runs = [["passed"], ["passed", "error"], []]
print([exit_code(r) for r in runs])seen = {}
for run in [{"a": "passed", "b": "failed"}, {"a": "failed", "b": "failed"}]:
for test_id, outcome in run.items():
seen.setdefault(test_id, set()).add(outcome)
flaky = sorted(t for t, outs in seen.items() if "passed" in outs and outs & {"failed", "error"})
print(flaky)🎯 Your turn
The starter holds three parts. Quarry's code under test: parse_line
and error_rate. Quarry's suite: ten tests using @test, @fixture
and @parametrize, plus a skip(reason) helper. And the rig — which is
yours.
Make the decorators real (test registers, fixture registers by name,
parametrize attaches its rows), then write suite_report(keyword):
- Expand every registered test into IDs: a plain test's ID is its name; a
parametrised one gives
name[v1-v2]per row. - Keep only IDs containing
keyword(an empty keyword keeps everything). - Run each: fill parameters from the row, the rest from fresh fixtures. A
missing or crashing fixture is an error;
SkipTestis skipped;AssertionErroris failed; any other exception is an error. - Return
{"passed": [...], "failed": [...], "errors": [...], "skipped": [...]}, IDs in registration order.
suite_report("rate")
# → {"passed": ["test_rate_of_batch", "test_rate_counts_only_errors"],
# "failed": [], "errors": ["test_rate_of_empty_batch"], "skipped": []}Then read the full report with suite_report(""). Two of Quarry's own bugs
are in it — an empty batch that crashes and a newline the parser keeps —
and a test that asks for a fixture nobody wrote. Those are real findings.
That is what the rig is for.