Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
Testing & Debugging  ›  Lesson

Your First pytest Suite

pytest 22 minA test is a function that calls your code and raises if the answer is wrong
You're building a piece ofQuarry — the test rig
This piece — check_parse(): Runs a table of cases against the log-line parser and says which passed, failed or errored.
Scenario Someone tidied Quarry's log parser and the vault report lost its WARN lines; the rig's first job is a table of cases that would have caught it.
Your task
Build check_parse(cases). Each case is [line, expected], where expected is the dict parse_line should return or the string "ValueError". Run every case in its own try and return {"passed": count, "failed": [indices], "errors": [indices]} — failed when the result is wrong or nothing was raised, errors when something unexpected was raised. Example: check_parse([["junk", "ValueError"]]) → {"passed": 1, "failed": [], "errors": []}.

Your First pytest Suite

In short: a test is a function that calls your code and raises when the answer is wrong — assert condition does nothing when it's true and raises AssertionError when it's false, and that one rule is the whole idea. On your own machine the tool that collects, runs and reports tests is pytest; it has no pip and no terminal in this browser, so this lesson builds the part that matters — collector, runner, report — in a few lines of plain Python, and everything transfers. pytest.raises checks that code refuses bad input; Python's own unittest and doctest do the same job with a different shape, and both run right here.

Four functions on a shelf; only names starting "test" are picked up, each one run and sorted into a passed or failed bin — testing_helper counts, helper never does

Quarry has grown. It walks a yard of files, pulls fields out of log lines, stores records in a vault and reports on them. Last week someone "tidied" the log-line parser, every run still finished, and three days later the report was quietly missing every WARN line. Nothing crashed, nothing warned — the only way anyone could have known was to check. This section builds Quarry's test rig, and this lesson writes its first tests.

Every lesson in this section tests the same small function. Quarry's log lines look like WARN yard: disk 91% full — a level, a stage, a colon-space, then free text:

python

The second line's message itself contains : , and partition splits only at the first one, so the message survives intact. That is exactly the kind of detail a later "tidy-up" breaks — and exactly what a test pins down.

Catch the regression that slipped through silently

assert is Python's one built-in test primitive:

python

The rule is strict, which is the point:

What happens when this runs?
assert 0.1 + 0.2 == 0.3
print("passed")

Watch out: assert statements are removed when Python runs with -O. That never matters for tests, but it's why you must never use assert to validate input in production — raise a real exception there, the way parse_line does.

On your own machine you'd put tests like this in a file named test_*.py and run pytest. No classes, no special assertion methods, no registration: pytest collects every function starting test, runs each one separately so one failure never stops the rest, and reports — rewriting a failing assert before running it so you see the actual values, not just False:

    def test_keeps_colons_in_message():
>       assert parse_line("INFO vault: a: b")["msg"] == "a:b"
E       AssertionError: assert 'a: b' == 'a:b'

test_parse.py:9: AssertionError
========== 1 failed, 1 passed in 0.02s ==========

pytest does not run in this browser — no pip, no command line — so we build the same three jobs by hand and grade the tests you write by calling them. A module is, underneath, a dictionary from names to objects; globals() hands you that dictionary, so collection is a filter over it:

python

testing_helper was collected — pytest's default rule is the prefix test, not test_. And the report has three buckets, not two: a failure means the code ran and gave the wrong answer; an error means it never got that far, because something else raised first. The distinction tells you where to look.

step through it
1def test_ok():
2 assert 2 * 3 == 6
3 
4def test_bad():
5 assert "a,b".split(",") == ["a", "b", ""]
6 
7results = {}
8for name, fn in [("test_ok", test_ok), ("test_bad", test_bad)]:
9 try:
10 fn()
11 results[name] = "passed"
12 except AssertionError:
13 results[name] = "failed"
14print(results)
What does this print?
def test_lookup():
    levels = {"WARN": 2}
    assert levels["ERROR"] == 3

try:
    test_lookup()
    print("passed")
except AssertionError:
    print("failed")
except Exception as e:
    print("error:", type(e).__name__)

Prove a bad line really gets refused

Half of parse_line's job is saying no. pytest checks that with a context manager — the with block passes only if the code inside raises that exception (or a subclass), and fails if nothing is raised:

import pytest

def test_rejects_line_without_separator():
    with pytest.raises(ValueError):
        parse_line("no separator here")

That last sentence — the test fails if nothing is raised — is easy to lose when you write the check by hand:

broken — fix it

This parse_line has a regression — it accepts junk. The test exists to catch exactly that, so it should report fail. It reports pass. Fix the TEST (not parse_line) so it cannot pass when nothing is raised.

A test that cannot fail is worse than no test — it turns green, and green is a promise. Every time you write one, make it fail once on purpose: break the code, or the expectation, and watch it go red.

Python's own unittest does the identical collect/run/report, just worded differently — a method on a TestCase, self.assertEqual instead of assert — and it does run here:

python

pytest runs unittest classes unchanged, so nothing written this way is wasted. For small pure functions, doctest turns the docstring's own example into the test, so documentation can't drift away from the code it describes:

python

The next lesson, 05a, goes deeper into unittest and doctest, plus tox — the tool that runs a whole suite across several Python versions.

Habits that make a suite worth having

One behaviour per test, named for it. Arrange, act, assert — build the input, call the code once, check the result; a test needing three calls and five asserts is probably three tests. Check the value a function returns, never a printout. A bug becomes a test: the first thing you write when Quarry misbehaves is the failing test that reproduces it, which then stays forever — and if each test has some chance p of catching a particular bug, n independent tests let it slip through with chance (1 − p)ⁿ. Ten vague tests (p = 0.1) still miss a bug 35% of the time; a handful of sharp, specific tests beats a pile of loose ones.

Quick practice

What does this print?
def test_div():
    assert 1 / 0 == 0

try:
    test_div()
    print("passed")
except AssertionError:
    print("failed")
except Exception as e:
    print("error:", type(e).__name__)
What does this print?
class Suite:
    def test_inside_class(self):
        assert True

def test_top_level():
    assert True

def run_all(namespace):
    report = {"passed": [], "failed": [], "errors": []}
    for name, obj in list(namespace.items()):
        if not (name.startswith("test") and callable(obj)):
            continue
        try:
            obj()
            report["passed"].append(name)
        except AssertionError:
            report["failed"].append(name)
        except Exception:
            report["errors"].append(name)
    return report

print(run_all(globals())["passed"])
What does this print?
def test_always_true():
    result = parse_line("junk")
    assert True

LEVELS = ("DEBUG", "INFO", "WARN", "ERROR")

def parse_line(line):
    head, sep, msg = line.partition(": ")
    if not sep:
        raise ValueError("no separator")
    return {"level": head, "stage": "", "msg": msg}

try:
    test_always_true()
    print("passed")
except Exception as e:
    print("error:", type(e).__name__)

🎯 Your turn

Build check_parse(cases) — the first piece of the test rig. Each case is a pair [line, expected], where expected is either the dict parse_line should return or the string "ValueError" when the line must be refused. Run every case and return a report:

  • passed — how many cases behaved as expected
  • failed — indices of cases that returned the wrong thing, or returned when they should have raised
  • errors — indices of cases that raised something nobody expected
check_parse([["WARN yard: disk full", {"level": "WARN", "stage": "yard", "msg": "disk full"}],
             ["junk", "ValueError"],
             ["INFO vault: ok", {"level": "INFO", "stage": "yard", "msg": "ok"}]])
# → {"passed": 2, "failed": [2], "errors": []}

parse_line is already in the starter. Run each case in its own try, so one bad case can never stop the rest — that is the first thing every test runner promises.

Practice — level up

0/6 solved

Solve each in its own editor — Run to try, Check to grade. Stuck? Reveal a Hint, or the full Solution + walkthrough.

Warm-upWhich files get collected?

pytest imports files named test_*.py or *_test.py. Write is_test_file(filename) returning True for those and False for everything else.

pyteststrings
DrillThe summary line

Given a list of outcomes ("passed", "failed", "error"), write tally(outcomes) returning the summary line a runner prints last: counts in the order failed, passed, errors, zero counts left out, joined by ", ". Use error for one and errors for more. An empty list gives "no tests ran". Example: ["passed", "failed", "passed"] → "1 failed, 2 passed".

pytestdictionariesformatting
BuildSay what differs

pytest's report shows which part of a dict was wrong. Write diff_keys(got, expected) returning a sorted list of strings, one per differing key: "k: missing" if only expected has it, "k: unexpected" if only got has it, and "k: <got repr> != <expected repr>" if both have it with different values. Equal dicts give [].

pytestdictionariessorted
BossA hand-made pytest.raises

Write expect_raises(fn_name, arg, exc_name). Look up the function in FUNCS and the exception class by name in builtins, call FUNCS[fn_name](arg), and return "pass" if it raised that exception or a subclass of it, "fail: did not raise" if it returned, or "fail: raised <Name>" if it raised something else.

pytestexceptionsfirst-class-functions
CapstoneCollect and run a whole module

Write run_module(source). Execute the source text into a fresh namespace dict, collect every function whose name starts with test in the order it was defined, call each with no arguments, and return {"passed": [...], "failed": [...], "errors": [...]} of names. An AssertionError is a failure; any other exception is an error. Non-functions (like a list called test_data) are not collected.

pytestexceptionsdictionariesfunctions
ExplainWhy a test that cannot fail is worse than none

A teammate adds a test for Quarry's parser that wraps the call in try: ... except ValueError: pass and it goes green. Explain why a test like that can be worse than having no test at all, and describe the habit that would have caught it.

check_parse.py
Call the function you wrote — just like your app's frontend would. Edit the inputs and hit Call.
check_parse(cases) → dictRun a table of parse_line cases and report passed, failed and errored.
try:
returned
Press Run to execute your code.
Press Check to run the tests for this lesson.