Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
Failure & Observability  ›  Capstone Project

Capstone — The Black Box

40 minWrap a whole run so a failure is sorted, logged and recorded: what failed, what caused it, where, and the last things the program said before it did
You're building a piece ofQuarry — the black box
This piece — run_safely(): Runs a whole batch, survives bad rows and a broken source, and after a crash leaves a report: the error, its causes, where it happened and the last records before it.
Scenario Quarry failed at 03:12 with nobody watching; the black box says it was the summary dividing by zero after every row of a malformed export was skipped.
Your task
Build run_safely(lines): attach a BlackBox(BOX_SIZE) handler to the "quarry" logger at DEBUG and call run(lines). On success return {"status": "ok", "result": <run's result>, "error": None, "chain": [], "where": None, "dump": []}. If run raises, return status "failed" for a QuarryError or "crashed" for anything else, with "result": None, the error's class name, its cause chain, the innermost function name, and the black box's last records as "dump". Always remove the handler. Example: run_safely(["id=a;ms=100", "", "id=c;ms=3"]) → status "failed", error "SourceError", where "load_row", dump ending "DEBUG quarry.yard: line 1: ''".

Capstone — The Black Box

In short: an aircraft's flight recorder doesn't prevent crashes — after one, it says what the aircraft was doing in the minutes before. Quarry needs the same thing: a bounded recorder that keeps the last stretch of every log record, a console that only shows a person what matters, and one function, run_safely, that catches whatever escapes the pipeline, sorts it into "failed" (an anticipated QuarryError) or "crashed" (a bug), and leaves behind the error, its chain of causes, where it happened, and the recorder's last few lines — detaching the recorder on every way out, so the next run starts clean.

Every log record reaches two handlers: the console at WARNING, for a person, and the BlackBox at DEBUG, backed by a bounded deque that drops the oldest record as a new one arrives

Keep only the last few lines of a run

A list you trim by hand works; the standard library already has the exact structure. A deque with a maxlen drops the oldest item the moment a new one arrives:

python
What does this print?
from collections import deque

box = deque(["started"], maxlen=2)
box.append("line 0")
box.append("line 1")
box.appendleft("again")
print(list(box))

For the black box that means appending on the right and never thinking about size again — the buffer always holds exactly the most recent stretch.

Give the box everything, the console only what matters

In lesson 2 you wrote ListHandler, which kept every formatted line. The black box is the same idea with a bounded memory — and because it's a handler, the pipeline doesn't know it exists. The console gets WARNING and up; the box gets everything:

python

The console showed one line; the box holds the last three records at every level. Nobody wants to read the DEBUG lines of a successful run — but after a failure, those are exactly the lines that say what was going on. A handler's level can only narrow what reaches it, never widen it:

What does this print?
import logging

log = logging.getLogger("quarry.bb2")
log.handlers.clear()
log.propagate = False
log.setLevel(logging.WARNING)

seen = []
class Keep(logging.Handler):
    def emit(self, record):
        seen.append(record.levelname)

log.addHandler(Keep(logging.DEBUG))
log.debug("line 0")
log.warning("line 0 skipped")
print(seen)

That's why run_safely sets the quarry logger to DEBUG before attaching the box, and relies on each handler's level to decide who hears what.

Sort a crash from an anticipated failure, in the right order

run_safely is the one place allowed to catch everything, because it doesn't swallow anything — it turns the exception into a report. The order of the clauses still matters, the same rule as lesson 1:

broken — fix it

A broken source is an anticipated failure, not a crash — but it's being reported as "crashed". Fix the handler so a QuarryError is "failed" and anything else is "crashed".

In run_safely you can use either shape: two clauses in the right order, or one except Exception as exc: followed by isinstance(exc, QuarryError) — the second keeps the report-building code in one place.

The report also names the function the exception was raised in, found by walking the traceback: exc.__traceback__ starts at the frame that caught it, and each tb_next steps one call deeper to the innermost frame:

step through it
1def summarise(rows):
2 return sum(rows) // len(rows)
3 
4def run(rows):
5 return summarise(rows)
6 
7try:
8 run([])
9except ZeroDivisionError as exc:
10 tb = exc.__traceback__
11 names = []
12 while tb is not None:
13 names.append(tb.tb_frame.f_code.co_name)
14 tb = tb.tb_next
15print(names)

The first name is <module>, where the try ran; the last is summarise, where the division happened — run_safely wants only that last one. The cause chain is the lesson-1 loop: follow __cause__, or __context__ when it isn't suppressed, until nothing is left.

Drawing diagram…

Read a report the way you'd read it at 09:00: status crashed, so it's a bug, not bad data; raised in summarise; the recorder shows both rows were skipped just before. The fix isn't in summarise's arithmetic — it's deciding what a batch with no good rows should report.

Traps worth knowing

Detach in finally. A return inside except still runs finally first, so one cleanup line covers success, failure and crash alike. Copy the buffer out — list(box.records) — because the deque itself belongs to a handler that's about to be detached. Don't let the report crash: everything in it comes from the exception object you already hold, so building it can't raise a second error in the middle of explaining the first. Check the happy path too: a clean run returns "ok" and an empty dump — the box is only opened after an accident.

Quick practice

What does this print?
from collections import deque

log = deque(maxlen=2)
for word in ["open", "read", "parse", "close"]:
    log.append(word)
print(list(log))
What does this print?
class QuarryError(Exception): pass

def outcome(exc):
    try:
        if exc:
            raise exc
    except QuarryError:
        return "failed"
    except Exception:
        return "crashed"
    return "ok"

print(outcome(None), outcome(QuarryError("x")), outcome(KeyError("y")))
broken — fix it

The box should be marked detached whichever way run_once leaves — right now a failing run skips it entirely, because the return inside except exits before that line runs.

🎯 Your turn

Build run_safely(lines). The starter holds Quarry's errors and its pipeline: run(lines) logs through quarry.yard and quarry.vault, skips bad rows with a warning, and lets a SourceError, a bug or a failed summary escape. Finish BlackBox.emit, cause_chain and innermost, then:

  1. Get the quarry logger, set it to DEBUG, and attach a BlackBox(BOX_SIZE) formatted with FORMAT.
  2. Call run(lines). On success return {"status": "ok", "result": <run's result>, "error": None, "chain": [], "where": None, "dump": []}.
  3. If it raises, return "failed" for a QuarryError or "crashed" otherwise, with "result": None, the class name, cause_chain(exc), innermost(exc) and list(box.records) as the dump.
  4. Detach the box on every way out.
run_safely(["id=a;ms=100", "", "id=c;ms=3"])
# → {"status": "failed", "result": None, "error": "SourceError",
#    "chain": ["SourceError"], "where": "load_row",
#    "dump": ["INFO quarry.yard: run started: 3 lines",
#             "DEBUG quarry.yard: line 0: 'id=a;ms=100'",
#             "DEBUG quarry.yard: line 1: ''"]}

Press ▶ Run and hit ✓ Check. When it's green, the black box is done: a crash at 03:12 leaves an answer, not a mystery. ✅

Practice — level up

0/6 solved

Solve each in its own editor — Run to try, Check to grade. Stuck? Reveal a Hint, or the full Solution + walkthrough.

Warm-upA flight recorder's memory

Write ring(items, size): feed items one by one into a buffer that holds at most size things, dropping the oldest when full, and return what's left as a list, oldest first.

loggingdeque
DrillWhat does each failure mean?

The starter defines Quarry's error classes and lookup(name), which returns a Quarry class or a builtin exception class by name. Write policy(name): "skip" for a BadRowError, "retry" for a SourceTimeout, "stop" for any other SourceError, "abort" for any other QuarryError, and "crash" for anything else. Subclasses count.

exception-hierarchy
BuildNothing secret in the box

A black box that records everything will record secrets too. Complete the Redact filter so every record's final message has the value after token= or password= (up to the next & or whitespace) replaced with ***. Then write redacted(events): log each [level, template, args] through the logger "quarry.fetch" at DEBUG, with a ListHandler that carries the filter, remove the handler afterwards, and return the captured lines.

loggingregex
BossWhere was it, and which line?

Using Quarry's pipeline in the starter, write crash_report(lines): call run(lines); if it raises, return {"error": class name, "frames": [function names from run inward], "line": the value of run's local variable i, or None if it has none}. If nothing is raised, return {"error": None, "frames": [], "line": None}.

tracebacksdebuggingexceptions
CapstoneA week of batches, one report

Write supervise(batches): batches maps a batch name to its lines. Attach the starter's Collect handler (it keeps WARNING and above) to the "quarry" logger set to DEBUG. Run every batch with run, in order, each in its own try: a QuarryError makes the batch "failed" and is logged by "quarry.supervisor" as error("batch %s failed: %s", name, exc); any other exception makes it "crashed", logged with exception("batch %s crashed", name); otherwise it is "ok". One bad batch never stops the rest. Remove the handler at the end and return {"results": {name: status}, "log": lines}.

loggingexception-hierarchytracebacks
ExplainRecord everything, show little

The black box records DEBUG and above in memory, but Quarry's console shows only WARNING and above, and the box's contents are only written out when a run fails. Explain why that split makes sense, and what could go wrong if you simply logged everything at DEBUG to the console instead.

run_safely.py
Call the function you wrote — just like your app's frontend would. Edit the inputs and hit Call.
run_safely(lines) → dictRun one batch through Quarry and, if it fails, return a report built from the error and the black box.
try:
returned
Press Run to execute your code.
Press Check to run the tests for this lesson.