Capstone — The Extractor

In short: parse_log() turns a line from either of Quarry's two log formats into one typed record. The fixed part of each line — timestamp, level, component — is a head pattern, one per format, which is exactly what regex is good at. The variable part — an unknown number of key=value pairs mixed with plain words — is not: a repeated group can only keep its last repetition, so one giant pattern can't capture them all. A master- pattern scanner (named alternatives run with finditer) walks the tail one token at a time instead, and a small typed() helper turns each bare value into a number, a duration in milliseconds, or a string.

Four pieces of Quarry's extractor exist. The gate decides whether a line is a log line. extract_fields names its parts. redact makes it safe to show. The guard keeps operator patterns from freezing the tab. What's missing is the piece every later stage has been waiting for: something that turns any log line in the yard into one typed record.

And the yard has a complication. Quarry's own services write 2026-09-14 08:15:02 WARN [auth] login failed user=maya took=312ms, but the old web gateway, which nobody is allowed to touch, writes [14/Sep/2026:08:15:02] E gateway: upstream timeout took=2s retries=3 — same information, different shape: a bracketed day-first timestamp with a month name, a one-letter level, a colon instead of brackets, durations in seconds. The report must not care which service wrote a line. This capstone builds parse_log(). It should take 35 to 50 minutes, and nothing in it is new — it is the whole section, used together.

The shape of the record

Three rules make records from different services comparable: timestamps become YYYY-MM-DDTHH:MM:SS; levels are always DEBUG/INFO/WARN/ ERROR — the gateway's D/I/W/E are expanded; durations are whole milliseconds under a key ending in _ms. Lines matching neither format return None, exactly as extract_fields did.

First attempt: one enormous regex

The head of each format is fine — patterns like it all section. The trouble is the tail, a variable number of key=value pairs mixed with words. A repeated group looks like the answer:

You could keep going — match the tail with one pattern, findall pairs out of it, subtract the pairs to find the message, special-case quoted values — and each step adds another pattern with its own edge cases. The better design splits the work where the structure splits:

A head pattern per format, which is a fixed shape, and a scanner for the tail, which is a sequence of tokens.

The heads

Each head pattern ends with a tail group that takes the rest of the line:

The gateway's month is a name — that's meaning, not shape — so the pattern only checks it looks like Sep, and a dictionary does the rest:

MONTHS.get returns None for a month that doesn't exist — the signal to reject the line rather than crash on it.

Scanning the tail

Lesson four built a scanner one character at a time. There's a middle road just as linear and much shorter: a master pattern of named alternatives, one per token kind, run with finditer. Each match is one token, and m.lastgroup names which alternative matched.

The scanner is a generator — one token at a time, no list built up front. Whitespace is never matched, so finditer skips it between tokens. The fourth item says whether the value was quoted: a quoted "3" is text someone deliberately wrote; a bare 3 is a number.

Alternation order is load-bearing. The engine tries alternatives left to right and takes the first that fits, so a general alternative placed first shadows every specific one after it:

Watch the scanner walk the gateway tail from the drawing and sort each token into message or field:

Typing the values

The scanner produces strings; the record wants types. Each bare value gets one pass, most specific check first: a duration — \d+(?:\.\d+)?(ms|s) — becomes whole milliseconds under a _ms key; a number becomes an int or float; anything else stays a string. Each check is a fullmatch on one short token — anchored, nothing to backtrack into.

Putting it together

The finished function reads top to bottom, like the flowchart: try the Quarry head, then the gateway head, normalise the timestamp and level, scan the tail, type the values, assemble the record. Keep each step a small helper — head(), tokens(), typed() — so parse_log itself stays short. When a third format arrives, it's one more head pattern and one more branch in head(); the scanner and the typing never change.

Before you check, test the edges yourself: a line with no fields, a quoted value with spaces, a gateway line with an impossible month, a stack trace.

Quick practice

🎯 Your turn

Write parse_log(line) returning the typed record described above, or None if the line matches neither format.

• parse_log("2026-09-14 08:15:02 WARN [auth] login failed user=maya took=312ms") → {"ts": "2026-09-14T08:15:02", "level": "WARN", "component": "auth", "message": "login failed", "fields": {"user": "maya", "took_ms": 312}}
• parse_log("[14/Sep/2026:08:15:02] E gateway: upstream timeout took=2s retries=3") → {"ts": "2026-09-14T08:15:02", "level": "ERROR", "component": "gateway", "message": "upstream timeout", "fields": {"took_ms": 2000, "retries": 3}}
• parse_log("Traceback (most recent call last):") → None

Hint — head patterns for the fixed part, the master-pattern scanner for the tail, and a small typed() helper for the values. Quoted values stay strings.

Then press ▶ Run and hit ✓ Check. Green = Quarry's extractor is built. ✅