Capstone — The Yard
In short: this capstone wires lessons 01–05 into the yard — one
pipeline that takes a directory and reports, for every source file, what it
really is. Every file is read once, as bytes (lesson 02); sniff
weighs the evidence strongest-first — a telemetry magic (lesson 05), then a
byte-order mark, then a NUL byte, then everything else as text, decoded by
lesson 04's rules — and only text gets decoded, only once, at the edge. The
section's one idea pays off here: a filename is a claim; the bytes are
evidence. When they disagree, the bytes win, and the disagreement gets
recorded instead of silently ignored.
Weigh the evidence before deciding what a file is
Here is the real shared folder Quarry has been pointed at: a server log, a
Windows export, a UTF-16 spreadsheet, the hourly pump telemetry, a hidden
editor cache, a README — and one file somebody renamed. It says odd.log.
Its first four bytes say QRY1.
The obvious-looking rule, "a NUL byte means binary", is wrong on its own:
data = "id,name\n".encode("utf-16")
label = "binary" if b"\x00" in data else "text"
print(label)So sniff(data) weighs the evidence strongest-first:
- The telemetry magic. A valid
QRY1header with a record count is about as unambiguous as evidence gets. Ifheader_records(data)returns a number, the file is telemetry — whatever its name says. - A byte-order mark.
EF BB BF,FF FEorFE FFat the start was put there on purpose by the program that wrote the text. It means text, and it names the encoding. - A NUL byte, now that a BOM has been ruled out, means this isn't text in any encoding the yard handles — binary of some kind nobody told Quarry about.
- Everything else is text, decoded by lesson 04's rules and counted
with
splitlines(), which handles\n,\r\nand\ralike.
(The real sniff is stricter about telemetry than startswith(b"QRY1") —
it insists on a full 20-byte header, so a four-byte fragment isn't mistaken
for a sensor file.)
Catch the sensor file wearing a .log costume
The yard compares the suffix's claim against the bytes' evidence. Most of the time they agree. When they don't, the bytes win, and the mismatch is recorded — because it usually means something upstream broke:
from pathlib import PurePosixPath
name = "sensors/odd.log"
data = bytes.fromhex("51525931010000") + bytes(13)
print(PurePosixPath(name).suffix, data[:4] == b"QRY1")Here's the naive version — decode everything as UTF-8 and count lines. It works on the honest file and falls over on the renamed one. Fix it so telemetry is recognised before anything tries to decode it:
UnicodeDecodeErrorRun it and read which byte UTF-8 rejected. Then sniff first and only decode what is text.
The rule for lied is short once sniff exists: a text suffix
(.log, .csv, .json, .txt) lies if the content isn't text; a .qry
lies if the content isn't telemetry. A UTF-16 CSV does not lie — it's
text in an unusual encoding, and the encoding column already says so.
Read every file once, in binary
A detail that's easy to get wrong: the yard reads every file with
path.read_bytes() — binary mode, once. It never opens a file in text
mode "to have a look", for three reasons you already know: text mode needs
an encoding, and choosing it is the question being answered; text mode
translates line endings, destroying the evidence lesson 02 cared about; and
a telemetry file isn't text at all, so a text-mode read would either crash
or, worse, succeed with nonsense.
Once sniff has decided, decode(data) turns bytes into a str exactly
once, at the edge. Everything downstream of the yard — parsers, records, a
database in a later section — only ever sees text or typed records, never
raw bytes. An encoding bug in Quarry can only ever live in that one place.
Here's the whole yard, drawn as the pieces it's built from:
What you're given, and what you write
The starter already contains, compacted, the pieces from earlier lessons:
plant(root, files), find_sources(root), decode(data) and
header_records(data). You write two functions:
sniff(data)→(kind, encoding, count), weighing the evidence in the order above. Telemetry is("telemetry", "binary", records); unknown binary is("binary", "binary", 0); text is("text", encoding, number of lines).open_source(files)→ plant the yard, loopfind_sources(root), sniff each file's bytes, and build the report keyed bypath.relative_to(root).as_posix(), adding"lied"from the suffix and the kind.
Quick practice
import struct
HEADER = struct.Struct("<4sBBHI8s")
data = b"QRY1" + b"\x01"
print(len(data) < HEADER.size)import codecs
def sniff_simple(data):
if data.startswith(b"QRY1"):
return "telemetry"
if data.startswith((codecs.BOM_UTF8, codecs.BOM_UTF16_LE, codecs.BOM_UTF16_BE)):
return "text (bom)"
if b"\x00" in data:
return "binary"
return "text"
print(sniff_simple(b"boot ok\nwarm\n"))UnicodeDecodeErrorRun it and read which byte UTF-8 rejected. Sniff before you decode.
suffix, kind = ".csv", "text"
TEXT_SUFFIXES = {".log", ".csv", ".json", ".txt"}
def lied(suffix, kind):
if suffix in TEXT_SUFFIXES:
return kind != "text"
if suffix == ".qry":
return kind != "telemetry"
return False
print(lied(suffix, kind))🎯 Your turn
Build sniff(data) and open_source(files):
- a
.logwhose bytes start with aQRY1header recording 12 readings →{"kind": "telemetry", "encoding": "binary", "count": 12, "lied": True} - a UTF-16
.csvwith two lines →{"kind": "text", "encoding": "utf-16", "count": 2, "lied": False}
Hint — telemetry first, then a BOM, then a NUL byte, then decode; read each file
once with read_bytes(); decide lied from path.suffix.lower() and the kind.
Then press ▶ Run and hit ✓ Check. ✅