Pickle, and Why You Must Not Load Untrusted Data

In short: pickle.dumps(obj) turns almost any Python object into bytes and pickle.loads(data) rebuilds it exactly — tuples stay tuples, int keys stay ints, datetimes and sets need no hooks, and shared references survive. That's the opposite of JSON's lossy border crossing, and the price is steep: pickle.loads doesn't parse data, it runs a small program. The bytes name a callable and arguments, and loading them calls that callable. If the bytes came from anyone but you, they can name os.system as easily as datetime. Pickle is for a program talking to itself — never for data that crossed a trust boundary.

You'll build cache_batch(raws), the vault's batch cache: park a parsed batch as bytes, and get back exactly the records you had.

pickle.loads doesn't read data, it makes calls: GLOBAL imports a name, REDUCE calls it with arguments from the stream — fine for Quarry's own file, dangerous for one from a shared folder

Cache a parsed batch so tomorrow's run skips the parsing

Parsing is the slow part of Quarry's pipeline. If the source file hasn't changed, loading yesterday's parsed batch beats reparsing it:

python

back == record is True. The tuple is still a tuple, the int keys are still ints — every lossy edge from the JSON lesson is gone. Pickle even tracks which names point at the same object:

What does this print?

import pickle

shared = ["web"]
pair = [shared, shared]          # two references to ONE list
back = pickle.loads(pickle.dumps(pair))
back[0].append("disk")
print(back[1])

What pickle can't carry is code: a function is pickled by reference — module and name — and reimported at load time. A lambda has no name to look up. A generator is a paused stack frame with nothing to name. Fix this batch so it pickles, keeping the rows intact:

broken — fix it

Run it and read which object pickle refused. Then change the batch so it holds only data.

Open a batch cache written by someone you don't trust

Here's where the convenience turns dangerous. pickletools.dis shows what's really in the bytes:

python

That's not a data layout — it's instructions for a tiny stack machine. GLOBAL 'datetime date' imports datetime and pushes the date class. REDUCE pops a callable and a tuple of arguments, calls the callable, and pushes the result. Rebuilding a date means calling date(...) — and that's exactly how pickle restores anything: by calling whatever the stream names, with whatever arguments the stream supplies. A class controls its own rebuild with __reduce__:

What does this print?

import pickle

class Sneaky:
    def __reduce__(self):
        return (str.upper, ("owned",))

blob = pickle.dumps(Sneaky())
print(pickle.loads(blob))

str.upper is harmless; an attacker picks something that isn't. And the call happens inside loads, before your code ever sees the result — no check afterwards can help.

Drawing diagram…

Here's that machine cut down to five instructions you can trace by hand:

step through it

1CALLABLES = {"upper": str.upper, "len": len}

3program = [("GLOBAL", "upper"), ("PUSH", "owned"), ("TUPLE", 1), ("REDUCE", None), ("STOP", None)]

4stack = []

5for op, arg in program:

6 if op == "GLOBAL":

7 stack.append(CALLABLES[arg])

8 elif op == "PUSH":

9 stack.append(arg)

10 elif op == "TUPLE":

11 items = stack[-arg:]

12 del stack[-arg:]

13 stack.append(tuple(items))

14 elif op == "REDUCE":

15 args = stack.pop()

16 func = stack.pop()

17 stack.append(func(*args))

18 elif op == "STOP":

19 break

20print(stack.pop())

Sign a cache so a tampered file is refused

Never unpickle data you did not produce yourself, from a place nobody else can write to.

A cache in a shared directory, an upload, a queue message: each is "untrusted" the moment someone other than you could have written it. The real fix is to not use pickle across a trust boundary at all — reach for JSON or a database instead when data crosses between programs or people. When Quarry writes its own cache and reads it back, signing proves the bytes are yours: attach an HMAC, a keyed hash made with a secret only Quarry knows, and check it before calling loads:

python

The comparison uses hmac.compare_digest, not ==, so the time it takes doesn't leak how many bytes matched. A short checksum like CRC-16 is broken almost at once; HMAC-SHA256's 256 bits put a forged tag out of reach. Signing proves the bytes are yours — it does not make hostile bytes safe, which is why the real rule is still "don't cross a trust boundary with pickle".

Quick practice

What does this print?

import pickle

print(pickle.loads(pickle.dumps((1, 2, 3))) == (1, 2, 3))

broken — fix itTypeError

Run it, read which object pickle refuses, then pickle a realised list instead of the generator and print "ok".

What does this print?

import pickle

class Payload:
    def __reduce__(self):
        return (len, ([1, 2, 3],))

print(pickle.loads(pickle.dumps(Payload())))

🎯 Your turn

Write cache_batch(raws). The starter's build_record(raw) is the same stand-in record stage as in the JSON lesson. Your function should:

build records = [build_record(r) for r in raws]
pickle the list to bytes with pickle.HIGHEST_PROTOCOL, then load it back
return {"count": len(restored), "intact": restored == records, "kinds": sorted type names of every value in every restored record}

Example: one incident record → {"count": 1, "intact": True, "kinds": ["Decimal", "datetime", "int", "set", "str"]}. Every one of those types came back as itself, which JSON couldn't do.

Hint: a set comprehension, {type(v).__name__ for r in restored for v in r.values()}, collects the names without duplicates. Then sorted it.

Press ▶ Run and hit ✓ Check. ✅

Pickle, and Why You Must Not Load Untrusted Data

Pickle, and Why You Must Not Load Untrusted Data

Cache a parsed batch so tomorrow's run skips the parsing

Open a batch cache written by someone you don't trust

Sign a cache so a tampered file is refused

Quick practice

🎯 Your turn

Practice — level up