Modules, Packages & What import Really Does
In short: import is not a declaration — it's a small program. Python
finds a file on sys.path, creates an empty module object and stores
it in sys.modules before running anything, runs the file's code with
that object's dict as its namespace, then binds a name to it in your
namespace. The cache in sys.modules means a module's body runs exactly
once per program, however many times it's imported — and a package is a
folder with an __init__.py, so importing something deep inside one runs
every __init__.py above it first, outermost first.
Cache a slow module so it only runs once
Steps 2 to 4 of import need nothing but the standard library — done here by
hand, with types.ModuleType, the very type every imported module has:
The body printed once, though load ran twice. With a real file the same
thing happens — the browser has an in-memory filesystem, so this really
imports:
import importlib
import pathlib
import sys
import tempfile
sys.modules.pop("greeter", None)
root = tempfile.mkdtemp()
pathlib.Path(root, "greeter.py").write_text('print("loading", end=" ")\n')
sys.path.insert(0, root)
importlib.invalidate_caches()
import greeter
import greeter
import greeter
print(sys.modules["greeter"] is greeter)Guard a script so importing it doesn't rerun main()
Every module has a __name__, set before its first line runs. A file
started with python report.py gets the special name "__main__"; imported,
it gets its dotted name. That's the entire trick behind if __name__ == "__main__"::
Run as a script, the guarded block calls main(). Imported, the file only
defines main and leaves calling it to whoever imported it. Without the
guard, merely importing the module would build a report as a side effect,
every time:
import types
report = types.ModuleType("quarry.report")
exec("where = __name__.rpartition('.')[2]", report.__dict__)
print(report.where)Reach the vault from a sibling package
Fourteen modules in one flat folder is hard to navigate. A package
groups them — a folder with an __init__.py — and the folder's name becomes
part of every file's dotted name:
quarry/
├── __init__.py → quarry
├── cli.py → quarry.cli
├── stages/
│ ├── __init__.py → quarry.stages
│ └── parse.py → quarry.stages.parse
└── store/
├── __init__.py → quarry.store
└── vault.py → quarry.store.vaultA package is a module — quarry has a __dict__ and a body, which is its
__init__.py — and importing anything inside it runs every __init__.py
above it first, because quarry.store.vault cannot exist until quarry
and quarry.store do:
quarry/stages/parse.py can reach the vault two ways: from quarry.store import vault (absolute) or from ..store import vault (relative — each dot
climbs one package level). Step through the dot arithmetic:
Run cli.py directly and a relative import fails with ImportError: attempted relative import with no known parent package: the file is
__main__, belonging to no package, so the dot has nothing to be relative
to. python -m quarry.cli imports it by its dotted name instead, and the
same line works — never make a module work only by running it as a script.
Traps worth knowing
Circular imports. Module a imports b, and b imports a. A module
is cached before its body runs, so when b asks for a, Python hands
back the half-built module, stopped on the line that imported b — anything
below that line doesn't exist yet:
ImportErrorcli imports report, and report imports a name from cli that cli hasn't defined yet. Change report.py's source so it no longer needs VERSION at import time, and the program prints "quarry 0.3".
Shadowing. Name your own file random.py and import random in that
folder finds yours first, silently. Import-time cost is startup cost: a
CLI that eagerly imports 30 stage modules to print --help pays for all of
them — keep module bodies cheap, and import heavy things inside the function
that needs them.
Quick practice
import sys, importlib, pathlib, tempfile
sys.modules.pop("widget", None)
root = tempfile.mkdtemp()
pathlib.Path(root, "widget.py").write_text("COUNT = 0\n")
sys.path.insert(0, root)
importlib.invalidate_caches()
import widget
import widget as w2
w2.COUNT = 99
print(widget.COUNT)Importing this module prints "building report..." as a side effect every time. Move that print under a main guard so importing it stays silent and only 42 prints.
import sys, importlib, pathlib, tempfile
root = tempfile.mkdtemp()
pathlib.Path(root, "random.py").write_text("def choice(seq):\n return 'rigged'\n")
sys.path.insert(0, root)
importlib.invalidate_caches()
sys.modules.pop("random", None)
import random
print(random.choice(["a", "b", "c"]))🎯 Your turn
Build resolve_import(files, current, statement) — the first piece of
Quarry's shipment. It answers the question every confusing ImportError
hides: what does this line actually load?
fileslists the project's files, relative to the folder onsys.path.currentis the path of the file containing the statement.statementisimport Xorfrom X import Y, whereXmay be relative.
Return the dotted name of the deepest module loaded and the chain of
files Python runs for it, parents first. A folder's __init__.py wins over
a same-named .py file. For from X import Y, if X.Y is itself a module,
it is loaded too; otherwise Y is just a name inside X.
resolve_import(FILES, "quarry/stages/parse.py", "from ..store import vault")
# → {"module": "quarry.store.vault",
# "chain": ["quarry/__init__.py", "quarry/store/__init__.py", "quarry/store/vault.py"]}
resolve_import(FILES, "cli.py", "from . import quarry")
# → {"error": "ImportError"} # no parent package
resolve_import(FILES, "quarry/cli.py", "import quarry.yard.walk")
# → {"error": "ModuleNotFoundError", "name": "quarry.yard"}Press ▶ Run and hit ✓ Check. ✅