|$ curl https://forge-ai.dev/api/markdown?path=docs/python/stdlib-reference
$cat docs/python-stdlib-reference.md
updated Today·35 min read·published
Python Stdlib Reference
Introduction
This encyclopedia indexes standard library modules you must recognize fluently — the lookup layer beside the mastery curriculum. Scan tables, then open linked deep pages for narrative and production patterns.
Coverage spans builtins, pathlib, os/sys, json, csv, collections, itertools, functools, datetime/zoneinfo, argparse, logging, re, hashlib/secrets, urllib, http.client, sqlite3, concurrent.futures, typing, dataclasses, and contextlib.
ℹ
info
Start learning order from How to Master Python. Agents: curriculum=python.
How to Use This Reference
- Prefer pathlib over os.path for new code
- Prefer zoneinfo over pytz for timezones (3.9+)
- Prefer secrets over random for tokens
- Follow deep-page links for security and concurrency patterns
- Re-check docs.python.org for your minor version — APIs evolve
imports.py
Python
| 1 | from pathlib import Path |
| 2 | from collections import Counter, defaultdict, deque |
| 3 | from functools import lru_cache, partial, wraps |
| 4 | from concurrent.futures import ThreadPoolExecutor |
| 5 | import json, csv, argparse, logging, hashlib, secrets |
Module Index
| Module | Deep page | One-liner |
|---|---|---|
| builtins | — | print, len, open, iter, next, ... |
| pathlib | /docs/python/pathlib | Object-oriented filesystem paths |
| os / sys | — | Env, argv, platform, process |
| json | /docs/python/json-csv | JSON encode/decode |
| csv | /docs/python/json-csv | Delimited text tables |
| collections | /docs/python/collections-itertools | Counter, deque, defaultdict |
| itertools | /docs/python/collections-itertools | Lazy iterator algebra |
| functools | /docs/python/functools | cache, partial, wraps |
| datetime | /docs/python/datetime-time | Dates, times, timedeltas |
| zoneinfo | /docs/python/datetime-time | IANA time zones |
| argparse | /docs/python/argparse | CLI parsers |
| logging | /docs/python/logging | Structured log pipeline |
| re | /docs/python/regex | Regular expressions |
| hashlib | /docs/python/hashlib-secrets | Cryptographic hashes |
| secrets | /docs/python/hashlib-secrets | Secure tokens |
| urllib | /docs/python/httpx-requests | URL parse + basic HTTP |
| http.client | — | Low-level HTTP |
| sqlite3 | — | Embedded SQL database |
| concurrent.futures | /docs/python/concurrency | Thread/Process pools |
| typing | /docs/python/type-hints | Static type annotations |
| dataclasses | /docs/python/dataclasses | Boilerplate-free classes |
| contextlib | /docs/python/context-managers | Context manager helpers |
| abc | /docs/python/abc-enums | Abstract base classes |
| enum | /docs/python/abc-enums | Enumerations |
Builtins Overview
The builtins module is always available. Prefer explicit names; avoid shadowing list, id, type, or input.
| Category | Examples | Notes |
|---|---|---|
| Types | int, float, str, bytes, list, dict, set, tuple | Constructors + factories |
| Introspection | type, isinstance, issubclass, dir, vars, getattr | Prefer isinstance over type == |
| Iteration | iter, next, enumerate, zip, map, filter, reversed | zip(..., strict=True) on 3.10+ |
| Aggregates | len, sum, min, max, any, all, sorted | key= for min/max/sorted |
| I/O | print, open, input | Prefer Path.open / Path.read_text |
| Functional | callable, property, classmethod, staticmethod | See descriptors page |
| Numeric | abs, round, divmod, pow, complex | pow(a,b,mod) for modular |
builtins_tour.py
Python
| 1 | # zip strict catches length mismatches (3.10+) |
| 2 | names = ["a", "b"] |
| 3 | vals = [1, 2, 3] |
| 4 | try: |
| 5 | list(zip(names, vals, strict=True)) |
| 6 | except ValueError as e: |
| 7 | print(e) |
| 8 | |
| 9 | # enumerate with start |
| 10 | for i, ch in enumerate("py", start=1): |
| 11 | print(i, ch) |
| 12 | |
| 13 | # sorted with key |
| 14 | rows = [{"n": "b", "v": 2}, {"n": "a", "v": 1}] |
| 15 | print(sorted(rows, key=lambda r: r["n"])) |
pathlib
PurePath is path arithmetic without I/O; Path adds filesystem operations. Prefer Path for almost all new code.
pathlib_basics.py
Python
| 1 | from pathlib import Path |
| 2 | |
| 3 | root = Path.cwd() |
| 4 | cfg = root / "config" / "app.toml" |
| 5 | print(cfg.name, cfg.suffix, cfg.stem) |
| 6 | print(cfg.resolve()) |
| 7 | print(list(root.glob("**/*.py"))) |
| 8 | |
| 9 | text = Path("readme.md").read_text(encoding="utf-8") |
| 10 | Path("out.txt").write_text(text.upper(), encoding="utf-8") |
| 11 | Path("data.bin").write_bytes(b"\x00\x01") |
| 12 | |
| 13 | # relative_to raises if not a subpath |
| 14 | rel = Path("/a/b/c").relative_to("/a") |
| 15 | assert str(rel) == "b/c" |
📝
note
Deep dive: pathlib.
os & sys
Use os.environ / os.getenv for environment; sys.argv for raw CLI args (prefer argparse); sys.path for import path debugging.
os_sys.py
Python
| 1 | import os |
| 2 | import sys |
| 3 | |
| 4 | print(sys.version_info[:2]) # (3, 12) |
| 5 | print(sys.platform) # linux, darwin, win32 |
| 6 | print(os.getenv("HOME") or os.getenv("USERPROFILE")) |
| 7 | |
| 8 | # Never mutate os.environ casually in libraries |
| 9 | os.environ.setdefault("APP_ENV", "dev") |
| 10 | |
| 11 | # Exit codes |
| 12 | # sys.exit(0) success; non-zero = failure |
| 13 | if __name__ == "__main__": |
| 14 | if len(sys.argv) < 2: |
| 15 | print("usage: tool <file>", file=sys.stderr) |
| 16 | raise SystemExit(2) |
| API | Purpose |
|---|---|
| os.environ / getenv | Environment variables |
| os.getpid / getppid | Process ids |
| os.cpu_count() | Scheduler hint |
| sys.argv | Raw CLI arguments |
| sys.path | Module search path |
| sys.stdin/out/err | Standard streams |
| sys.getsizeof | Object shallow size |
json
json.dumps / loads for strings; dump / load for file objects. Use default= for non-JSON types.
json_mod.py
Python
| 1 | import json |
| 2 | from datetime import datetime, timezone |
| 3 | from pathlib import Path |
| 4 | |
| 5 | payload = {"ok": True, "n": 1, "tags": ["a", "b"]} |
| 6 | s = json.dumps(payload, indent=2, sort_keys=True) |
| 7 | data = json.loads(s) |
| 8 | |
| 9 | def default(o): |
| 10 | if isinstance(o, datetime): |
| 11 | return o.isoformat() |
| 12 | raise TypeError(type(o)) |
| 13 | |
| 14 | doc = {"ts": datetime.now(timezone.utc)} |
| 15 | Path("out.json").write_text( |
| 16 | json.dumps(doc, default=default), encoding="utf-8" |
| 17 | ) |
📝
note
Deep dive: json-csv.
csv
csv_mod.py
Python
| 1 | import csv |
| 2 | from pathlib import Path |
| 3 | |
| 4 | path = Path("users.csv") |
| 5 | with path.open("w", newline="", encoding="utf-8") as f: |
| 6 | w = csv.DictWriter(f, fieldnames=["id", "name"]) |
| 7 | w.writeheader() |
| 8 | w.writerow({"id": 1, "name": "Ada"}) |
| 9 | |
| 10 | with path.open(newline="", encoding="utf-8") as f: |
| 11 | for row in csv.DictReader(f): |
| 12 | print(row["id"], row["name"]) |
collections
| Type | Use when |
|---|---|
| Counter | Tallies / frequencies |
| defaultdict | Auto-create missing keys |
| deque | Queues, sliding windows |
| namedtuple | Lightweight immutable records |
| ChainMap | Layered lookups (defaults) |
| OrderedDict | Rare — dict is ordered since 3.7 |
collections_mod.py
Python
| 1 | from collections import Counter, defaultdict, deque, namedtuple, ChainMap |
| 2 | |
| 3 | Point = namedtuple("Point", "x y") |
| 4 | p = Point(1, 2) |
| 5 | print(p.x, p._asdict()) |
| 6 | |
| 7 | c = Counter("abracadabra") |
| 8 | print(c.most_common(3)) |
| 9 | |
| 10 | dd = defaultdict(list) |
| 11 | dd["a"].append(1) |
| 12 | |
| 13 | q = deque(maxlen=3) |
| 14 | for i in range(5): |
| 15 | q.append(i) |
| 16 | print(list(q)) # [2, 3, 4] |
| 17 | |
| 18 | defaults = {"host": "localhost", "port": 5432} |
| 19 | override = {"port": 5433} |
| 20 | cfg = ChainMap(override, defaults) |
| 21 | print(cfg["host"], cfg["port"]) |
itertools
Lazy combinatorics and slicing over iterators. Compose instead of materializing large lists.
itertools_mod.py
Python
| 1 | from itertools import chain, groupby, islice, product, combinations, count, cycle, repeat |
| 2 | |
| 3 | print(list(chain([1, 2], (3, 4)))) |
| 4 | print(list(islice(count(10), 5))) # 10..14 |
| 5 | print(list(product("AB", repeat=2))) |
| 6 | print(list(combinations(range(4), 2))) |
| 7 | |
| 8 | rows = sorted([("a", 1), ("a", 2), ("b", 3)], key=lambda r: r[0]) |
| 9 | for key, group in groupby(rows, key=lambda r: r[0]): |
| 10 | print(key, list(group)) |
🔥
pro tip
Third-party more-itertools adds chunked, windowed, unique_everseen, and dozens more recipes.
functools
functools_mod.py
Python
| 1 | from functools import lru_cache, cache, partial, reduce, wraps, singledispatch |
| 2 | |
| 3 | @lru_cache(maxsize=128) |
| 4 | def fib(n: int) -> int: |
| 5 | return n if n < 2 else fib(n - 1) + fib(n - 2) |
| 6 | |
| 7 | @cache # 3.9+ unbounded |
| 8 | def expensive(x: int) -> int: |
| 9 | return x * x |
| 10 | |
| 11 | add = partial(int.__add__, 10) |
| 12 | print(add(5)) # 15 |
| 13 | |
| 14 | print(reduce(lambda a, b: a + b, [1, 2, 3], 0)) |
| 15 | |
| 16 | @singledispatch |
| 17 | def dump(obj): |
| 18 | return repr(obj) |
| 19 | |
| 20 | @dump.register |
| 21 | def _(obj: list): |
| 22 | return "[" + ", ".join(map(dump, obj)) + "]" |
📝
note
Deep dive: functools.
datetime & zoneinfo
datetime_mod.py
Python
| 1 | from datetime import datetime, timedelta, timezone |
| 2 | from zoneinfo import ZoneInfo |
| 3 | |
| 4 | utc_now = datetime.now(timezone.utc) |
| 5 | ny = utc_now.astimezone(ZoneInfo("America/New_York")) |
| 6 | print(ny.isoformat()) |
| 7 | |
| 8 | delta = timedelta(days=1, hours=3) |
| 9 | print(utc_now + delta) |
| 10 | |
| 11 | # Prefer aware datetimes — never compare naive across zones |
| 12 | assert utc_now.tzinfo is not None |
argparse
argparse_mod.py
Python
| 1 | import argparse |
| 2 | from pathlib import Path |
| 3 | |
| 4 | parser = argparse.ArgumentParser(prog="tool", description="Demo CLI") |
| 5 | parser.add_argument("input", type=Path, help="input file") |
| 6 | parser.add_argument("-v", "--verbose", action="store_true") |
| 7 | parser.add_argument("--count", type=int, default=1, choices=range(1, 6)) |
| 8 | sub = parser.add_subparsers(dest="cmd", required=True) |
| 9 | p_run = sub.add_parser("run") |
| 10 | p_run.add_argument("--dry-run", action="store_true") |
| 11 | p_ls = sub.add_parser("list") |
| 12 | args = parser.parse_args() |
| 13 | print(args) |
logging
logging_mod.py
Python
| 1 | import logging |
| 2 | |
| 3 | logging.basicConfig( |
| 4 | level=logging.INFO, |
| 5 | format="%(asctime)s %(levelname)s %(name)s %(message)s", |
| 6 | ) |
| 7 | log = logging.getLogger("app") |
| 8 | log.info("started", extra={"request_id": "abc"}) |
| 9 | log.warning("slow query %.2fs", 1.23) |
| 10 | try: |
| 11 | 1 / 0 |
| 12 | except ZeroDivisionError: |
| 13 | log.exception("boom") |
re
re_mod.py
Python
| 1 | import re |
| 2 | |
| 3 | pat = re.compile(r"(?P<user>[\w.]+)@(?P<domain>[\w.-]+)", re.I) |
| 4 | m = pat.search("Mail: Ada.Lovelace@Example.COM") |
| 5 | if m: |
| 6 | print(m["user"], m["domain"]) |
| 7 | |
| 8 | print(re.sub(r"\s+", " ", " a b ").strip()) |
| 9 | print(re.split(r"[,;]\s*", "a, b; c")) |
hashlib & secrets
hash_secrets.py
Python
| 1 | import hashlib |
| 2 | import hmac |
| 3 | import secrets |
| 4 | |
| 5 | data = b"payload" |
| 6 | print(hashlib.sha256(data).hexdigest()) |
| 7 | |
| 8 | key = secrets.token_bytes(32) |
| 9 | sig = hmac.new(key, data, hashlib.sha256).hexdigest() |
| 10 | print(hmac.compare_digest(sig, sig)) |
| 11 | |
| 12 | token = secrets.token_urlsafe(16) |
| 13 | print(token) |
✕
danger
Never use random for passwords, tokens, or session ids. Use secrets.
urllib & http.client
For production HTTP clients prefer httpx/requests. Stdlib is fine for simple scripts and URL parsing.
urllib_mod.py
Python
| 1 | from urllib.parse import urlparse, urlencode, parse_qs |
| 2 | from urllib.request import urlopen, Request |
| 3 | |
| 4 | u = urlparse("https://example.com/path?q=1#frag") |
| 5 | print(u.scheme, u.netloc, u.path, parse_qs(u.query)) |
| 6 | |
| 7 | qs = urlencode({"q": "python pathlib", "page": 1}) |
| 8 | req = Request(f"https://httpbin.org/get?{qs}", headers={"User-Agent": "forgelearn"}) |
| 9 | # with urlopen(req, timeout=5) as resp: |
| 10 | # print(resp.status, resp.read()[:100]) |
sqlite3
sqlite3_mod.py
Python
| 1 | import sqlite3 |
| 2 | from pathlib import Path |
| 3 | |
| 4 | db = Path("app.db") |
| 5 | with sqlite3.connect(db) as conn: |
| 6 | conn.row_factory = sqlite3.Row |
| 7 | conn.execute( |
| 8 | "CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)" |
| 9 | ) |
| 10 | conn.execute("INSERT INTO users (name) VALUES (?)", ("Ada",)) |
| 11 | row = conn.execute("SELECT * FROM users WHERE name = ?", ("Ada",)).fetchone() |
| 12 | print(dict(row)) |
✓
best practice
Always use parameterized queries (? placeholders). Never format SQL with f-strings.
concurrent.futures
futures_mod.py
Python
| 1 | from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed |
| 2 | |
| 3 | def fetch(n: int) -> int: |
| 4 | return n * n |
| 5 | |
| 6 | with ThreadPoolExecutor(max_workers=4) as pool: |
| 7 | futs = {pool.submit(fetch, i): i for i in range(8)} |
| 8 | for fut in as_completed(futs): |
| 9 | print(futs[fut], fut.result()) |
| 10 | |
| 11 | # CPU-bound: ProcessPoolExecutor (bypasses GIL) |
| 12 | # with ProcessPoolExecutor() as pool: |
| 13 | # list(pool.map(fetch, range(8))) |
typing
typing_mod.py
Python
| 1 | from typing import TypeVar, Iterable, overload, Literal |
| 2 | from collections.abc import Sequence, Mapping, Callable |
| 3 | |
| 4 | T = TypeVar("T") |
| 5 | |
| 6 | def first(xs: Sequence[T]) -> T: |
| 7 | return xs[0] |
| 8 | |
| 9 | def apply(fn: Callable[[int], str], n: int) -> str: |
| 10 | return fn(n) |
| 11 | |
| 12 | Mode = Literal["r", "w", "a"] |
| 13 | |
| 14 | @overload |
| 15 | def read(path: str, binary: Literal[False] = False) -> str: ... |
| 16 | @overload |
| 17 | def read(path: str, binary: Literal[True]) -> bytes: ... |
| 18 | def read(path: str, binary: bool = False) -> str | bytes: |
| 19 | return b"" if binary else "" |
dataclasses
dataclasses_mod.py
Python
| 1 | from dataclasses import dataclass, field, asdict |
| 2 | |
| 3 | @dataclass(frozen=True, slots=True) |
| 4 | class User: |
| 5 | id: int |
| 6 | name: str |
| 7 | tags: tuple[str, ...] = () |
| 8 | meta: dict[str, str] = field(default_factory=dict) |
| 9 | |
| 10 | u = User(1, "Ada", tags=("admin",)) |
| 11 | print(asdict(u)) |
contextlib
contextlib_mod.py
Python
| 1 | from contextlib import contextmanager, suppress, ExitStack, nullcontext |
| 2 | from pathlib import Path |
| 3 | |
| 4 | @contextmanager |
| 5 | def chdir(path: Path): |
| 6 | import os |
| 7 | prev = Path.cwd() |
| 8 | os.chdir(path) |
| 9 | try: |
| 10 | yield |
| 11 | finally: |
| 12 | os.chdir(prev) |
| 13 | |
| 14 | with suppress(FileNotFoundError): |
| 15 | Path("missing.txt").unlink() |
| 16 | |
| 17 | with ExitStack() as stack: |
| 18 | files = [stack.enter_context(Path(f"f{i}.txt").open("w")) for i in range(3)] |
| 19 | for f in files: |
| 20 | f.write("ok\n") |
pathlib Recipes
pathlib_recipes.py
Python
| 1 | from pathlib import Path |
| 2 | import shutil |
| 3 | |
| 4 | def ensure_dir(p: Path) -> Path: |
| 5 | p.mkdir(parents=True, exist_ok=True) |
| 6 | return p |
| 7 | |
| 8 | def iter_py_files(root: Path): |
| 9 | yield from root.rglob("*.py") |
| 10 | |
| 11 | def safe_rel(path: Path, root: Path) -> Path | None: |
| 12 | try: |
| 13 | resolved = path.resolve() |
| 14 | return resolved.relative_to(root.resolve()) |
| 15 | except ValueError: |
| 16 | return None # path escape attempt |
| 17 | |
| 18 | def copy_tree(src: Path, dst: Path) -> None: |
| 19 | if dst.exists(): |
| 20 | shutil.rmtree(dst) |
| 21 | shutil.copytree(src, dst) |
| 22 | |
| 23 | # Walk with depth control via glob |
| 24 | root = Path(".") |
| 25 | for p in root.glob("*/*.md"): |
| 26 | print(p) |
Quick Decision Table
| Task | Reach for |
|---|---|
| Filesystem paths | pathlib.Path |
| CLI | argparse (or typer) |
| JSON/CSV | json / csv |
| Frequencies / queues | Counter / deque |
| Memoization | functools.cache / lru_cache |
| Time zones | zoneinfo |
| Tokens / secrets | secrets |
| Hashes / HMAC | hashlib / hmac |
| Thread/process pools | concurrent.futures |
| DTOs | dataclasses (+ Pydantic at boundaries) |
| HTTP client (prod) | httpx (see deep page) |
| Embedded DB | sqlite3 |
$Blueprint — Engineering Documentation·Section ID: PYTHON-STDLIB-REF·Revision: 1.0
Community
Get help on Slack, Discord or VIP
Stuck on a guide? Join the community and ask.