|$ curl https://forge-ai.dev/api/markdown?path=docs/python/multiprocessing

$cat docs/python-—-multiprocessing.md

updated Recently·20 min read·published

Python — Multiprocessing

◆Python◆Advanced

Introduction

The multiprocessing module enables parallel execution by spawning separate OS processes, each with its own Python interpreter and memory space. Unlike threads, processes bypass the Global Interpreter Lock (GIL), making them ideal for CPU-bound workloads.

Each process runs independently with its own GIL, so multiple processes can execute Python bytecode simultaneously on multiple CPU cores. This comes at a cost: processes have higher memory overhead and require inter-process communication (IPC) mechanisms to share data.

Feature	Threading	Multiprocessing
GIL	Bound by GIL	Bypasses GIL
Memory	Shared (same process)	Separate (each process)
Overhead	Low (~1 MB per thread)	High (~10-50 MB per process)
Best for	I/O-bound tasks	CPU-bound tasks
Data sharing	Direct (needs locks)	IPC (Queue, Pipe, Manager)
Start time	Fast	Slower (fork/exec)

The module mirrors the threading API in many ways — Process mirrors Thread, and synchronization primitives like Lock, Semaphore, and Event are available in multiprocessing as well. The key difference is that each primitive operates across process boundaries rather than within a single process.

Process Class

The Process class represents an OS-level process. Its API is almost identical to threading.Thread: pass a target callable and args, then call start() and join(). Each process runs in its own memory space with a fresh Python interpreter.

process.py

Python

1	from multiprocessing import Process
2	import os
3	import time
4
5	# Simple process
6	def worker(name: str, delay: float = 1.0) -> None:
7	pid = os.getpid()
8	ppid = os.getppid()
9	print(f"[{name}] pid={pid}, ppid={ppid}")
10	time.sleep(delay)
11	print(f"[{name}] done")
12
13	if __name__ == "__main__":
14	p = Process(target=worker, args=("A", 1.5))
15	p.start()
16	print(f"Main: spawned process {p.pid}")
17	p.join()
18	print("Main: joined")
19
20	# Multiple processes — runs on separate cores
21	def cpu_work(n: int) -> int:
22	total = 0
23	for i in range(n):
24	total += i * i
25	return total
26
27	if __name__ == "__main__":
28	processes = []
29	for i in range(4):
30	p = Process(target=cpu_work, args=(10_000_000,), name=f"Worker-{i}")
31	processes.append(p)
32	p.start()
33
34	for p in processes:
35	p.join()
36	print("all processes complete")
37
38	# Custom Process subclass
39	class ComputeProcess(Process):
40	def __init__(self, n: int):
41	super().__init__()
42	self.n = n
43	self.result: int = 0
44
45	def run(self) -> None:
46	self.result = sum(i * i for i in range(self.n))
47
48	if __name__ == "__main__":
49	p = ComputeProcess(5_000_000)
50	p.start()
51	p.join()
52	print(p.result) # accessed after join
53
54	# Process identity
55	if __name__ == "__main__":
56	print(f"main pid: {os.getpid()}")
57	p = Process(target=worker, args=("X",))
58	p.start()
59	print(f"child pid: {p.pid}")
60	print(f"alive: {p.is_alive()}")
61	p.join()
62	print(f"exitcode: {p.exitcode}") # 0 = success
63	p.close() # release resources

⚠

warning

Always guard process-spawning code with if __name__ == "__main__":. On Windows and with the spawn start method, the module is re-imported by each child process. Without the guard, processes would be spawned recursively, causing a crash.

Process Pool (Pool)

The Pool class manages a fixed set of worker processes and distributes tasks among them. It provides multiple mapping methods: map (blocking, ordered), apply (single task), starmap (unpacking arguments), imap (lazy iterator, ordered), and imap_unordered (lazy iterator, any order). Use Pool when you have many independent tasks that can run in parallel.

pool.py

Python

1	from multiprocessing import Pool, cpu_count
2	import time
3
4	def square(n: int) -> int:
5	time.sleep(0.1) # simulate CPU work
6	return n * n
7
8	if __name__ == "__main__":
9	with Pool(processes=cpu_count()) as pool:
10	# map — blocking, results in order
11	results = pool.map(square, range(10))
12	print(results) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
13
14	# map with chunksize — reduces IPC overhead
15	results = pool.map(square, range(100), chunksize=10)
16
17	# apply — single task, returns result
18	with Pool(4) as pool:
19	result = pool.apply(square, args=(5,))
20	print(result) # 25
21
22	# apply_async — non-blocking, returns AsyncResult
23	with Pool(4) as pool:
24	async_result = pool.apply_async(square, args=(7,))
25	result = async_result.get(timeout=5)
26	print(result) # 49
27
28	# starmap — unpack argument tuples
29	def multiply(a: int, b: int) -> int:
30	return a * b
31
32	if __name__ == "__main__":
33	with Pool(4) as pool:
34	results = pool.starmap(multiply, [(2, 3), (4, 5), (6, 7)])
35	print(results) # [6, 20, 42]
36
37	# starmap_async — non-blocking starmap
38	with Pool(4) as pool:
39	async_result = pool.starmap_async(
40	multiply, [(2, 3), (4, 5), (6, 7)]
41	)
42	results = async_result.get()
43	print(results) # [6, 20, 42]
44
45	# imap — lazy, ordered, returns iterator
46	if __name__ == "__main__":
47	with Pool(4) as pool:
48	it = pool.imap(square, range(100), chunksize=10)
49	for val in it:
50	print(val, end=" ") # flows as results arrive
51
52	# imap_unordered — lazy, any order (faster)
53	if __name__ == "__main__":
54	with Pool(4) as pool:
55	it = pool.imap_unordered(square, range(100))
56	for val in it:
57	print(val, end=" ") # results arrive in arbitrary order

ℹ

info

Use imap_unordered when you don't need results in order — it is faster because workers return results as soon as they finish without waiting for earlier tasks. Use chunksize to reduce IPC overhead for large iterables. A good default is len(iterable) // (processes * 4).

Inter-Process Communication

Processes cannot share Python objects directly because each process has its own memory space. The multiprocessing module provides Queue and Pipe for inter-process communication. Queue is thread- and process-safe (uses locks internally) and is the preferred way to pass data between processes. Pipe is faster for two-way communication between two endpoints.

ipc.py

Python

1	from multiprocessing import Process, Queue, Pipe
2	import time
3
4	# ----- Queue (process-safe, producer-consumer) -----
5	def producer(q: Queue, count: int) -> None:
6	for i in range(count):
7	item = f"item-{i}"
8	q.put(item)
9	print(f"produced: {item}")
10	time.sleep(0.1)
11	q.put(None) # sentinel — signals "done"
12
13	def consumer(q: Queue, name: str) -> None:
14	while True:
15	item = q.get()
16	if item is None:
17	q.put(None) # pass sentinel to other consumers
18	break
19	print(f"[{name}] consumed: {item}")
20	time.sleep(0.2)
21
22	if __name__ == "__main__":
23	q: Queue = Queue()
24	procs = [
25	Process(target=producer, args=(q, 10)),
26	Process(target=consumer, args=(q, "A")),
27	Process(target=consumer, args=(q, "B")),
28	]
29	for p in procs: p.start()
30	for p in procs: p.join()
31
32	# ----- Pipe (bidirectional, two endpoints) -----
33	def pipe_worker(conn):
34	"""Worker sends and receives on one pipe end."""
35	msg = conn.recv() # receive from main
36	print(f"worker got: {msg}")
37	conn.send("pong") # send back
38	conn.close()
39
40	if __name__ == "__main__":
41	parent_conn, child_conn = Pipe()
42	p = Process(target=pipe_worker, args=(child_conn,))
43	p.start()
44	parent_conn.send("ping")
45	response = parent_conn.recv()
46	print(f"main got: {response}") # "pong"
47	p.join()
48
49	# Pipe with duplex=False — one-directional
50	if __name__ == "__main__":
51	receiver, sender = Pipe(duplex=False)
52	sender.send("one way")
53	print(receiver.recv()) # "one way"
54
55	# Queue vs Pipe decision
56	# Queue: multi-producer, multi-consumer, thread-safe, higher overhead
57	# Pipe: 2 endpoints only, faster, not thread-safe without locks

✓

best practice

Prefer Queue over Pipe for most use cases. Queue is simpler to use correctly — it handles locking internally and supports multiple producers/consumers. Use Pipe only for high-performance two-way communication between a single sender and receiver.

Shared Memory (Value & Array)

For primitive data types, multiprocessing.Value and multiprocessing.Array provide shared memory that is accessible by all processes. They use ctypes internally and synchronize access with an optional lock. Shared memory is faster than Queue or Pipe for simple numeric data but is limited to primitive types.

shared_memory.py

Python

1	from multiprocessing import Process, Value, Array, Lock
2
3	# ----- Value (single shared value) -----
4	def increment(counter: Value, lock: Lock, n: int) -> None:
5	for _ in range(n):
6	with lock:
7	counter.value += 1
8
9	if __name__ == "__main__":
10	counter = Value("i", 0) # "i" = signed int
11	lock = Lock()
12	processes = [
13	Process(target=increment, args=(counter, lock, 100_000))
14	for _ in range(4)
15	]
16	for p in processes: p.start()
17	for p in processes: p.join()
18	print(counter.value) # → 400_000 (correct with lock)
19
20	# Without lock — race condition
21	def unsafe_increment(counter: Value, n: int) -> None:
22	for _ in range(n):
23	counter.value += 1
24
25	if __name__ == "__main__":
26	counter = Value("i", 0)
27	processes = [
28	Process(target=unsafe_increment, args=(counter, 100_000))
29	for _ in range(4)
30	]
31	for p in processes: p.start()
32	for p in processes: p.join()
33	print(counter.value) # → likely less than 400_000
34
35	# cttypes type codes
36	# "i" = int, "f" = float, "d" = double, "c" = char
37	# "b" = signed char, "B" = unsigned char, "h" = short, "H" = unsigned short
38	# "l" = long, "L" = unsigned long
39
40	# ----- Array (shared array of primitives) -----
41	def worker(arr: Array, lock: Lock, index: int, value: int) -> None:
42	with lock:
43	arr[index] = value
44
45	if __name__ == "__main__":
46	arr = Array("i", 10) # array of 10 ints
47	lock = Lock()
48	processes = [
49	Process(target=worker, args=(arr, lock, i, i * 10))
50	for i in range(10)
51	]
52	for p in processes: p.start()
53	for p in processes: p.join()
54	print(list(arr)) # [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
55
56	# Array with typecode and initializer
57	if __name__ == "__main__":
58	arr = Array("d", [1.0, 2.0, 3.0]) # double array
59	print(arr[:]) # [1.0, 2.0, 3.0]
60
61	# RawArray (no lock) — faster but no synchronization
62	from multiprocessing.sharedctypes import RawArray
63
64	if __name__ == "__main__":
65	raw = RawArray("i", 100)

⚠

warning

Shared memory with Value and Array only works for primitive ctypes. You cannot share Python objects (dicts, lists, custom classes) this way — use a Manager instead. Always protect concurrent writes with a Lock to avoid race conditions.

Manager (Shared Complex Objects)

The Manager creates a separate server process that holds Python objects, making them accessible to other processes via proxies. This allows sharing complex data structures (lists, dicts, queues, Namespace) across processes without manual serialization. The manager process handles all synchronization, but there is overhead from IPC for every access.

manager.py

Python

1	from multiprocessing import Process, Manager, Lock
2
3	# Shared dictionary via Manager
4	def worker_dict(d: dict, lock: Lock, key: str, value: int) -> None:
5	with lock:
6	d[key] = value
7
8	if __name__ == "__main__":
9	with Manager() as manager:
10	d = manager.dict()
11	lock = Lock()
12	processes = [
13	Process(target=worker_dict, args=(d, lock, f"k{i}", i))
14	for i in range(5)
15	]
16	for p in processes: p.start()
17	for p in processes: p.join()
18	print(dict(d)) # {"k0": 0, "k1": 1, "k2": 2, ...}
19
20	# Shared list
21	def worker_list(lst, index: int, value: int) -> None:
22	lst.append(value)
23
24	if __name__ == "__main__":
25	with Manager() as manager:
26	lst = manager.list()
27	processes = [
28	Process(target=worker_list, args=(lst, i, i * 10))
29	for i in range(5)
30	]
31	for p in processes: p.start()
32	for p in processes: p.join()
33	print(list(lst)) # [0, 10, 20, 30, 40]
34
35	# Manager.Namespace — attribute-based shared state
36	def worker_ns(ns, key: str, value: int) -> None:
37	setattr(ns, key, value)
38
39	if __name__ == "__main__":
40	with Manager() as manager:
41	ns = manager.Namespace()
42	ns.counter = 0
43	processes = [
44	Process(target=worker_ns, args=(ns, f"attr{i}", i))
45	for i in range(4)
46	]
47	for p in processes: p.start()
48	for p in processes: p.join()
49	print(ns.attr0, ns.attr1, ns.attr2, ns.attr3)
50
51	# Manager.Value and Manager.Array
52	if __name__ == "__main__":
53	with Manager() as manager:
54	val = manager.Value("i", 0)
55	arr = manager.list([1, 2, 3])
56	print(val.value) # 0
57	print(arr) # [1, 2, 3]
58
59	# Custom class via Manager
60	class Counter:
61	def __init__(self):
62	self.value = 0
63	def increment(self):
64	self.value += 1
65
66	if __name__ == "__main__":
67	with Manager() as manager:
68	counter = manager.Namespace()
69	counter.value = 0
70	# Custom objects need explicit registration with Manager
71	# or use Namespace for simple attribute-based state

📝

note

Manager proxies are slower than shared memory (Value/Array) because every attribute access goes through IPC to the manager server process. For high-frequency counters or numeric arrays, use Value or Array. Use Manager for complex data structures where convenience matters more than raw throughput.

Synchronization Primitives

The multiprocessing module provides the same synchronization primitives as threading: Lock, RLock, Semaphore, BoundedSemaphore, Event, Condition, and Barrier. These work across processes, using OS-level synchronization objects rather than Python-level ones.

Primitive	Purpose	Cross-process
Lock	Exclusive access (mutex)	Yes
RLock	Reentrant exclusive access	Yes
Semaphore	Limit concurrent access count	Yes
Event	Signal between processes	Yes
Condition	Wait/notify pattern	Yes
Barrier	Sync N processes at a point	Yes

synchronization.py

Python

1	from multiprocessing import Process, Lock, Semaphore, Event
2	import time
3
4	# ----- Lock (mutex) -----
5	def safe_work(lock: Lock, shared_list: list) -> None:
6	with lock:
7	shared_list.append(os.getpid())
8
9	if __name__ == "__main__":
10	from multiprocessing import Manager
11	with Manager() as m:
12	lock = Lock()
13	data = m.list()
14	procs = [Process(target=safe_work, args=(lock, data))
15	for _ in range(5)]
16	for p in procs: p.start()
17	for p in procs: p.join()
18	print(len(data)) # 5
19
20	# ----- Semaphore (limit concurrency) -----
21	def limited_task(sem: Semaphore, n: int) -> None:
22	with sem:
23	print(f"[P{n}] running")
24	time.sleep(1)
25	print(f"[P{n}] done")
26
27	if __name__ == "__main__":
28	sem = Semaphore(2) # max 2 concurrent
29	procs = [Process(target=limited_task, args=(sem, i))
30	for i in range(6)]
31	for p in procs: p.start()
32	for p in procs: p.join()
33	# At most 2 processes run simultaneously
34
35	# ----- Event (signal across processes) -----
36	def waiter(event: Event, name: str) -> None:
37	print(f"[{name}] waiting for event")
38	event.wait()
39	print(f"[{name}] starting work")
40
41	def signaller(event: Event) -> None:
42	time.sleep(1)
43	print("signaller: setting event")
44	event.set()
45
46	if __name__ == "__main__":
47	event = Event()
48	procs = [
49	Process(target=waiter, args=(event, f"W{i}"))
50	for i in range(3)
51	] + [Process(target=signaller, args=(event,))]
52	for p in procs: p.start()
53	for p in procs: p.join()
54
55	# ----- Barrier (sync N processes) -----
56	from multiprocessing import Barrier
57
58	def worker_barrier(b: Barrier, name: str) -> None:
59	print(f"[{name}] before barrier")
60	b.wait() # waits until all N processes reach barrier
61	print(f"[{name}] after barrier")
62
63	if __name__ == "__main__":
64	b = Barrier(3) # 3 processes must sync
65	procs = [
66	Process(target=worker_barrier, args=(b, f"W{i}"))
67	for i in range(3)
68	]
69	for p in procs: p.start()
70	for p in procs: p.join()
71
72	# Synchronization primitives are picklable
73	# — can be passed as arguments to Process

ℹ

info

Synchronization primitives in multiprocessing use OS-level mutexes/semaphores, making them more expensive than their threading counterparts. Minimise the time spent holding locks across processes. Consider lock-free designs using Queue or Pool.map where possible.

ProcessPoolExecutor

The concurrent.futures.ProcessPoolExecutor provides a high-level API for process-based parallelism. It mirrors ThreadPoolExecutor but uses processes instead of threads, bypassing the GIL. You submit callables and receive Future objects. The API is simpler than multiprocessing.Pool and is the recommended starting point for CPU-bound parallelism.

process_pool_executor.py

Python

1	from concurrent.futures import (
2	ProcessPoolExecutor,
3	as_completed,
4	wait,
5	FIRST_COMPLETED,
6	)
7	import time
8
9	# CPU-bound task — benefits from multiple processes
10	def is_prime(n: int) -> tuple[int, bool]:
11	if n < 2:
12	return n, False
13	for i in range(2, int(n ** 0.5) + 1):
14	if n % i == 0:
15	return n, False
16	return n, True
17
18	if __name__ == "__main__":
19	numbers = [n for n in range(10_000, 10_100)]
20
21	# Basic map — results in order
22	with ProcessPoolExecutor(max_workers=4) as executor:
23	results = list(executor.map(is_prime, numbers))
24	for n, prime in results:
25	print(f"{n}: {'prime' if prime else 'composite'}")
26
27	# submit + as_completed — process as results arrive
28	with ProcessPoolExecutor(max_workers=4) as executor:
29	futures = {
30	executor.submit(is_prime, n): n
31	for n in numbers
32	}
33	for future in as_completed(futures):
34	n, prime = future.result()
35	print(f"[done] {n}: {'prime' if prime else 'composite'}")
36
37	# wait — block until condition
38	with ProcessPoolExecutor(max_workers=4) as executor:
39	futures = [executor.submit(is_prime, n) for n in numbers[:10]]
40	done, pending = wait(futures, timeout=5.0,
41	return_when=FIRST_COMPLETED)
42	print(f"{len(done)} done, {len(pending)} pending")
43
44	# Exception handling
45	def risky(n: int) -> int:
46	if n < 0:
47	raise ValueError(f"negative: {n}")
48	return n * 2
49
50	if __name__ == "__main__":
51	with ProcessPoolExecutor(max_workers=2) as executor:
52	future = executor.submit(risky, -1)
53	try:
54	result = future.result()
55	except ValueError as e:
56	print(f"caught: {e}")
57
58	# Context manager ensures clean shutdown
59	with ProcessPoolExecutor(max_workers=4) as executor:
60	future = executor.submit(is_prime, 97)
61	print(future.result()) # (97, True)
62
63	# Max workers — defaults to cpu_count()
64	from multiprocessing import cpu_count
65	print(cpu_count()) # e.g., 8 (logical cores)
66
67	# Tuning max_workers
68	# CPU-bound: worker_count <= cpu_count()
69	# Mixed I/O+CPU: worker_count = cpu_count() * 2

✓

best practice

Use ProcessPoolExecutor over multiprocessing.Pool when possible. The API matches ThreadPoolExecutor, making it easy to switch between threads and processes by changing one class name. Use multiprocessing.Pool when you need imap_unordered, starmap, or advanced features.

Start Methods: fork, spawn, forkserver

The multiprocessing module supports three start methods for creating new processes. The method determines how the child process's memory is initialized. Each has tradeoffs for speed, safety, and compatibility. The default method depends on the platform: fork on Linux/macOS (before 3.14), spawn on Windows and macOS (3.14+).

Method	How it works	Speed	Thread-safe
fork	Copies parent memory (copy-on-write)	Fast	No
spawn	Starts fresh interpreter, imports parent module	Slow	Yes
forkserver	Fork from dedicated server process	Medium	Yes

start_methods.py

Python

1	import multiprocessing as mp
2	import os
3	import sys
4
5	# Check current start method
6	print(mp.get_start_method()) # e.g., "fork" or "spawn"
7	print(mp.get_all_start_methods()) # available on this platform
8
9	# Set start method — must be called BEFORE any Process creation
10	if __name__ == "__main__":
11	mp.set_start_method("spawn", force=True)
12	# force=True overrides previous set (if any)
13
14	# Using context manager for method scope
15	if __name__ == "__main__":
16	with mp.get_context("spawn") as ctx:
17	p = ctx.Process(target=worker, args=(42,))
18	p.start()
19	p.join()
20
21	# Fork — classic Unix fork
22	# Pros: fastest, parent state inherited (no re-import)
23	# Cons: not thread-safe, can deadlock with threads
24	# macOS 3.14+: spawn is default; fork deprecated
25
26	if __name__ == "__main__":
27	print(f"start method: {mp.get_start_method()}")
28
29	# Spawn — always safe
30	# Pros: thread-safe, works everywhere
31	# Cons: slower (re-imports module), # requires __main__ guard
32	# Pickles args to send to child
33
34	if __name__ == "__main__":
35	ctx = mp.get_context("spawn")
36	q = ctx.Queue()
37	p = ctx.Process(target=consumer, args=(q,))
38	p.start()
39	q.put("data")
40	p.join()
41
42	# Forkserver — best of both
43	# Pros: thread-safe, faster than spawn
44	# Cons: Unix-only, requires forking server
45	# Server starts once, then forks for each new process
46
47	# Platform-specific defaults
48	# Linux: fork (default), spawn, forkserver
49	# macOS 3.13: fork (default; deprecated in 3.14)
50	# macOS 3.14: spawn (default)
51	# Windows: spawn (only option)
52
53	# Deprecation note
54	if sys.platform == "darwin":
55	import warnings
56	warnings.filterwarnings("ignore",
57	message=".fork start method is deprecated.")
58
59	# Safety check — avoid fork with threads
60	if mp.get_start_method() == "fork" and threading.active_count() > 1:
61	print("WARNING: fork with active threads is unsafe")

⚠

warning

On macOS with Python 3.14+, the default start method changed from fork to spawn. Code relying on fork-inherited global state will break. Always use if __name__ == "__main__": guards for compatibility with spawn. Use forkserver for a good balance of speed and safety on Unix systems.

Common Pitfalls

Multiprocessing introduces unique challenges that differ from threading. Understanding these pitfalls saves hours of debugging. The most common issues involve pickling (serialization), global state, and platform-specific behavior.

pitfalls.py

Python

1	# ----- Pitfall 1: Pickling Errors -----
2	# Arguments sent to processes must be picklable (serializable).
3	# Lambdas, nested functions, and instance methods are NOT picklable.
4
5	def bad():
6	# This will raise AttributeError:
7	# Can't pickle local object 'bad.<locals>.nested'
8	def nested():
9	return 42
10	with Pool(4) as pool:
11	pool.map(nested, range(10))
12
13	# Fix: use module-level functions
14	def module_level(x: int) -> int:
15	return x * 2
16
17	if __name__ == "__main__":
18	with Pool(4) as pool:
19	print(pool.map(module_level, range(5)))
20
21	# ----- Pitfall 2: Global State -----
22	# Each process has its own copy of global variables.
23	# Modifications in child processes do NOT affect the parent.
24
25	COUNTER = 0
26
27	def increment_global():
28	global COUNTER
29	COUNTER += 1
30
31	if __name__ == "__main__":
32	procs = [Process(target=increment_global) for _ in range(5)]
33	for p in procs: p.start()
34	for p in procs: p.join()
35	print(COUNTER) # 0 — parent's global unchanged!
36
37	# Fix: use shared memory or Manager
38	if __name__ == "__main__":
39	counter = Value("i", 0)
40	lock = Lock()
41	procs = [Process(target=safe_increment, args=(counter, lock))
42	for _ in range(5)]
43	for p in procs: p.start()
44	for p in procs: p.join()
45	print(counter.value) # 5
46
47	# ----- Pitfall 3: Fork + Threads Deadlock -----
48	# With fork start method, forking from a process
49	# with active threads can cause deadlocks.
50	# The child inherits locks held in unknown state.
51
52	# Fix: use spawn or forkserver start method
53
54	# ----- Pitfall 4: File Descriptor Leaks -----
55	# Each process opens its own file descriptors.
56	# Always close file handles in child processes.
57	import resource
58
59	def leak_example():
60	# Open files in parent, use in child —
61	# child inherits FDs with fork method
62	pass # use 'with open()' for auto-close
63
64	# ----- Pitfall 5: Too Many Processes -----
65	# More processes than CPU cores causes context-switching
66	# overhead without throughput gain.
67
68	if __name__ == "__main__":
69	from multiprocessing import cpu_count
70	N_WORKERS = cpu_count() # optimal for CPU-bound
71	# For I/O-bound, N_WORKERS can be higher
72	# For memory-bound, N_WORKERS should be lower
73
74	# ----- Pitfall 6: Non-deterministic Output -----
75	# Print statements from different processes interleave.
76	# Use a Queue for structured logging.
77
78	def safe_logger(q: Queue, msg: str) -> None:
79	q.put((os.getpid(), msg))
80
81	if __name__ == "__main__":
82	with Manager() as m:
83	log_q = m.Queue()
84	# Workers put to log_q instead of print()
85	# Main process drains log_q sequentially
86
87	# ----- Pitfall 7: Deadlock with Queue -----
88	# Queue.put() blocks if the queue is full.
89	# Queue.join() blocks if tasks are not marked done.
90	# Deadlock can occur if join() is called before
91	# all items are consumed.
92
93	# Fix: always call q.task_done() after each q.get()
94	# and ensure producer calls q.join() after all items queued
95
96	# ----- Pitfall 8: Shared Resource Cleanup -----
97	# Manager and Pool resources must be properly cleaned up.
98	# Always use context managers ('with' statement).
99
100	# Bad:
101	pool = Pool(4)
102	pool.map(work, data)
103	# pool not closed — processes may linger
104
105	# Good:
106	with Pool(4) as pool:
107	pool.map(work, data)
108	# processes terminated on block exit

danger

Pickling errors are the most common multiprocessing frustration. If you hit AttributeError: Can't pickle local object, move the function to module level or use dill as the serializer (multiprocessing.set_start_method("spawn") does not fix pickle issues). For complex callables, consider structuring your code as top-level functions or using pathos.multiprocessing which uses dill.

Additional debugging tips: use maxtasksperchild in Pool to limit memory leaks from long-running workers, set Pool(processes, maxtasksperchild=100). For hanging processes, use pool.map(..., timeout=...) or future.result(timeout=...). On Linux, use strace -f -p <pid> to trace system calls of child processes.

$Blueprint — Engineering Documentation·Section ID: PYTHON-MP·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor