|$ curl https://forge-ai.dev/api/markdown?path=docs/python/packaging

$cat docs/python-—-packaging-&-distribution.md

updated Recently·16 min read·published

Python — Packaging & Distribution

◆Python◆Intermediate

Introduction

Python packaging is the process of bundling your code so others can install it with a single command. The ecosystem has evolved significantly: pyproject.toml is now the standard, replacing the older setup.py-only approach. This guide covers the modern workflow — from declaring dependencies to publishing on PyPI.

pyproject.toml — The Modern Standard

PEP 517 and PEP 518 established pyproject.toml as the definitive configuration file for Python projects. It replaces both setup.py and setup.cfg for most use cases. Tools like setuptools, flit, pdm, and hatch all read from it.

pyproject.toml

TOML

1	[build-system]
2	requires = ["setuptools>=64", "wheel"]
3	build-backend = "setuptools.build_meta"
4
5	[project]
6	name = "mypackage"
7	version = "0.1.0"
8	description = "A short description of my package"
9	readme = "README.md"
10	requires-python = ">=3.10"
11	license = { text = "MIT" }
12	authors = [
13	{ name = "Your Name", email = "you@example.com" },
14	]
15
16	classifiers = [
17	"Development Status :: 4 - Beta",
18	"Programming Language :: Python :: 3",
19	"License :: OSI Approved :: MIT License",
20	]
21
22	dependencies = [
23	"requests>=2.28",
24	"click>=8.0",
25	]
26
27	[project.optional-dependencies]
28	dev = ["pytest>=7", "ruff", "mypy"]
29	docs = ["mkdocs", "mkdocs-material"]
30	test = ["pytest-cov", "pytest-asyncio"]
31
32	[project.urls]
33	Homepage = "https://github.com/you/mypackage"
34	Documentation = "https://mypackage.readthedocs.io"
35	Source = "https://github.com/you/mypackage"

📝

note

setup.py is still supported for dynamic logic (e.g., C extensions), but static metadata belongs in pyproject.toml. Modern build backends like hatchling and flit_core require no setup.py at all.

pyproject.toml

TOML

1	# Minimal pyproject.toml using Hatch
2	[build-system]
3	requires = ["hatchling"]
4	build-backend = "hatchling.build"
5
6	[project]
7	name = "mypackage"
8	version = "0.1.0"
9	description = "Minimal example"
10	requires-python = ">=3.10"
11	dependencies = []

pip install & Editable Mode

pip is Python's default package installer. It reads the [project] table from pyproject.toml to resolve dependencies and install your package into the current environment.

terminal

Bash

1	# Install from local directory
2	pip install .
3
4	# Install in editable (development) mode
5	# Changes to source take effect immediately
6	pip install -e .
7
8	# Install from a Git repository
9	pip install git+https://github.com/user/repo.git
10
11	# Install a specific tag/branch
12	pip install git+https://github.com/user/repo.git@v1.0.0
13
14	# Install from a wheel file
15	pip install mypackage-1.0.0-py3-none-any.whl
16
17	# Install with extras (optional-dependencies)
18	pip install "mypackage[dev,test]"

ℹ

info

Use pip install -e . during development. It creates a symlink-like link so your package always reflects the latest source without reinstalling. This is essential when working on packages used by other projects.

terminal

Bash

1	# requirements.txt — flat list of pins
2	# Typically used for apps/deployments, not libraries
3	pip freeze > requirements.txt
4
5	# Install from requirements.txt
6	pip install -r requirements.txt

requirements.txt is a flat list of exact version pins, typically generated by pip freeze. It's suitable for applications where reproducibility matters, but libraries should declare loose version bounds in pyproject.toml instead.

Dependencies — Groups & Management

Modern pyproject.toml supports three levels of dependency specification: core, optional, and (via tools) dev-only. Neither requirements.txt nor setup.py are needed for dependency management.

pyproject.toml

TOML

1	[project]
2	# Core — installed with the package always
3	dependencies = [
4	"httpx>=0.27,<1",
5	"pydantic>=2.0",
6	"rich>=13",
7	]
8
9	# Optional — installed only when requested
10	[project.optional-dependencies]
11	cli = ["typer>=0.9"]
12	db = ["sqlalchemy>=2", "alembic>=1.12"]
13	async = ["httpx", "anyio>=4"]
14	all = ["mypackage[cli,db,async]"] # meta-extras
15
16	# Tool-specific dev deps — not in standard [project]
17	[tool.pdm.dev-dependencies]
18	dev = ["pytest", "ruff", "pre-commit"]
19
20	[tool.hatch.envs.default]
21	dependencies = ["pytest", "coverage"]

terminal

Bash

1	# Install optional extras
2	pip install "mypackage[cli]"
3	pip install "mypackage[all]"
4
5	# For dev dependencies with PDM
6	pdm add --dev pytest ruff
7
8	# With Hatch environments
9	hatch env create
10	hatch run pytest

⚠

warning

Never pin exact versions in library dependencies. Use upper/lower bounds (e.g. "httpx>=0.27,<1") to avoid conflicts with downstream projects. Exact pins belong in lock files.

Building — Wheel vs sdist

Python packages are distributed in two formats: wheels (.whl) and source distributions (.tar.gz sdist). Wheels are the preferred format — they're pre-built and install faster.

terminal

Bash

1	# Build both wheel and sdist (modern tool)
2	python -m build
3
4	# Or using setuptools directly
5	python setup.py sdist bdist_wheel
6
7	# Check the built artifacts
8	ls dist/
9	# mypackage-0.1.0-py3-none-any.whl
10	# mypackage-0.1.0.tar.gz

Wheel (.whl)

Pre-built ZIP archive. No build step during install. Faster, smaller, preferred.

sdist (.tar.gz)

Source archive with pyproject.toml. Requires build on install. Needed for platforms without wheel support.

dist

Python

1	# dist/ structure after python -m build
2	dist/
3	├── mypackage-0.1.0-py3-none-any.whl
4	│ # - py3: Python 3 only
5	│ # - none: no ABI tag (pure Python)
6	│ # - any: platform-independent
7	│
8	└── mypackage-0.1.0.tar.gz
9	# sdist — contains source + pyproject.toml

Publishing to PyPI

The Python Package Index (PyPI) is the official repository. Use build to create distributions and twine to upload them. For testing, use TestPyPI first.

terminal

Bash

1	# 1. Install publishing tools
2	pip install build twine
3
4	# 2. Build the package
5	python -m build
6
7	# 3. Upload to TestPyPI first
8	twine upload --repository-url https://test.pypi.org/legacy/ dist/*
9
10	# 4. Upload to real PyPI
11	twine upload dist/*
12
13	# You'll be prompted for your PyPI API token
14	# Or use a .pypirc file for automation

.pypirc

INI

1	# ~/.pypirc — store credentials for automation
2	[distutils]
3	index-servers =
4	pypi
5	testpypi
6
7	[pypi]
8	username = __token__
9	password = pypi-xxxxxxxxxxxxxxxxxxxx
10
11	[testpypi]
12	repository = https://test.pypi.org/legacy/
13	username = __token__
14	password = pypi-yyyyyyyyyyyyyyyyyyyy

⚠

warning

Never commit .pypirc or hardcode tokens in scripts. Use environment variables like TWINE_PASSWORD or GitHub Actions secrets. Always upload to TestPyPI before releasing to production.

.github/workflows/publish.yml

Bash

1	# GitHub Actions workflow for publishing
2	name: Publish to PyPI
3	on:
4	release:
5	types: [published]
6	jobs:
7	deploy:
8	runs-on: ubuntu-latest
9	steps:
10	- uses: actions/checkout@v4
11	- uses: actions/setup-python@v5
12	with:
13	python-version: "3.12"
14	- run: pip install build twine
15	- run: python -m build
16	- run: twine upload dist/*
17	env:
18	TWINE_USERNAME: __token__
19	TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

Versioning — Semver & Single Source

Python packages typically follow semantic versioning (MAJOR.MINOR.PATCH). Pre-release tags use a (alpha), b (beta), or rc (release candidate): 1.0.0a1, 2.3.0rc2.

src/mypackage/__init__.py

Python

1	# Option 1: importlib.metadata (Python 3.8+)
2	# Reads version from installed package metadata
3	# No need to maintain __version__ manually
4	from importlib.metadata import version
5
6	try:
7	__version__ = version("mypackage")
8	except Exception:
9	__version__ = "0.0.0"
10
11	# Option 2: __version__ in __init__.py
12	# Simple but requires manual sync with pyproject.toml
13	__version__ = "0.1.0"
14
15	# Option 3: Single-source from pyproject.toml
16	# Tools like hatch-vcs or setuptools-scm derive
17	# version from Git tags automatically
18	# pyproject.toml: version = "0.0.0" (placeholder)

pyproject.toml

TOML

1	# Using setuptools-scm — auto version from Git tags
2	[build-system]
3	requires = ["setuptools>=64", "setuptools-scm>=8"]
4	build-backend = "setuptools.build_meta"
5
6	[tool.setuptools_scm]
7	# No [project.version] needed — reads from git tag
8	# git tag v0.1.0 → version = "0.1.0"

📝

note

importlib.metadata.version() is the most reliable approach for libraries. It always reflects what's actually installed, avoiding drift between __version__ and the declared pyproject.toml version.

Entry Points & Console Scripts

Console scripts expose your package's functions as CLI commands. When the package is installed, pip creates a wrapper script in the environment's bin directory.

pyproject.toml

TOML

1	[project.scripts]
2	# Command name = "module:function"
3	mycli = "mypackage.cli:main"
4	mypkg = "mypackage.__main__:run"
5
6	# GUI scripts (no terminal popup on Windows)
7	[project.gui-scripts]
8	myapp = "mypackage.gui:launch"
9
10	# Entry points for plugins (extensible by other packages)
11	[project.entry-points."mypackage.plugins"]
12	formatter = "mypackage_plugin_format:Plugin"

cli.py

Python

1	# src/mypackage/cli.py
2	import click
3
4	@click.command()
5	@click.option("--name", default="world")
6	def main(name: str) -> None:
7	"""Greet someone from the CLI."""
8	click.echo(f"Hello, {name}!")
9
10	if __name__ == "__main__":
11	main()

terminal

Bash

1	# After pip install -e . or pip install .
2	mycli --name Alice
3	# → Hello, Alice!
4
5	# The script lives in your environment
6	which mycli
7	# → /path/to/venv/bin/mycli

ℹ

info

Use click or typer for CLI argument parsing. They integrate naturally with console scripts and provide help text, validation, and shell completion out of the box.

Best Practices

These patterns keep your package maintainable, reproducible, and easy to consume.

MANIFEST.in — Including Extra Files

By default, only files tracked by version control are included in the sdist. Use MANIFEST.in to add documentation, config files, or data.

MANIFEST.in

TEXT

1	# MANIFEST.in — extra files for sdist
2	include README.md
3	include LICENSE
4	include CHANGELOG.md
5	recursive-include docs/ *.md
6	recursive-include tests/ *.py
7	prune .github
8	prune .git

Lock Files — pip freeze, pip-compile & uv

requirements.txt generated by pip freeze is the simplest lock file. For more control, pip-compile (from pip-tools) resolves dependencies from pyproject.toml into a pinned lock file. uv provides a faster Rust-based alternative.

terminal

Bash

1	# pip freeze — snapshot of current environment
2	pip freeze > requirements.txt
3
4	# pip-compile — resolve from pyproject.toml
5	pip install pip-tools
6	pip-compile pyproject.toml # generates requirements.txt
7	pip-compile --extra dev pyproject.toml # with extras
8	pip-compile --output-file requirements-dev.txt \
9	--extra dev pyproject.toml
10
11	# uv — fast Rust-based alternative
12	uv pip compile pyproject.toml -o requirements.txt
13	uv lock # generates uv.lock (like package-lock.json)

ℹ

info

For libraries, commit a lock file only if you want reproducible CI/dev environments. Library dependencies should remain loose in pyproject.toml. For applications, always commit the lock file.

Private Package Indexes

Organizations often host internal packages on private indexes like devpi, Azure Artifacts, AWS CodeArtifact, or a self-hosted PyPI mirror.

terminal

Bash

1	# Install from a private index
2	pip install --index-url https://user:pass@private.example.com/simple/ mypackage
3
4	# Use multiple indexes (primary + fallback)
5	pip install --index-url https://private.example.com/simple/ \
6	--extra-index-url https://pypi.org/simple/ mypackage
7
8	# pip.conf — persistent index configuration
9	# ~/.config/pip/pip.conf (Linux/macOS)
10	# %APPDATA%\pip\pip.ini (Windows)
11
12	[global]
13	index-url = https://private.example.com/simple/
14	extra-index-url = https://pypi.org/simple/
15	trusted-host = private.example.com
16
17	# devpi — self-hosted PyPI server
18	pip install devpi-client
19	devpi use https://devpi.example.com
20	devpi login user --password
21	devpi index -c dev bases=root/pypi
22	devpi upload dist/*

Project Structure Checklist

tree

TEXT

1	mypackage/
2	├── pyproject.toml # Project metadata & dependencies
3	├── MANIFEST.in # Extra files for sdist
4	├── README.md
5	├── LICENSE
6	├── .github/
7	│ └── workflows/
8	│ ├── ci.yml # Test on push/PR
9	│ └── publish.yml # Publish on release
10	├── src/ # src layout (recommended)
11	│ └── mypackage/
12	│ ├── __init__.py
13	│ ├── __main__.py
14	│ ├── cli.py
15	│ └── core.py
16	├── tests/
17	│ └── test_core.py
18	└── docs/
19	└── index.md

$Blueprint — Engineering Documentation·Section ID: PYTHON-PKG·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor