|$ curl https://forge-ai.dev/api/markdown?path=docs/python/logging

$cat docs/python-—-logging.md

updated Recently·14 min read·published

Python — Logging

◆Python◆Intermediate

Introduction

Python's logging module provides a flexible event-logging system for applications and libraries. Unlike print(), logging supports severity levels, output routing, structured data, and runtime configuration — making it essential for debugging and monitoring in production.

Feature	print()	logging
Severity Levels	None	DEBUG through CRITICAL
Output Routing	stdout only	Files, sockets, syslog, HTTP
Runtime Control	Edit source code	Config files, API calls
Structured Data	Manual formatting	Formatters, extra params, JSON
Filtering	Manual if/else	Built-in filter system

Log Levels

The logging module defines five standard severity levels in ascending order. Setting a level on a logger means it only processes messages at that level or higher.

log_levels.py

Python

1	import logging
2
3	# Numeric values (ascending severity)
4	# CRITICAL = 50 — program may be about to crash
5	# ERROR = 40 — something failed
6	# WARNING = 30 — unexpected but recoverable
7	# INFO = 20 — confirmation that things work as expected
8	# DEBUG = 10 — detailed diagnostic info
9
10	# All-in-one convenience function (configures root logger)
11	logging.basicConfig(level=logging.DEBUG)
12	logging.debug("detailed diagnostic")
13	logging.info("operation succeeded")
14	logging.warning("disk usage at 87%%")
15	logging.error("failed to connect to database")
16	logging.critical("out of memory — shutting down")
17
18	# Output:
19	# DEBUG:root:detailed diagnostic
20	# INFO:root:operation succeeded
21	# WARNING:root:disk usage at 87%%
22	# ERROR:root:failed to connect to database
23	# CRITICAL:root:out of memory — shutting down
24
25	# NOTSET = 0 — special level that propagates to parent

ℹ

info

Use DEBUG for variable values and control flow. INFO for high-level events (startup, requests handled). WARNING for degraded but functioning state. ERROR for failures that need attention. CRITICAL for imminent shutdown scenarios.

basicConfig

logging.basicConfig() is the simplest way to configure logging. It sets up a StreamHandler on stderr with the root logger. It only takes effect the first time it is called — subsequent calls are ignored unless force=True is passed.

basic_config.py

Python

1	import logging
2
3	# Basic setup — stdout, DEBUG level, default format
4	logging.basicConfig(level=logging.DEBUG)
5
6	# Custom format
7	logging.basicConfig(
8	level=logging.INFO,
9	format="%(asctime)s [%(levelname)s] %(message)s",
10	datefmt="%Y-%m-%d %H:%M:%S",
11	)
12
13	# Write to file instead of stderr
14	logging.basicConfig(
15	level=logging.DEBUG,
16	filename="app.log",
17	filemode="w", # "w" = overwrite, "a" = append (default)
18	format="%(asctime)s %(name)s %(levelname)s %(message)s",
19	)
20
21	# Force reconfiguration (useful in scripts / notebooks)
22	logging.basicConfig(
23	level=logging.DEBUG,
24	format="%(levelname)s: %(message)s",
25	force=True,
26	)
27
28	# Log to both file and console (basicConfig can't do this — use root logger directly)
29	root = logging.getLogger()
30	root.setLevel(logging.DEBUG)
31
32	console = logging.StreamHandler()
33	console.setLevel(logging.INFO)
34	file_handler = logging.FileHandler("app.log")
35	file_handler.setLevel(logging.DEBUG)
36
37	fmt = logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s")
38	console.setFormatter(fmt)
39	file_handler.setFormatter(fmt)
40
41	root.addHandler(console)
42	root.addHandler(file_handler)

Logger Objects

Logger instances are the main interface for logging. Always use logging.getLogger(__name__) to get a named logger — never instantiate Logger() directly. Each named logger is a singleton; calling getLogger with the same name returns the same object.

logger_objects.py

Python

1	import logging
2
3	# Get a named logger (returns same instance each time)
4	logger = logging.getLogger("myapp.db")
5
6	# Set level on specific logger (overrides parent)
7	logger.setLevel(logging.WARNING)
8
9	# Log messages — format strings are lazy (only interpolated if level enabled)
10	logger.debug("query: SELECT * FROM users") # skipped if level > DEBUG
11	logger.info("connected to database") # skipped at WARNING level
12	logger.warning("slow query detected (2.3s)")
13	logger.error("connection pool exhausted")
14	logger.critical("database unreachable")
15
16	# Log with context via % formatting (lazy — avoids string interpolation cost)
17	logger.info("user %s logged in from %s", username, ip)
18
19	# Log with exception traceback
20	try:
21	result = 1 / 0
22	except ZeroDivisionError:
23	logger.exception("math error occurred")
24	# Automatically appends traceback to the log message
25
26	# Logger methods and their equivalent
27	# logger.debug(msg) == logging.getLogger().debug(msg)
28	# logger.info(msg) == logging.info(msg)
29	# logger.warning(msg) == logging.warning(msg)
30	# logger.error(msg) == logging.error(msg)
31	# logger.critical(msg) == logging.critical(msg)
32
33	# Disable all logging on a logger (and its children)
34	logger.disabled = True

✓

best practice

Always use logger.warning() not logger.warn() — the latter is deprecated. Use %s formatting in log calls rather than f-strings, so the string is only constructed when the message is actually emitted. For exception logging, use logger.exception() inside except blocks to automatically capture the traceback.

Handlers

Handlers determine where log records go. A logger can have multiple handlers, each with its own level and formatter — enabling simultaneous console output, file logging, and remote aggregation.

handlers.py

Python

1	import logging
2	from logging.handlers import (
3	StreamHandler,
4	FileHandler,
5	RotatingFileHandler,
6	TimedRotatingFileHandler,
7	)
8
9	# StreamHandler — writes to sys.stderr (or a stream)
10	stream = StreamHandler()
11	stream.setLevel(logging.WARNING)
12	stream.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
13
14	# FileHandler — writes to a file
15	file_h = FileHandler("app.log", mode="a", encoding="utf-8")
16	file_h.setLevel(logging.DEBUG)
17
18	# RotatingFileHandler — rotates by file size
19	rotating = RotatingFileHandler(
20	"app.log",
21	maxBytes=5_000_000, # 5 MB per file
22	backupCount=3, # keep 3 old files (app.log.1, .2, .3)
23	)
24	rotating.setLevel(logging.INFO)
25
26	# TimedRotatingFileHandler — rotates by time interval
27	timed = TimedRotatingFileHandler(
28	"app.log",
29	when="midnight", # rotate at midnight
30	interval=1,
31	backupCount=30, # keep 30 days
32	encoding="utf-8",
33	)
34
35	# Alternative 'when' values: 'S' (seconds), 'M' (minutes), 'H' (hours),
36	# 'D' (days), 'W0'-'W6' (weekday 0=Mon), 'midnight'
37
38	# Attach handlers to a logger
39	logger = logging.getLogger("myapp")
40	logger.setLevel(logging.DEBUG)
41	logger.addHandler(stream)
42	logger.addHandler(file_h)
43	logger.addHandler(rotating)
44	logger.addHandler(timed)

handlers_advanced.py

Python

1	import logging
2	from logging.handlers import SocketHandler, SysLogHandler, HTTPHandler
3
4	# SocketHandler — send to a remote log server (TCP)
5	socket_h = SocketHandler("logserver.example.com", port=9020)
6
7	# SysLogHandler — send to system syslog
8	syslog_h = SysLogHandler(address="/dev/log") # Unix
9	syslog_h = SysLogHandler(address=("localhost", 514)) # UDP
10
11	# HTTPHandler — send to a web server via POST
12	http_h = HTTPHandler(
13	"https://log.example.com",
14	method="POST",
15	secure=True,
16	credentials=("user", "pass"),
17	)
18
19	# NullHandler — prevents "No handler found" warnings in libraries
20	# Libraries should use this, not basicConfig
21	lib_logger = logging.getLogger("mylib")
22	lib_logger.addHandler(logging.NullHandler())

📝

note

NullHandler is the correct handler for libraries. It absorbs all log records, preventing “No handlers could be found for logger X” warnings. Application code is responsible for configuring actual handlers.

Formatters

Formatters control the layout of log output. They use %(attribute)s interpolation with standard LogRecord attributes, and support %(asctime)s, %(name)s, %(levelname)s, and many others.

formatters.py

Python

1	import logging
2
3	# Default format: "%(levelname)s:%(name)s:%(message)s"
4
5	# Custom formatter with full context
6	fmt = logging.Formatter(
7	fmt="%(asctime)s \| %(name)-20s \| %(levelname)-8s \| %(message)s",
8	datefmt="%Y-%m-%d %H:%M:%S",
9	)
10
11	# Attach to a handler
12	handler = logging.StreamHandler()
13	handler.setFormatter(fmt)
14
15	# Useful LogRecord attributes:
16	# %(asctime)s — human-readable timestamp
17	# %(name)s — logger name
18	# %(levelname)s — level name (DEBUG, INFO, ...)
19	# %(levelno)s — numeric level (10, 20, 30, ...)
20	# %(message)s — the logged message
21	# %(module)s — module name (filename without .py)
22	# %(funcName)s — function name
23	# %(lineno)d — line number
24	# %(pathname)s — full file path
25	# %(process)d — process ID
26	# %(thread)d — thread ID
27	# %(threadName)s — thread name
28	# %(filename)s — filename
29	# %(relativeCreated)d — time in ms since logging module loaded
30
31	# Different formatters per handler
32	console_fmt = logging.Formatter("%(levelname)s: %(message)s")
33	file_fmt = logging.Formatter(
34	"%(asctime)s [%(levelname)s] %(name)s.%(funcName)s:%(lineno)d — %(message)s"
35	)
36
37	console = logging.StreamHandler()
38	console.setFormatter(console_fmt)
39
40	file_h = logging.FileHandler("debug.log")
41	file_h.setFormatter(file_fmt)
42
43	logger = logging.getLogger("app")
44	logger.addHandler(console)
45	logger.addHandler(file_h)

Logger Hierarchy

Loggers form a hierarchy separated by dots. logging.getLogger("a.b") is a child of logging.getLogger("a"). Children inherit handlers and levels from their parent by default unless propagate=False is set.

hierarchy.py

Python

1	import logging
2
3	# Logger hierarchy
4	# root (empty name)
5	# └── myapp
6	# ├── myapp.db
7	# ├── myapp.api
8	# └── myapp.api.auth
9
10	root = logging.getLogger() # root logger
11	myapp = logging.getLogger("myapp")
12	db = logging.getLogger("myapp.db")
13	api = logging.getLogger("myapp.api")
14	auth = logging.getLogger("myapp.api.auth")
15
16	# Setting level — children propagate to parent if no explicit level set
17	root.setLevel(logging.WARNING)
18	db.setLevel(logging.DEBUG) # overrides inherited WARNING
19
20	# Propagation — child messages bubble up to parent handlers
21	# By default propagate=True
22
23	# Scenario: child has own handler AND parent has handler
24	db = logging.getLogger("myapp.db")
25	db.setLevel(logging.DEBUG)
26	db_handler = logging.StreamHandler()
27	db_handler.setFormatter(logging.Formatter("[DB] %(message)s"))
28	db.addHandler(db_handler)
29
30	# If root also has a handler, the message goes to BOTH
31	# To prevent duplicate output, set propagate=False:
32	db.propagate = False # stop bubbling to root
33
34	# Disable an entire hierarchy
35	for name in logging.Logger.manager.loggerDict:
36	logging.getLogger(name).setLevel(logging.CRITICAL)
37
38	# List all loggers
39	for name, logger in sorted(logging.Logger.manager.loggerDict.items()):
40	print(name, getattr(logger, "level", "default"))

⚠

warning

The most common logging bug is duplicate messages: a child logger emits a record, it propagates to the parent, and both have handlers writing the same message. Fix this by setting propagate=False on the child or removing duplicate handlers.

Configuration

For complex applications, configure logging programmatically with dictConfig() rather than calling basicConfig() multiple times. This gives you full control over loggers, handlers, formatters, and levels in a single declaration.

dict_config.py

Python

1	import logging.config
2
3	# dictConfig — the recommended approach for applications
4	LOGGING_CONFIG = {
5	"version": 1,
6	"disable_existing_loggers": False, # keep loggers created before config
7
8	"formatters": {
9	"standard": {
10	"format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
11	"datefmt": "%Y-%m-%d %H:%M:%S",
12	},
13	"detailed": {
14	"format": "%(asctime)s [%(levelname)s] %(name)s.%(funcName)s:%(lineno)d — %(message)s",
15	},
16	},
17
18	"handlers": {
19	"console": {
20	"class": "logging.StreamHandler",
21	"level": "INFO",
22	"formatter": "standard",
23	},
24	"file": {
25	"class": "logging.handlers.RotatingFileHandler",
26	"filename": "app.log",
27	"maxBytes": 10_000_000,
28	"backupCount": 5,
29	"level": "DEBUG",
30	"formatter": "detailed",
31	},
32	"error_file": {
33	"class": "logging.FileHandler",
34	"filename": "errors.log",
35	"level": "ERROR",
36	"formatter": "detailed",
37	},
38	},
39
40	"loggers": {
41	"myapp": {
42	"level": "DEBUG",
43	"handlers": ["console", "file"],
44	"propagate": False,
45	},
46	"myapp.db": {
47	"level": "DEBUG",
48	"handlers": ["file"],
49	"propagate": True, # bubble up to myapp's console handler
50	},
51	},
52
53	"root": {
54	"level": "WARNING",
55	"handlers": ["console"],
56	},
57	}
58
59	logging.config.dictConfig(LOGGING_CONFIG)
60
61	# Now get loggers — they inherit the config above
62	logger = logging.getLogger("myapp")
63	logger.debug("app started") # goes to file only
64	logger.info("ready") # goes to console + file
65	logger.warning("low memory") # goes to console + file
66
67	db_logger = logging.getLogger("myapp.db")
68	db_logger.debug("SELECT * FROM users") # file only + console via myapp

file_config.py

Python

1	# fileConfig — simpler file-based alternative
2	# logging.conf (INI format):
3	# [loggers]
4	# keys=root,myapp
5	#
6	# [handlers]
7	# keys=console,file
8	#
9	# [formatters]
10	# keys=standard
11	#
12	# [logger_root]
13	# level=WARNING
14	# handlers=console
15	#
16	# [logger_myapp]
17	# level=DEBUG
18	# handlers=console,file
19	# propagate=0
20	#
21	# [handler_console]
22	# class=StreamHandler
23	# level=INFO
24	# formatter=standard
25	# args=(sys.stderr,)
26	#
27	# [handler_file]
28	# class=FileHandler
29	# level=DEBUG
30	# formatter=standard
31	# args=("app.log", "a")
32	#
33	# [formatter_standard]
34	# format=%(asctime)s [%(levelname)s] %(name)s: %(message)s
35
36	# Load from file
37	import logging.config
38	logging.config.fileConfig("logging.conf", disable_existing_loggers=False)

✓

best practice

Always use "disable_existing_loggers": False in dictConfig unless you specifically want to silence loggers created before configuration. This is the most common footgun — existing loggers silently lose their handlers when dictConfig is called.

Structured Logging

Structured logging outputs machine-parseable records (typically JSON). This enables log aggregation, filtering, and analysis in tools like Elasticsearch, Datadog, and CloudWatch. Use the extra parameter or a custom JSON formatter.

structured_logging.py

Python

1	import logging
2	import json
3	from datetime import datetime, timezone
4
5	# JSON formatter — produces structured log records
6	class JSONFormatter(logging.Formatter):
7	def format(self, record):
8	log_data = {
9	"timestamp": datetime.now(timezone.utc).isoformat(),
10	"level": record.levelname,
11	"logger": record.name,
12	"message": record.getMessage(),
13	"module": record.module,
14	"function": record.funcName,
15	"line": record.lineno,
16	}
17	# Include extra fields passed via logger.info(msg, extra={...})
18	for key in ["user_id", "request_id", "duration_ms", "status_code"]:
19	if hasattr(record, key):
20	log_data[key] = getattr(record, key)
21
22	# Include exception info if present
23	if record.exc_info and record.exc_info[0] is not None:
24	log_data["exception"] = self.formatException(record.exc_info)
25
26	return json.dumps(log_data, default=str)
27
28	# Setup
29	logger = logging.getLogger("myapp")
30	handler = logging.StreamHandler()
31	handler.setFormatter(JSONFormatter())
32	logger.addHandler(handler)
33	logger.setLevel(logging.DEBUG)
34
35	# Log with extra context
36	logger.info("user logged in", extra={"user_id": "u_abc123", "request_id": "req_42"})
37	logger.info("request completed", extra={"duration_ms": 142, "status_code": 200})
38
39	# Output:
40	# {"timestamp": "2026-07-09T14:30:00+00:00", "level": "INFO", "logger": "myapp",
41	# "message": "user logged in", "user_id": "u_abc123", "request_id": "req_42", ...}
42
43	# LoggerAdapter — attach context to all messages from a logger
44	class ContextLoggerAdapter(logging.LoggerAdapter):
45	def process(self, msg, kwargs):
46	extra = kwargs.get("extra", {})
47	extra.update(self.extra)
48	kwargs["extra"] = extra
49	return msg, kwargs
50
51	# Usage
52	base_logger = logging.getLogger("myapp.requests")
53	adapter = ContextLoggerAdapter(base_logger, {"request_id": "req_42"})
54	adapter.info("processing started") # request_id automatically included
55	adapter.warning("slow response") # also has request_id
56
57	# Custom Filter
58	class LevelFilter(logging.Filter):
59	"""Only allow messages at exactly the given level."""
60	def __init__(self, level):
61	self.level = level
62
63	def filter(self, record):
64	return record.levelno == self.level
65
66	# Use filter to only emit ERROR messages to error_file handler
67	error_filter = LevelFilter(logging.ERROR)
68	error_handler = logging.FileHandler("errors.log")
69	error_handler.addFilter(error_filter)

ℹ

info

For production systems, consider the python-json-logger package for battle-tested JSON formatting, or use structlog which integrates with the standard logging module and provides context managers, processors, and middleware for request-scoped logging.

Best Practices

best_practices.py

Python

1	# 1. Module-level logger — one per module
2	import logging
3	logger = logging.getLogger(__name__)
4
5	# __name__ gives you: "myapp" for app entry point,
6	# "myapp.db.connection" for myapp/db/connection.py
7	# This automatically creates the correct hierarchy
8
9	# 2. Don't use print() — use logging
10	# BAD
11	print("Starting server...")
12	print(f"User {username} connected")
13
14	# GOOD
15	logger.info("Starting server")
16	logger.info("User %s connected", username)
17
18	# 3. Libraries — use NullHandler, never basicConfig
19	# BAD (in a library)
20	logging.basicConfig() # hijacks root logger config!
21
22	# GOOD (in a library)
23	logger = logging.getLogger(__name__)
24	logger.addHandler(logging.NullHandler())
25
26	# 4. Lazy string formatting with %
27	# BAD — f-string always interpolates (cost even if level disabled)
28	logger.debug(f"Processing {len(items)} items")
29
30	# GOOD — % formatting only interpolates if message is emitted
31	logger.debug("Processing %d items", len(items))
32
33	# 5. Log the right level
34	logger.debug("raw data: %s", data) # diagnostic detail
35	logger.info("processed %d records", count) # normal operation
36	logger.warning("cache miss for key %s", key) # degraded but working
37	logger.error("API returned %d", status) # failure needing attention
38	logger.critical("data corruption detected") # system failure
39
40	# 6. Exception logging — always inside except blocks
41	try:
42	result = risky_operation()
43	except ConnectionError as e:
44	logger.error("connection failed: %s", e)
45	# OR for full traceback:
46	logger.exception("connection failed")
47	# OR with context:
48	logger.error("connection failed to %s:%d", host, port, exc_info=True)
49
50	# 7. Use logger.exception() or exc_info=True for tracebacks
51	# DON'T do this:
52	logger.error(str(e)) # loses traceback
53	logger.error(f"error: {e}") # same
54
55	# DO this:
56	logger.exception("error occurred") # full traceback
57	logger.error("error occurred", exc_info=True) # same result
58
59	# 8. Avoid logging sensitive data
60	logger.info("user %s logged in", user_id) # OK
61	logger.debug("password: %s", password) # NEVER
62	logger.debug("credit card: %s", card_number) # NEVER

danger

Never log passwords, API keys, credit card numbers, or other sensitive credentials. Even DEBUG-level logs can end up in production systems. Always use a secrets scanner and filter sensitive fields from log output. Consider using a Filter subclass to automatically redact known patterns.

$Blueprint — Engineering Documentation·Section ID: PYTHON-LOG·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor