Monitoring & Observability

◆Operations◆Monitoring◆Intermediate

Introduction

Monitoring and observability answer the question: "What is happening in my system?" Monitoring tracks known failure modes with dashboards and alerts, while observability enables you to understand unknown failure modes by querying telemetry data.

A production system is blind without the three pillars of observability: logs, metrics, and traces. This guide covers how to implement all three effectively.

Structured Logging (JSON)

Structured logs output data in JSON format, making them machine-parseable and queryable. Unlike plain text logs, structured logs include fields like level, timestamp, service, and request_id for filtering and aggregation.

logging.ts

TypeScript

1	// Pino — Structured logging for Node.js
2	import pino from 'pino';
3
4	const logger = pino({
5	level: process.env.LOG_LEVEL \|\| 'info',
6	formatters: {
7	level(label) {
8	return { level: label };
9	},
10	},
11	serializers: {
12	req: pino.stdSerializers.req,
13	err: pino.stdSerializers.err,
14	},
15	redact: {
16	paths: ['req.headers.authorization', 'req.headers.cookie'],
17	censor: '***',
18	},
19	});
20
21	// Usage
22	logger.info({ userId: 123, action: 'login' }, 'User logged in');
23	logger.warn({ rateLimit: { remaining: 5 } }, 'Rate limit warning');
24	logger.error({ err, requestId: 'abc-123' }, 'Database connection failed');
25
26	// Express middleware for request logging
27	app.use((req, res, next) => {
28	const start = Date.now();
29	res.on('finish', () => {
30	logger.info({
31	method: req.method,
32	url: req.url,
33	status: res.statusCode,
34	duration: Date.now() - start,
35	}, 'request completed');
36	});
37	next();
38	});
39
40	// Log levels and their use:
41	// fatal — service is unusable, immediate attention
42	// error — request failed, user-visible error
43	// warn — potential issue, needs investigation
44	// info — normal operations
45	// debug — detailed diagnostic info (not in production)

✓

best practice

Use structured JSON logging everywhere — including your database queries, API calls, and background jobs. Multi-line log aggregation tools (ELK, Datadog, Grafana Loki) cannot effectively parse unstructured text logs. Include correlation IDs for tracking requests across services.

Key Metrics

Metrics are numerical measurements of system behavior over time. Track the Four Golden Signals recommended by Google SRE.

Signal	What It Measures	Example Metrics
Latency	Time to serve a request	p50, p95, p99 response times
Traffic	Requests per second	RPS, active users, bandwidth
Errors	Rate of failed requests	5xx rate, 4xx rate, exception count
Saturation	How "full" the service is	CPU, memory, disk I/O, connection pool

metrics.ts

TypeScript

1	// Prometheus client — Exposing metrics from Node.js
2	import client from 'prom-client';
3
4	// Create a Registry
5	const register = new client.Registry();
6	client.collectDefaultMetrics({ register });
7
8	// HTTP request duration histogram
9	const httpDuration = new client.Histogram({
10	name: 'http_request_duration_ms',
11	help: 'Duration of HTTP requests in ms',
12	labelNames: ['method', 'route', 'status'],
13	buckets: [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000],
14	registers: [register],
15	});
16
17	// Request counter
18	const requestCount = new client.Counter({
19	name: 'http_requests_total',
20	help: 'Total number of HTTP requests',
21	labelNames: ['method', 'route', 'status'],
22	registers: [register],
23	});
24
25	// Active users gauge
26	const activeUsers = new client.Gauge({
27	name: 'active_users',
28	help: 'Number of active users',
29	registers: [register],
30	});
31
32	// Express middleware
33	app.use((req, res, next) => {
34	const end = httpDuration.startTimer();
35	res.on('finish', () => {
36	const labels = { method: req.method, route: req.route?.path \|\| 'unknown', status: res.statusCode };
37	end(labels);
38	requestCount.inc(labels);
39	});
40	next();
41	});
42
43	// Metrics endpoint
44	app.get('/metrics', async (req, res) => {
45	res.set('Content-Type', register.contentType);
46	res.end(await register.metrics());
47	});

Distributed Tracing (OpenTelemetry)

Tracing follows a single request as it travels through multiple services. OpenTelemetry is the industry standard for generating, collecting, and exporting telemetry data.

opentelemetry.ts

TypeScript

1	// OpenTelemetry setup for Node.js
2	import { NodeSDK } from '@opentelemetry/sdk-node';
3	import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
4	import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
5	import { Resource } from '@opentelemetry/resources';
6	import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
7
8	const sdk = new NodeSDK({
9	resource: new Resource({
10	[SemanticResourceAttributes.SERVICE_NAME]: 'my-app',
11	[SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
12	}),
13	traceExporter: new OTLPTraceExporter({
14	url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
15	}),
16	instrumentations: [getNodeAutoInstrumentations()],
17	});
18
19	sdk.start();
20
21	// Custom spans for manual instrumentation
22	import { trace } from '@opentelemetry/api';
23
24	const tracer = trace.getTracer('my-app');
25
26	async function processOrder(orderId: string) {
27	const span = tracer.startSpan('process-order', {
28	attributes: { 'order.id': orderId },
29	});
30
31	try {
32	// Do work...
33	const result = await db.query('SELECT * FROM orders WHERE id = $1', [orderId]);
34
35	span.setAttribute('order.items', result.length);
36	span.setStatus({ code: SpanStatusCode.OK });
37	} catch (error) {
38	span.recordException(error);
39	span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
40	throw error;
41	} finally {
42	span.end();
43	}
44	}

🔥

pro tip

OpenTelemetry auto-instrumentation covers most common libraries (Express, HTTP, gRPC, database drivers) without code changes. Enable it during application startup. Traces are essential for debugging performance issues in microservice architectures — they show exactly where time is spent across service boundaries.

Uptime Monitoring & Alerting

Uptime monitors periodically check your application from multiple global locations. They detect outages before users do and alert the on-call team.

Service	Check Interval	Price	Best For
Pingdom	1 min	From $15/mo	Multi-location, enterprise
Better Stack (ex. Better Uptime)	30s	Free tier available	Developer-friendly, status pages
Checkly	30s	Free tier available	Browser checks, Playwright-based
UptimeRobot	5 min	Free (50 monitors)	Simple, cost-effective

uptime-alerting.ts

TypeScript

1	// Better Stack — Heartbeat monitoring (from your app)
2	import fetch from 'node-fetch';
3
4	const HEARTBEAT_URL = process.env.BETTERSTACK_HEARTBEAT_URL;
5
6	async function sendHeartbeat() {
7	// Call this from your cron job or after critical operations
8	await fetch(HEARTBEAT_URL);
9	}
10
11	// Set up a recurring heartbeat
12	setInterval(sendHeartbeat, 60_000); // every minute
13
14	// Slack webhook alerting
15	async function sendAlert(message, severity = 'warning') {
16	await fetch(process.env.SLACK_WEBHOOK_URL, {
17	method: 'POST',
18	headers: { 'Content-Type': 'application/json' },
19	body: JSON.stringify({
20	text: `[${severity.toUpperCase()}] ${message}`,
21	}),
22	});
23	}
24
25	// PagerDuty integration
26	async function triggerPagerDutyIncident(summary, source, severity = 'critical') {
27	await fetch('https://events.pagerduty.com/v2/enqueue', {
28	method: 'POST',
29	headers: {
30	'Content-Type': 'application/json',
31	'Authorization': `Token token=${process.env.PAGERDUTY_ROUTING_KEY}`,
32	},
33	body: JSON.stringify({
34	routing_key: process.env.PAGERDUTY_ROUTING_KEY,
35	event_action: 'trigger',
36	payload: {
37	summary,
38	source,
39	severity,
40	},
41	}),
42	});
43	}

⚠

warning

Avoid alert fatigue — only alert on actionable signals. A pager should not ring for every 4xx error. Use alert for user-visible errors, increased 5xx rates, high latency, and saturation. Log warn-level events to dashboards but do not page people for them.

Dashboard Example (Grafana)

A well-designed dashboard shows the health of your system at a glance. Here is the structure of a typical service dashboard in Grafana:

Row	Panels	Description
1. Service Health	Uptime, 5xx rate, p99 latency	Top-level status at a glance
2. Traffic	RPS by route, active users	Understand load patterns
3. Latency	p50, p95, p99 by endpoint	Identify slow endpoints
4. Saturation	CPU, memory, disk, connections	Capacity planning
5. Dependencies	Database, cache, external APIs	Downstream service health

$Blueprint — Engineering Documentation·Section ID: OPS-MONITORING-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor