|$ curl https://forge-ai.dev/api/markdown?path=docs/python/data-science
$cat docs/python-—-data-science.md
updated Last week·28 min read·published
Python — Data Science
Introduction
Python is the dominant language for data science thanks to three core libraries: NumPy for numerical computation, Pandas for data manipulation, and Matplotlib for visualization. Together they form the foundation of nearly every data workflow.
The typical data science workflow is: load data, clean and transform it, explore patterns with statistics and plots, build models, and communicate results. Jupyter notebooks provide an interactive environment for each step.
NumPy Arrays
NumPy provides the ndarray, a fast, fixed-type multidimensional array. Vectorized operations on NumPy arrays are 10-100x faster than equivalent Python loops because they run in compiled C code.
numpy_basics.py
Python
| 1 | import numpy as np |
| 2 | |
| 3 | # Creating arrays |
| 4 | a = np.array([1, 2, 3, 4, 5]) # from list |
| 5 | b = np.zeros((3, 4)) # 3x4 zeros |
| 6 | c = np.ones((2, 3, 4)) # 3D ones |
| 7 | d = np.arange(0, 10, 0.5) # like range() but float |
| 8 | e = np.linspace(0, 1, 100) # 100 evenly spaced [0,1] |
| 9 | f = np.random.randn(3, 4) # 3x4 normal distribution |
| 10 | |
| 11 | # Shape and dtype |
| 12 | print(a.shape) # (5,) |
| 13 | print(f.shape) # (3, 4) |
| 14 | print(a.dtype) # int64 |
| 15 | print(b.dtype) # float64 |
| 16 | |
| 17 | # Reshaping |
| 18 | matrix = np.arange(12).reshape(3, 4) |
| 19 | print(matrix) |
| 20 | # [[ 0 1 2 3] |
| 21 | # [ 4 5 6 7] |
| 22 | # [ 8 9 10 11]] |
| 23 | |
| 24 | # Indexing and slicing |
| 25 | print(matrix[0, :]) # first row → [0 1 2 3] |
| 26 | print(matrix[:, 2]) # third column → [2 6 10] |
| 27 | print(matrix[1:3, 1:3]) # submatrix → [[5 6] [9 10]] |
numpy_ops.py
Python
| 1 | # Vectorized operations — no loops needed |
| 2 | a = np.array([1, 2, 3, 4]) |
| 3 | b = np.array([10, 20, 30, 40]) |
| 4 | |
| 5 | print(a + b) # [11 22 33 44] |
| 6 | print(a * b) # [10 40 90 160] |
| 7 | print(a ** 2) # [1 4 9 16] |
| 8 | print(np.sqrt(a)) # [1.0 1.414 1.732 2.0] |
| 9 | |
| 10 | # Aggregations |
| 11 | data = np.random.randn(1000) |
| 12 | print(np.mean(data)) # ~0.0 |
| 13 | print(np.std(data)) # ~1.0 |
| 14 | print(np.median(data)) # ~0.0 |
| 15 | print(np.percentile(data, [25, 50, 75])) # quartiles |
| 16 | |
| 17 | # Broadcasting — operations between different shapes |
| 18 | matrix = np.arange(12).reshape(3, 4) |
| 19 | row = np.array([1, 0, 1, 0]) |
| 20 | print(matrix + row) # adds row to each row of matrix |
| 21 | |
| 22 | # Boolean indexing |
| 23 | data = np.array([1, 5, 3, 8, 2, 9, 4]) |
| 24 | mask = data > 4 |
| 25 | print(data[mask]) # [5 8 9] |
| 26 | print(np.sum(mask)) # 3 elements > 4 |
| 27 | |
| 28 | # Linear algebra |
| 29 | A = np.array([[1, 2], [3, 4]]) |
| 30 | B = np.array([[5, 6], [7, 8]]) |
| 31 | print(A @ B) # matrix multiply |
| 32 | print(np.linalg.inv(A)) # inverse |
| 33 | print(np.linalg.det(A)) # determinant |
ℹ
info
Always prefer vectorized NumPy operations over Python loops. If you find yourself writing a for-loop over array elements, there is almost certainly a NumPy function that does it faster.
Pandas DataFrames
Pandas builds on NumPy to provide labeled, heterogeneous tabular data via DataFrame and Series. It is the primary tool for data cleaning, transformation, and exploration.
pandas_basics.py
Python
| 1 | import pandas as pd |
| 2 | |
| 3 | # Creating DataFrames |
| 4 | df = pd.DataFrame({ |
| 5 | "name": ["Alice", "Bob", "Charlie", "Diana"], |
| 6 | "age": [28, 35, 42, 31], |
| 7 | "city": ["NYC", "SF", "NYC", "LA"], |
| 8 | "salary": [85000, 120000, 95000, 110000], |
| 9 | }) |
| 10 | |
| 11 | # From CSV |
| 12 | # df = pd.read_csv("data.csv") |
| 13 | # From JSON |
| 14 | # df = pd.read_json("data.json") |
| 15 | # From Excel |
| 16 | # df = pd.read_excel("data.xlsx") |
| 17 | |
| 18 | # Basic inspection |
| 19 | print(df.shape) # (4, 4) |
| 20 | print(df.head(2)) # first 2 rows |
| 21 | print(df.dtypes) # column types |
| 22 | print(df.describe()) # summary statistics |
| 23 | print(df.info()) # non-null counts and types |
pandas_select.py
Python
| 1 | # Selection and filtering |
| 2 | print(df["name"]) # Series |
| 3 | print(df[["name", "age"]]) # DataFrame subset |
| 4 | |
| 5 | # loc — label-based |
| 6 | print(df.loc[0:2, "name":"city"]) # rows 0-2, name through city |
| 7 | |
| 8 | # iloc — integer position |
| 9 | print(df.iloc[0:2, 0:3]) # first 2 rows, first 3 columns |
| 10 | |
| 11 | # Boolean filtering |
| 12 | seniors = df[df["age"] > 35] |
| 13 | nyc = df[df["city"] == "NYC"] |
| 14 | complex_filter = df[(df["age"] > 30) & (df["salary"] > 100000)] |
| 15 | |
| 16 | # Query syntax |
| 17 | result = df.query("age > 30 and city == 'NYC'") |
| 18 | |
| 19 | # Adding columns |
| 20 | df["bonus"] = df["salary"] * 0.1 |
| 21 | df["senior"] = df["age"] > 35 |
| 22 | |
| 23 | # apply — row or column-wise operations |
| 24 | df["name_upper"] = df["name"].str.upper() |
| 25 | df["age_group"] = df["age"].apply(lambda x: "senior" if x > 35 else "junior") |
pandas_group.py
Python
| 1 | # Grouping and aggregation |
| 2 | city_stats = df.groupby("city").agg( |
| 3 | avg_age=("age", "mean"), |
| 4 | avg_salary=("salary", "mean"), |
| 5 | count=("name", "count"), |
| 6 | ).reset_index() |
| 7 | print(city_stats) |
| 8 | # city avg_age avg_salary count |
| 9 | # 0 LA 31.0 110000.0 1 |
| 10 | # 1 NYC 35.0 90000.0 2 |
| 11 | # 2 SF 35.0 120000.0 1 |
| 12 | |
| 13 | # Multiple groupby columns |
| 14 | # df.groupby(["city", "senior"]).agg(...) |
| 15 | |
| 16 | # Sorting |
| 17 | print(df.sort_values("salary", ascending=False)) |
| 18 | print(df.sort_values(["city", "salary"], ascending=[True, False])) |
| 19 | |
| 20 | # Merging |
| 21 | employees = pd.DataFrame({ |
| 22 | "name": ["Alice", "Bob"], |
| 23 | "dept": ["Engineering", "Marketing"], |
| 24 | }) |
| 25 | merged = pd.merge(df, employees, on="name", how="left") |
| 26 | |
| 27 | # Pivot tables |
| 28 | pivot = df.pivot_table( |
| 29 | values="salary", |
| 30 | index="city", |
| 31 | aggfunc=["mean", "count"], |
| 32 | ) |
Data Cleaning
cleaning.py
Python
| 1 | import pandas as pd |
| 2 | import numpy as np |
| 3 | |
| 4 | # Load messy data |
| 5 | df = pd.read_csv("sales.csv") |
| 6 | |
| 7 | # Check for missing values |
| 8 | print(df.isnull().sum()) |
| 9 | |
| 10 | # Drop rows with missing values |
| 11 | df_clean = df.dropna(subset=["revenue", "date"]) |
| 12 | |
| 13 | # Fill missing values |
| 14 | df["category"] = df["category"].fillna("Unknown") |
| 15 | df["price"] = df["price"].fillna(df["price"].median()) |
| 16 | |
| 17 | # Remove duplicates |
| 18 | df = df.drop_duplicates(subset=["order_id"]) |
| 19 | |
| 20 | # Fix data types |
| 21 | df["date"] = pd.to_datetime(df["date"]) |
| 22 | df["price"] = pd.to_numeric(df["price"], errors="coerce") |
| 23 | |
| 24 | # String cleanup |
| 25 | df["product"] = df["product"].str.strip().str.lower() |
| 26 | |
| 27 | # Outlier detection with IQR |
| 28 | Q1 = df["price"].quantile(0.25) |
| 29 | Q3 = df["price"].quantile(0.75) |
| 30 | IQR = Q3 - Q1 |
| 31 | lower = Q1 - 1.5 * IQR |
| 32 | upper = Q3 + 1.5 * IQR |
| 33 | df = df[(df["price"] >= lower) & (df["price"] <= upper)] |
| 34 | |
| 35 | # Rename columns |
| 36 | df = df.rename(columns={"price": "unit_price", "qty": "quantity"}) |
| 37 | |
| 38 | # Replace values |
| 39 | df["status"] = df["status"].replace({ |
| 40 | "cancelled": "canceled", |
| 41 | "shipped": "completed", |
| 42 | }) |
✓
best practice
Always inspect your data before cleaning. Use df.info(), df.describe(), and df.isnull().sum() to understand what you are working with before making changes.
Matplotlib Plotting
plotting.py
Python
| 1 | import matplotlib.pyplot as plt |
| 2 | import numpy as np |
| 3 | |
| 4 | # Line plot |
| 5 | x = np.linspace(0, 10, 100) |
| 6 | plt.figure(figsize=(10, 6)) |
| 7 | plt.plot(x, np.sin(x), label="sin(x)", color="#3776AB") |
| 8 | plt.plot(x, np.cos(x), label="cos(x)", color="#FF6B35") |
| 9 | plt.title("Trigonometric Functions") |
| 10 | plt.xlabel("x") |
| 11 | plt.ylabel("y") |
| 12 | plt.legend() |
| 13 | plt.grid(True, alpha=0.3) |
| 14 | plt.savefig("trig_plot.png", dpi=150, bbox_inches="tight") |
| 15 | plt.close() |
| 16 | |
| 17 | # Bar chart |
| 18 | categories = ["A", "B", "C", "D"] |
| 19 | values = [23, 45, 56, 78] |
| 20 | plt.figure(figsize=(8, 5)) |
| 21 | plt.bar(categories, values, color="#3776AB") |
| 22 | plt.title("Category Values") |
| 23 | plt.savefig("bar_chart.png") |
| 24 | plt.close() |
| 25 | |
| 26 | # Histogram |
| 27 | data = np.random.randn(1000) |
| 28 | plt.figure(figsize=(8, 5)) |
| 29 | plt.hist(data, bins=30, edgecolor="white", alpha=0.7, color="#3776AB") |
| 30 | plt.title("Normal Distribution") |
| 31 | plt.savefig("histogram.png") |
| 32 | plt.close() |
| 33 | |
| 34 | # Scatter plot |
| 35 | x = np.random.randn(200) |
| 36 | y = 2 * x + np.random.randn(200) * 0.5 |
| 37 | plt.figure(figsize=(8, 6)) |
| 38 | plt.scatter(x, y, alpha=0.5, c="#3776AB", s=20) |
| 39 | plt.title("Scatter Plot with Correlation") |
| 40 | plt.savefig("scatter.png") |
| 41 | plt.close() |
subplots.py
Python
| 1 | # Subplots — multiple charts in one figure |
| 2 | fig, axes = plt.subplots(2, 2, figsize=(12, 10)) |
| 3 | |
| 4 | # Top-left: line plot |
| 5 | axes[0, 0].plot(x, np.sin(x), color="#3776AB") |
| 6 | axes[0, 0].set_title("Sine Wave") |
| 7 | |
| 8 | # Top-right: bar chart |
| 9 | axes[0, 1].bar(["A", "B", "C"], [3, 7, 5], color="#FF6B35") |
| 10 | axes[0, 1].set_title("Bar Chart") |
| 11 | |
| 12 | # Bottom-left: histogram |
| 13 | axes[1, 0].hist(np.random.randn(500), bins=25, color="#3776AB") |
| 14 | axes[1, 0].set_title("Distribution") |
| 15 | |
| 16 | # Bottom-right: scatter |
| 17 | axes[1, 1].scatter(np.random.randn(100), np.random.randn(100), alpha=0.6) |
| 18 | axes[1, 1].set_title("Scatter") |
| 19 | |
| 20 | plt.tight_layout() |
| 21 | plt.savefig("subplots.png", dpi=150) |
| 22 | plt.close() |
| 23 | |
| 24 | # Pandas integration — plot directly from DataFrame |
| 25 | df.groupby("city")["salary"].mean().plot( |
| 26 | kind="bar", title="Average Salary by City", figsize=(8, 5) |
| 27 | ) |
| 28 | plt.tight_layout() |
| 29 | plt.savefig("pandas_plot.png") |
| 30 | plt.close() |
Complete Workflow
workflow.py
Python
| 1 | # End-to-end data analysis workflow |
| 2 | import pandas as pd |
| 3 | import numpy as np |
| 4 | import matplotlib.pyplot as plt |
| 5 | |
| 6 | # 1. Load data |
| 7 | df = pd.read_csv("sales_2024.csv", parse_dates=["date"]) |
| 8 | |
| 9 | # 2. Initial exploration |
| 10 | print(f"Shape: {df.shape}") |
| 11 | print(f"Columns: {df.columns.tolist()}") |
| 12 | print(f"Missing:\n{df.isnull().sum()}") |
| 13 | print(f"Date range: {df['date'].min()} to {df['date'].max()}") |
| 14 | |
| 15 | # 3. Clean data |
| 16 | df = df.dropna(subset=["revenue"]) |
| 17 | df = df[df["revenue"] > 0] |
| 18 | df["month"] = df["date"].dt.to_period("M") |
| 19 | |
| 20 | # 4. Transform and analyze |
| 21 | monthly = df.groupby("month").agg( |
| 22 | total_revenue=("revenue", "sum"), |
| 23 | avg_order=("revenue", "mean"), |
| 24 | num_orders=("revenue", "count"), |
| 25 | ).reset_index() |
| 26 | monthly["month_str"] = monthly["month"].astype(str) |
| 27 | |
| 28 | # 5. Visualize |
| 29 | fig, ax1 = plt.subplots(figsize=(12, 6)) |
| 30 | ax1.bar(monthly["month_str"], monthly["total_revenue"], color="#3776AB", alpha=0.7) |
| 31 | ax1.set_xlabel("Month") |
| 32 | ax1.set_ylabel("Total Revenue", color="#3776AB") |
| 33 | ax1.tick_params(axis="y", labelcolor="#3776AB") |
| 34 | ax1.set_xticklabels(monthly["month_str"], rotation=45) |
| 35 | |
| 36 | ax2 = ax1.twinx() |
| 37 | ax2.plot(monthly["month_str"], monthly["num_orders"], color="#FF6B35", marker="o") |
| 38 | ax2.set_ylabel("Number of Orders", color="#FF6B35") |
| 39 | ax2.tick_params(axis="y", labelcolor="#FF6B35") |
| 40 | |
| 41 | plt.title("Monthly Revenue and Orders") |
| 42 | plt.tight_layout() |
| 43 | plt.savefig("monthly_report.png", dpi=150) |
| 44 | plt.close() |
| 45 | |
| 46 | # 6. Export results |
| 47 | monthly.to_csv("monthly_summary.csv", index=False) |
| 48 | print("Report generated: monthly_report.png") |
ℹ
info
In Jupyter notebooks, add %matplotlib inline at the top to display plots directly in the notebook. Use %timeit to benchmark code and %load_ext autoreload to auto-reload changed modules.
$Blueprint — Engineering Documentation·Section ID: PYTHON-DS·Revision: 1.0
Community
Get help on Slack, Discord or VIP
Stuck on a guide? Join the community and ask.