AI Engineering — Open Source Models

◆AI Engineering◆Models◆Intermediate to Advanced

Introduction

Open source language models have reached parity with proprietary models on many benchmarks. Leading models like Llama 3, Mistral, Qwen, DeepSeek, and Gemma offer competitive performance with the benefits of local deployment, data privacy, custom fine-tuning, and no per-token costs.

The open source ecosystem includes model hubs (Hugging Face), inference engines (vLLM, Ollama, TGI, llama.cpp), quantization techniques (GGUF, AWQ, GPTQ), and tooling for fine-tuning, evaluation, and deployment. This ecosystem enables self-hosted AI that rivals cloud API quality.

Leading Open Models

The open source LLM landscape evolves rapidly. Here are the current leading model families and their key characteristics.

Model	Developer	Parameters	Context	License
Llama 3.1 / 3.2	Meta	8B, 70B, 405B	128K	Llama 3 Community
Mistral / Mixtral	Mistral AI	7B, 8x7B (MoE), 12B, 123B	32K-128K	Apache 2.0 / MIT
Qwen 2.5	Alibaba	0.5B, 1.5B, 7B, 14B, 32B, 72B (MoE)	32K-128K	Apache 2.0
DeepSeek V2 / V3	DeepSeek (幻方)	16B (MoE), 67B, 236B (MoE)	128K	MIT
Gemma 2	Google	2B, 9B, 27B	8K	Gemma License
CodeLlama / Llama-Code	Meta	7B, 13B, 34B, 70B	16K-100K	Llama 3 Community

ℹ

info

For most applications, a 7B-8B parameter model quantized to 4-bit offers the best quality-to-performance ratio. The 70B+ models are 5-10x slower and require multi-GPU setups. Start small, benchmark your use case, then scale up if needed.

Hugging Face Hub

Hugging Face is the primary hub for open source models, datasets, and spaces. The transformers library provides a unified API for loading and running thousands of models. Use pipeline for quick inference or direct model loading for more control.

huggingface-transformers.py

Python

1	# Load any model with Hugging Face Transformers
2	# pip install transformers torch accelerate
3
4	from transformers import pipeline
5
6	# Quick pipeline API
7	generator = pipeline(
8	"text-generation",
9	model="meta-llama/Llama-3.2-8B-Instruct",
10	device="mps", # or "cuda", "cpu"
11	torch_dtype="auto"
12	)
13
14	result = generator(
15	"What is machine learning?",
16	max_new_tokens=256,
17	do_sample=True,
18	temperature=0.7
19	)
20
21	print(result[0]["generated_text"])
22
23	# Direct model loading for more control
24	from transformers import AutoModelForCausalLM, AutoTokenizer
25
26	model_name = "mistralai/Mistral-7B-Instruct-v0.3"
27
28	tokenizer = AutoTokenizer.from_pretrained(model_name)
29	model = AutoModelForCausalLM.from_pretrained(
30	model_name,
31	device_map="auto",
32	load_in_4bit=True, # 4-bit quantization
33	bnb_4bit_compute_dtype="float16"
34	)
35
36	messages = [
37	{"role": "user", "content": "Explain quantum computing in 3 sentences."}
38	]
39
40	inputs = tokenizer.apply_chat_template(
41	messages,
42	add_generation_prompt=True,
43	return_tensors="pt"
44	).to(model.device)
45
46	outputs = model.generate(
47	inputs,
48	max_new_tokens=256,
49	temperature=0.7
50	)
51
52	response = tokenizer.decode(outputs[0], skip_special_tokens=True)
53	print(response)

Self-Hosting Inference Engines

Several inference engines are optimized for running open models efficiently. Choose based on your hardware, throughput needs, and feature requirements.

vLLM — High-Throughput Production

vLLM uses PagedAttention for efficient memory management, achieving 10-20x higher throughput than naive implementations. Supports continuous batching, tensor parallelism, and an OpenAI-compatible API.

vllm.py

Python

1	# pip install vllm
2
3	# Start an OpenAI-compatible API server
4	# python -m vllm.entrypoints.openai.api_server \
5	# --model meta-llama/Llama-3.2-8B-Instruct \
6	# --port 8000 \
7	# --tensor-parallel-size 2 \
8	# --max-model-len 8192
9
10	# Client code (identical to OpenAI SDK)
11	from openai import OpenAI
12
13	client = OpenAI(
14	base_url="http://localhost:8000/v1",
15	api_key="not-needed" # vLLM doesn't require auth
16	)
17
18	response = client.chat.completions.create(
19	model="meta-llama/Llama-3.2-8B-Instruct",
20	messages=[
21	{"role": "user", "content": "Hello!"}
22	]
23	)
24
25	print(response.choices[0].message.content)
26
27	# Programmatic usage
28	from vllm import LLM, SamplingParams
29
30	llm = LLM(
31	model="mistralai/Mistral-7B-Instruct-v0.3",
32	tensor_parallel_size=1,
33	max_model_len=8192
34	)
35
36	params = SamplingParams(
37	temperature=0.7,
38	max_tokens=512,
39	top_p=0.95
40	)
41
42	outputs = llm.generate(
43	["Explain AI in one paragraph."],
44	params
45	)
46
47	for output in outputs:
48	print(output.outputs[0].text)

Ollama — Local Development

Ollama provides the simplest local setup. Download, run ollama pull llama3.2, and you have a running model. Ideal for development, prototyping, and personal use.

ollama-setup.sh

Bash

1	# Install: brew install ollama
2	# Download a model
3	ollama pull llama3.2
4	ollama pull mistral
5	ollama pull qwen2.5:7b
6
7	# Run interactive
8	ollama run llama3.2
9
10	# API (runs on port 11434 by default)
11	curl http://localhost:11434/api/generate -d '{
12	"model": "llama3.2",
13	"prompt": "What is machine learning?",
14	"stream": false
15	}'
16
17	# Python client
18	# pip install ollama
19	import ollama
20
21	response = ollama.chat(
22	model="llama3.2",
23	messages=[
24	{"role": "user", "content": "Hello!"}
25	]
26	)
27	print(response["message"]["content"])

TGI & llama.cpp

Hugging Face TGI (Text Generation Inference) is optimized for production with continuous batching and flash attention. llama.cpp is optimized for CPU and Apple Silicon with GGUF quantization.

tgi-llamacpp.sh

Bash

1	# TGI — Docker deployment
2	# docker run -p 8080:80 \
3	# ghcr.io/huggingface/text-generation-inference:latest \
4	# --model-id meta-llama/Llama-3.2-8B-Instruct
5
6	# llama.cpp — CPU/Apple Silicon optimized
7	# git clone https://github.com/ggerganov/llama.cpp
8	# cd llama.cpp && make
9
10	# Download GGUF model
11	# wget https://huggingface.co/TheBloke/Llama-3.2-8B-Instruct-GGUF/resolve/main/llama-3.2-8b-instruct.Q4_K_M.gguf
12
13	# Run server
14	./llama-server \
15	-m models/llama-3.2-8b-instruct.Q4_K_M.gguf \
16	--port 8080 \
17	-ngl 99 # GPU layers (99 = all on GPU)

Quantization

Quantization reduces model precision (e.g., 16-bit to 4-bit) to decrease memory usage and increase inference speed with minimal quality loss. The most common formats are GGUF (llama.cpp), AWQ (performance-focused), and GPTQ (GPU-optimized).

Format	Engine	Precision	Memory Reduction	Best For
GGUF	llama.cpp, Ollama	2-8 bit	4-6x at Q4_K_M	CPU, Apple Silicon, edge
AWQ	vLLM, TGI, AutoAWQ	4 bit	~3x	GPU production
GPTQ	vLLM, TGI, AutoGPTQ	2-8 bit	~3-4x	GPU, older models
FP8 / FP16	All engines	8 or 16 bit	1-2x	Maximum quality
BitsAndBytes	Transformers, PEFT	4 or 8 bit	~4x	Fine-tuning, flexible loading

quantization-examples.py

Python

1	# GGUF with llama.cpp Python bindings
2	# pip install llama-cpp-python
3
4	from llama_cpp import Llama
5
6	# Load a GGUF model
7	llm = Llama(
8	model_path="models/llama-3.2-8b-instruct.Q4_K_M.gguf",
9	n_ctx=8192, # Context length
10	n_threads=8, # CPU threads
11	n_gpu_layers=-1, # -1 = all layers on GPU
12	verbose=False
13	)
14
15	output = llm.create_chat_completion(
16	messages=[
17	{"role": "user", "content": "What is quantization?"}
18	],
19	max_tokens=256,
20	temperature=0.7
21	)
22
23	print(output["choices"][0]["message"]["content"])
24
25	# AWQ with vLLM
26	# No special loading needed — vLLM detects AWQ automatically
27	from vllm import LLM
28
29	llm = LLM(
30	model="TheBloke/Llama-3.2-8B-Instruct-AWQ",
31	quantization="awq",
32	max_model_len=8192
33	)

✓

best practice

Use Q4_K_M GGUF as your default quantization for the best quality-to-size trade-off. For GPU-only deployment, AWQ offers slightly better quality. Q2 and Q3 quantizations degrade quality noticeably — avoid them for instruction-following tasks. Always benchmark your specific use case as quality loss varies by task.

Hardware Requirements

Hardware requirements vary dramatically by model size and quantization. Here are rough guidelines for running models locally.

Model Size	FP16 VRAM	INT4 VRAM	Minimum GPU
1-3B	2-6 GB	1-2 GB	Any GPU, Apple Silicon, or CPU
7-8B	14-16 GB	4-6 GB	RTX 3060+, M1 Pro+, or CPU (4-bit)
13-14B	26-28 GB	8-10 GB	RTX 3090+, M2 Max+, or dual GPU
32-34B	64+ GB	16-20 GB	RTX 4090, A100, or multi-GPU
70-72B	140+ GB	35-40 GB	2x RTX 3090+, A100, or CPU offloading
236-405B	470+ GB	120-200 GB	Multi-GPU server cluster

🔥

pro tip

Apple Silicon Macs (M1/M2/M3) are excellent for local LLMs. Unified memory allows running 7-8B models at Q4 on 16GB machines and 70B models on 128GB+ machines. Metal acceleration through llama.cpp or MLX provides competitive token generation speeds.

$Blueprint — Engineering Documentation·Section ID: AI-OS-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor