RUNTIME

Inference Engines

A model is just a file of weights. An inference engine is the program that loads those weights, manages memory, and turns your prompt into tokens — llama.cpp, Ollama and vLLM are three different answers to that job.

Where the Engine Sits

Between the weights on disk and the application asking for text

Model Weights

Billions of numbers in a file — the trained parameters, often quantized to shrink them.

model.gguf

→

Inference Engine

Loads the weights, allocates the KV cache, schedules requests and runs the maths on your hardware.

llama.cpp · Ollama · vLLM

→

Your Application

A chat UI, an agent, a script — talking to the engine over HTTP, usually an OpenAI-compatible API.

POST /v1/chat/completions

What an Engine Actually Does

Four jobs that sit between "a file of weights" and "text on screen"

Memory Management

Fits the weights into VRAM or RAM, offloads layers that do not fit, and allocates the KV cache that grows with every token.

Batching & Scheduling

Decides which requests run together on each forward pass. Good scheduling is the difference between serving one user and serving hundreds.

Hardware Kernels

Ships optimised matrix-multiply code per backend — CUDA, Metal, Vulkan, ROCm or plain CPU. Same model, very different speed.

API Surface

Exposes the model over HTTP. Most engines speak the OpenAI chat format, so client code moves between them with a URL change.

Three Engines, Three Priorities

They solve the same problem for very different users

Portability first

llama.cpp

A C/C++ implementation of transformer inference with no heavy runtime dependencies. The engine that made running LLMs on a laptop normal.

Written in C / C++

Format GGUF

Runs on CPU, CUDA, Metal, Vulkan, ROCm

Sweet spot One user, modest hardware

Runs on machines with no GPU at all, and offloads only the layers that fit when there is one.
Wide range of quantization levels, from roughly 2-bit up to 8-bit, chosen per file.
Ships a server binary with an OpenAI-compatible endpoint alongside the CLI.
Powers many higher-level tools rather than being used directly.

Ergonomics first

Ollama

A local runtime that wraps the hard parts — downloading, storing and configuring models — behind a package-manager-style workflow.

Written in Go (llama.cpp-based core)

Format GGUF, via a model registry

Runs on macOS, Linux, Windows

Sweet spot Local development, quick starts

One command pulls and runs a model — no flags, quantization choices or file paths to reason about.
Background service with a REST API plus an OpenAI-compatible route.
Modelfiles bundle a base model with a system prompt and parameter defaults.
Inherits llama.cpp's hardware reach, and its single-stream performance ceiling.

Throughput first

vLLM

A GPU serving engine built around PagedAttention and continuous batching, designed to keep costly accelerators saturated under load.

Written in Python + CUDA

Format safetensors (Hugging Face)

Runs on Datacentre GPUs (NVIDIA, plus others)

Sweet spot Many users, production serving

PagedAttention stores the KV cache in non-contiguous blocks, cutting the fragmentation that caps batch size.
Continuous batching swaps finished sequences out mid-flight instead of waiting for the slowest one.
Tensor parallelism splits a model that does not fit across several GPUs.
Needs real GPU memory — it is the wrong tool for a laptop.

Side by Side


	llama.cpp	Ollama	vLLM
Needs a GPU	No	No	Yes
Model format	GGUF	GGUF	safetensors
Setup effort	Moderate	Minimal	Moderate
Single-user latency	Strong	Strong	Good
Concurrent throughput	Limited	Limited	Strong
Multi-GPU model split	Limited	Limited	Built in
OpenAI-compatible API	Yes	Yes	Yes
Typical home	Laptop, edge device	Developer machine	GPU server

Ratings are relative to each other for typical use, not measured benchmarks — your model, hardware and settings decide the real numbers

One User vs Many Users

The single biggest reason these engines differ

Request at a time

Each request runs to completion before the next starts. Latency for the person waiting is excellent, but the accelerator idles between and during requests.

req A

req B

req C

Fine for one person at a keyboard — llama.cpp and Ollama live here

Continuous batching

Requests join and leave the running batch between token steps. The GPU stays busy, so total tokens per second across all users climbs sharply.

req A

req B

req C

Built for a queue of users — this is what vLLM optimises for

Why You Cannot Just Swap Model Files

Engines disagree about how weights should be packaged

GGUF

A single-file format holding weights, tokenizer and metadata together, with the quantization baked in. Built for loading fast on mixed CPU/GPU machines.

Used by: llama.cpp, Ollama, and the tools built on them

safetensors

The Hugging Face standard — weights in full or half precision, with the config and tokenizer as separate files. Quantization is applied by the serving engine at load time, using schemes such as GPTQ or AWQ.

Used by: vLLM, TensorRT-LLM, SGLang, most GPU servers

Which One Should You Reach For?

Trying a model on your laptop

Ollama — one command, sensible defaults, nothing to configure.

Squeezing a big model onto small hardware

llama.cpp — pick the exact quantization and layer offload you need.

Serving an app with real traffic

vLLM — continuous batching keeps cost per token down as users arrive.

Embedding in a desktop or edge app

llama.cpp — a library with no Python runtime to ship alongside it.

Others Worth Knowing

The same job, tuned for narrower situations

TensorRT-LLM

NVIDIA's compiled-kernel engine. Fastest on NVIDIA hardware, at the cost of a build step per model.

SGLang

Serving engine focused on structured generation and aggressive prefix-cache reuse.

TGI

Hugging Face's Text Generation Inference, a production server close to the Hub ecosystem.

MLX

Apple's array framework for Apple silicon, using unified memory rather than a separate VRAM pool.

LM Studio

A desktop GUI over local engines, for people who would rather not touch a terminal.