Why Run Locally?
Local LLMs offer privacy (data never leaves your machine), zero cost per token, offline capability, and the ability to run uncensored models for research. With quantisation (GGUF format), a 7B model runs acceptably fast on a 16GB MacBook.
Option 1: Ollama (Recommended)
Ollama is the simplest way to get started — one command to install, one command per model:
# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
# Pull models
ollama pull llama3.1 # 4.7GB — best all-rounder
ollama pull phi3 # 2.2GB — fast and smart for its size
ollama pull mistral # 4.1GB — great for code and reasoning
# Chat
ollama run llama3.1
>>> Who invented the transformer architecture?Ollama also exposes a REST API at http://localhost:11434 compatible with the OpenAI client library:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(model="llama3.1", messages=[{"role":"user","content":"Hello!"}])Option 2: LM Studio (GUI)
LM Studio provides a polished desktop app for downloading, managing, and chatting with models. It auto-selects the right GGUF quantisation for your hardware and includes a built-in chat UI.
Understanding GGUF Quantisation
| Quantisation | Size (7B) | Quality vs FP16 | Best For |
|---|---|---|---|
| Q8_0 | ~7GB | Near-identical | High-quality, have VRAM |
| Q5_K_M | ~5GB | Excellent | Best balance |
| Q4_K_M | ~4GB | Very good | Recommended default |
| Q3_K_M | ~3GB | Decent | RAM-constrained |
Start with Q4_K_M. It is the sweet spot: 4GB on disk, fast inference, and quality indistinguishable from Q8 for most tasks.
