I have been using premium models for quite some time now, Claude Code, Codex, the usual suspects. And for just as long I wanted to try running an open weight model on my own machine for real work, not just to play with it for an afternoon and forget about it. So last weekend I finally sat down and spent a couple of hours on it.

The goal was fairly specific. Not "can I get a model to say hello", which takes five minutes and proves nothing. I wanted a setup where the day to day experience looks like what I already get from a premium coding agent: a terminal, a TUI, file editing, tool calls, sessions. Something I can genuinely fall back to when I run out of credits in the middle of the month.

That distinction matters more than it sounds. Most local LLM write ups stop at the chat window, which is the easy half. The interesting half is everything after: choosing a model your hardware can actually hold, splitting it between GPU and RAM so it does not crawl, and getting tool calling to work so an agent can edit files instead of just describing edits to you.

This post is what worked, in the order it worked, including the parts that broke. If you are starting from zero you should be able to follow it straight through. I used a Qwen model as the worked example, but nothing here is specific to it, swap in whichever model suits your hardware and the rest is identical.

Everything below was done on Fedora 44, RTX 4070 Laptop (8 GB VRAM), Core Ultra 7 155H, 30 GB RAM. Your numbers will differ, the process will not.

Contents

1. The two tools, and why these ones
2. Install llama.cpp
3. Check what llama.cpp can see
4. Pick a model and a quant
5. Download it
6. First chat in the terminal
7. Run the server
8. Wire up your coding agent
9. Making it actually fit
10. When things break
11. What to actually expect

---

The two tools, and why these ones

Only two pieces of software do the work here. One runs the model, the other gives you an agentic coding interface on top of it.

llama.cpp is the inference engine. It is a C/C++ implementation that runs large language models on ordinary hardware, on your CPU, your GPU, or split across both. It reads models in a format called GGUF, which packs the weights and everything needed to run them into a single file. Practically every open weight model shows up as a GGUF within days of release. It is also what most of the friendlier local LLM apps are quietly built on top of.

OpenCode is an open source terminal coding agent, same shape of tool as Claude Code, with a TUI, file editing, tool calling and sessions. What makes it fit here is that it treats model providers as configuration rather than a fixed list. Point it at any OpenAI compatible endpoint and it works, which is exactly what llama.cpp serves. No adapter, no proxy, no bridge process. If you prefer a different agent, that is fine, anything that lets you set a custom base URL will drop into step 7 the same way.

Why not Ollama? Ollama is genuinely nicer to start with, ollama run qwen3 and you are chatting. It wraps llama.cpp with a model registry and a daemon that loads and unloads models for you. The trade is control. On a machine where the model does not comfortably fit in VRAM, which is most consumer hardware, you end up needing to control exactly how the model is split between GPU and system RAM, how the KV cache is quantized, how many threads are used. llama.cpp exposes all of that as flags. With a wrapper you are negotiating with someone else's defaults. Since the whole point was a setup I could tune rather than one I hoped would work, going direct made sense. If you just want something running in two minutes, Ollama is a perfectly fine answer and this post is not an argument against it.

One more thing before we start. Modern llama.cpp ships as a single binary with subcommands instead of the pile of separate executables it used to be:

Command	What it does
llama download	Pull a GGUF from Hugging Face into a local cache
llama cli	Interactive chat in the terminal
llama serve	HTTP server with a web UI and an OpenAI compatible API
llama fit-params	Estimate how much memory a model needs before running it
llama bench	Benchmark prompt processing and generation speed

Run llama help all for the rest, and llama <command> --help for the flags of any one of them. That flag list is long, the server alone has over two hundred options, so treat the server README as the reference and this post as the opinionated subset.

---

Step 1: Install llama.cpp

Prebuilt binaries for every platform are on the releases page, and there are Homebrew, winget and Nix packages listed in the README. Pick whichever suits your system.

The one choice that matters is the backend build, because it decides what hardware llama.cpp can actually use. Release assets are named after it:

Build	Use it when
-cuda-	NVIDIA GPU with the proprietary driver installed
-vulkan-	Any reasonably modern GPU, NVIDIA, AMD or Intel
-hip-	AMD GPU with ROCm
no suffix	CPU only

Grab the Vulkan build if you are unsure. It is the most broadly compatible option and, as we will see later, sometimes the more reliable one even on NVIDIA hardware.

# check the releases page for the current build number
mkdir -p ~/.local/opt/llama && cd ~/.local/opt/llama
curl -sL -O https://github.com/ggml-org/llama.cpp/releases/download/b10453/llama-b10453-bin-ubuntu-vulkan-x64.tar.gz
tar xzf llama-b10453-bin-ubuntu-vulkan-x64.tar.gz
ln -sf ~/.local/opt/llama/llama-b10453/llama-server ~/.local/bin/llama-server

NOTE 💡 If you have an NVIDIA GPU, install the proprietary driver before anything else, otherwise llama.cpp will not see your card at all. On Fedora that is akmod-nvidia and xorg-x11-drv-nvidia-cuda from RPM Fusion, followed by a reboot. Confirm with nvidia-smi, if that prints a table with your GPU in it you are good.

Step 2: Check what llama.cpp can see

Before downloading twenty gigabytes of model, confirm the GPU is actually visible. This one command saves a lot of confusion later:

llama-server --list-devices

Available devices:
  Vulkan0: Intel(R) Arc(tm) Graphics (MTL) (23691 MiB, 21322 MiB free)
  Vulkan1: NVIDIA GeForce RTX 4070 Laptop GPU (8188 MiB, 7819 MiB free)

Two things to take from this output. First, the amount of free VRAM, this is the single number that governs every decision that follows. Second, the device names. Laptops routinely have two GPUs, an integrated one and a discrete one, and you almost always want the discrete one. Note its identifier so you can pin to it explicitly later with --device.

If nothing but CPU shows up here, your driver or backend build is wrong. Fix that before continuing, everything after this point assumes the GPU is visible.

Step 3: Pick a model and a quant

This is the step that decides whether the whole thing is pleasant or miserable, so it is worth a minute of thought rather than grabbing whatever is trending.

Where to get models. The ggml-org organisation on Hugging Face is maintained by the llama.cpp team, which makes it the safest place to get GGUFs, they are built and tested against the engine you are running. Unsloth is the other well regarded publisher and often has more quant options for a given model.

Dense or mixture of experts. This is the choice that actually matters on constrained hardware. A dense model activates all of its parameters for every token. A mixture of experts (MoE) model has a large total parameter count but only activates a small slice of it per token, which is what makes it possible to run something big on a small card at all.

Your hardware	Reasonable pick
8 GB VRAM or less	A dense 7B to 12B at Q4, fits entirely in VRAM and stays fast
8 to 16 GB VRAM, plenty of RAM	A ~30B MoE, split between GPU and RAM, slower but far more capable
24 GB VRAM or more	A ~30B MoE fully on GPU, or a dense 30B+

I went with Qwen3.6-35B-A3B as the worked example. It has 35 billion parameters in total but only around 3 billion are active for any given token, and that sparsity is the only reason a model this size runs on an 8 GB card. Qwen3-Coder-30B-A3B is the more coding specific sibling and behaves the same way. Substitute freely, every command below takes a file path and does not care what is in it.

Then pick a quantization, which is how aggressively the weights have been compressed. The same model is published at several sizes:

Quant	Size	Notes
BF16	69.4 GB	Full precision. Not happening on consumer hardware.
Q8_0	36.9 GB	8 bit. Excellent quality, still very large.
Q4_K_M	20.4 GB	4 bit. The practical sweet spot.

Read the names as bits per weight plus a quality profile. Q4_K_M is roughly 4.5 bits per weight using llama.cpp's K-quant scheme, in its medium variant. K-quants do not compress every tensor equally, they keep the error sensitive ones at higher precision and squeeze the bulk of the feed forward weights harder, which buys back most of the quality that naive 4 bit loses.

Start with Q4_K_M. It is the best quality you can get without doubling your memory budget, and quantization noise hurts code and tool call JSON more than it hurts prose, so going lower is a real risk when the model is meant to do actual work.

NOTE 💡 Rough sizing rule: your budget is VRAM plus free system RAM, minus a few gigabytes of headroom for the KV cache and the OS. On an 8 GB card with 30 GB of RAM, a 20 GB model fits. A 37 GB one does not.

Step 4: Download it

The CLI pulls straight from Hugging Face. The -hf flag takes a repo, and optionally a quant after a colon:

llama download -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Files land in the standard Hugging Face cache at ~/.cache/huggingface/hub/. Downloads resume if interrupted, and llama serve --cache-list shows what is already there.

I prefer keeping a tidy ~/models directory of symlinks instead of hunting through cache hashes every time. It costs no extra disk:

mkdir -p ~/models
ln -sf ~/.cache/huggingface/hub/models--ggml-org--Qwen3.6-35B-A3B-GGUF/snapshots/*/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  ~/models/local-model.gguf

Some repos also ship a vision projector file alongside the model. If you only want text, pass --no-mmproj to skip downloading it.

Step 5: First chat in the terminal

Before wiring anything up, confirm the model actually runs. llama cli gives you an interactive prompt:

llama cli -m ~/models/local-model.gguf --device Vulkan1 -ngl 99 -c 8192

Three flags worth understanding right away, because they show up everywhere after this:

• --device pins to a specific GPU using the name from step 2. Skip it and llama.cpp picks for you, which on a dual GPU laptop is often the wrong one.
• -ngl 99 means put up to 99 layers on the GPU. The number is clamped to the model's real layer count, so 99 is just shorthand for as much as possible. This is the flag people mean when they talk about offloading.
• -c 8192 sets the context window in tokens. Larger contexts eat meaningfully more memory, so keep it modest while testing.

If this responds, everything downstream will work. If it crashes or silently falls back to CPU, fix it here rather than debugging through two more layers of tooling.

Step 6: Run the server

This is the piece everything else talks to. It serves a browser UI and an OpenAI compatible API from the same port:

llama-server \
  -m ~/models/local-model.gguf \
  --device Vulkan1 \
  -ngl 99 \
  -c 32768 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -np 1 \
  -t 16 \
  --jinja \
  --api-key local-dev-key

Flag	What it does
-c 32768	32K token context window
-fa on	Flash attention. Faster at long context, and required for a quantized KV cache
-ctk q8_0 -ctv q8_0	Store the KV cache at 8 bit instead of 16 bit, roughly halving its memory for a barely noticeable quality cost
-np 1	One request slot. Correct for solo use, extra slots each reserve their own KV cache
-t 16	Thread count. Use your physical core count, not the logical one
--jinja	Use the model's real chat template. Required for tool calling to work
--api-key	A token clients must send. Any string will do

Now open http://127.0.0.1:8080. There is a complete chat interface built in, conversation history, markdown rendering, temperature and sampling settings. It is the fastest way to sanity check the model and, honestly, good enough for plenty of everyday use on its own.

NOTE 💡 You might wonder why set an API key on a local server at all. The server binds to localhost but allows all cross origin requests by default, and it warns you about this on startup. That means a webpage you have open could in principle send requests to it. The key is not protecting state secrets, it just closes that gap, and clients tend to expect an auth token anyway.

Step 7: Wire up your coding agent

The chat UI is where most guides stop. This step is the one that turns the setup into something you can actually work in.

Install OpenCode from opencode.ai, then create ~/.config/opencode/opencode.json to register your local server as a provider:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama-cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1",
        "apiKey": "local-dev-key"
      },
      "models": {
        "local-model": {
          "name": "Local model",
          "limit": { "context": 32768, "output": 8192 }
        }
      }
    }
  },
  "model": "llama-cpp/local-model"
}

Four details that will cost you time if you get them wrong:

1. The baseURL must end in /v1. OpenCode uses the AI SDK's OpenAI compatible provider, which appends /chat/completions to whatever you give it.
2. baseURL and apiKey go inside options, not at the provider root.
3. The apiKey must match the --api-key you passed the server.
4. Use a custom provider id like llama-cpp rather than reusing a built in name like openai.

Verify the config parses before launching anything:

opencode debug config
opencode

Inside the TUI, /models opens the model picker. The full config schema is documented at opencode.ai/docs/providers.

The same three values, base URL, API key and model name, are all any other agent needs. If you would rather use something else, the shape of the config changes but the values do not.

---

Making it actually fit

If your model comfortably fits in VRAM, skip this section, -ngl 99 is all you need. If it does not, which is the common case on most machines, this is where the real work is.

llama.cpp tries to size things automatically, but its auto fit logic has a known weakness with large MoE models: it can plan to put nearly the whole model on the GPU and only discover the problem when it runs out of memory partway through a request. Do not trust it on anything big, check first.

Measure before you guess

llama fit-params estimates memory placement without running anything. It is probably the single most useful command in this whole post:

llama fit-params -m ~/models/local-model.gguf \
  -c 32768 -fa on -ctk q8_0 -ctv q8_0 --fit-print on

(device, model, context, compute) in MiB
CUDA0 19190 402 525
Host    272   0  40

19 GB planned for an 8 GB card. That is never going to work.

Move the experts to system RAM

Here is the trick that makes MoE models viable on small GPUs. In a mixture of experts model, the expert weights are most of the file but are only active a small fraction of the time. The attention layers, which run for every single token, are comparatively tiny.

So you put the attention layers on the GPU where speed matters, and park the experts in system RAM. Two flags do this:

• --cpu-moe keeps all expert weights on the CPU.
• -ncmoe N keeps only the first N layers' experts on the CPU, letting the rest use the GPU.

Sweep N with fit-params until the device total leaves you about a gigabyte of headroom:

for n in 0 20 32 40; do
  llama fit-params -m ~/models/local-model.gguf -c 32768 -fa on \
    -ctk q8_0 -ctv q8_0 --fit-print on -ncmoe $n
done

-ncmoe  0  ->  GPU 19190 MiB    # way over
-ncmoe 20  ->  GPU 10550 MiB    # still over
-ncmoe 32  ->  GPU  5366 MiB    # fits, with room to spare
-ncmoe 40  ->  GPU  1910 MiB    # fits, but wasting the GPU

On my machine -ncmoe 32 was the answer, about 6.2 GB of the 8 GB card in use once context and compute buffers are included. Add that flag to the server command from step 6 and you are done.

One note on threads. Once experts live in system RAM the CPU does real work on every token, so thread count starts to matter. Set -t to your physical core count. llama.cpp's auto detection picked 6 threads on a 16 core CPU for me, and setting it explicitly was worth a noticeable improvement in prompt processing. That said, do not expect miracles, generation with CPU resident experts is limited by memory bandwidth rather than cores, so past a point more threads buy you nothing.

---

When things break

These are the problems you are most likely to hit, and what they actually mean.

Only CPU shows in --list-devices. Either your GPU driver is missing or the backend build does not match your hardware. Confirm the driver first (nvidia-smi for NVIDIA, vulkaninfo for Vulkan), then check you downloaded the right release asset.

CUDA error: the resource allocation failed. If this points at cublasCreate_v2, it is not your configuration. It is a known regression in the CUDA backend where the cuBLAS workspace is allocated lazily on first inference, after the model has already claimed the VRAM. The model loads fine and then dies on your first prompt. Reducing context and forcing more CPU offload did not help me, what did was switching to the Vulkan build, which does not go through cuBLAS at all and so cannot hit this path. It still runs on the NVIDIA GPU through the normal driver. You lose some raw throughput compared to a healthy CUDA build, but a slightly slower server that works beats a fast one that crashes.

It loads, then the machine grinds to a halt. You are swapping. The model plus KV cache exceeds your actual free RAM and the OS is paging to disk. Check with free -h, then reduce -c, use a smaller quant, or close other applications.

Tool calling does not work. Almost always a missing --jinja on the server. Without it llama.cpp does not apply the model's real chat template, so tool call syntax is never parsed correctly, and an agent that cannot call tools is just a chat window. Also check the server log at startup for a warning about the template failing to determine a tool format, that means the GGUF's embedded template is broken and you may need a different build of the same model.

Responses arrive empty or get cut off mid thought. Reasoning models spend tokens thinking before they answer, and that thinking counts against your output limit. If max_tokens is low you can burn the entire budget on internal reasoning and get an empty response back. Raise the limit, or constrain the thinking with --reasoning-budget.

The agent cannot reach the server. Check the three usual suspects in order: the /v1 suffix on baseURL, the apiKey matching the server's --api-key, and whether the server is actually listening (curl http://127.0.0.1:8080/v1/models).

---

What to actually expect

Worth being straight about this, because it is the part most write ups skip.

A 35B MoE model at Q4_K_M, split across an 8 GB laptop GPU and system RAM, generated at roughly 4 to 5 tokens per second for me, with prompt processing around 12 to 13 tokens per second. That is about reading speed. It is usable and it is genuinely free, but it is not the experience of a hosted frontier model and pretending otherwise helps nobody.

Two things move that number meaningfully. First is fitting more of the model into VRAM, which is why the tuning section matters. Second is simply choosing a smaller model, anything that fits entirely in VRAM with no CPU offload at all runs dramatically faster because it never pays the memory bandwidth cost. A dense model in the 7B to 12B range at Q4 fits in 8 GB and feels far more responsive.

So a big MoE model on constrained hardware buys you capability at the cost of speed. That is a reasonable trade for a model you reach for when the credits run out and you want something thorough. It is a poor trade for quick edits and autocomplete, where a small fast model wins easily. Running both and switching based on the task is the setup that actually holds up, and llama.cpp's router mode makes serving two models at once straightforward once you get there.

Is it a full replacement for a premium model? Honestly, not yet, not on a machine like this. But it is a real fallback rather than a toy, and the gap has been closing fast enough that it is worth having the setup ready.

Reference

• llama.cpp, the engine, builds and install options.
• Server README, complete flag and endpoint reference.
• ggml-org on Hugging Face, GGUFs maintained by the llama.cpp team.
• OpenCode docs and provider config.
• Model management in llama.cpp, router mode for serving several models at once.

---

Thanks for reading, if you would like to hear more from me, feel free to reach out via email or follow me on Twitter.