FR
Download CV ↓
CV in French · pick a theme Dark theme Light theme
← Back to projects Project 04 / 04
Homelab · Local AI

A 100% local AI agent

The open-source Qwen3 model runs on my own graphics card with Ollama, and OpenCode gives it tools: reading files, running commands, proposing changes. No data ever leaves the machine.

See the diagram →
Type
Personal project · homelab
Hardware
NVIDIA RTX 4060 Ti · CUDA
Model
Qwen3 8B / 14B via Ollama
Harness
OpenCode, in the terminal
Architecture of my local AI In the terminal, I give OpenCode a task. OpenCode sends the request and its list of tools to Ollama's local API, which runs Qwen3 on the RTX 4060 Ti graphics card. The model answers with text or a tool call; OpenCode runs the tool on the files in the working folder and sends the result back to the model, until the final answer. No request goes out to the Internet. MY MACHINE · EVERYTHING STAYS LOCAL INTERNET no data sent 01 · ME Terminal I describe the task 02 · AGENT HARNESS OpenCode handles context, tools and the loop request + tools the model decides runs the tool returns the result loop read / write shell commands file search can ask before running a command WORKING FOLDER Files logs · scripts · configs request response 03 · MODEL SERVER Ollama local OpenAI-compatible API loads the model into memory 04 · OPEN-SOURCE MODEL Qwen3 · 8B / 14B 4-bit quantized · tool calling 05 · GRAPHICS CARD RTX 4060 Ti · CUDA VRAM
01 — Goal

Understanding what sits
behind an AI agent.

Cloud AI assistants are handy, but you can't send them just anything: production logs, configurations or client data must not leave the company. I wanted to see what can be done with a model that runs entirely locally.

The other goal was to understand how an agent works end to end: a model on its own only produces text. The harness, the program around the model, gives it tools, runs what it asks for and sends the results back.

100 %Inference on my graphics card
0Data sent to the cloud
0 €Cost per request
02 — Architecture

Four layers,
one machine.

The diagram at the top of the page reads from left to right. Each layer has a clear role:

  • OpenCode (the harness): receives my request, keeps track of the conversation, describes its tools to the model and runs the ones the model asks for.
  • Ollama (the model server): loads the model onto the graphics card and exposes a local OpenAI-compatible API, which OpenCode uses.
  • Qwen3 (the model): reads the context and answers either with text or with a structured tool call.
  • The RTX 4060 Ti: does the computation through CUDA. Its video memory (VRAM) decides which models I can run.

How a request flows

  1. I describe the taskFor example: “analyse this log file and summarise the errors”.
  2. OpenCode builds the requestIt sends Ollama my request, the history and the list of available tools.
  3. Qwen3 decidesIt answers directly, or asks for a tool: read a file, run a command, search a folder.
  4. OpenCode runs the toolOn the files in the working folder; it can ask for my confirmation before a command.
  5. The result goes back to the modelThe loop continues until the model has enough to give its final answer.

No web interface: everything happens in the terminal. No extra service is exposed on the network.

03 — Technical choices

Simple,
open-source building blocks.

Ollamamodel server
Simple setup, automatic GPU handling, models pulled with one command and an OpenAI-compatible API that most tools can connect to.
Qwen3model
Open-source model that handles tool calling well, which an agent needs. The 8B and 14B sizes fit on a consumer graphics card once quantized.
8B or 14Btrade-off
The 8B answers faster and leaves more memory for context; the 14B reasons better on multi-step tasks. I pick one depending on the task.
OpenCodeagent harness
Open-source terminal agent that connects to any OpenAI-compatible API, including Ollama. It provides the tools (files, shell, search) and the agent loop.
04 — Limits

What I took away.

  • A local model is more limited than a large cloud model: on long tasks it loses track sooner and its tool calls are sometimes malformed. I always review what it proposes.
  • Video memory is the real constraint: it limits both model size and context length.
  • The harness matters as much as the model: it is what makes the model useful, by giving it tools and limiting what it can do.

Next steps

  • Connect the agent to my homelab logs to help with troubleshooting.
  • Compare other open-source models on the same admin tasks.

Skills used

  • Ollama
  • Qwen3
  • OpenCode
  • CUDA
  • LLMs & quantization
  • Agents & tool calling