Running Ollama in a container takes one docker run command. That command also decides two things the docs bury: whether your models survive a rebuild, and whether your GPU gets used. This Ollama Docker setup guide covers both. I ran it on Docker 29.4.3 with Ollama 0.40.13 and hit three problems the one-liner doesn’t warn you about.
What Is the Minimum Ollama Docker Setup?
The minimum Ollama Docker setup is one container with a named volume and port 11434 published. The official Ollama Docker docs give this CPU-only command, and it worked as written:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaFirst, check that the API answers, then pull and run a model inside the container:
curl http://localhost:11434/
docker exec -it ollama ollama run llama3.2The first curl returns Ollama is running. The server logs show what hardware it found, and on my Mac the line read inference compute id=cpu. That log line is your first GPU check. The second check is docker exec ollama ollama ps, whose PROCESSOR column read 100% CPU for a loaded model.
Inside the container, Ollama binds to 0.0.0.0:11434 by default. That is why the published port works without setting OLLAMA_HOST yourself.
How Do Volumes Keep Ollama Models Between Containers?
A named volume mounted at /root/.ollama keeps your models when you delete and recreate the container. Without it, every docker rm throws away every model you pulled, and the next container starts empty.
I tested both cases with a 270 MB model, smollm2:135m:
# Same volume, new container: the model is still there
docker rm -f ollama
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec ollama ollama list # smollm2:135m listed
# No volume, new container: nothing
docker rm -f ollama
docker run -d -p 11434:11434 --name ollama ollama/ollama
docker exec ollama ollama list # emptyHowever, the volume holds more than models. Mounting a bind directory showed four entries in it: models, cache, id_ed25519 and id_ed25519.pub. Ollama generates that key pair on first start (“Couldn’t find ‘/root/.ollama/id_ed25519’. Generating new private key” in the log). So if you lose the volume, you lose the key too, not just the downloads.
⚠️ Note: the volume name changes between docker run and Compose. Compose prefixes volume names with the project name, so a volume called ollama in compose.yaml became comp_ollama on my machine. Models you pulled with the plain docker run command above won’t show up. Set name: ollama under the volume to pin it, as in the file further down.
How Do You Enable GPU in an Ollama Docker Setup?
For NVIDIA, install the NVIDIA Container Toolkit on the host and register it with Docker. Then add --gpus=all to the run command. AMD uses the rocm image tag with the /dev/kfd and /dev/dri devices. Docker Desktop on macOS has no GPU passthrough at all.
NVIDIA Ollama Docker Setup
Ollama’s docs walk through the toolkit install for apt and yum, and the NVIDIA install guide has the current repository steps. After installing the package, point Docker at it and restart the daemon:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerThen start the container with every GPU visible:
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaOn top of that, your card needs compute capability 5.0 or higher and a driver version 550 or newer, according to the Ollama GPU docs. Cards on compute capability 5.0 through 6.2 need driver 570 or newer.
AMD Ollama Docker Setup
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocmPlus, the rocm tag is the smaller download. Docker Hub lists the 0.40.13 amd64 image at 3.39 GB and 0.40.13-rocm at 1.45 GB.
Ollama Docker Setup on macOS
Ollama’s FAQ is blunt: GPU acceleration is not available for Docker Desktop on macOS. On my Mac, --gpus=all failed with failed to discover GPU vendor from CDI: no known GPU vendor found. So if you want Apple GPU speed, run Ollama natively and keep Docker for everything else. Ollama vs LM Studio covers the native install.
How do you confirm the container is really using the GPU?
Run a prompt, then check docker exec ollama ollama ps. A PROCESSOR value of 100% GPU means the model sits entirely in VRAM. 100% CPU means no GPU was found. A split like 48%/52% CPU/GPU means the model sits partly in each. Because of that, a healthy status proves nothing about the GPU.
How Do You Write an Ollama Docker Compose File?
An Ollama Docker Compose file needs four things beyond the image. Those are a named volume, a localhost port binding, a healthcheck, and an optional one-shot service that pulls your model. This is the file I tested, with the GPU block split out into its own file so the base file runs anywhere:
# compose.yaml
services:
ollama:
image: ollama/ollama:0.40.13
container_name: ollama
restart: unless-stopped
ports:
- "127.0.0.1:11434:11434"
environment:
OLLAMA_KEEP_ALIVE: 30m
OLLAMA_NUM_PARALLEL: 2
volumes:
- ollama:/root/.ollama
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
pull-model:
image: ollama/ollama:0.40.13
depends_on:
ollama:
condition: service_healthy
environment:
OLLAMA_HOST: ollama:11434
entrypoint: ["ollama", "pull", "smollm2:135m"]
restart: "no"
volumes:
ollama:
name: ollama# compose.gpu.yaml (NVIDIA hosts only)
services:
ollama:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]Start it on a CPU host with docker compose up -d. On a GPU host, layer the second file on top:
docker compose -f compose.yaml -f compose.gpu.yaml up -dWhy use ollama list as the healthcheck in an Ollama Docker setup?
The image ships without curl or wget, so the usual curl -f http://localhost:11434 healthcheck fails with a missing binary. ollama list talks to the local server, so it works as a probe. With it, pull-model waited for the Healthy state, pulled the model, and exited with code 0.
Why does the GPU block get its own file?
Because Compose refuses to start the service on a host without the NVIDIA runtime. With the deploy block in the single file, docker compose up on my Mac stopped with could not select device driver "nvidia" with capabilities: [[gpu]]. The Docker docs say the capabilities field is required. They also say count and device_ids are mutually exclusive. Check those two first if your GPU block fails on a real GPU host. See the Compose GPU support page for the full field list.
Why bind the port to 127.0.0.1?
Because -p 11434:11434 publishes on every interface. After a plain run, docker port showed 0.0.0.0:11434 and [::]:11434. The 127.0.0.1:11434:11434 form in the file above showed 127.0.0.1:11434 only. Docker also publishes ports ahead of ufw rules, which its own docs cover under Docker and ufw. If this host faces the internet, read the Ollama VPS hardening guide before you publish anything.
Which Environment Variables Matter in Docker?
Five variables cover most tuning, and you set them with -e on docker run or under environment: in Compose. Defaults below come from the server config line Ollama prints at startup and from the Ollama FAQ.
| Variable | Default | What it does |
|---|---|---|
OLLAMA_KEEP_ALIVE |
5m |
How long an idle model stays in memory |
OLLAMA_NUM_PARALLEL |
1 |
Parallel requests per model; RAM scales with this times the context length |
OLLAMA_MAX_LOADED_MODELS |
3 per GPU, or 3 on CPU | Models loaded at once, if they fit |
OLLAMA_MODELS |
/root/.ollama/models |
Where models live inside the container |
OLLAMA_VULKAN |
true |
Vulkan GPU support, bundled in the image |
One more detail from the startup log: with no GPU, Ollama picked a default context of 4096 tokens (default_num_ctx=4096). Set OLLAMA_CONTEXT_LENGTH if your prompts need more, and size memory for it. Ollama hardware requirements has the RAM and VRAM math.
What Breaks in an Ollama Docker Setup?
Three things broke or surprised me, and the first one costs the most.
⚠️ docker compose down -v deletes the models volume, and the -v flag is the only difference. Plain docker compose down stopped the containers and left the ollama volume in place. Adding -v removed it. Models and key pair went with it. Use it only when you want a clean slate:
docker compose down # keeps the ollama volume
docker compose down -v # deletes the volume and every model in itThe other two:
latestmoves fast. Docker Hub showed0.40.12on 27 October, release candidates on the 28th, and0.40.13on the 29th. Pin the tag in Compose so a restart doesn’t pull a new version while you’re debugging.- But the image is big. The amd64 image is 3.39 GB compressed on Docker Hub. The arm64 image used 4.29 GB on my disk once pulled. Budget for that before the first model.
What I Did Not Test
I tested everything on a Mac with no NVIDIA, AMD or Vulkan GPU. So none of the GPU paths ran against real hardware. I did not run the NVIDIA toolkit install, the rocm image, Vulkan, Windows with WSL2 or Jetson. Multi-GPU selection with device_ids is untested too. Those steps come from the official docs I linked, and the only GPU behavior I observed was the failure messages above. Everything else in this guide ran as written.
Frequently Asked Questions
Q: How do I run Ollama in Docker with GPU support?
A: On NVIDIA, install the NVIDIA Container Toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, restart Docker, then start the container with --gpus=all. On AMD, use the ollama/ollama:rocm image with --device /dev/kfd --device /dev/dri. Confirm with ollama ps.
Q: Where does Ollama store models in Docker?
A: Inside the container at /root/.ollama/models. Mount a named volume at /root/.ollama so models survive container removal. The same directory holds the id_ed25519 key pair Ollama generates on first start.
Q: Why is Ollama using the CPU inside Docker?
A: Ollama found no GPU the container can access. Check that you passed --gpus=all or the AMD devices. Check that the host driver meets the version requirement. Then run docker exec ollama ollama ps and look for 100% CPU. Docker Desktop on macOS can never pass a GPU through.
Q: Can I use Ollama in Docker on a Mac?
A: Yes, on the CPU only. Ollama’s FAQ states GPU acceleration is unavailable for Docker Desktop on macOS. Run Ollama natively for Apple GPU performance.
Quick Summary:
– Mount a named volume at /root/.ollama, or every container removal deletes your models and the key pair.
– Compose prefixes volume names, so set name: ollama to share one volume between docker run and Compose.
– Check the GPU with docker exec ollama ollama ps, not the container’s health status.
– Put the NVIDIA deploy block in a separate override file, because the single-file version fails on hosts without the runtime.
– The image has no curl, so use ollama list as the healthcheck, and bind the port to 127.0.0.1 unless you mean to expose it.
Your Ollama Docker setup is now running, so point your apps at it with the Ollama API guide.