Do I need a GPU to run this?

No, Ollama runs on CPU by default and the generated compose file works without the GPU block. Enabling the GPU reservation only makes sense if you have an NVIDIA GPU and the NVIDIA Container Toolkit installed on the Docker host; without that toolkit, the deploy.resources.reservations block will fail to schedule the container.

Why is the model cache a named volume instead of a bind mount?

A named Docker volume persists model weights (which can be several gigabytes each) across container restarts and image upgrades without depending on a specific host path, which is what the official Ollama docs recommend for the /root/.ollama directory.

How do I actually load a model after `docker compose up`?

Run the `docker compose exec ollama ollama pull ` command shown in the generated file's trailing comment; this downloads the model into the named volume so it persists for future runs.

What does OLLAMA_HOST do here?

It controls the address and port Ollama's API server binds to inside the container. Setting it to 0.0.0.0:11434 (the default) makes the API reachable from outside the container via the mapped port; restricting it to 127.0.0.1 would make it unreachable through the port mapping.

Ollama docker-compose.yml Generator

Overview

Introduction

Running Ollama locally is usually one `docker run` command, until you want persistent model storage, GPU acceleration, and a reproducible setup you can check into a repo, at which point a docker-compose.yml is the better fit.

This tool generates that compose file: the official image, the exposed API port, a persistent named volume, an optional GPU reservation, and a reminder of the pull command to run once the container is up.

What Is Ollama Docker Compose Generator?

A generator for a single-service docker-compose.yml running `ollama/ollama:latest`, with port 11434 mapped, a named volume mounted at /root/.ollama for model persistence, and an OLLAMA_HOST environment variable.

It optionally adds a `deploy.resources.reservations.devices` block requesting an NVIDIA GPU, matching the configuration the NVIDIA Container Toolkit expects for GPU-accelerated inference.

How Ollama Docker Compose Generator Works

You name the cache volume, set the OLLAMA_HOST value, toggle GPU reservation, and specify which model you plan to pull. The generator validates the volume name and required fields, then assembles a valid compose file.

The GPU toggle inserts a `deploy.resources.reservations.devices` entry with `driver: nvidia`, `count: all`, and `capabilities: [gpu]`, the standard shape Docker Compose uses to hand a container access to the host's GPU(s) via the NVIDIA Container Toolkit.

When To Use Ollama Docker Compose Generator

Use it when standing up a local or self-hosted Ollama instance for development, testing a RAG pipeline, or serving models to other containers on the same Docker network.

It's also useful when you need GPU acceleration and don't want to hand-write the NVIDIA reservation block from memory each time.

Often used alongside Hugging Face Config Generator, OpenAI Environment Generator and LangChain Config Generator.

Features

Advantages

Persists downloaded models in a named volume so they survive container recreation and image upgrades.
Includes the exact `docker compose exec ollama ollama pull <model>` command you need to run after startup, rather than leaving you to guess it.
GPU support is a single toggle away, using the standard Compose device-reservation shape rather than a bespoke script.

Limitations

GPU acceleration additionally requires the NVIDIA Container Toolkit installed on the host; the compose file alone doesn't install it.
Only configures a single Ollama service; multi-model routing, load balancing across replicas, or a reverse proxy in front of it are out of scope.

Examples

GPU-enabled setup for llama3.1

Input

volumeName: ollama_data, gpuEnabled: true, ollamaHost: 0.0.0.0:11434, pullModel: llama3.1

Output

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama_data:

# docker compose exec ollama ollama pull llama3.1

The GPU device reservation only appears when the toggle is on; the pull command in the trailing comment always matches the model field.

Best Practices & Notes

Best Practices

Pull the model once after the first `docker compose up -d`, since the volume persists it for subsequent restarts without re-downloading.
Keep OLLAMA_HOST bound to 0.0.0.0 inside the container and control external exposure through the Docker port mapping or a reverse proxy, rather than trying to restrict it from inside the container.
Size the host's disk for the models you plan to run; a 70B-parameter model can require tens of gigabytes in the volume.

Developer Notes

The GPU reservation follows the Compose Specification's `deploy.resources.reservations.devices` schema (`driver: nvidia`, `capabilities: [gpu]`), which `docker compose up` only honors when the NVIDIA Container Toolkit's runtime is registered with the Docker daemon; without it, Compose will start the container without GPU access rather than failing outright on most setups.

Ollama Docker Compose Generator Use Cases

Standing up a local Ollama server for development against a RAG or agent pipeline
Serving a self-hosted model to other containers on the same Docker Compose network
Reproducing a GPU-accelerated Ollama setup consistently across machines via a checked-in compose file

Common Mistakes

Enabling the GPU block without having the NVIDIA Container Toolkit installed, then being confused why inference still runs on CPU.
Forgetting to pull a model after the first startup and then getting a 404 from the API when trying to generate against it.

Tips

Add this service into a larger docker-compose.yml alongside your app container and reference it by service name, e.g. http://ollama:11434, over the shared Compose network instead of localhost.

Ollama Docker Compose Generator