Unlock AI Potential with Docker Model Runner: A Quick Guide

Amaresh Pelleti

The Docker Model Runner is a powerful new plugin that brings AI models directly to your fingertips. Designed for developers and AI enthusiasts alike, it allows you to pull, run, and manage AI models seamlessly from the command line. Whether you’re building a generative AI application or experimenting with pre-trained models, Docker Model Runner simplifies the process by integrating AI capabilities into your local development environment.

Currently in Beta, this plugin is available for Docker Desktop 4.40 and later, specifically optimized for Mac with Apple Silicon. Let’s dive into what makes Docker Model Runner a game-changer and how you can use it effectively.

What Can Docker Model Runner Do?

With Docker Model Runner, you can:

  • Pull AI models directly from Docker Hub
  • Run models interactively or with predefined prompts
  • Manage models locally (add, list, remove)
  • Interact with models using OpenAI-compatible APIs

Models are downloaded from Docker Hub the first time they’re used and cached locally for faster access. This ensures efficient resource management—models are only loaded into memory when actively in use and unloaded afterward.

Getting Started: Enable Docker Model Runner

Before you start running AI models, you’ll need to enable the Docker Model Runner feature in Docker Desktop:

  1. Open Settings in Docker Desktop.
  2. Navigate to the Features in development tab.
  3. Under the Experimental features section, check Access experimental features.
  4. Go to the Beta tab and enable Docker Model Runner.
  5. Restart Docker Desktop to apply the changes.

Once enabled, you’re ready to explore the capabilities of Docker Model Runner!

Key Commands and Examples

1. Check the Status of Docker Model Runner

To confirm that the plugin is active, run:

docker model status

This will return whether the Docker Model Runner is up and running.

2. Pull a Model

Download a model from Docker Hub to your local machine using:

docker model pull <model>

For example:

docker model pull ai/largemodel1

Output:

Downloaded: 1.2 GB  
Model ai/largemodel1 pulled successfully

3. List Available Models

View all models currently downloaded to your system:

docker model list

Sample output:

+MODEL         PARAMETERS  QUANTIZATION    ARCHITECTURE  MODEL ID      CREATED     SIZE  
+ai/largemodel1  7B          Q8_0            llama         abcdef123456  5 days ago  1.2 GiB

4. Run a Model

You can interact with a model either by submitting a one-time prompt or entering interactive chat mode.

One-Time Prompt

docker model run ai/largemodel1 "What is the capital of France?"

Output:

The capital of France is Paris.

Interactive Chat Mode

docker model run ai/largemodel1

Output:

Interactive chat mode started. Type '/exit' to quit.  
> Hi  
Hello! How can I assist you today?  
> /exit  
Chat session ended.

5. Remove a Model

To free up space, you can delete a model from your system:

docker model rm <model>

Example:

docker model rm ai/largemodel1

Output:

Model ai/largemodel1 removed successfully

Integrating Docker Model Runner into Your Workflow

Ready to take your AI projects to the next level? Docker Model Runner makes it easy to integrate AI models into your software development lifecycle. For instance, you can build a custom GenAI application by pulling a model and running it alongside your app.

Here’s how:

  1. Clone a sample repository:
   git clone https://github.com/docker/hello-genai.git
  1. Navigate to the project directory:
   cd hello-genai
  1. Run the setup script to pull the model and launch the app:
   ./run.sh
  1. Open the app in your browser at the address specified in the README file.

Now, you can interact with your own AI-powered app, running entirely on your local machine thanks to Docker Model Runner.

Frequently Asked Questions

What Models Are Available?

All supported models are hosted in the public Docker Hub namespace ai. Explore the available options here.

How Do I Use OpenAI-Compatible APIs?

Docker Model Runner supports OpenAI API endpoints for interacting with models. Here’s an example of making a request via curl:

curl http://model-runner.docker.internal/engines/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "ai/largemodel1",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Explain quantum mechanics in simple terms."
            }
        ]
    }'

Known Issues and Troubleshooting

While Docker Model Runner is still in Beta, there are a few known issues to be aware of:

  1. Command Not Recognized: If you encounter an error like docker: 'model' is not a docker command, create a symlink to fix it:
   ln -s /Applications/Docker.app/Contents/Resources/cli-plugins/docker-model ~/.docker/cli-plugins/docker-model
  1. Running Oversized Models: Be cautious when running large models—they may exceed your system’s resources, leading to slowdowns or crashes.
  2. Misleading Progress Reports: If a pull fails partway through, subsequent attempts may show “0 bytes” downloaded even though the process is working in the background.

Share Your Feedback

We’d love to hear about your experience with Docker Model Runner! Whether you’ve encountered a bug or have suggestions for improvement, use the Give feedback link in Docker Desktop settings to share your thoughts.

Disable Docker Model Runner

If you no longer wish to use the plugin, you can disable it by:

  1. Opening Settings in Docker Desktop.
  2. Navigating to the Beta tab under Features in development.
  3. Clearing the Enable Docker Model Runner checkbox.
  4. Restarting Docker Desktop.

Conclusion

Docker Model Runner is a revolutionary tool that bridges the gap between AI models and local development environments. By enabling seamless interaction with AI models directly from the command line, it empowers developers to experiment, build, and deploy AI-driven applications with ease.

So why wait? Dive into the world of AI with Docker Model Runner today and unlock endless possibilities!

Have you tried Docker Model Runner yet? Share your thoughts and experiences in the comments below!

Leave a Reply