Installation

Install Harbor and set up your evaluation environment

System Requirements

Before installing Harbor, ensure your system meets these requirements:

Python Version

Python 3.12 or higher is required.

Check your Python version:

python --version

If you need to upgrade, visit python.org or use a version manager like pyenv.

Docker (for local execution)

Docker is required for running evaluations locally.

Check if Docker is installed:

docker --version

Install Docker:

• macOS/Windows: Docker Desktop
• Linux: Follow the official Docker Engine installation guide

Ensure Docker daemon is running before executing Harbor evaluations. Test with docker ps.

Operating System

Harbor supports:

• Linux (recommended for production)
• macOS (great for development)
• Windows (via WSL2 with Docker Desktop)

For Windows users, we recommend using WSL2 with Ubuntu.

Memory & Storage

Recommended resources:

• RAM: 8GB minimum, 16GB+ recommended for concurrent evaluations
• Storage: 20GB+ free space for Docker images and task environments
• CPU: Multi-core processor for parallel execution

Memory requirements scale with --n-concurrent. Each concurrent trial typically uses 1-2GB RAM.

Installation Methods

Harbor can be installed using uv (recommended) or pip.

Using uv (Recommended)

uv is a fast Python package installer and resolver. It’s the recommended way to install Harbor.

1. Install uv

   If you don’t have uv installed:

   macOS/Linux

   curl -LsSf https://astral.sh/uv/install.sh | sh
   

   powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
   

   pip install uv
   
2. Install Harbor

   Install Harbor as a tool:

   uv tool install harbor
   

   This installs Harbor and its dependencies in an isolated environment.

3. Verify installation

   Check that Harbor is installed correctly:

   harbor --version
   

   You should see output like:

   0.1.45
   

Using pip

You can also install Harbor using standard pip:

1. Create a virtual environment (optional but recommended)

   python -m venv harbor-env
   source harbor-env/bin/activate  # On Windows: harbor-env\Scripts\activate
   
2. Install Harbor

   pip install harbor
   
3. Verify installation

   harbor --version
   

Command-Line Aliases

Harbor provides convenient command aliases for faster typing:

harbor    # Full command
hr        # Short alias
hb        # Alternative alias

All three are equivalent:

harbor run --dataset terminal-bench@2.0 --agent claude-code
hr run --dataset terminal-bench@2.0 --agent claude-code
hb run --dataset terminal-bench@2.0 --agent claude-code

API Keys Setup

To use AI agents, you’ll need API keys from the respective providers.

Anthropic (Claude)

For Claude Code and Claude-based agents:

1. Get an API key from Anthropic Console
2. Set the environment variable:

export ANTHROPIC_API_KEY=sk-ant-...

Make it persistent by adding to your shell profile (~/.bashrc, ~/.zshrc, etc.):

echo 'export ANTHROPIC_API_KEY=sk-ant-...' >> ~/.bashrc
source ~/.bashrc

OpenAI

For OpenAI models and agents:

1. Get an API key from OpenAI Platform
2. Set the environment variable:

export OPENAI_API_KEY=sk-...

Google (Gemini)

For Gemini CLI agent:

1. Get an API key from Google AI Studio
2. Set the environment variable:

export GEMINI_API_KEY=...

Other Providers

Harbor supports any LiteLLM-compatible provider:

• OpenRouter: export OPENROUTER_API_KEY=...
• Together AI: export TOGETHER_API_KEY=...
• Replicate: export REPLICATE_API_KEY=...
• Hugging Face: export HUGGINGFACE_API_KEY=...

See LiteLLM docs for the complete list.

Store API keys in a .env file in your project directory. Harbor automatically loads environment variables from .env files.

Cloud Provider Setup

For cloud-based evaluation execution, configure your provider credentials.

Daytona

Daytona provides managed development environments.

1. Sign up at daytona.io
2. Get your API key from the dashboard
3. Set the environment variable:

export DAYTONA_API_KEY=...

4. Run evaluations:

harbor run --dataset terminal-bench@2.0 --agent claude-code --env daytona --n-concurrent 100

Modal

Modal provides serverless infrastructure.

1. Install Modal CLI:

pip install modal

2. Authenticate:

modal token new

3. Run evaluations:

harbor run --dataset terminal-bench@2.0 --agent claude-code --env modal --n-concurrent 50

E2B

E2B provides code execution sandboxes.

1. Sign up at e2b.dev
2. Get your API key
3. Set the environment variable:

export E2B_API_KEY=...

4. Run evaluations:

harbor run --dataset terminal-bench@2.0 --agent claude-code --env e2b

Runloop

Runloop provides DevOps automation.

1. Set up Runloop credentials according to their documentation
2. Run evaluations:

harbor run --dataset terminal-bench@2.0 --agent claude-code --env runloop

GKE

Google Kubernetes Engine for large-scale deployments.

1. Set up GCP credentials and kubectl access
2. Configure GKE cluster
3. Run evaluations:

harbor run --dataset terminal-bench@2.0 --agent claude-code --env gke

Docker Configuration

For local execution, optimize Docker settings for Harbor:

Resource Limits

Configure Docker Desktop resource limits:

1. Open Docker Desktop Settings
2. Go to Resources
3. Adjust:
   • CPUs: Allocate enough for --n-concurrent trials
   • Memory: At least 2GB per concurrent trial
   • Disk space: 50GB+ recommended

For Docker Engine on Linux, modify /etc/docker/daemon.json:

{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 64000,
      "Soft": 64000
    }
  }
}

Image Caching

Harbor caches Docker images for faster subsequent runs. To manage cache:

# View cache usage
harbor cache info

# Clear cache
harbor cache clear

# Prune old Docker images
docker system prune -a

Network Configuration

Some tasks require internet access. Ensure Docker containers can reach external networks:

# Test network connectivity
docker run --rm alpine ping -c 1 google.com

If you’re behind a proxy, configure Docker proxy settings in Docker Desktop or daemon configuration.

Verification

Verify your Harbor installation is working correctly:

1. Check version and help

   harbor --version
   harbor --help
   
2. List available datasets

   harbor datasets list
   

   This should show available benchmarks from the Harbor registry.

3. Run a test evaluation

   Test with the Oracle agent (always succeeds):

   # Create a simple task
   mkdir -p test-task/tests
   echo 'Create a file called hello.txt with "Hello, world!"' > test-task/instruction.md
   cat > test-task/task.toml << EOF
   version = "1.0"
   [agent]
   timeout_sec = 60.0
   [verifier]
   timeout_sec = 60.0
   EOF
   
   cat > test-task/tests/test_state.py << EOF
   from pathlib import Path
   def test_hello():
       assert Path("/app/hello.txt").read_text().strip() == "Hello, world!"
   EOF
   
   # Run evaluation
   harbor run --path test-task --agent oracle
   

   If successful, you should see a completion message with reward 1.0.

Troubleshooting

Python version error

Error: Python 3.12 or higher is required

Solution: Upgrade Python using:

• pyenv: pyenv install 3.12 && pyenv global 3.12
• Official installer
• System package manager: brew install python@3.12 (macOS)

Docker daemon not running

Error: Cannot connect to the Docker daemon

Solution:

• Start Docker Desktop (macOS/Windows)
• Start Docker daemon: sudo systemctl start docker (Linux)
• Verify: docker ps

Permission denied (Docker)

Error: Permission denied while trying to connect to Docker

Solution (Linux):

sudo usermod -aG docker $USER
newgrp docker

Or run Docker commands with sudo.

Module import errors

Error: ImportError: No module named 'harbor'

Solution:

• Ensure you’re in the correct virtual environment
• Reinstall: pip install --force-reinstall harbor
• Check installation: pip show harbor

API key not found

Error: API key not found for provider

Solution:

• Set environment variable: export ANTHROPIC_API_KEY=...
• Check it’s set: echo $ANTHROPIC_API_KEY
• Add to shell profile for persistence

Out of memory during evaluation

Error: Docker containers killed or system freezes

Solution:

• Reduce --n-concurrent flag
• Increase Docker memory limits
• Use cloud provider for large-scale runs

Development Installation

For contributing to Harbor or developing custom extensions:

1. Clone the repository

   git clone https://github.com/laude-institute/harbor.git
   cd harbor
   
2. Install with development dependencies

   uv sync --all-extras --dev
   
3. Run tests

   uv run pytest tests/
   
4. Format and lint

   uvx ruff format .
   uvx ruff check --fix .
   uvx ty check
   

Uninstallation

To remove Harbor:

uv

uv tool uninstall harbor

pip uninstall harbor

Optionally clean up cached data:

rm -rf ~/.cache/harbor
rm -rf ~/.harbor

Next Steps

Quickstart Guide

Run your first evaluation in minutes

Core Concepts

Learn about Harbor’s architecture

CLI Reference

Explore all available commands

Configuration

Advanced configuration options

Need help? Join our Discord community for support.