harbor trials

Run and manage individual trials

The harbor trials command group provides commands for running and managing individual trials. A trial is a single execution of an agent on a task.

Commands

harbor trials start

Start a single trial.

harbor trials start [OPTIONS]

Configuration

-c, --config · Path

Path to a trial configuration file in YAML or JSON format. Should implement the schema of harbor.models.trial.config:TrialConfig. Allows for more granular control over the trial configuration.

Trial Settings

-p, --path · Path

Path to a local task directory, or path within git repo if --task-git-url is specified.

--trial-name · string

Name of the trial. Default: auto-generated

--trials-dir · Path

Directory to store trial results. Default: ./trials

--timeout-multiplier · float

Multiplier for task timeouts. Default: 1.0

--agent-timeout-multiplier · float

Multiplier for agent execution timeout. Overrides --timeout-multiplier.

--verifier-timeout-multiplier · float

Multiplier for verifier timeout. Overrides --timeout-multiplier.

--agent-setup-timeout-multiplier · float

Multiplier for agent setup timeout. Overrides --timeout-multiplier.

--environment-build-timeout-multiplier · float

Multiplier for environment build timeout. Overrides --timeout-multiplier.

Agent Options

-a, --agent · AgentName

Agent name. Default: oracle

--agent-import-path · string

Import path for custom agent.

-m, --model · string

Model name for the agent.

--agent-timeout · float

Agent execution timeout in seconds. Overrides task default.

--agent-setup-timeout · float

Agent setup timeout in seconds. Overrides default.

--agent-kwarg · list[string]

Additional agent kwarg in the format key=value. You can view available kwargs by looking at the agent’s __init__ method. Can be set multiple times to set multiple kwargs.

Common kwargs include: version, prompt_template, etc.

--ae, --agent-env · list[string]

Environment variable to pass to the agent in KEY=VALUE format. Can be used multiple times.

Example: --ae AWS_REGION=us-east-1

Environment Options

--environment-type · EnvironmentType

Environment type. Default: docker

--environment-import-path · string

Import path for custom environment (e.g., module.path:ClassName).

--force-build/--no-force-build · boolean

Whether to force rebuild the environment. Default: --no-force-build

--delete/--no-delete · boolean

Whether to delete the environment after completion. Default: --delete

--override-cpus · int

Override the number of CPUs for the environment.

--override-memory-mb · int

Override the memory (in MB) for the environment.

--override-storage-mb · int

Override the storage (in MB) for the environment.

--override-gpus · int

Override the number of GPUs for the environment.

--environment-kwarg · list[string]

Environment kwarg in key=value format. Can be used multiple times.

Verifier Options

--verifier-timeout · float

Verifier execution timeout in seconds. Overrides task default.

Task Options

--task-git-url · string

Git URL for a task repository.

--task-git-commit · string

Git commit ID for the task. Requires --task-git-url.

Examples

Run a single trial on a local task:

harbor trials start \
  --path ./my-task \
  --agent claude-code \
  --model anthropic/claude-opus-4-1

Run with custom timeout:

harbor trials start \
  --path ./my-task \
  --agent claude-code \
  --model anthropic/claude-opus-4-1 \
  --agent-timeout 3600

Run on Daytona:

harbor trials start \
  --path ./my-task \
  --agent claude-code \
  --model anthropic/claude-opus-4-1 \
  --environment-type daytona

Use a configuration file:

harbor trials start --config trial-config.yaml

Example trial-config.yaml:

trial_name: my-trial
trials_dir: ./my-trials
timeout_multiplier: 2.0
agent:
  name: claude-code
  model_name: anthropic/claude-opus-4-1
  kwargs:
    version: "1.0"
task:
  path: ./my-task
environment:
  type: docker
  force_build: false
  delete: true

harbor trials summarize

Summarize a single trial using Claude Agent SDK.

harbor trials summarize <TRIAL_PATH> [OPTIONS]

Arguments

TRIAL_PATH · Path · required

Path to the trial directory to summarize.

Options

-m, --model · string

Model to use for summarization (e.g., haiku, sonnet, opus). Default: haiku

--overwrite · boolean

Overwrite existing summary.md file.

Examples

Summarize a failed trial:

harbor trials summarize ./trials/my-task__claude-code__attempt-1

Use a different model:

harbor trials summarize \
  ./trials/my-task__claude-code__attempt-1 \
  --model sonnet

Regenerate summary:

harbor trials summarize \
  ./trials/my-task__claude-code__attempt-1 \
  --overwrite

Trial Directory Structure

A typical trial directory structure:

./trials/my-task__claude-code__attempt-1/
├── result.json              # Trial result with rewards and timing
├── trajectory.json          # Agent trajectory (if ATIF supported)
├── summary.md               # Generated summary (if using summarize)
├── logs/
│   ├── agent/
│   │   ├── stdout.txt       # Agent stdout
│   │   └── stderr.txt       # Agent stderr
│   ├── environment/
│   │   ├── build.log        # Environment build log
│   │   └── runtime.log      # Environment runtime log
│   └── verifier/
│       ├── stdout.txt       # Verifier stdout
│       ├── stderr.txt       # Verifier stderr
│       └── reward.txt       # Reward value
└── artifacts/               # Downloaded artifacts (if specified)

Trial Results

The result.json file contains:

{
  "trial_name": "my-task__claude-code__attempt-1",
  "task_name": "my-task",
  "started_at": "2026-03-03T12:00:00Z",
  "finished_at": "2026-03-03T12:15:00Z",
  "verifier_result": {
    "rewards": {"reward": 1.0},
    "stdout": "...",
    "stderr": "..."
  },
  "exception_info": null,
  "metadata": {...}
}

Use Cases

Development and Debugging

Trials are useful for:

• Testing a single task during development
• Debugging agent behavior
• Iterating on task definitions
• Testing custom agents or environments

Running Quick Tests

Quickly test a task before running a full job:

harbor trials start --path ./new-task --agent oracle

Analyzing Individual Failures

After a job completes, deep dive into specific failures:

harbor trials summarize ~/.cache/harbor/jobs/my-job/task1__agent__attempt-1

See Also

• harbor run - Run complete evaluation jobs
• harbor jobs - Manage jobs
• harbor tasks - Manage task definitions