Job Configuration

Configuration models for Harbor jobs and orchestration

JobConfig

Defines the complete configuration for a Harbor evaluation job.

Import: from harbor.models.job.config import JobConfig

Fields

job_name · str · default: "datetime.now()"

Name of the job. Defaults to current timestamp in format YYYY-MM-DD__HH-MM-SS.

jobs_dir · Path · default: "Path('jobs')"

Directory where job results are stored.

n_attempts · int · default: "1"

Number of attempts to run each trial (for measuring variance).

timeout_multiplier · float · default: "1.0"

Global multiplier for all timeout values.

agent_timeout_multiplier · float | None · default: "None"

Specific multiplier for agent timeouts. Overrides timeout_multiplier for agents.

verifier_timeout_multiplier · float | None · default: "None"

Specific multiplier for verifier timeouts. Overrides timeout_multiplier for verification.

agent_setup_timeout_multiplier · float | None · default: "None"

Specific multiplier for agent setup timeouts.

environment_build_timeout_multiplier · float | None · default: "None"

Specific multiplier for environment build timeouts.

debug · bool · default: "False"

Enable debug logging.

orchestrator · OrchestratorConfig · default: "OrchestratorConfig()"

Configuration for trial orchestration.

environment · EnvironmentConfig · default: "EnvironmentConfig()"

Default environment configuration applied to all trials.

verifier · VerifierConfig · default: "VerifierConfig()"

Default verifier configuration.

metrics · list[MetricConfig] · default: "[]"

Metrics to compute across trials.

agents · list[AgentConfig] · default: "[AgentConfig()]"

List of agents to evaluate.

datasets · list[LocalDatasetConfig | RegistryDatasetConfig] · default: "[]"

Datasets to evaluate against.

tasks · list[TaskConfig] · default: "[]"

Individual tasks to run (alternative to datasets).

artifacts · list[str | ArtifactConfig] · default: "[]"

Files or directories to collect from trial environments.

Example

from harbor.models.job.config import JobConfig, OrchestratorConfig
from harbor.models.trial.config import AgentConfig, EnvironmentConfig
from harbor.models.metric.config import MetricConfig
from harbor.models.metric.type import MetricType

config = JobConfig(
    job_name="benchmark-evaluation",
    n_attempts=3,
    timeout_multiplier=1.5,
    debug=True,
    agents=[
        AgentConfig(
            name="claude-code",
            model_name="anthropic/claude-opus-4-1"
        ),
        AgentConfig(
            name="openhands",
            model_name="anthropic/claude-sonnet-4-1"
        )
    ],
    datasets=[
        {"name": "terminal-bench", "version": "2.0"}
    ],
    orchestrator=OrchestratorConfig(
        n_concurrent_trials=8,
        quiet=False
    ),
    metrics=[
        MetricConfig(type=MetricType.MEAN),
        MetricConfig(type=MetricType.MAX)
    ]
)

OrchestratorConfig

Configuration for how trials are orchestrated and executed.

Import: from harbor.models.job.config import OrchestratorConfig

Fields

type · OrchestratorType · default: "OrchestratorType.LOCAL"

Type of orchestrator to use (LOCAL, MODAL, etc.).

n_concurrent_trials · int · default: "4"

Maximum number of trials to run concurrently.

quiet · bool · default: "False"

Suppress trial progress displays.

retry · RetryConfig · default: "RetryConfig()"

Retry configuration for failed trials.

kwargs · dict[str, Any] · default: "{}"

Additional orchestrator-specific configuration.

Example

from harbor.models.job.config import OrchestratorConfig, RetryConfig
from harbor.models.orchestrator_type import OrchestratorType

orchestrator = OrchestratorConfig(
    type=OrchestratorType.LOCAL,
    n_concurrent_trials=16,
    quiet=False,
    retry=RetryConfig(
        max_retries=2,
        min_wait_sec=5.0,
        max_wait_sec=60.0
    )
)

RetryConfig

Configuration for retrying failed trials.

Import: from harbor.models.job.config import RetryConfig

Fields

max_retries · int · default: "0"

Maximum number of retry attempts. Must be >= 0.

include_exceptions · set[str] | None · default: "None"

Exception types to retry on. If None, retries all exceptions (subject to exclude_exceptions).

exclude_exceptions · set[str] | None

Exception types to NOT retry on. Takes precedence over include_exceptions. Defaults to:

• AgentTimeoutError
• VerifierTimeoutError
• RewardFileNotFoundError
• RewardFileEmptyError
• VerifierOutputParseError

wait_multiplier · float · default: "1.0"

Multiplier for exponential backoff wait time.

min_wait_sec · float · default: "1.0"

Minimum wait time in seconds between retries.

max_wait_sec · float · default: "60.0"

Maximum wait time in seconds between retries.

Example

from harbor.models.job.config import RetryConfig

retry_config = RetryConfig(
    max_retries=3,
    include_exceptions={"RateLimitError", "NetworkError"},
    wait_multiplier=2.0,
    min_wait_sec=2.0,
    max_wait_sec=120.0
)

Dataset Configurations

LocalDatasetConfig

Configuration for loading tasks from a local directory.

Import: from harbor.models.job.config import LocalDatasetConfig

Fields

path · Path · required

Path to the local dataset directory.

task_names · list[str] | None · default: "None"

Tasks to include from the dataset. Supports glob patterns (e.g., test_*).

exclude_task_names · list[str] | None · default: "None"

Tasks to exclude from the dataset. Supports glob patterns.

n_tasks · int | None · default: "None"

Maximum number of tasks to include. Applied after filtering.

Example

from pathlib import Path
from harbor.models.job.config import LocalDatasetConfig

dataset = LocalDatasetConfig(
    path=Path("./my-tasks"),
    task_names=["task_001", "task_002"],
    n_tasks=10
)

RegistryDatasetConfig

Configuration for loading tasks from a Harbor registry.

Import: from harbor.models.job.config import RegistryDatasetConfig

Fields

registry · LocalRegistryInfo | RemoteRegistryInfo · required

Registry information (local or remote).

name · str · required

Name of the dataset in the registry.

version · str | None · default: "None"

Specific version to use. If None, uses the latest version.

overwrite · bool · default: "False"

Whether to overwrite cached remote tasks.

download_dir · Path | None · default: "None"

Directory to cache remote tasks to.

task_names · list[str] | None · default: "None"

Tasks to include. Supports glob patterns.

exclude_task_names · list[str] | None · default: "None"

Tasks to exclude. Supports glob patterns.

n_tasks · int | None · default: "None"

Maximum number of tasks to include.

Example

from harbor.models.job.config import RegistryDatasetConfig
from harbor.models.registry import RemoteRegistryInfo

dataset = RegistryDatasetConfig(
    registry=RemoteRegistryInfo(),
    name="terminal-bench",
    version="2.0",
    task_names=["test_*"],
    n_tasks=50
)

Complete Example

from pathlib import Path
from harbor.models.job.config import (
    JobConfig,
    OrchestratorConfig,
    RetryConfig,
    LocalDatasetConfig,
)
from harbor.models.trial.config import AgentConfig, EnvironmentConfig
from harbor.models.environment_type import EnvironmentType

config = JobConfig(
    job_name="comprehensive-eval",
    jobs_dir=Path("./evaluation-results"),
    n_attempts=5,
    timeout_multiplier=2.0,
    agent_timeout_multiplier=1.5,
    debug=True,
    
    orchestrator=OrchestratorConfig(
        n_concurrent_trials=12,
        retry=RetryConfig(
            max_retries=2,
            wait_multiplier=1.5
        )
    ),
    
    environment=EnvironmentConfig(
        type=EnvironmentType.DOCKER,
        force_build=False,
        delete=True
    ),
    
    agents=[
        AgentConfig(
            name="claude-code",
            model_name="anthropic/claude-opus-4-1",
            env={"LOG_LEVEL": "DEBUG"}
        )
    ],
    
    datasets=[
        LocalDatasetConfig(
            path=Path("./custom-tasks"),
            task_names=["priority_*"],
            n_tasks=20
        )
    ],
    
    artifacts=["*.log", "/workspace/output"]
)

Related Types

• TrialConfig - Individual trial configuration
• TaskConfig - Task-level configuration
• MetricConfig - Metric configuration