Metrics

Metric configuration and base classes for computing evaluation metrics

Overview

Harbor’s metric system allows you to compute aggregate statistics across trial results. Metrics process lists of rewards and produce summary values like mean, max, min, or custom computations.

MetricConfig

Configuration for a metric to be computed.

Import: from harbor.models.metric.config import MetricConfig

Fields

type · MetricType · default: "MetricType.MEAN"

The type of metric to compute.

kwargs · dict[str, Any] · default: "{}"

Additional keyword arguments passed to the metric constructor.

Example

from harbor.models.metric.config import MetricConfig
from harbor.models.metric.type import MetricType
from harbor.models.job.config import JobConfig

config = JobConfig(
    job_name="metrics-example",
    agents=[{"name": "claude-code"}],
    datasets=[{"name": "terminal-bench", "version": "2.0"}],
    metrics=[
        MetricConfig(type=MetricType.MEAN),
        MetricConfig(type=MetricType.MAX),
        MetricConfig(type=MetricType.MIN),
        MetricConfig(
            type=MetricType.UV_SCRIPT,
            kwargs={"script_path": "./custom_metric.py"}
        )
    ]
)

MetricType

Enum defining available metric types.

Import: from harbor.models.metric.type import MetricType

Values

MEAN · str

Compute the mean (average) of rewards.

MAX · str

Compute the maximum reward value.

MIN · str

Compute the minimum reward value.

SUM · str

Compute the sum of all rewards.

UV_SCRIPT · str

Run a custom Python script to compute metrics.

Example

from harbor.models.metric.type import MetricType
from harbor.models.metric.config import MetricConfig

# Built-in metrics
mean_metric = MetricConfig(type=MetricType.MEAN)
max_metric = MetricConfig(type=MetricType.MAX)
min_metric = MetricConfig(type=MetricType.MIN)
sum_metric = MetricConfig(type=MetricType.SUM)

# Custom script metric
custom_metric = MetricConfig(
    type=MetricType.UV_SCRIPT,
    kwargs={
        "script_path": "./metrics/custom.py",
        "param1": "value1"
    }
)

BaseMetric

Abstract base class for implementing custom metrics.

Import: from harbor.metrics.base import BaseMetric

Type Parameters

T · TypeVar

The type of values the metric processes (typically dict[str, float | int] for rewards).

Abstract Methods

compute

@abstractmethod
def compute(self, rewards: list[T | None]) -> dict[str, float | int]

Compute metric values from a list of rewards.

rewards · list[T | None] · required

List of reward values. None values represent failed trials or missing rewards.

metrics · dict[str, float | int] · required

Dictionary of computed metric name-value pairs.

Example Implementation

from harbor.metrics.base import BaseMetric

class PassRateMetric(BaseMetric[dict[str, float]]):
    """Compute the pass rate (percentage of rewards == 1.0)."""
    
    def compute(self, rewards: list[dict[str, float] | None]) -> dict[str, float | int]:
        # Filter out None values
        valid_rewards = [r for r in rewards if r is not None]
        
        if not valid_rewards:
            return {"pass_rate": 0.0, "n_trials": 0}
        
        # Count how many have score == 1.0
        passes = sum(1 for r in valid_rewards if r.get("score") == 1.0)
        pass_rate = passes / len(valid_rewards)
        
        return {
            "pass_rate": pass_rate,
            "n_trials": len(valid_rewards),
            "n_passed": passes
        }

# Use the metric
metric = PassRateMetric()
rewards = [
    {"score": 1.0},
    {"score": 0.5},
    {"score": 1.0},
    None,  # Failed trial
    {"score": 0.0}
]

result = metric.compute(rewards)
print(result)
# {'pass_rate': 0.5, 'n_trials': 4, 'n_passed': 2}

Built-in Metrics

Harbor includes several built-in metrics:

Mean

from harbor.metrics.mean import Mean

metric = Mean()
rewards = [
    {"score": 1.0, "efficiency": 0.8},
    {"score": 0.5, "efficiency": 0.9},
    {"score": 0.0, "efficiency": 0.7}
]

result = metric.compute(rewards)
# {'score': 0.5, 'efficiency': 0.8}

Max

from harbor.metrics.max import Max

metric = Max()
rewards = [
    {"score": 0.8},
    {"score": 0.95},
    {"score": 0.6}
]

result = metric.compute(rewards)
# {'score': 0.95}

Min

from harbor.metrics.min import Min

metric = Min()
rewards = [
    {"score": 0.8},
    {"score": 0.95},
    {"score": 0.6}
]

result = metric.compute(rewards)
# {'score': 0.6}

Sum

from harbor.metrics.sum import Sum

metric = Sum()
rewards = [
    {"points": 10},
    {"points": 20},
    {"points": 15}
]

result = metric.compute(rewards)
# {'points': 45}

VerifierResult

Contains the verification results including rewards.

Import: from harbor.models.verifier.result import VerifierResult

Fields

rewards · dict[str, float | int] | None · default: "None"

Dictionary mapping reward names to numeric values. None if verification failed or was disabled.

Example

from harbor.models.verifier.result import VerifierResult

# Single reward
result = VerifierResult(
    rewards={"score": 1.0}
)

# Multiple rewards
result = VerifierResult(
    rewards={
        "correctness": 1.0,
        "efficiency": 0.85,
        "style": 0.9
    }
)

# No rewards (verification failed)
result = VerifierResult(
    rewards=None
)

Writing Custom Metrics

Simple Aggregation Metric

from harbor.metrics.base import BaseMetric

class MedianMetric(BaseMetric[dict[str, float]]):
    """Compute the median of each reward."""
    
    def compute(self, rewards: list[dict[str, float] | None]) -> dict[str, float | int]:
        import statistics
        
        # Filter out None values
        valid_rewards = [r for r in rewards if r is not None]
        
        if not valid_rewards:
            return {}
        
        # Collect all reward keys
        all_keys = set()
        for r in valid_rewards:
            all_keys.update(r.keys())
        
        # Compute median for each reward
        result = {}
        for key in all_keys:
            values = [r[key] for r in valid_rewards if key in r]
            if values:
                result[key] = statistics.median(values)
        
        return result

Metric with Parameters

class ThresholdMetric(BaseMetric[dict[str, float]]):
    """Compute percentage of trials above a threshold."""
    
    def __init__(self, threshold: float = 0.8, reward_key: str = "score"):
        self.threshold = threshold
        self.reward_key = reward_key
    
    def compute(self, rewards: list[dict[str, float] | None]) -> dict[str, float | int]:
        valid_rewards = [r for r in rewards if r is not None and self.reward_key in r]
        
        if not valid_rewards:
            return {"above_threshold": 0.0, "n_trials": 0}
        
        above = sum(1 for r in valid_rewards if r[self.reward_key] >= self.threshold)
        percentage = above / len(valid_rewards)
        
        return {
            "above_threshold": percentage,
            "threshold": self.threshold,
            "n_above": above,
            "n_trials": len(valid_rewards)
        }

# Use with custom parameters
metric = ThresholdMetric(threshold=0.9, reward_key="accuracy")

Multi-Reward Analysis Metric

class CorrelationMetric(BaseMetric[dict[str, float]]):
    """Compute correlation between two reward metrics."""
    
    def __init__(self, reward1: str, reward2: str):
        self.reward1 = reward1
        self.reward2 = reward2
    
    def compute(self, rewards: list[dict[str, float] | None]) -> dict[str, float | int]:
        import statistics
        
        # Get pairs where both rewards exist
        pairs = []
        for r in rewards:
            if r and self.reward1 in r and self.reward2 in r:
                pairs.append((r[self.reward1], r[self.reward2]))
        
        if len(pairs) < 2:
            return {"correlation": 0.0, "n_pairs": len(pairs)}
        
        # Compute Pearson correlation
        x = [p[0] for p in pairs]
        y = [p[1] for p in pairs]
        
        mean_x = statistics.mean(x)
        mean_y = statistics.mean(y)
        
        numerator = sum((xi - mean_x) * (yi - mean_y) for xi, yi in zip(x, y))
        denominator = (
            sum((xi - mean_x) ** 2 for xi in x) ** 0.5 *
            sum((yi - mean_y) ** 2 for yi in y) ** 0.5
        )
        
        correlation = numerator / denominator if denominator != 0 else 0.0
        
        return {
            "correlation": correlation,
            "n_pairs": len(pairs)
        }

Using Metrics in Jobs

from harbor import Job
from harbor.models.job.config import JobConfig
from harbor.models.metric.config import MetricConfig
from harbor.models.metric.type import MetricType

config = JobConfig(
    job_name="comprehensive-metrics",
    agents=[{"name": "claude-code", "model_name": "anthropic/claude-opus-4-1"}],
    datasets=[{"name": "terminal-bench", "version": "2.0"}],
    n_attempts=5,
    metrics=[
        MetricConfig(type=MetricType.MEAN),
        MetricConfig(type=MetricType.MAX),
        MetricConfig(type=MetricType.MIN),
    ]
)

job = Job(config)
result = await job.run()

# Access computed metrics
for eval_key, agent_stats in result.stats.evals.items():
    print(f"\n{eval_key}:")
    for metric in agent_stats.metrics:
        print(f"  {metric}")

Related Types

• JobConfig - Configure metrics for a job
• JobResult - Access computed metrics
• TrialResult - Individual trial rewards