> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/harbor-framework/harbor/llms.txt
> Use this file to discover all available pages before exploring further.

# Creating custom benchmarks

> Build your own evaluation benchmarks with Harbor's flexible task system

Harbor makes it easy to create custom benchmarks for evaluating AI agents on domain-specific tasks. This guide walks you through creating a complete benchmark from scratch.

## Overview

A custom benchmark in Harbor consists of:

* **Task directories**: Each containing instructions, environment, and tests
* **Dataset configuration**: Organizing tasks into a cohesive benchmark
* **Verification scripts**: Automated grading of agent solutions
* **Optional adapter**: Converting existing datasets to Harbor format

## Quick Example

Let's create a simple Python coding benchmark with 3 tasks.

<Steps>
  <Step title="Initialize task structure">
    ```bash theme={null}
    mkdir -p my-benchmark/tasks
    cd my-benchmark/tasks

    # Create first task
    harbor tasks init fibonacci
    harbor tasks init palindrome
    harbor tasks init sorting
    ```
  </Step>

  <Step title="Configure task metadata">
    Edit each `task.toml`:

    ```toml title="fibonacci/task.toml" theme={null}
    version = "1.0"

    [metadata]
    author_name = "Your Name"
    author_email = "you@example.com"
    difficulty = "easy"
    category = "programming"
    tags = ["python", "algorithms"]

    [verifier]
    timeout_sec = 60.0

    [agent]
    timeout_sec = 300.0

    [environment]
    build_timeout_sec = 300.0
    cpus = 2
    memory = "4G"
    ```
  </Step>

  <Step title="Write instructions">
    Create clear task descriptions:

    ````markdown title="fibonacci/instruction.md" theme={null}
    Write a Python function `fibonacci(n)` that returns the nth Fibonacci number.

    Requirements:
    - Handle n >= 0
    - Use efficient implementation
    - Include the function in a file called `fibonacci.py`

    Example:
    ```python
    fibonacci(0) -> 0
    fibonacci(1) -> 1
    fibonacci(10) -> 55
    ````

    ````
    </Step>

    <Step title="Create verification tests">
    Write automated tests:

    ```python title="fibonacci/tests/test_solution.py"
    import pytest
    from fibonacci import fibonacci

    def test_base_cases():
        assert fibonacci(0) == 0
        assert fibonacci(1) == 1

    def test_small_values():
        assert fibonacci(5) == 5
        assert fibonacci(10) == 55

    def test_large_values():
        assert fibonacci(20) == 6765
    ````

    ```bash title="fibonacci/tests/test.sh" theme={null}
    #!/bin/bash
    set -e

    # Install dependencies
    pip install pytest pytest-json-ctrf

    # Run tests
    if pytest --ctrf /logs/verifier/ctrf.json /tests/test_solution.py -rA; then
      echo 1 > /logs/verifier/reward.txt
    else
      echo 0 > /logs/verifier/reward.txt
    fi
    ```
  </Step>

  <Step title="Run evaluation">
    ```bash theme={null}
    harbor run -p my-benchmark/tasks \
      -a claude-code \
      -m anthropic/claude-opus-4-1 \
      --n-concurrent 3
    ```
  </Step>
</Steps>

## Benchmark Design Best Practices

### Task Variety

Include diverse task types:

```
my-benchmark/
├── tasks/
│   ├── algorithms/          # Algorithmic challenges
│   │   ├── sorting/
│   │   ├── search/
│   │   └── graphs/
│   ├── debugging/           # Bug fixing tasks
│   │   ├── syntax-errors/
│   │   └── logic-bugs/
│   ├── refactoring/         # Code improvement
│   │   ├── performance/
│   │   └── readability/
│   └── testing/             # Test writing
│       ├── unit-tests/
│       └── integration-tests/
```

### Difficulty Levels

Balance task difficulty:

```toml theme={null}
# Easy tasks - 40%
[metadata]
difficulty = "easy"

# Medium tasks - 40%
[metadata]
difficulty = "medium"

# Hard tasks - 20%
[metadata]
difficulty = "hard"
```

### Clear Instructions

<Accordion title="Good vs Bad Instructions">
  **Bad:**

  ```
  Implement a function.
  ```

  **Good:**

  ```
  Implement a function `merge_sorted_arrays(arr1, arr2)` that:
  - Takes two sorted arrays as input
  - Returns a single sorted array containing all elements
  - Maintains O(n + m) time complexity
  - Handles empty arrays

  Save your solution in `merge.py`.
  ```
</Accordion>

## Advanced: Creating an Adapter

For converting existing datasets, create a custom adapter:

<Steps>
  <Step title="Initialize adapter structure">
    ```bash theme={null}
    mkdir -p adapters/my-benchmark
    cd adapters/my-benchmark

    # Create adapter files
    touch adapter.py
    touch run_adapter.py
    mkdir template
    ```
  </Step>

  <Step title="Implement adapter logic">
    ```python title="adapter.py" theme={null}
    from pathlib import Path
    import json
    from typing import Iterator

    class MyBenchmarkAdapter:
        def __init__(self, source_path: Path):
            self.source_path = source_path
        
        def load_instances(self) -> Iterator[dict]:
            """Load benchmark instances from source."""
            with open(self.source_path / "data.json") as f:
                data = json.load(f)
            
            for instance in data:
                yield {
                    "id": instance["id"],
                    "instruction": instance["problem"],
                    "test_cases": instance["tests"],
                    "metadata": {
                        "difficulty": instance.get("difficulty", "medium"),
                        "category": instance.get("category", "programming"),
                    }
                }
        
        def generate_task(self, instance: dict, output_dir: Path):
            """Generate Harbor task from instance."""
            task_dir = output_dir / instance["id"]
            task_dir.mkdir(parents=True, exist_ok=True)
            
            # Write instruction
            (task_dir / "instruction.md").write_text(instance["instruction"])
            
            # Write task.toml
            task_config = f"""
            version = "1.0"
            
            [metadata]
            difficulty = "{instance['metadata']['difficulty']}"
            category = "{instance['metadata']['category']}"
            
            [verifier]
            timeout_sec = 120.0
            
            [agent]
            timeout_sec = 600.0
            """
            (task_dir / "task.toml").write_text(task_config)
            
            # Generate test script
            self._generate_tests(instance, task_dir / "tests")
        
        def _generate_tests(self, instance: dict, test_dir: Path):
            test_dir.mkdir(exist_ok=True)
            
            # Generate pytest file
            test_code = "import pytest\n\n"
            for i, test in enumerate(instance["test_cases"]):
                test_code += f"def test_case_{i}():\n"
                test_code += f"    {test['assertion']}\n\n"
            
            (test_dir / "test_solution.py").write_text(test_code)
            
            # Generate test.sh
            test_script = """
            #!/bin/bash
            set -e
            pip install pytest pytest-json-ctrf
            if pytest --ctrf /logs/verifier/ctrf.json /tests/test_solution.py; then
              echo 1 > /logs/verifier/reward.txt
            else
              echo 0 > /logs/verifier/reward.txt
            fi
            """
            (test_dir / "test.sh").write_text(test_script)
            (test_dir / "test.sh").chmod(0o755)
    ```
  </Step>

  <Step title="Create CLI runner">
    ```python title="run_adapter.py" theme={null}
    import argparse
    from pathlib import Path
    from adapter import MyBenchmarkAdapter

    def main():
        parser = argparse.ArgumentParser()
        parser.add_argument("--source", type=Path, required=True)
        parser.add_argument("--output", type=Path, required=True)
        args = parser.parse_args()
        
        adapter = MyBenchmarkAdapter(args.source)
        
        for instance in adapter.load_instances():
            adapter.generate_task(instance, args.output)
            print(f"Generated task: {instance['id']}")

    if __name__ == "__main__":
        main()
    ```
  </Step>

  <Step title="Run adapter">
    ```bash theme={null}
    python run_adapter.py \
      --source /path/to/source/data \
      --output ../../datasets/my-benchmark
    ```
  </Step>
</Steps>

## Partial Credit and Scoring

Implement partial credit for nuanced evaluation:

```python title="tests/test_solution.py" theme={null}
import pytest
import json
from pathlib import Path

def calculate_score():
    results = []
    
    # Test 1: Correctness (50%)
    try:
        from solution import solve
        assert solve(test_input) == expected_output
        results.append({"name": "correctness", "score": 0.5})
    except:
        results.append({"name": "correctness", "score": 0.0})
    
    # Test 2: Performance (30%)
    import time
    start = time.time()
    result = solve(large_input)
    elapsed = time.time() - start
    
    if elapsed < 1.0:
        results.append({"name": "performance", "score": 0.3})
    elif elapsed < 5.0:
        results.append({"name": "performance", "score": 0.15})
    else:
        results.append({"name": "performance", "score": 0.0})
    
    # Test 3: Code quality (20%)
    # (simplified - use static analysis tools)
    results.append({"name": "quality", "score": 0.2})
    
    total = sum(r["score"] for r in results)
    
    # Write JSON reward
    Path("/logs/verifier/reward.json").write_text(json.dumps({
        "reward": total,
        "components": results
    }))

if __name__ == "__main__":
    calculate_score()
```

## Publishing Your Benchmark

### Create Registry Entry

```json title="registry-entry.json" theme={null}
{
  "name": "my-benchmark",
  "version": "1.0",
  "description": "A benchmark for evaluating Python coding abilities",
  "tasks": [
    {
      "name": "fibonacci",
      "git_url": "https://github.com/you/my-benchmark-tasks.git",
      "git_commit_id": "abc123...",
      "path": "tasks/fibonacci"
    }
  ]
}
```

### Share on GitHub

1. Push tasks to repository
2. Create release with version tag
3. Document usage in README
4. Submit PR to Harbor registry

## Example Benchmarks

Study these real examples:

* **Aider Polyglot**: Multi-language code editing
* **LiveCodeBench**: Recent competitive programming
* **AutoCodeBench**: Complex software engineering
* **HumanEvalFix**: Bug fixing tasks

See `/integrations/benchmarks` for full list.

## Next Steps

<CardGroup cols={2}>
  <Card title="Creating Tasks" icon="hammer" href="/guides/creating-tasks">
    Detailed task creation guide
  </Card>

  <Card title="Benchmark Adapters" icon="plug" href="/guides/benchmark-adapters">
    Advanced adapter patterns
  </Card>

  <Card title="Terminal-Bench" icon="terminal" href="/examples/terminal-bench">
    Study a real benchmark
  </Card>

  <Card title="Verifiers" icon="check" href="/concepts/verifiers">
    Learn about verification
  </Card>
</CardGroup>