Benchmark data your models have never seen

Private, execution-verified tasks for agentic coding evals like Terminal-Bench, Frontier-SWE and Deep-SWE. Written by engineers, never published, licensed to one lab at a time.

Request a sample setSee what we build
A human hand and a robotic hand reaching toward each other, fingertips almost touching, in the manner of Michelangelo's Creation of Adam

Task formats built for

Terminal-BenchFrontier-SWEDeep-SWEyour internal evals

Public benchmarks leak. Private ones don’t.

Once a task is on GitHub it ends up in someone’s training mix. Scores inflate and the signal disappears. We write fresh tasks from scratch, keep them off the internet and give you exclusive rights, so the score you see is the score your model earned.

What we build

Terminal tasks
A containerised environment, a precise instruction, a hidden test suite and a reference solution. Difficulty is calibrated against current frontier models.
Long-horizon SWE tasks
Real-repository issues that need multi-file changes, with fail-to-pass and pass-to-pass tests, in the shape Frontier-SWE and Deep-SWE expect.
RL environments
Reproducible Docker images with deterministic graders and clean reward signals, ready to drop into a post-training pipeline.
Rubrics and trajectories
Expert-written grading rubrics, annotated agent trajectories and failure write-ups that explain why a model broke, not just that it did.

Every task ships complete and verified

Each task runs end to end in its own container before it reaches you. The reference solution passes, a naive attempt fails, and a second engineer reviews both.

  • Dockerfile with pinned dependencies
  • An instruction written for an agent, not a person
  • Hidden tests the model never sees
  • Reference solution and difficulty notes
tasks/fix-log-rotation-race/
├── task.yaml
├── Dockerfile
├── solution.sh
└── tests/
    └── test_outputs.py

# task.yaml
id:          fix-log-rotation-race
category:    systems / concurrency
difficulty:  hard
timeout:     1800s
instruction: |
  The rotation daemon drops lines under
  concurrent writers. Find the race and
  fix it without changing the CLI.
reviewed_by: 2
An example of the structure. Real tasks are shared under NDA.

From first call to delivered dataset

  1. 1

    Scope

    We agree on the target benchmark, domains, difficulty and volume under NDA.

  2. 2

    Pilot

    You run a small sample set against your own models before committing.

  3. 3

    Produce

    Engineers write tasks in batches, and each one is executed and peer-reviewed.

  4. 4

    Deliver

    Secure handover in your format, with exclusive rights and regular refreshes.

Your eval stays yours

Never published

No task appears in a public repo, forum or paper.

Exclusive licence

Each dataset goes to one lab. We don’t resell it.

Contamination checks

Tasks are screened against public benchmarks and code before delivery.

NDA first

Contributors sign NDAs and only see their own batch.

Let’s build your next eval

Tell us which benchmark you’re targeting and we’ll come back with a scoped pilot within one business day.

Email
hello@acrolabs.ai