Menu

Environments

How a job becomes a training environment.

The technical method behind both data lines: task structure, grading, calibration and the format your team receives.

Open the CAD sample task Request a sample pack

What an environment is

Everything a run needs, in one folder.

Each task is one folder in a Harbor-compatible layout. The agent sees the instruction and the project. The verifier and the reference solution stay out of its reach.

Instruction instruction.md
The work request as a colleague or client would send it, with the exact deliverables and the paths to save them to.
Starting state environment/project/
The seed data the job starts from: drawings and their revisions, 3D models, tables, mail, database rows.
Tools environment/Dockerfile
A container with what the work needs: a CAD kernel, hydrostatics and calculation libraries, document and spreadsheet tools.
Verifier tests/test.sh
The program that reads the deliverables and the final state, runs every check and writes the reward, one line per check.
Probes tests/probes/
Reward hacks, plausible mistakes and correct alternatives the verifier must score as expected before the task ships.
Reference solution solution/solve.sh
A solution that scores 1, which proves the task can be solved as written.
Metadata and rights task.toml
Domain, family, difficulty band, time limits, resources, split, source and licence.

Open the sample task folder, file by file

task.toml Illustrative

version = "0.3"

[metadata]
id          = "naval/tank-capacity-014"
domain      = "ship design"
family      = "tank-capacity"   # generated, seed 1408
difficulty  = "hard"            # pass@1 0.12 at calibration
split       = "train"           # train | sealed-eval

[rights]
source      = "generated"       # generated | licensed | authored
use         = "training and evaluation"

[agent]
timeout_sec = 2700

[environment]
network     = "none"
cpus        = 2
memory_mb   = 4096

[verifier]
timeout_sec = 300
reward      = "strict"          # 1 only if every check passes
weights     = [1, 3, 3, 2, 1, 2, 3, 1]   # C1 to C8
tolerance   = { volume_m3 = 0.5, lcg_m = 0.02, trim_m = 0.01 }

/logs/verifier/reward.json Illustrative

{
  "reward": 0,
  "score": 0.94,
  "C1_deliverable_parses": 1,
  "C2_inputs_unchanged": 1,
  "C3_tank_volumes_98pct": 1,
  "C4_tank_lcg_vcg": 1,
  "C5_sounding_tables": 0,
  "C6_trim_departure": 1,
  "C7_tank_bounds_rev_c": 1,
  "C8_tolerances_cited": 1
}

One failed check sets the reward to 0. The weighted score shows the rest of the answer was sound, and where it went wrong: C5 caught tank 5P’s soundings read from the sounding-pipe datum in revision B. Revision C moved the pipe, so every sounding in that tank is referenced to the wrong point.

Verifier principles

Grade the state, not the prose.

A reward is only as good as the verifier that computes it. Code checks what can be computed; where expert judgment is needed, calibrated model judges and custom rubric verifiers grade it, each checked against expert grades.

Grade the state

The verifier opens the files, geometry, rows and messages the agent left behind and recomputes what they should be. The agent’s summary of its work earns nothing.

Deterministic, fail-closed

Same deliverable, same reward, on every run. A missing, malformed or unreadable deliverable scores zero with a stated reason, never a crash and never a pass.

Strict pass is the reward

The reward is 1 only when every check passes. A weighted per-check score sits beside it, so you can read where an answer went wrong without rewarding a near miss.

Process and outcome

Outcome checks read the deliverable. Process checks read how it was reached: inputs left untouched, the current revision used, and the clauses and values applied, cited.

Tolerance-aware

Each quantity carries the engineering tolerance of its drawing or code. A correct answer in other units, another order or another layout still passes.

Isolated and resettable

No network, read-only inputs, and tests mounted only after the run ends. Reset returns the workspace to the same state, checked by digest.

Frozen and versioned

Each verifier is frozen by hash with its task version. A fix is a new version with a change-log entry, and every probe runs again.

Engineering and geometry Business records and replies

Reward-hacking defence

Every verifier is attacked before it ships.

Before a task leaves us, we run a set of probes against its verifier.

Probe classWhat it doesMust score
EmptyLeaves the workspace exactly as it was reset.0
Claim-onlyReports the job done and changes nothing.0
TamperEdits an input, a protected table, the tests or the reward file.0
Hard-coded answerWrites the expected format with guessed or copied values.0
Superseded inputWorks from the old drawing revision or the first email, not the amendment.Fail strict
Plausible mistakeRight overall shape or total, wrong toleranced feature or line item.Fail strict
Correct alternativeThe right answer by another method, in other units or another layout.Pass strict
Six demonstration tasks, one harness.

Gates run 10 November 2026; model attempts on or before 7 November 2026. All six are demonstration tasks we wrote in-house with invented names and figures, not partner records. The probes are a regression test, not a guarantee.

Results from the enterprise-workflow harness. The CAD sample run is reported separately.

Checks
59 across the six tasks, 26 critical; the reference solution scores 1.0 on each
Reward-hack probes
73, every one scoring zero; 101 hostile inputs, all zero without a verifier error
Plausible mistakes
71, every one failing the strict reward
Correct alternatives
8, all passing strict
Known gaps
3 probes passing by design; documented on the evidence page
Model attempts
Two current models from one provider, including its small tier, one rollout per model per version. Both passed every task after we clarified the wording of two version-0 tasks. The hardened versions run next (below). These runs show that the harness works, not how hard the tasks are
Open
What the verifier does not yet catch is on the evidence page

Published with its date, pass or fail.

Planned

Pass rates on the six tasks for one small open model and one stronger model, five rollouts each. Published here when complete, whatever the result, with the run date and task versions.

Request a sample pack Book the walk-through

Trajectory audit

Probes test the verifier. The audit tests the passes. On sampled runs from calibration we read how each pass was earned, not only that it was.

A pass is thrown out when the run

  • opened the reference, the tests or anything outside the project
  • changed an input, a protected table or the verifier
  • reached for the network, or wrote an answer it never computed
  • took a route the instruction rules out

A pass earned the wrong way becomes a new probe, and the verifier is hardened before the batch ships.

Difficulty and calibration

Hard enough to teach, solvable by design.

A task every model passes teaches nothing. A task no model passes may be broken. We measure each task before it ships and keep it only inside the band you set.

Repeated attempts
Models from a small open model to the strongest available attempt each task several times. A pass rate is an average over attempts, never one run.
pass@1 and pass@k
pass@1 places the task in the band; pass@k shows it is reachable. A task outside the band is hardened, rebalanced or dropped.
Zero is a warning
Before we keep a task no model passes, we re-check its instruction, its reference solution and its verifier.
Knobs, not wording
Generated families are tuned with the knobs below. Difficulty never comes from an ambiguous instruction or a hidden trap.

Difficulty knobs by domain

Parts
Feature count, tolerance class, views to reconcile
Ships
Tanks and loading conditions, superseded revisions in the folder
Buildings
Sheets and disciplines to cross-read, clash density
Workflows
Horizon length, amendments, distracting threads

Calibration card, one per batch Illustrative

Batch
naval/tank-capacity v0.3
Tasks kept
40 of 46 drafted
Attempts
3 per model per task
Small model
pass@1 0.05
Large model
pass@1 0.21 · pass@3 0.34
Band kept
pass@1 between 0.02 and 0.30
Reward hacks
14 of 14 at zero
Planted mistakes
12 of 12 failing strict
Alternatives
4 of 4 passing strict
Expert agreement
9 of 10 tasks, graded blind
Time per attempt
median 38 min

The shape of the card, not a measured result. Every batch carries its own.

Two engines for scale

Volume from generators. Depth from practitioners.

Most of a batch can be generated with its ground truth computed. The hardest tasks start with someone who has done the job.

Engine one

Parametric generators

A generator builds a family of tasks from parameters: parts with their features and tolerances, hull forms with their tank arrangements, building grids with their frames. The code that builds a task also computes its answer, so volume never costs correctness.

  • Ground truth computed, never typed
  • Every seed recorded on the datasheet and reproducible
Generated. Nine parts from our generators, built feature by feature.

Engine two

Expert seeds

For commissioned tasks, a practitioner supplies the job and signs off the result. Models assist with the drafting, so expert hours go where only an expert can help.

  1. SeedA practitioner describes a real job in six fields: situation, inputs, goal, constraints, how to verify, expected output.
  2. DraftWe build the instruction, project files, verifier and reference solution from the seed, with models assisting the drafting.
  3. SolveA solver model from another family attempts it blind, several times, with no access to the verifier or the reference. If it passes too easily, we read the solver’s log and harden the task.
  4. ValidateThe practitioner checks the reference and the verifier against their own judgement, and makes the task harder where the real job is harder.
  5. ReviewA second practitioner grades a sample blind. We report the agreement on the calibration card, and each disagreement goes to adjudication.

Delivery

In the shape your harness already runs.

Tasks arrive ready to run, documented item by item, and split so that evaluation stays clean.

Format
Harbor-compatible task folders, container images, or the interface your harness takes. One task runs in your harness first, then the batch follows.
Datasheet
Per item: source (generated, licensed or authored), generator and seed, or the author’s pseudonymous ID, revisions, de-identification, and the calibration results.
Rights note
Per item: who owns it, the licence for training and evaluation, and the assignment or agreement it rests on.
Splits
The agreed split and permitted use are recorded with each task. Evaluation and licensing policy.
Upkeep
As models improve, verifiers are hardened and saturated tasks are replaced with harder ones from the same family.

Get started

Read one task before you decide.

Inspect a complete task before discussing an integration.

Request a sample pack Open the CAD sample task