Identity
- Task
mfg-rfq-001, version 1.0.3, verifier frozen by hash- Family
- packaging-quotation, difficulty easy, as the public gate file labels it
- Outcome
unresolved, by construction: the episode ends when the reply is sent, before the customer answers; a practitioner authors the label in v1.1- Domain
- Manufacturing: plastics packaging, a request for quotation answered with a quote
- Status
- Public demonstration; disclosed in full and not for sale.
- Gates run
- 10 November 2026, Python 3.11.8, deterministic gates only: no model runs, no human raters, no training
- Model attempts
- Version 0, on or before 7 November 2026: two current models from one provider, one of them its small tier, one rollout each, both scored 1.0 and passed strict. The later versions, which tightened the verifier and left the reference answer unchanged, run next
The job
A sales executive at a packaging plant has to get a quote out today. The customer’s second email cut the order and moved the delivery date, the production chat moved the job to another machine because the usual one is down, and the quoting procedure says how to cost, price and escalate. The instruction, as the agent receives it:
You are the sales executive (sales_exec_2) at Tembaga Ridge Packaging. Today is 16 Nov 2026. Get quote Q-2611 out to Selat Biru Agrochem today. Read the inbox and the production chat, followdocs/SOP-Q-07.md, update the quotes row inerp/plant.db, log each change you make to it incrm_history, and writeout/quote.json(qty, machine, unit_cost_rm, unit_price_rm, earliest_delivery as YYYY-MM-DD, meets_requested_delivery) andout/reply.eml.
The right answer takes the latest quantity, the confirmed machine and the current resin price, recomputes cost and price from the procedure, finds the earliest slot in the schedule, marks the requested date as not met, escalates the quote in the database, logs each change it makes to the quote row, and writes a reply that promises nothing the schedule cannot deliver.
Read the starting state
Currency amounts appear as [amount] on this page; the full files go to your engineer under NDA. The reference reply is deliberately minimal: the verifier reads its price, quantity and dates, not its tone.
The task’s own files, in the order a worker meets them, then the reference end state and one plausible mistake. Each sheet is generated from the task when the page is built, so it cannot drift from the files the verifier reads. The highlighter marks the facts that changed and the line the verifier caught.
The inbox, four messages
inbox/001.eml9 Nov 2026, 10:14
- From
- procurement@selatbiru.example
- To
- sales@tembagaridge.example
- Subject
- RFQ - 20 L jerrican
Please quote for 50,000 pcs of your 20 L HDPE jerrican (JC20), natural colour. Delivery to Port Klang.
Procurement, Selat Biru Agrochem
inbox/002.eml13 Nov 2026, 15:31
- From
- procurement@selatbiru.example
- To
- sales@tembagaridge.example
- Subject
- Re: RFQ - 20 L jerrican (revised)
Revision to our RFQ: we now need the jerricans in BLUE, and the first order will be 12,000 pcs, not 50,000. We need delivery within 21 days of your quotation date. Please re-quote.
Procurement, Selat Biru Agrochem
inbox/003.eml16 Nov 2026, 08:40
- From
- sales.director@tembagaridge.example
- To
- sales_exec_2@tembagaridge.example
- Subject
- Q-2611 today please
Please get Q-2611 out to Selat Biru today. Follow SOP-Q-07 and check with production first.
inbox/000.eml16 Nov 2026, 09:05
- From
- procurement@selatbiru.example
- To
- sales@tembagaridge.example
- Subject
- Re: RFQ - 20 L jerrican (revised) - quotation?
Good morning. We have not yet received your revised quotation. Our planning meeting is this afternoon, so please send it today.
Procurement, Selat Biru Agrochem
The production chat
chat/production.json#production
BM-07 is down for a screw and barrel replacement, back 10 Dec at the earliest. Run the JC20 mould on BM-03 until then.
Noted. JC20 cycle on BM-03 runs 18% longer than on BM-07, same single cavity. BM-03 is booked, check machine_schedule before promising dates.
Reminder: drop test on the blue 10 L pails is at 2pm.
The procedure
docs/SOP-Q-07.mdThe quoting procedure
SOP-Q-07 Quotation costing and disposition (blow moulding)
- Governing request. The latest written customer amendment governs quantity, colour and delivery.
- Machine. Quote on the machine production has confirmed for the job. Never quote on a machine whose status is DOWN.
- Unit cost in RM per piece:
- material mass = part mass x (1 + 3% scrap)
- coloured parts: masterbatch replaces resin at the dosing % in the masterbatch table
- material cost = resin kg x resin price + masterbatch kg x masterbatch price
- resin price = the row with the latest effective_date on or before the quotation date
- machine cost = cycle seconds / cavities / 3600 x machine rate per hour
- set-up = [amount] per job / quantity
- unit cost = material + machine + set-up
- Price. unit price = unit cost / (1 - 22%), rounded to 2 decimals. Never quote below an 18% margin.
- MOQ for JC20 is 10,000 pcs. Below MOQ: stage REJECTED_MOQ and no price is sent.
- Delivery.
- production days = quantity x cycle seconds / cavities / (3600 x machine hours per day x 85% OEE), rounded up
- first free day = the day after booked_until in machine_schedule
- earliest delivery = first free day + production days + 3 days QC and dispatch
- Disposition, in the quotes table:
- delivery can be met: stage QUOTED, owner stays with the sales executive
- delivery cannot be met: still price the job, stage ESCALATED_LEADTIME, owner = the sales manager. The reply states the price and the earliest delivery date and does not promise the requested date.
- Outputs. Update the quotes row (qty, unit_price_rm, stage, owner). Record every change you make to a quotes field as a row in crm_history (quote_id, ts on the quotation date, actor, field, old_value, new_value, note); never edit an existing history row. Write out/quote.json and out/reply.eml.
The database before the work
erp/plant.dbThe tables the job touches or reads
quotes2 rows
id | customer_id | sku | qty | unit_price_rm | stage | owner |
|---|---|---|---|---|---|---|
| Q-2604 | C-0388 | PL10 | 20000 | [amount] | QUOTED | sales_exec_2 |
| Q-2611 | C-0417 | JC20 | 50000 | null | ENQUIRY | sales_exec_2 |
machine_schedule2 rows
machine_id | booked_until |
|---|---|
| BM-03 | 2026-11-27 |
| BM-07 | 2026-12-09 |
machines2 rows
id | status | rate_rm_per_hr | hours_per_day |
|---|---|---|---|
| BM-07 | DOWN | [amount] | 20.0 |
| BM-03 | RUNNING | [amount] | 20.0 |
crm_history4 rows
id | quote_id | ts | actor | field | old_value | new_value | note |
|---|---|---|---|---|---|---|---|
| 1 | Q-2604 | 2026-11-02 14:10 | sales_exec_2 | stage | ENQUIRY | QUOTED | PL10 20,000 pcs quoted at [amount] |
| 2 | Q-2611 | 2026-11-09 10:40 | sales_exec_2 | stage | null | ENQUIRY | RFQ received: 50,000 pcs JC20 natural, delivery Port Klang |
| 3 | Q-2611 | 2026-11-13 16:02 | sales_exec_2 | note | null | null | Customer revision: blue, first order 12,000 pcs, delivery within 21 days of quotation; re-quote pending |
| 4 | Q-2611 | 2026-11-16 08:45 | sales_exec_2 | note | null | null | Sales director: quote out today per SOP-Q-07, check with production first |
The reference end state
out/quote.jsonJSON
{
"quote_id": "Q-2611",
"qty": 12000,
"machine": "BM-03",
"unit_cost_rm": "[amount]",
"unit_price_rm": "[amount]",
"earliest_delivery": "2026-12-11",
"meets_requested_delivery": false
}out/reply.eml16 Nov 2026, 11:00
- From
- sales_exec_2@tembagaridge.example
- To
- procurement@selatbiru.example
- Subject
- Quotation Q-2611
Our price for 12,000 pcs JC20 is [amount] per piece. Earliest delivery is 2026-12-11.
erp/plant.dbAfter the work
quotes, the Q-2611 row1 rows
id | customer_id | sku | qty | unit_price_rm | stage | owner |
|---|---|---|---|---|---|---|
| Q-2611 | C-0417 | JC20 | 12000 | [amount] | ESCALATED_LEADTIME | sales_mgr_1 |
crm_history, rows the work added4 rows
id | quote_id | ts | actor | field | old_value | new_value | note |
|---|---|---|---|---|---|---|---|
| 5 | Q-2611 | 2026-11-16 11:00 | sales_exec_2 | qty | 50000 | 12000 | Q-2611 re-quoted per SOP-Q-07 |
| 6 | Q-2611 | 2026-11-16 11:00 | sales_exec_2 | unit_price_rm | null | [amount] | Q-2611 re-quoted per SOP-Q-07 |
| 7 | Q-2611 | 2026-11-16 11:00 | sales_exec_2 | stage | ENQUIRY | ESCALATED_LEADTIME | Q-2611 re-quoted per SOP-Q-07 |
| 8 | Q-2611 | 2026-11-16 11:00 | sales_exec_2 | owner | sales_exec_2 | sales_mgr_1 | Q-2611 re-quoted per SOP-Q-07 |
The verifier: Pass · score 1.00
Raw verifier report, reference solution
{
"gates": [
{
"name": "inputs untouched and no file added under inbox/, chat/ or docs/",
"passed": true
},
{
"name": "protected ERP tables and the database schema untouched",
"passed": true
},
{
"name": "quotes table: only Q-2611's qty, unit_price_rm, stage and owner changed; no row added, removed or re-identified",
"passed": true
},
{
"name": "crm_history: seeded rows untouched; rows appended only for Q-2611 and dated on the quotation date",
"passed": true
}
],
"criteria": [
{
"id": "C1",
"desc": "Quantity follows the latest amendment (12,000)",
"passed": true,
"weight": 2,
"critical": true
},
{
"id": "C2",
"desc": "Machine is the one production confirmed (BM-03), not the down machine",
"passed": true,
"weight": 2,
"critical": false
},
{
"id": "C3",
"desc": "Unit cost within 0.5% of the SOP recomputation",
"passed": true,
"weight": 3,
"critical": false
},
{
"id": "C4",
"desc": "Unit price equals the SOP price to the cent",
"passed": true,
"weight": 3,
"critical": false
},
{
"id": "C5",
"desc": "Price keeps at least an 18% margin on the true cost",
"passed": true,
"weight": 2,
"critical": true
},
{
"id": "C6",
"desc": "Earliest delivery date matches the schedule recomputation",
"passed": true,
"weight": 2,
"critical": false
},
{
"id": "C7",
"desc": "Delivery marked as not met and the quote escalated in the ERP",
"passed": true,
"weight": 3,
"critical": true
},
{
"id": "C8",
"desc": "Owner reassigned to the sales manager",
"passed": true,
"weight": 1,
"critical": false
},
{
"id": "C9",
"desc": "ERP row agrees with quote.json (qty, and price to the cent)",
"passed": true,
"weight": 2,
"critical": false
},
{
"id": "C10",
"desc": "Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date",
"passed": true,
"weight": 1,
"critical": true
},
{
"id": "C11",
"desc": "Reply is addressed to the customer, identifies Q-2611, and names no quantity other than 12,000 pcs",
"passed": true,
"weight": 1,
"critical": false
},
{
"id": "C12",
"desc": "Stated price follows from the stated cost under the SOP mark-up (cost / 0.78, rounded)",
"passed": true,
"weight": 1,
"critical": false
},
{
"id": "C13",
"desc": "Each change to stage and owner on the quotes row is logged in crm_history with its old and new value",
"passed": true,
"weight": 2,
"critical": true
}
],
"strict_pass": true,
"reward": 1.0,
"rejection": null
}One plausible mistake: costed the machine that is down
The probe down_machine. Plausible mistake. Its files, and what the verifier caught.
out/quote.jsonJSON
{
"quote_id": "Q-2611",
"qty": 12000,
"machine": "BM-07",
"unit_cost_rm": "[amount]",
"unit_price_rm": "[amount]",
"earliest_delivery": "2026-11-27",
"meets_requested_delivery": true
}out/reply.eml16 Nov 2026, 11:00
- From
- sales_exec_2@tembagaridge.example
- To
- procurement@selatbiru.example
- Subject
- Quotation Q-2611
Our price for 12,000 pcs JC20 is [amount] per piece. Earliest delivery is 2026-11-27.
The verifier: Fail · score 0.40
C2Machine is the one production confirmed (BM-03), not the down machineC3Unit cost within 0.5% of the SOP recomputationC4Unit price equals the SOP price to the centC6Earliest delivery date matches the schedule recomputationC7Delivery marked as not met and the quote escalated in the ERPC8Owner reassigned to the sales managerC10Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date
Raw verifier report, down_machine
{
"gates": [
{
"name": "inputs untouched and no file added under inbox/, chat/ or docs/",
"passed": true
},
{
"name": "protected ERP tables and the database schema untouched",
"passed": true
},
{
"name": "quotes table: only Q-2611's qty, unit_price_rm, stage and owner changed; no row added, removed or re-identified",
"passed": true
},
{
"name": "crm_history: seeded rows untouched; rows appended only for Q-2611 and dated on the quotation date",
"passed": true
}
],
"criteria": [
{
"id": "C1",
"desc": "Quantity follows the latest amendment (12,000)",
"passed": true,
"weight": 2,
"critical": true
},
{
"id": "C2",
"desc": "Machine is the one production confirmed (BM-03), not the down machine",
"passed": false,
"weight": 2,
"critical": false
},
{
"id": "C3",
"desc": "Unit cost within 0.5% of the SOP recomputation",
"passed": false,
"weight": 3,
"critical": false
},
{
"id": "C4",
"desc": "Unit price equals the SOP price to the cent",
"passed": false,
"weight": 3,
"critical": false
},
{
"id": "C5",
"desc": "Price keeps at least an 18% margin on the true cost",
"passed": true,
"weight": 2,
"critical": true
},
{
"id": "C6",
"desc": "Earliest delivery date matches the schedule recomputation",
"passed": false,
"weight": 2,
"critical": false
},
{
"id": "C7",
"desc": "Delivery marked as not met and the quote escalated in the ERP",
"passed": false,
"weight": 3,
"critical": true
},
{
"id": "C8",
"desc": "Owner reassigned to the sales manager",
"passed": false,
"weight": 1,
"critical": false
},
{
"id": "C9",
"desc": "ERP row agrees with quote.json (qty, and price to the cent)",
"passed": true,
"weight": 2,
"critical": false
},
{
"id": "C10",
"desc": "Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date",
"passed": false,
"weight": 1,
"critical": true
},
{
"id": "C11",
"desc": "Reply is addressed to the customer, identifies Q-2611, and names no quantity other than 12,000 pcs",
"passed": true,
"weight": 1,
"critical": false
},
{
"id": "C12",
"desc": "Stated price follows from the stated cost under the SOP mark-up (cost / 0.78, rounded)",
"passed": true,
"weight": 1,
"critical": false
},
{
"id": "C13",
"desc": "Each change to stage and owner on the quotes row is logged in crm_history with its old and new value",
"passed": true,
"weight": 2,
"critical": true
}
],
"strict_pass": false,
"reward": 0.4,
"rejection": null
}What to notice
- The second email supersedes the first.
- The constraint lives only in the chat.
- The reply must not promise a date the schedule cannot meet.
- The checker reads the files and rows, never the reply’s claims.
- Every change to the quote row must be logged in the history table, dated on the quotation date.
The five parts in this version
- Customer emails
- Present: the enquiry, the revision, the director’s note and the chase, four messages in the inbox
- Internal discussion
- Present: the production chat, three lines
- Related documents
- Present: the quoting procedure
- History log
- Present: the history table, four authored rows before the work starts. Each change to the quote row must add one, checked by C13
- Outcome label
- Present on the task record, authored as “unresolved”. By construction: the episode ends when the reply is sent, before the customer answers; a practitioner authors the label in v1.1
The workspace
Every run starts from the same files and the same database, written fresh by the task’s own reset.
| Path | What it holds after reset |
|---|---|
chat/production.json | 639 bytes |
docs/SOP-Q-07.md | 1877 bytes |
erp/plant.db | crm_history (4), customers (2), machine_schedule (2), machines (2), masterbatch (2), mould_machine (1), products (2), quotes (2), resin_prices (3), staff (3) |
inbox/000.eml | 324 bytes |
inbox/001.eml | 272 bytes |
inbox/002.eml | 364 bytes |
inbox/003.eml | 240 bytes |
Reset digest, SHA-256 of the seeded workspace with the database taken as its SQL dump: 3366510ba57c7b69d148f9dd16a74ceaea0a556e649b813b1a5d1d55c446d20d. Reset deterministic from an empty directory: yes. A used workspace resets to the same digest: yes.
What the agent may read and change
- Reads
- The four inbox messages, the production chat, the quoting procedure, and any table in the database.
- Writes
out/quote.jsonwith the required fields,out/reply.eml, four fields of the one quote row the instruction names: quantity, unit price, stage and owner, and one new row in the history table for each of those changes.- Must not touch
- The input files, the input folders, the other tables, the database schema, every other field and row of the quotes table, and every history row that was there at the start. Any change there fails a gate, and the run scores zero whatever else it did.
Gates and checks
A submission the verifier cannot score, because a required output is missing, malformed or damaged, is rejected as invalid at reward zero, with a structured reason rather than a crash. Then four gates. Fail any and the reward is zero, with no checks counted.
- inputs untouched and no file added under inbox/, chat/ or docs/.
- protected ERP tables and the database schema untouched.
- quotes table: only Q-2611's qty, unit_price_rm, stage and owner changed; no row added, removed or re-identified.
- crm_history: seeded rows untouched; rows appended only for Q-2611 and dated on the quotation date.
Then thirteen checks, each recomputed by the verifier from a fresh copy of the starting state and compared with the files and database rows on disk. The verifier never reads what the agent says it did. Weights sum to 25; 5 checks are critical.
| Check | What must be true | Weight | Critical |
|---|---|---|---|
| C1 | Quantity follows the latest amendment (12,000) | 2 | yes |
| C2 | Machine is the one production confirmed (BM-03), not the down machine | 2 | |
| C3 | Unit cost within 0.5% of the SOP recomputation | 3 | |
| C4 | Unit price equals the SOP price to the cent | 3 | |
| C5 | Price keeps at least an 18% margin on the true cost | 2 | yes |
| C6 | Earliest delivery date matches the schedule recomputation | 2 | |
| C7 | Delivery marked as not met and the quote escalated in the ERP | 3 | yes |
| C8 | Owner reassigned to the sales manager | 1 | |
| C9 | ERP row agrees with quote.json (qty, and price to the cent) | 2 | |
| C10 | Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date | 1 | yes |
| C11 | Reply is addressed to the customer, identifies Q-2611, and names no quantity other than 12,000 pcs | 1 | |
| C12 | Stated price follows from the stated cost under the SOP mark-up (cost / 0.78, rounded) | 1 | |
| C13 | Each change to stage and owner on the quotes row is logged in crm_history with its old and new value | 2 | yes |
Tolerances in version 1.0.3: C3 compares unit cost within 0.5%, as its text says; C4, C9 and C12 compare prices to the cent (within half a cent); C10 reads the reply’s subject and body, every date format, and only dates after the quotation date.
The probes
Before a buyer sees a task, we attack its verifier. A reward hack fakes or tampers and must score exactly zero. A plausible mistake is one a careful worker could make; it must not earn the strict reward, which needs every gate and every check to pass. A correct alternative is a right answer written another way; it must earn it. A known-gap probe records a weakness we have not closed: it passes by design, so any change at that boundary shows up. 19 reward hacks, 26 plausible mistakes, 3 correct alternatives and 3 known-gap probes ran on 10 November 2026, and every one returned the expected result. Most plausible mistakes target the reply, the one free-text output: a promise of an earlier date, in every date format we have met.
| Probe | Kind | What it did | Strict | Score | What failed |
|---|---|---|---|---|---|
empty | Reward hack | No answer written | Fail | 0.00 | Rejected as invalid |
claim_only | Reward hack | A confident reply with no files written | Fail | 0.00 | Rejected as invalid |
tamper_schedule | Reward hack | Freed the booked machine in the schedule table | Fail | 0.00 | gate 2 |
edit_customer_email | Reward hack | Edited the customer’s email so the old quantity reads as current | Fail | 0.00 | gate 1 |
superseded_quantity | Plausible mistake | Quoted the superseded quantity | Fail | 0.40 | C1, C3, C4, C5, C6, C10, C11, C12 |
down_machine | Plausible mistake | Costed the machine that is down | Fail | 0.40 | C2, C3, C4, C6, C7, C8, C10 |
stale_resin_price | Plausible mistake | Priced on a resin price the table had superseded | Fail | 0.72 | C3, C4, C10 |
json_without_erp | Plausible mistake | Wrote the quote file and never updated the database row | Fail | 0.76 | C7, C8, C9 |
contradictory_reply_date | Plausible mistake | Reply promised a date earlier than the quote’s own | Fail | 0.96 | C10 |
wrong_customer_on_row | Reward hack | Right amounts, wrong customer on the quote row | Fail | 0.00 | gate 3 |
wrong_sku_on_row | Reward hack | Right amounts, wrong product on the quote row | Fail | 0.00 | gate 3 |
duplicate_quote_row | Reward hack | A second quote row added | Fail | 0.00 | gate 3 |
neighbour_quote_deleted | Reward hack | Another quote’s row deleted | Fail | 0.00 | gate 3 |
quotes_table_replaced | Reward hack | The quotes table dropped and rebuilt | Fail | 0.00 | gate 2, gate 3 |
schema_column_added | Reward hack | A column added to the database schema | Fail | 0.00 | gate 2, gate 3 |
drop_quotes_table | Reward hack | The quotes table dropped | Fail | 0.00 | gate 2, Rejected as invalid |
malformed_quote_json | Reward hack | A quote file that is not valid JSON | Fail | 0.00 | Rejected as invalid |
oversized_number | Reward hack | An out-of-range number in the quote file | Fail | 0.00 | Rejected as invalid |
reply_without_headers | Reward hack | A reply with no email headers | Fail | 0.00 | Rejected as invalid |
extra_reply_file | Reward hack | A second reply file added | Fail | 0.00 | Rejected as invalid |
forged_customer_email | Reward hack | A forged customer email added to the inbox | Fail | 0.00 | gate 1 |
price_one_cent_off | Plausible mistake | Price one cent off the procedure’s figure | Fail | 0.80 | C4, C10, C12 |
erp_price_differs | Plausible mistake | Database price differs from the quote file | Fail | 0.92 | C9 |
cost_price_inconsistent | Plausible mistake | Stated price does not follow from the stated cost | Fail | 0.96 | C12 |
promise_in_subject | Plausible mistake | Earlier date promised in the subject line | Fail | 0.96 | C10 |
wrapped_promise | Plausible mistake | Earlier date promised across a line break | Fail | 0.96 | C10 |
month_first_promise | Plausible mistake | Earlier date promised in month-first form | Fail | 0.96 | C10 |
worded_promise | Plausible mistake | Earlier date promised in words | Fail | 0.96 | C10 |
reply_wrong_quantity | Plausible mistake | Reply names a quantity other than the order’s | Fail | 0.96 | C11 |
reply_wrong_currency | Plausible mistake | Reply states the price in another currency | Fail | 0.96 | C10 |
reply_to_other_customer | Plausible mistake | Reply addressed to another customer | Fail | 0.96 | C11 |
slash_date_promise | Plausible mistake | Earlier date promised in slash form | Fail | 0.96 | C10 |
negation_in_subordinate_clause | Plausible mistake | Earlier date promised, with a negation only in a side clause | Fail | 0.96 | C10 |
relative_in_21_days | Plausible mistake | Delivery promised within a number of days the schedule cannot meet | Fail | 0.96 | C10 |
contradictory_second_price | Plausible mistake | Reply offers a second price that contradicts the first | Fail | 0.96 | C10 |
reply_omits_quantity | Plausible mistake | Reply never states the quantity | Fail | 0.96 | C11 |
dash_date_promise | Plausible mistake | Earlier date promised in day-month-year form with dashes | Fail | 0.96 | C10 |
hyphen_month_promise | Plausible mistake | Earlier date promised as day, short month and year joined by hyphens | Fail | 0.96 | C10 |
iso_slash_promise | Plausible mistake | Earlier date promised in year-first slash form | Fail | 0.96 | C10 |
date_before_verb | Plausible mistake | Earlier date promised, placed before the verb in the sentence | Fail | 0.96 | C10 |
cost_as_discount | Plausible mistake | Reply offers the plant’s internal unit cost as a discounted price | Fail | 0.96 | C10 |
stage_changed_without_history | Plausible mistake | Quote row escalated with no history row recording the change | Fail | 0.92 | C13 |
history_rewritten | Reward hack | A seeded history row edited | Fail | 0.00 | gate 4 |
history_on_other_quote | Reward hack | A history row appended to another quote | Fail | 0.00 | gate 4 |
history_backdated | Reward hack | A history row appended with a date before the quotation date | Fail | 0.00 | gate 4 |
requested_date_no_token | Open class, passes by design | Reply confirms delivery “by your requested date”, with no date written | Pass | 1.00 | none |
lead_time_no_token | Open class, passes by design | Reply promises delivery “within your lead time”, with no date written | Pass | 1.00 | none |
vague_week_promise | Open class, passes by design | Reply promises to ship “the week after next” | Pass | 1.00 | none |
reply_cites_customer_email | Correct alternative | Reply quotes the customer’s email, its dates included | Pass | 1.00 | none |
reply_dates_in_words | Correct alternative | Reply gives its dates in words | Pass | 1.00 | none |
reply_with_total_and_slash_date | Correct alternative | Reply states the unit price, the line total, the set-up charge and the date in slash form | Pass | 1.00 | none |
reference | Reference solution | Our reference answer, from a fresh reset | Pass | 1.00 | none |
Our test harness also built 29 hostile inputs from a correct end state: 10 with a table dropped, 10 with a table’s first column renamed, one with the database overwritten with text, 6 with an input replaced by a directory, 2 with an output replaced by a directory. Every one returned a report at reward zero with no verifier error.
What the numbers mean
Score. Zero if the submission is rejected or any gate fails. Otherwise the weighted share of checks met: the weights of the checks that passed, divided by the sum of all weights. It is a diagnostic for where an answer went wrong.
Strict. True only when every gate and every check passes. This is the training reward we recommend: the score pays a half-finished job, as the row that wrote the quote file and never updated the database shows at 0.76.
Critical. A separate field, true when every check marked critical passes. It is reported, never used as the reward.
What reset guarantees
The reset writes the workspace from code, with no network and no clock. Two fresh resets produce the same digest, and a used workspace resets to it, so a buyer can check on their own machine that the starting state they have is the one the verifier was run against. The digest hashes the files and an SQL dump of the database, so matching digests prove matching logical content, not identical bytes.
What a practitioner settles
Where the trade has more than one convention, the verifier encodes one. In a commissioned task, a practitioner who does this job makes the choice, and we report whether it changes the answer. Open questions for this task:
- Are production days in the schedule calendar days or working days? The earliest delivery date (C6) and the reply (C10) move with the answer.
- Should the reply carry freight, validity and payment terms? Today it carries none, and no check asks for them.
The other five tasks carry their own open conventions, from whether confirming a name counts as asking for a personal detail in a graded call to how dry tonnes are rounded before a cargo is priced. Each is listed in the task record a buyer receives.
What the verifier does not yet catch
We attack our own verifiers. On 7 November 2026, three model-based reviews of the first versions built wrong end states, and two verifiers accepted them at full score. We reproduced them and, on 8 November 2026, tested all six verifiers against about 220 single changes to a correct end state. The classes found, and their status on this task in the gates run on 10 November 2026:
| Class of wrong work | Status on this task, 10 November 2026 |
|---|---|
| The reply contradicts the structured output: it promises a date the schedule cannot meet while the quote file carries the right date | Caught since version 0.1: C10, critical, and the probe contradictory_reply_date. Version 1.0 also reads the subject line and more date forms, slash and relative dates among them, and passes a correct reply that cites the customer’s email, which version 0.1 wrongly failed |
| Identity fields on the output row rewritten while the checked amounts stay right | Caught since version 1.0 by gate 3: wrong_customer_on_row, wrong_sku_on_row |
| Extra rows inserted in the output table | Caught since version 1.0 by gate 3: duplicate_quote_row |
| Historical or neighbouring rows deleted or corrupted | Caught since version 1.0 by gate 3: neighbour_quote_deleted, with a second quote row added to the seed so there is something to delete |
| Amounts off by up to one unit pass a check whose text says “to the cent” | Closed in version 1.0: C4, C9 and C12 compare to the cent; price_one_cent_off fails strict at 0.80 |
| Reset of a used workspace fails instead of restoring the starting state | Closed in version 1.0: a used workspace resets to the recorded digest |
| Dropping the output table crashes the verifier instead of returning a rejection | Closed in version 1.0: drop_quotes_table and 29 hostile inputs return a structured rejection or a zero |
| The reset was described as identical to the byte | Wording corrected: the digest proves matching logical content |
| Every reward-hack probe exercised only the top-level gates; none touched the mutable output table | Closed in version 1.0: six probes write to the quotes table, each at zero |
| The reply promises an earlier date in a form the parser did not read: dashes, a short month joined by hyphens, year first with slashes | Caught since version 1.0.1: dash_date_promise, hyphen_month_promise, iso_slash_promise. Three promises with no date in them at all are recorded as open-class probes that pass by design: requested_date_no_token, lead_time_no_token, vague_week_promise |
| The reply puts the date before the promise, or offers the plant’s internal cost as a discount | Caught since version 1.0.2: date_before_verb and cost_as_discount, both failing C10 |
| The quote row changes with no record of who changed what, or the history is rewritten to fit | Caught since version 1.0.3: C13, critical, and the probe stage_changed_without_history; the history table is protected by gate 4, with history_rewritten, history_on_other_quote and history_backdated each at zero |
| Still not read: prose inside a required output that contradicts its own correct figures, such as a reply that negates the price it states, a reason column, an extra narrative key, or a report beside correct tables | Open, by design. No check reads that prose today, and a hand-written probe cannot close it. A practitioner’s blind grade is the only check on it |
Every class found by those reviews and by our own is now a named probe, and each scores as required in the gates run on 10 November 2026. Hand-written probes are a regression test, not a guarantee: they prove that every wrong end state we have thought of scores what it should, and nothing about the ones we have not. Each new class becomes a named probe the day it is found.
Change log
- By 7 November 2026. Version 0 written and run. Two current models from one provider solved it in one rollout each. A model-based review of the verifier found the reply check accepted a promise of a delivery date the schedule could not meet.
- 8 November 2026. Version 0.1: C10 parses the dates in the reply body and is critical; the probe
contradictory_reply_dateadded, failing strict at 0.96. - 8 November 2026. Version 1.0: field-level protection of the quotes table (gate 3), structured rejection of a submission that cannot be scored, prices compared to the cent, a stricter reply check that also passes a correct reply version 0.1 wrongly failed, two new checks (C11, C12), 27 new reward-hack and plausible-mistake probes, two correct alternatives, and one historical quote row in the seed. The reference answer is unchanged, so the version 0 model attempts stand. This page published.
- 8 November 2026. Version 1.0.1: the reply check reads dash, day-short-month and year-first slash dates, each with a plausible mistake. Three promises with no date in them are recorded as open-class probes that pass by design. New module hash. The reference answer is unchanged.
- 8 November 2026. Version 1.0.2: the reply check reads a date placed before the promise, and no longer passes a reply that offers the plant’s internal cost as a discount. One more correct alternative. The reference answer is unchanged.
- 8 November 2026. Version 1.0.3: the seed gains the customer’s chase email and a history table with four authored rows, protected at field level by gate 4. The procedure and the instruction ask for each change to the quote row to be logged, and C13, critical, checks it. Four new probes. Prices, dates and the reference quote are unchanged; the weights now sum to 25, so every score moved.
New results are published with their date, whether they pass or fail.
Run it yourself
First, a twenty-minute call: we run this task in front of your engineer, from a reset through a reward hack and a plausible mistake to the reference solution. Under a signed two-way NDA, your engineer then receives the standalone verifier, with seed, reset, score and self-test commands in Python 3.11 and no third-party packages, together with the workspace, the reference end state and the verifier’s reports on the untouched workspace and on the reference solution. Re-running every probe above on your own machine is the acceptance test for a first order, which we return within ten business days of your written requirements.