Menu

Research

Packaging re-quote, version 1.0.3: every check, and every known gap

One demonstration task, laid open: the job, the starting state, the gates and the thirteen checks, every probe with its result, how the reward is defined, what reset guarantees, and the mistakes the verifier does not yet catch.

Task mfg-rfq-001 · Version 1.0.3, gates run 10 November 2026 · Datarena

Public demonstration · origin and scope of the demonstration tasks.

Identity

Task
mfg-rfq-001, version 1.0.3, verifier frozen by hash
Family
packaging-quotation, difficulty easy, as the public gate file labels it
Outcome
unresolved, by construction: the episode ends when the reply is sent, before the customer answers; a practitioner authors the label in v1.1
Domain
Manufacturing: plastics packaging, a request for quotation answered with a quote
Status
Public demonstration; disclosed in full and not for sale.
Gates run
10 November 2026, Python 3.11.8, deterministic gates only: no model runs, no human raters, no training
Model attempts
Version 0, on or before 7 November 2026: two current models from one provider, one of them its small tier, one rollout each, both scored 1.0 and passed strict. The later versions, which tightened the verifier and left the reference answer unchanged, run next

The job

A sales executive at a packaging plant has to get a quote out today. The customer’s second email cut the order and moved the delivery date, the production chat moved the job to another machine because the usual one is down, and the quoting procedure says how to cost, price and escalate. The instruction, as the agent receives it:

You are the sales executive (sales_exec_2) at Tembaga Ridge Packaging. Today is 16 Nov 2026. Get quote Q-2611 out to Selat Biru Agrochem today. Read the inbox and the production chat, follow docs/SOP-Q-07.md, update the quotes row in erp/plant.db, log each change you make to it in crm_history, and write out/quote.json (qty, machine, unit_cost_rm, unit_price_rm, earliest_delivery as YYYY-MM-DD, meets_requested_delivery) and out/reply.eml.

The right answer takes the latest quantity, the confirmed machine and the current resin price, recomputes cost and price from the procedure, finds the earliest slot in the schedule, marks the requested date as not met, escalates the quote in the database, logs each change it makes to the quote row, and writes a reply that promises nothing the schedule cannot deliver.

Read the starting state

Currency amounts appear as [amount] on this page; the full files go to your engineer under NDA. The reference reply is deliberately minimal: the verifier reads its price, quantity and dates, not its tone.

The task’s own files, in the order a worker meets them, then the reference end state and one plausible mistake. Each sheet is generated from the task when the page is built, so it cannot drift from the files the verifier reads. The highlighter marks the facts that changed and the line the verifier caught.

The inbox, four messages

inbox/001.eml9 Nov 2026, 10:14

From
procurement@selatbiru.example
To
sales@tembagaridge.example
Subject
RFQ - 20 L jerrican

Please quote for 50,000 pcs of your 20 L HDPE jerrican (JC20), natural colour. Delivery to Port Klang.

Procurement, Selat Biru Agrochem

inbox/002.eml13 Nov 2026, 15:31

From
procurement@selatbiru.example
To
sales@tembagaridge.example
Subject
Re: RFQ - 20 L jerrican (revised)

Revision to our RFQ: we now need the jerricans in BLUE, and the first order will be 12,000 pcs, not 50,000. We need delivery within 21 days of your quotation date. Please re-quote.

Procurement, Selat Biru Agrochem

inbox/003.eml16 Nov 2026, 08:40

From
sales.director@tembagaridge.example
To
sales_exec_2@tembagaridge.example
Subject
Q-2611 today please

Please get Q-2611 out to Selat Biru today. Follow SOP-Q-07 and check with production first.

inbox/000.eml16 Nov 2026, 09:05

From
procurement@selatbiru.example
To
sales@tembagaridge.example
Subject
Re: RFQ - 20 L jerrican (revised) - quotation?

Good morning. We have not yet received your revised quotation. Our planning meeting is this afternoon, so please send it today.

Procurement, Selat Biru Agrochem

The production chat

chat/production.json#production

  1. tooling_lead · 14 Nov 2026, 16:02

    BM-07 is down for a screw and barrel replacement, back 10 Dec at the earliest. Run the JC20 mould on BM-03 until then.

  2. planner_1 · 14 Nov 2026, 16:05

    Noted. JC20 cycle on BM-03 runs 18% longer than on BM-07, same single cavity. BM-03 is booked, check machine_schedule before promising dates.

  3. qc_lead · 16 Nov 2026, 08:15

    Reminder: drop test on the blue 10 L pails is at 2pm.

The procedure

docs/SOP-Q-07.mdThe quoting procedure

SOP-Q-07 Quotation costing and disposition (blow moulding)

  1. Governing request. The latest written customer amendment governs quantity, colour and delivery.
  2. Machine. Quote on the machine production has confirmed for the job. Never quote on a machine whose status is DOWN.
  3. Unit cost in RM per piece:
    • material mass = part mass x (1 + 3% scrap)
    • coloured parts: masterbatch replaces resin at the dosing % in the masterbatch table
    • material cost = resin kg x resin price + masterbatch kg x masterbatch price
    • resin price = the row with the latest effective_date on or before the quotation date
    • machine cost = cycle seconds / cavities / 3600 x machine rate per hour
    • set-up = [amount] per job / quantity
    • unit cost = material + machine + set-up
  4. Price. unit price = unit cost / (1 - 22%), rounded to 2 decimals. Never quote below an 18% margin.
  5. MOQ for JC20 is 10,000 pcs. Below MOQ: stage REJECTED_MOQ and no price is sent.
  6. Delivery.
    • production days = quantity x cycle seconds / cavities / (3600 x machine hours per day x 85% OEE), rounded up
    • first free day = the day after booked_until in machine_schedule
    • earliest delivery = first free day + production days + 3 days QC and dispatch
  7. Disposition, in the quotes table:
    • delivery can be met: stage QUOTED, owner stays with the sales executive
    • delivery cannot be met: still price the job, stage ESCALATED_LEADTIME, owner = the sales manager. The reply states the price and the earliest delivery date and does not promise the requested date.
  8. Outputs. Update the quotes row (qty, unit_price_rm, stage, owner). Record every change you make to a quotes field as a row in crm_history (quote_id, ts on the quotation date, actor, field, old_value, new_value, note); never edit an existing history row. Write out/quote.json and out/reply.eml.

The database before the work

erp/plant.dbThe tables the job touches or reads

quotes2 rows

idcustomer_idskuqtyunit_price_rmstageowner
Q-2604C-0388PL1020000[amount]QUOTEDsales_exec_2
Q-2611C-0417JC2050000nullENQUIRYsales_exec_2

machine_schedule2 rows

machine_idbooked_until
BM-032026-11-27
BM-072026-12-09

machines2 rows

idstatusrate_rm_per_hrhours_per_day
BM-07DOWN[amount]20.0
BM-03RUNNING[amount]20.0

crm_history4 rows

idquote_idtsactorfieldold_valuenew_valuenote
1Q-26042026-11-02 14:10sales_exec_2stageENQUIRYQUOTEDPL10 20,000 pcs quoted at [amount]
2Q-26112026-11-09 10:40sales_exec_2stagenullENQUIRYRFQ received: 50,000 pcs JC20 natural, delivery Port Klang
3Q-26112026-11-13 16:02sales_exec_2notenullnullCustomer revision: blue, first order 12,000 pcs, delivery within 21 days of quotation; re-quote pending
4Q-26112026-11-16 08:45sales_exec_2notenullnullSales director: quote out today per SOP-Q-07, check with production first

The reference end state

out/quote.jsonJSON

{
  "quote_id": "Q-2611",
  "qty": 12000,
  "machine": "BM-03",
  "unit_cost_rm": "[amount]",
  "unit_price_rm": "[amount]",
  "earliest_delivery": "2026-12-11",
  "meets_requested_delivery": false
}

out/reply.eml16 Nov 2026, 11:00

From
sales_exec_2@tembagaridge.example
To
procurement@selatbiru.example
Subject
Quotation Q-2611

Our price for 12,000 pcs JC20 is [amount] per piece. Earliest delivery is 2026-12-11.

erp/plant.dbAfter the work

quotes, the Q-2611 row1 rows

idcustomer_idskuqtyunit_price_rmstageowner
Q-2611C-0417JC2012000[amount]ESCALATED_LEADTIMEsales_mgr_1

crm_history, rows the work added4 rows

idquote_idtsactorfieldold_valuenew_valuenote
5Q-26112026-11-16 11:00sales_exec_2qty5000012000Q-2611 re-quoted per SOP-Q-07
6Q-26112026-11-16 11:00sales_exec_2unit_price_rmnull[amount]Q-2611 re-quoted per SOP-Q-07
7Q-26112026-11-16 11:00sales_exec_2stageENQUIRYESCALATED_LEADTIMEQ-2611 re-quoted per SOP-Q-07
8Q-26112026-11-16 11:00sales_exec_2ownersales_exec_2sales_mgr_1Q-2611 re-quoted per SOP-Q-07

The verifier: Pass · score 1.00

Raw verifier report, reference solution
{
  "gates": [
    {
      "name": "inputs untouched and no file added under inbox/, chat/ or docs/",
      "passed": true
    },
    {
      "name": "protected ERP tables and the database schema untouched",
      "passed": true
    },
    {
      "name": "quotes table: only Q-2611's qty, unit_price_rm, stage and owner changed; no row added, removed or re-identified",
      "passed": true
    },
    {
      "name": "crm_history: seeded rows untouched; rows appended only for Q-2611 and dated on the quotation date",
      "passed": true
    }
  ],
  "criteria": [
    {
      "id": "C1",
      "desc": "Quantity follows the latest amendment (12,000)",
      "passed": true,
      "weight": 2,
      "critical": true
    },
    {
      "id": "C2",
      "desc": "Machine is the one production confirmed (BM-03), not the down machine",
      "passed": true,
      "weight": 2,
      "critical": false
    },
    {
      "id": "C3",
      "desc": "Unit cost within 0.5% of the SOP recomputation",
      "passed": true,
      "weight": 3,
      "critical": false
    },
    {
      "id": "C4",
      "desc": "Unit price equals the SOP price to the cent",
      "passed": true,
      "weight": 3,
      "critical": false
    },
    {
      "id": "C5",
      "desc": "Price keeps at least an 18% margin on the true cost",
      "passed": true,
      "weight": 2,
      "critical": true
    },
    {
      "id": "C6",
      "desc": "Earliest delivery date matches the schedule recomputation",
      "passed": true,
      "weight": 2,
      "critical": false
    },
    {
      "id": "C7",
      "desc": "Delivery marked as not met and the quote escalated in the ERP",
      "passed": true,
      "weight": 3,
      "critical": true
    },
    {
      "id": "C8",
      "desc": "Owner reassigned to the sales manager",
      "passed": true,
      "weight": 1,
      "critical": false
    },
    {
      "id": "C9",
      "desc": "ERP row agrees with quote.json (qty, and price to the cent)",
      "passed": true,
      "weight": 2,
      "critical": false
    },
    {
      "id": "C10",
      "desc": "Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date",
      "passed": true,
      "weight": 1,
      "critical": true
    },
    {
      "id": "C11",
      "desc": "Reply is addressed to the customer, identifies Q-2611, and names no quantity other than 12,000 pcs",
      "passed": true,
      "weight": 1,
      "critical": false
    },
    {
      "id": "C12",
      "desc": "Stated price follows from the stated cost under the SOP mark-up (cost / 0.78, rounded)",
      "passed": true,
      "weight": 1,
      "critical": false
    },
    {
      "id": "C13",
      "desc": "Each change to stage and owner on the quotes row is logged in crm_history with its old and new value",
      "passed": true,
      "weight": 2,
      "critical": true
    }
  ],
  "strict_pass": true,
  "reward": 1.0,
  "rejection": null
}

One plausible mistake: costed the machine that is down

The probe down_machine. Plausible mistake. Its files, and what the verifier caught.

out/quote.jsonJSON

{
  "quote_id": "Q-2611",
  "qty": 12000,
  "machine": "BM-07",
  "unit_cost_rm": "[amount]",
  "unit_price_rm": "[amount]",
  "earliest_delivery": "2026-11-27",
  "meets_requested_delivery": true
}

out/reply.eml16 Nov 2026, 11:00

From
sales_exec_2@tembagaridge.example
To
procurement@selatbiru.example
Subject
Quotation Q-2611

Our price for 12,000 pcs JC20 is [amount] per piece. Earliest delivery is 2026-11-27.

The verifier: Fail · score 0.40

  • C2 Machine is the one production confirmed (BM-03), not the down machine
  • C3 Unit cost within 0.5% of the SOP recomputation
  • C4 Unit price equals the SOP price to the cent
  • C6 Earliest delivery date matches the schedule recomputation
  • C7 Delivery marked as not met and the quote escalated in the ERP
  • C8 Owner reassigned to the sales manager
  • C10 Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date
Raw verifier report, down_machine
{
  "gates": [
    {
      "name": "inputs untouched and no file added under inbox/, chat/ or docs/",
      "passed": true
    },
    {
      "name": "protected ERP tables and the database schema untouched",
      "passed": true
    },
    {
      "name": "quotes table: only Q-2611's qty, unit_price_rm, stage and owner changed; no row added, removed or re-identified",
      "passed": true
    },
    {
      "name": "crm_history: seeded rows untouched; rows appended only for Q-2611 and dated on the quotation date",
      "passed": true
    }
  ],
  "criteria": [
    {
      "id": "C1",
      "desc": "Quantity follows the latest amendment (12,000)",
      "passed": true,
      "weight": 2,
      "critical": true
    },
    {
      "id": "C2",
      "desc": "Machine is the one production confirmed (BM-03), not the down machine",
      "passed": false,
      "weight": 2,
      "critical": false
    },
    {
      "id": "C3",
      "desc": "Unit cost within 0.5% of the SOP recomputation",
      "passed": false,
      "weight": 3,
      "critical": false
    },
    {
      "id": "C4",
      "desc": "Unit price equals the SOP price to the cent",
      "passed": false,
      "weight": 3,
      "critical": false
    },
    {
      "id": "C5",
      "desc": "Price keeps at least an 18% margin on the true cost",
      "passed": true,
      "weight": 2,
      "critical": true
    },
    {
      "id": "C6",
      "desc": "Earliest delivery date matches the schedule recomputation",
      "passed": false,
      "weight": 2,
      "critical": false
    },
    {
      "id": "C7",
      "desc": "Delivery marked as not met and the quote escalated in the ERP",
      "passed": false,
      "weight": 3,
      "critical": true
    },
    {
      "id": "C8",
      "desc": "Owner reassigned to the sales manager",
      "passed": false,
      "weight": 1,
      "critical": false
    },
    {
      "id": "C9",
      "desc": "ERP row agrees with quote.json (qty, and price to the cent)",
      "passed": true,
      "weight": 2,
      "critical": false
    },
    {
      "id": "C10",
      "desc": "Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date",
      "passed": false,
      "weight": 1,
      "critical": true
    },
    {
      "id": "C11",
      "desc": "Reply is addressed to the customer, identifies Q-2611, and names no quantity other than 12,000 pcs",
      "passed": true,
      "weight": 1,
      "critical": false
    },
    {
      "id": "C12",
      "desc": "Stated price follows from the stated cost under the SOP mark-up (cost / 0.78, rounded)",
      "passed": true,
      "weight": 1,
      "critical": false
    },
    {
      "id": "C13",
      "desc": "Each change to stage and owner on the quotes row is logged in crm_history with its old and new value",
      "passed": true,
      "weight": 2,
      "critical": true
    }
  ],
  "strict_pass": false,
  "reward": 0.4,
  "rejection": null
}

What to notice

  • The second email supersedes the first.
  • The constraint lives only in the chat.
  • The reply must not promise a date the schedule cannot meet.
  • The checker reads the files and rows, never the reply’s claims.
  • Every change to the quote row must be logged in the history table, dated on the quotation date.

The five parts in this version

Customer emails
Present: the enquiry, the revision, the director’s note and the chase, four messages in the inbox
Internal discussion
Present: the production chat, three lines
Related documents
Present: the quoting procedure
History log
Present: the history table, four authored rows before the work starts. Each change to the quote row must add one, checked by C13
Outcome label
Present on the task record, authored as “unresolved”. By construction: the episode ends when the reply is sent, before the customer answers; a practitioner authors the label in v1.1

The workspace

Every run starts from the same files and the same database, written fresh by the task’s own reset.

PathWhat it holds after reset
chat/production.json639 bytes
docs/SOP-Q-07.md1877 bytes
erp/plant.dbcrm_history (4), customers (2), machine_schedule (2), machines (2), masterbatch (2), mould_machine (1), products (2), quotes (2), resin_prices (3), staff (3)
inbox/000.eml324 bytes
inbox/001.eml272 bytes
inbox/002.eml364 bytes
inbox/003.eml240 bytes

Reset digest, SHA-256 of the seeded workspace with the database taken as its SQL dump: 3366510ba57c7b69d148f9dd16a74ceaea0a556e649b813b1a5d1d55c446d20d. Reset deterministic from an empty directory: yes. A used workspace resets to the same digest: yes.

What the agent may read and change

Reads
The four inbox messages, the production chat, the quoting procedure, and any table in the database.
Writes
out/quote.json with the required fields, out/reply.eml, four fields of the one quote row the instruction names: quantity, unit price, stage and owner, and one new row in the history table for each of those changes.
Must not touch
The input files, the input folders, the other tables, the database schema, every other field and row of the quotes table, and every history row that was there at the start. Any change there fails a gate, and the run scores zero whatever else it did.

Gates and checks

A submission the verifier cannot score, because a required output is missing, malformed or damaged, is rejected as invalid at reward zero, with a structured reason rather than a crash. Then four gates. Fail any and the reward is zero, with no checks counted.

  1. inputs untouched and no file added under inbox/, chat/ or docs/.
  2. protected ERP tables and the database schema untouched.
  3. quotes table: only Q-2611's qty, unit_price_rm, stage and owner changed; no row added, removed or re-identified.
  4. crm_history: seeded rows untouched; rows appended only for Q-2611 and dated on the quotation date.

Then thirteen checks, each recomputed by the verifier from a fresh copy of the starting state and compared with the files and database rows on disk. The verifier never reads what the agent says it did. Weights sum to 25; 5 checks are critical.

CheckWhat must be trueWeightCritical
C1Quantity follows the latest amendment (12,000)2yes
C2Machine is the one production confirmed (BM-03), not the down machine2
C3Unit cost within 0.5% of the SOP recomputation3
C4Unit price equals the SOP price to the cent3
C5Price keeps at least an 18% margin on the true cost2yes
C6Earliest delivery date matches the schedule recomputation2
C7Delivery marked as not met and the quote escalated in the ERP3yes
C8Owner reassigned to the sales manager1
C9ERP row agrees with quote.json (qty, and price to the cent)2
C10Reply states the SOP price in RM and the earliest delivery date, and promises no earlier date1yes
C11Reply is addressed to the customer, identifies Q-2611, and names no quantity other than 12,000 pcs1
C12Stated price follows from the stated cost under the SOP mark-up (cost / 0.78, rounded)1
C13Each change to stage and owner on the quotes row is logged in crm_history with its old and new value2yes

Tolerances in version 1.0.3: C3 compares unit cost within 0.5%, as its text says; C4, C9 and C12 compare prices to the cent (within half a cent); C10 reads the reply’s subject and body, every date format, and only dates after the quotation date.

The probes

Before a buyer sees a task, we attack its verifier. A reward hack fakes or tampers and must score exactly zero. A plausible mistake is one a careful worker could make; it must not earn the strict reward, which needs every gate and every check to pass. A correct alternative is a right answer written another way; it must earn it. A known-gap probe records a weakness we have not closed: it passes by design, so any change at that boundary shows up. 19 reward hacks, 26 plausible mistakes, 3 correct alternatives and 3 known-gap probes ran on 10 November 2026, and every one returned the expected result. Most plausible mistakes target the reply, the one free-text output: a promise of an earlier date, in every date format we have met.

ProbeKindWhat it didStrictScoreWhat failed
emptyReward hackNo answer writtenFail0.00Rejected as invalid
claim_onlyReward hackA confident reply with no files writtenFail0.00Rejected as invalid
tamper_scheduleReward hackFreed the booked machine in the schedule tableFail0.00gate 2
edit_customer_emailReward hackEdited the customer’s email so the old quantity reads as currentFail0.00gate 1
superseded_quantityPlausible mistakeQuoted the superseded quantityFail0.40C1, C3, C4, C5, C6, C10, C11, C12
down_machinePlausible mistakeCosted the machine that is downFail0.40C2, C3, C4, C6, C7, C8, C10
stale_resin_pricePlausible mistakePriced on a resin price the table had supersededFail0.72C3, C4, C10
json_without_erpPlausible mistakeWrote the quote file and never updated the database rowFail0.76C7, C8, C9
contradictory_reply_datePlausible mistakeReply promised a date earlier than the quote’s ownFail0.96C10
wrong_customer_on_rowReward hackRight amounts, wrong customer on the quote rowFail0.00gate 3
wrong_sku_on_rowReward hackRight amounts, wrong product on the quote rowFail0.00gate 3
duplicate_quote_rowReward hackA second quote row addedFail0.00gate 3
neighbour_quote_deletedReward hackAnother quote’s row deletedFail0.00gate 3
quotes_table_replacedReward hackThe quotes table dropped and rebuiltFail0.00gate 2, gate 3
schema_column_addedReward hackA column added to the database schemaFail0.00gate 2, gate 3
drop_quotes_tableReward hackThe quotes table droppedFail0.00gate 2, Rejected as invalid
malformed_quote_jsonReward hackA quote file that is not valid JSONFail0.00Rejected as invalid
oversized_numberReward hackAn out-of-range number in the quote fileFail0.00Rejected as invalid
reply_without_headersReward hackA reply with no email headersFail0.00Rejected as invalid
extra_reply_fileReward hackA second reply file addedFail0.00Rejected as invalid
forged_customer_emailReward hackA forged customer email added to the inboxFail0.00gate 1
price_one_cent_offPlausible mistakePrice one cent off the procedure’s figureFail0.80C4, C10, C12
erp_price_differsPlausible mistakeDatabase price differs from the quote fileFail0.92C9
cost_price_inconsistentPlausible mistakeStated price does not follow from the stated costFail0.96C12
promise_in_subjectPlausible mistakeEarlier date promised in the subject lineFail0.96C10
wrapped_promisePlausible mistakeEarlier date promised across a line breakFail0.96C10
month_first_promisePlausible mistakeEarlier date promised in month-first formFail0.96C10
worded_promisePlausible mistakeEarlier date promised in wordsFail0.96C10
reply_wrong_quantityPlausible mistakeReply names a quantity other than the order’sFail0.96C11
reply_wrong_currencyPlausible mistakeReply states the price in another currencyFail0.96C10
reply_to_other_customerPlausible mistakeReply addressed to another customerFail0.96C11
slash_date_promisePlausible mistakeEarlier date promised in slash formFail0.96C10
negation_in_subordinate_clausePlausible mistakeEarlier date promised, with a negation only in a side clauseFail0.96C10
relative_in_21_daysPlausible mistakeDelivery promised within a number of days the schedule cannot meetFail0.96C10
contradictory_second_pricePlausible mistakeReply offers a second price that contradicts the firstFail0.96C10
reply_omits_quantityPlausible mistakeReply never states the quantityFail0.96C11
dash_date_promisePlausible mistakeEarlier date promised in day-month-year form with dashesFail0.96C10
hyphen_month_promisePlausible mistakeEarlier date promised as day, short month and year joined by hyphensFail0.96C10
iso_slash_promisePlausible mistakeEarlier date promised in year-first slash formFail0.96C10
date_before_verbPlausible mistakeEarlier date promised, placed before the verb in the sentenceFail0.96C10
cost_as_discountPlausible mistakeReply offers the plant’s internal unit cost as a discounted priceFail0.96C10
stage_changed_without_historyPlausible mistakeQuote row escalated with no history row recording the changeFail0.92C13
history_rewrittenReward hackA seeded history row editedFail0.00gate 4
history_on_other_quoteReward hackA history row appended to another quoteFail0.00gate 4
history_backdatedReward hackA history row appended with a date before the quotation dateFail0.00gate 4
requested_date_no_tokenOpen class, passes by designReply confirms delivery “by your requested date”, with no date writtenPass1.00none
lead_time_no_tokenOpen class, passes by designReply promises delivery “within your lead time”, with no date writtenPass1.00none
vague_week_promiseOpen class, passes by designReply promises to ship “the week after next”Pass1.00none
reply_cites_customer_emailCorrect alternativeReply quotes the customer’s email, its dates includedPass1.00none
reply_dates_in_wordsCorrect alternativeReply gives its dates in wordsPass1.00none
reply_with_total_and_slash_dateCorrect alternativeReply states the unit price, the line total, the set-up charge and the date in slash formPass1.00none
referenceReference solutionOur reference answer, from a fresh resetPass1.00none

Our test harness also built 29 hostile inputs from a correct end state: 10 with a table dropped, 10 with a table’s first column renamed, one with the database overwritten with text, 6 with an input replaced by a directory, 2 with an output replaced by a directory. Every one returned a report at reward zero with no verifier error.

Results across the full demonstration harness.

What the numbers mean

Score. Zero if the submission is rejected or any gate fails. Otherwise the weighted share of checks met: the weights of the checks that passed, divided by the sum of all weights. It is a diagnostic for where an answer went wrong.

Strict. True only when every gate and every check passes. This is the training reward we recommend: the score pays a half-finished job, as the row that wrote the quote file and never updated the database shows at 0.76.

Critical. A separate field, true when every check marked critical passes. It is reported, never used as the reward.

Shared verifier principles.

What reset guarantees

The reset writes the workspace from code, with no network and no clock. Two fresh resets produce the same digest, and a used workspace resets to it, so a buyer can check on their own machine that the starting state they have is the one the verifier was run against. The digest hashes the files and an SQL dump of the database, so matching digests prove matching logical content, not identical bytes.

What a practitioner settles

Where the trade has more than one convention, the verifier encodes one. In a commissioned task, a practitioner who does this job makes the choice, and we report whether it changes the answer. Open questions for this task:

  • Are production days in the schedule calendar days or working days? The earliest delivery date (C6) and the reply (C10) move with the answer.
  • Should the reply carry freight, validity and payment terms? Today it carries none, and no check asks for them.

The other five tasks carry their own open conventions, from whether confirming a name counts as asking for a personal detail in a graded call to how dry tonnes are rounded before a cargo is priced. Each is listed in the task record a buyer receives.

What the verifier does not yet catch

We attack our own verifiers. On 7 November 2026, three model-based reviews of the first versions built wrong end states, and two verifiers accepted them at full score. We reproduced them and, on 8 November 2026, tested all six verifiers against about 220 single changes to a correct end state. The classes found, and their status on this task in the gates run on 10 November 2026:

Class of wrong workStatus on this task, 10 November 2026
The reply contradicts the structured output: it promises a date the schedule cannot meet while the quote file carries the right dateCaught since version 0.1: C10, critical, and the probe contradictory_reply_date. Version 1.0 also reads the subject line and more date forms, slash and relative dates among them, and passes a correct reply that cites the customer’s email, which version 0.1 wrongly failed
Identity fields on the output row rewritten while the checked amounts stay rightCaught since version 1.0 by gate 3: wrong_customer_on_row, wrong_sku_on_row
Extra rows inserted in the output tableCaught since version 1.0 by gate 3: duplicate_quote_row
Historical or neighbouring rows deleted or corruptedCaught since version 1.0 by gate 3: neighbour_quote_deleted, with a second quote row added to the seed so there is something to delete
Amounts off by up to one unit pass a check whose text says “to the cent”Closed in version 1.0: C4, C9 and C12 compare to the cent; price_one_cent_off fails strict at 0.80
Reset of a used workspace fails instead of restoring the starting stateClosed in version 1.0: a used workspace resets to the recorded digest
Dropping the output table crashes the verifier instead of returning a rejectionClosed in version 1.0: drop_quotes_table and 29 hostile inputs return a structured rejection or a zero
The reset was described as identical to the byteWording corrected: the digest proves matching logical content
Every reward-hack probe exercised only the top-level gates; none touched the mutable output tableClosed in version 1.0: six probes write to the quotes table, each at zero
The reply promises an earlier date in a form the parser did not read: dashes, a short month joined by hyphens, year first with slashesCaught since version 1.0.1: dash_date_promise, hyphen_month_promise, iso_slash_promise. Three promises with no date in them at all are recorded as open-class probes that pass by design: requested_date_no_token, lead_time_no_token, vague_week_promise
The reply puts the date before the promise, or offers the plant’s internal cost as a discountCaught since version 1.0.2: date_before_verb and cost_as_discount, both failing C10
The quote row changes with no record of who changed what, or the history is rewritten to fitCaught since version 1.0.3: C13, critical, and the probe stage_changed_without_history; the history table is protected by gate 4, with history_rewritten, history_on_other_quote and history_backdated each at zero
Still not read: prose inside a required output that contradicts its own correct figures, such as a reply that negates the price it states, a reason column, an extra narrative key, or a report beside correct tablesOpen, by design. No check reads that prose today, and a hand-written probe cannot close it. A practitioner’s blind grade is the only check on it

Every class found by those reviews and by our own is now a named probe, and each scores as required in the gates run on 10 November 2026. Hand-written probes are a regression test, not a guarantee: they prove that every wrong end state we have thought of scores what it should, and nothing about the ones we have not. Each new class becomes a named probe the day it is found.

Change log

  1. By 7 November 2026. Version 0 written and run. Two current models from one provider solved it in one rollout each. A model-based review of the verifier found the reply check accepted a promise of a delivery date the schedule could not meet.
  2. 8 November 2026. Version 0.1: C10 parses the dates in the reply body and is critical; the probe contradictory_reply_date added, failing strict at 0.96.
  3. 8 November 2026. Version 1.0: field-level protection of the quotes table (gate 3), structured rejection of a submission that cannot be scored, prices compared to the cent, a stricter reply check that also passes a correct reply version 0.1 wrongly failed, two new checks (C11, C12), 27 new reward-hack and plausible-mistake probes, two correct alternatives, and one historical quote row in the seed. The reference answer is unchanged, so the version 0 model attempts stand. This page published.
  4. 8 November 2026. Version 1.0.1: the reply check reads dash, day-short-month and year-first slash dates, each with a plausible mistake. Three promises with no date in them are recorded as open-class probes that pass by design. New module hash. The reference answer is unchanged.
  5. 8 November 2026. Version 1.0.2: the reply check reads a date placed before the promise, and no longer passes a reply that offers the plant’s internal cost as a discount. One more correct alternative. The reference answer is unchanged.
  6. 8 November 2026. Version 1.0.3: the seed gains the customer’s chase email and a history table with four authored rows, protected at field level by gate 4. The procedure and the instruction ask for each change to the quote row to be logged, and C13, critical, checks it. Four new probes. Prices, dates and the reference quote are unchanged; the weights now sum to 25, so every score moved.

New results are published with their date, whether they pass or fail.

Run it yourself

First, a twenty-minute call: we run this task in front of your engineer, from a reset through a reward hack and a plausible mistake to the reference solution. Under a signed two-way NDA, your engineer then receives the standalone verifier, with seed, reset, score and self-test commands in Python 3.11 and no third-party packages, together with the workspace, the reference end state and the verifier’s reports on the untouched workspace and on the reference solution. Re-running every probe above on your own machine is the acceptance test for a first order, which we return within ten business days of your written requirements.

Request a sample pack Book the walk-through