PAIR-20: what happens when real procurement tasks meet the AI platforms
PAIR-20 is an open benchmark of realistic procurement case files — run on ChatGPT, Claude, Gemini and Copilot. Every verdict has a dated screenshot behind it, and a fabricated fact fails the whole task.
The name is the target: twenty procurement realities. Each task is a synthetic case file built to carry the same traps a real file carries — a buried mandatory fee, an inconsistent unit, a superseded quote left in the pile. The platforms get the file on their free or standard web tiers — whatever a procurement team actually has without an IT ticket — with both a generic prompt and a structured brief.
The runs are single and dated, not averages. A model that missed a trap on this run might catch it on a re-roll. That limitation is stated on every result, because the honesty is the point: this is a field note, not a leaderboard.
The fabrication rule
One rule sits above the others: a fabricated fact fails the task, whatever else the output got right. An invented contract expiry, a guessed pack quantity written in as a hard number — these flow into a deliverable as facts a reader cannot see are invented. That is the one failure a procurement team cannot defend against, so in PAIR-20 it is an automatic fail, not a deduction.
The July 2026 scoreboard
Twenty tasks are on the roadmap. Their status is graded by evidence, and the grade is stated per task — we never blur an authored draft into an evidenced result.
Six tasks · dated cross-platform screenshots on disk
v1.0. Each result rests on a real ChatGPT, Claude, Gemini and Copilot session with a screenshot saved and dated. These are the tasks the July Reality Check draws on.
| ID | Codename | Job | The trap it tests |
|---|---|---|---|
| T01 | Unit Flip | SaaS quote normalization | A unit conversion (per 100 seats vs per seat) that survives the expert prompt. |
| T02 | Blind Clock | SaaS MSA review | Two clauses 26 pages apart that together remove the buyer’s ability to reject a price rise. |
| T03 | Landed Mirage | Freight landed cost | The headline-cheapest carrier is the most expensive once the Incoterms stack is rebuilt. |
| T04 | Liar Fields | Supplier risk register | Data fields that lie — and a bait that invites fabricating one that doesn’t. |
| T05 | Hidden Twelve | Consulting rate comparison | A summary table that contradicts its own arithmetic; a fee outside the pricing table. |
| T06 | False Twin | MRO catalog merge | A false duplicate priced on a guessed pack quantity. |
Five tasks · pending a fresh cross-platform re-run
Full task folders on disk — case files copied verbatim, both prompts, and a sealed hand-verified answer key. They carry partial evidence from the July runs; what gates publication is a single dated pass across all four platforms, because Gemini and Copilot were left uncaptured on several. Candidates for the August 2026 issue. Authored is not evidenced — these are not yet results.
| ID | Codename | Job | The trap it tests |
|---|---|---|---|
| T07 | Blind Index | Commodity price counter | An external index the AI cannot know — does it name its blind spot or invent the number? |
| T08 | Locked Seats | SaaS license reclamation | A headline savings number the contracts (ratchet, bundle floor, seat minimum) kill. |
| T09 | Trigger Clause | SaaS consolidation | A repricing clause (§12.3) that fires on the exact headcount jump consolidation causes. |
| T10 | Silent Floor | Fleet bid scorecard | A mandatory coverage cost read as optional, flipping the recommended vendor. |
| T11 | Split Precedent | Services SOW audit | A precedence conflict between two files, and a closed-SOW distractor that baits fabrication. |
The remaining nine tasks (T12–T20) stay queued: drafted case files with hand-verified answer keys, awaiting authoring into full task folders and a real cross-platform run before they publish. No task ships without a dated real-platform run and a sealed answer key.
How a task is run
Case files. Synthetic. Fictional companies, fictional suppliers, fictional prices, built to carry the same traps a real file carries.
Platforms. ChatGPT, Claude, Gemini, Copilot. Free or standard web tiers, guest or logged-in — whatever a procurement team can reach without an IT ticket.
Prompts. Two per platform: a generic one (“review this and tell me if there are concerns”) and a structured brief that names the fields to normalize and the output format.
Evidence. A screenshot or transcript for every claim, saved and dated. Simulated output does not count and never enters a result. No screenshot, no claim.
Answer keys. Sealed and hand-verified before the run, so the grade is measured against a fixed ground truth, not back-fitted to what the models produced.
The tasks and rubric are open
Every task folder — case files, verbatim prompts, sealed answer keys, and the scoring rubric — lives in the open PAIR-20 repository, so anyone can re-run it and check our working.
The course works these same traps, end to end
Twenty lessons, each one a procurement task worked on four AI platforms with the evidence shown. Lesson 2 is free.