Is there a benchmark of AI performance on procurement tasks?
PAIR-20 is the closest published reference: six synthetic procurement tasks, four platforms, one dated July 2026 run. It shows where models agree and disagree, not a ranking—one run cannot be a leaderboard.
PAIR-20 is the closest published reference for testing AI on procurement work. The July 2026 run used ChatGPT, Claude, Gemini, and Copilot on six synthetic tasks—quote normalization, contract review, landed cost, supplier risk, consulting bids, and catalog deduplication—on free or standard web tiers. The runs were single dated runs, not averages. Each platform passed some tasks and missed others, and the result is a disagreement map rather than a score table.
The run surfaced useful coverage alongside specific failures. All four platforms found the contract renewal clauses; only one connected the interacting deadlines on the generic prompt. All four rebuilt the landed-cost stack under the structured brief; none reached the EUR 7,940 case answer. Fabrication appeared in two tasks—an invented contract expiry and a guessed catalog quantity. Structured briefs made outputs easier to audit but did not prevent a unit miss, a ranking contradiction, or stripped hidden costs.
One dated run is not a frequency. The July 2026 PAIR-20 run was a single pass per platform, not a repeated sample. A platform that missed a trap in this run might catch it next time, and one that caught it might miss it later. Do not read the results as stable rates of accuracy or fabrication.
Synthetic files are not production files. The tasks used fictional companies, invented figures, and controlled ground truths so each finding could be checked against a known answer. That is useful for surfacing patterns, but it does not predict how a model will handle a real supplier file with missing data, formatting quirks, and no answer key.
This is not a leaderboard. PAIR-20 does not name a best platform for procurement. In the July 2026 run, the useful signal was where platforms disagreed—unit handling, clause interaction, cost layering, blank-field discipline—not which one scored highest. For a working procurement team, the benchmark's value is the list of verification points it surfaced, not a ranking to shop by.
Where this comes from
Last checked Sun Jul 12 2026 00:00:00 GMT+0000 (Coordinated Universal Time). Evidence comes from dated, single-run platform sessions with screenshots on file — read each finding as “this happened,” not “this always happens.”
Work this yourself — from the course
Related questions
See what the platforms caught — and missed
Twenty procurement tasks, four AI platforms, real dated runs. Lesson 2 is free to read, no account needed.