Start free trial

TRUE, BUTScope Swap

The 18.5% is real.

It means 37 of 200 programs cloned by running the original and passing a finite test suite.

“Claude Opus 5.5 can now rebuild real programs from scratch: 18.5% on ProgramBench, up from 0% in May.”

Trending r/artificial post (October 2026), citing the ProgramBench leaderboard

Recorded 29 October 2026. Our ruling means: literally true, materially misleading.

Read the story →

Receipts pack

The evidence file for this verdict

9 receipts: 3 support the claim, 2 contradict it, 4 add context. 5 figures on the record, 2 traced to a quoted receipt.

The receipts

Each source, the line we quote from it, its date and which way it cuts. Free to read and to cite.

  1. 1Supports the claimvals.ai ↗

    Claude Opus 5.5 ranks first on ProgramBench with 18.50%

    Look up an archived copy ↗

  2. 2Supports the claimprogrambench.com ↗

    Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior.

    Look up an archived copy ↗

  3. 3Supports the claimarxiv.org ↗

    We evaluate 9 LMs and find that none fully resolve any task

    Look up an archived copy ↗

  4. 4Contextvals.ai ↗

    a task is counted as Fully Resolved only when the submitted implementation passes all tests.

    Look up an archived copy ↗

  5. 5Contextvals.ai ↗

    Claude Opus 5.5 87.0% 130 37

    Look up an archived copy ↗

  6. 6Contextprogrambench.com ↗

    The agent can run the given program with any input and observe exactly what it does

    Look up an archived copy ↗

  7. 7Contradicts the claimprogrambench.com ↗

    Note that behavioral tests can never cover all possible inputs.

    Look up an archived copy ↗

  8. 8Contextprogrambench.com ↗

    ranging from small terminal utilities like jq and ripgrep to massive software projects like the PHP compiler

    Look up an archived copy ↗

  9. 9Contradicts the claimarxiv.org ↗

    Models favor monolithic, single-file implementations that diverge sharply from human-written code.

    Look up an archived copy ↗

Members' extras

What each receipt proves, and what would change our mind.

  • What each receipt above proves, in our words
  • The figures table, each number traced to the receipt that states it
  • The trick behind the claim, explained, with the guide to spotting it
  • What would change our verdict
  • Our call on the record, with its due date

The verdict, the opening and every receipt above stay free. Membership adds our analysis to the evidence file for every checked story.

See Membership →Already a member? Sign in →

First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge.