New post All of MMLU in one request

One call. Whole job.

Pip reads the input and every question together, and returns every decision with its evidence in a single pass. Built on Phronic’s Rook architecture, trained with JCT.

Ask Pip

Customer message

Hi, I'm Dana. I ordered an oak bookshelf on March 3 and a walnut desk on March 20, and both were delivered together on March 28. The bookshelf is perfect. The desk arrived with its left leg snapped, so I haven't used it. Your site says returns are free within 30 days of delivery, but my brother insists it's 30 days from the order date. I'm away from April 10 to April 30 and can't post anything while I'm gone. Can I still send the desk back for free?

Pip decides · highlights 2

Oak bookshelf Walnut desk Both
Every answer is one of the options given; every highlight is already in the message. An illustration, not a recorded run.

One pass: 1,000 decisions, read together, answered in one response

  1. 01 · The job

    1,000 decisions.

    100 support tickets, 10 questions each. Every point of light is one decision.

  2. 02 · The pass

    Read once, all together.

    Pip takes every ticket and every question into a single pass.

  3. 03 · The answer

    1,000 answers, one response.

    Ten orbits, one per question. Each answer comes back with the words it rests on.

Same job, three ways.

The job: 100 support tickets with 10 questions each, so 1,000 answers. Watch how many calls each model needs to give them.

Elapsed

0.0 s

Ready when you are.

Pip 0 / 1 call
Jev 0 / 100 calls
GPT 6.0 Astra 0 / 100 calls
  1. Pip

    1 call with every ticket and every question in it

    0.0 s

    0 / 1 calls · waiting

  2. Jev

    100 calls: one ticket per call, read once, then one branch per question

    How Jev works here is our assumption, from its public behaviour. The times will be real: we are testing them and will update this.

    0.0 s

    0 / 100 calls · waiting

  3. GPT 6.0 Astra

    100 calls: one ticket per call, every question answered in a long reasoning reply

    0.0 s

    0 / 100 calls · waiting

Each dot is one call to the model, and a filled dot is a finished call. The dot counts are exact; the times are placeholders until we measure them.

Benchmarks

being measured
  • Smarter[Reasoning benchmark]
  • Faster[Workload]
  • Cheaper[MMLU]

Scores land here once they are real.

Every answer alone. Every answer together.

Jev1 prefill · 6 branches
Pip1 pass

Jev. The ticket is read once, then each question is answered on its own. No answer can see another.

Pip. The ticket and every question go through together. Each answer is informed by the others.

Our reading of Jev, from public behaviour.

It reasons. Every answer shows its evidence.

Give Pip the ticket, the customer's photo, the order's state, the policy and every question, with its options, at once. Each answer is one of those options or a span of the input, and points at what it rests on.

Worked example

Placeholder example, not a recorded run

Ticket

“My blender arrived smashed. Can I get my money back?”

Order state

Order 4471 record

Status
Delivered
Carrier
Courier B
Total
$240.00
Condition reported
Damaged, photo attached
Refunds issued
None

Policy

  1. P1 Refunds over $200 need a manager's sign-off.
  2. P2 Orders shipped with Courier B are insured.
  3. P3 Insured orders that arrive damaged are refunded in full.
  4. P4 A damage refund needs a photo that shows the item itself is damaged.

Asked in one pass

6 answers · 1 pass · [X] ms

  • Does the image show undeniable damage to the object?

    Yes No

    rests on PhotoP4

  • Does the image show damage to the packaging?

    Yes No

    rests on Photo

  • Is the order insured?

    Yes No

    rests on CarrierP2

  • Refund amount if approved

    $240.00

    span from the order · rests on TotalRefunds issuedP3

  • Does it need a manager?

    Yes No

    rests on TotalP1

  • Next step for the agent

    Refund Ask for item photo Decline

    rests on PhotoCondition reportedP4

Hand it the whole job.

Anything you can ask as questions with options. A few of the jobs we are building it for.

Build a request. Ask anything at once.

Start from a preset or write your own state, then configure questions of the three decision types: choice, score and binary. The request builds as you type.

Presets
Questions · 3

Levels, lowest first. The answer is a score between them.

012

Press Run.

Every answer comes back in one pass, each pointing at the words it rests on.

This is a sketch of the real thing. The full playground, with live runs on your own input and your API key, lives in the Phronic Console.

Console · opening soon

From the blog

All of MMLU in one request.

All 14,042 questions, about 1.5M tokens, sent as one API request. How we ran it, how long it took, and how it compares with other leading models.

Read the post
Questions
14,042
Request
1
Wall-clock
[X] s
Accuracy
[X]%

Stump Pip.

Send a job you think it gets wrong. We run it and publish every failure, credited to you.

Every failure we have published

Opening soon

Submissions are not open yet. Until they are, every failure we have published is in the ledger.

Start building.

Create an account to get an API key and use Pip in the console.

Opening soon

Accounts are not open yet.

esc
↑↓ move↵ open⌘K