24 Sep 2026Benchmarks[Author name] · @[handle]1 min read

Pip vs three models. Five benchmarks.

We ran Pip, Jev, Opus 5.5 and GPT 6.0 Astra on the same five jobs and measured accuracy, wall-clock time and cost.

Faster than the next fastest
[X]×
Benchmarks where Pip is most accurate
[X] / 5
Cheaper than Jev, the next cheapest
[X]×

Speed vs accuracy

[One sentence on the headline. Averaged across all five benchmarks. Up and to the left is better.]

Fig. 1 Speed vs accuracy

Not plotted yet: a point is drawn once its accuracy and speed are measured.

Pip
Average [X]%
Jev
Average [X]%
GPT 6.0 Astra
Average [X]%
Opus 5.5
Average [X]%

The benchmarks

Fig. 2 The benchmarks
Invoice workflow
Our TypeSafe-format invoice job: extract fields, check the PO match, abstain on missing values. [N] documents.
MuSiQue
Multi-hop questions where each answer depends on facts found in earlier steps.
CORD
Receipt understanding: pull line items, totals and taxes from scanned receipts.
DocILE
Key information and line-item extraction from business documents.
Code review
Our own job: two implementations of the same function, and four typed questions about them (which is cleaner, whether they return the same result, which expression changes it, how risky the swap is). Scored against [three] reviewers’ labels. [N] pairs.

Code review: a worked example

Which is cleaner?

One pair from the set: two implementations of the same refund function, and four questions about them, sent as one request. Pip answers the question you asked (B is cleaner) and the one you didn’t: B is not the same function. Every answer points at the characters it rests on, in either implementation. Hover an answer to see its spans; press it to keep them.

A The original
function refundCents(order) {
  let total = 0;
  for (let i = 0; i < order.lines.length; i++) {
    const line = order.lines[i];
    if (line.status !== 'damaged') {
      continue;
    }
    let qty = line.qty;
    if (qty === undefined || qty === null) {
      qty = 1;
    }
    total = total + line.cents * qty;
  }
  return total;
}
B The refactor
const refundCents = (order) =>
  order.lines
    .filter((line) => line.status === 'damaged')
    .reduce(
      (sum, line) => sum + line.cents * (line.qty || 1),
      0,
    );

One request, 4 questions. Every answer points into A and B.

same_total rests on two spans, one in each implementation: A’s null check matters only because of B’s ||, and B’s || matters only because of A’s null check. Neither is evidence on its own. swap_risk is asked given the answers above, so it reads the other three in the same pass. The difference is one value: a damaged line with qty: 0 (logged, then replaced rather than refunded) is refunded once by B and not at all by A. The fix is two characters, ??. Pip doesn’t write it; it points at where it goes.

The request and the response

Request: POST /v1/decide

{
  "model": "pip-latest",
  "state": "// A\nfunction refundCents(order) {\n  let total = 0;\n  for (let i = 0; i < order.lines.length; i++) {\n    const line = order.lines[i];\n    if (line.status !== 'damaged') {\n      continue;\n    }\n    let qty = line.qty;\n    if (qty === undefined || qty === null) {\n      qty = 1;\n    }\n    total = total + line.cents * qty;\n  }\n  return total;\n}\n\n// B\nconst refundCents = (order) =>\n  order.lines\n    .filter((line) => line.status === 'damaged')\n    .reduce(\n      (sum, line) => sum + line.cents * (line.qty || 1),\n      0,\n    );",
  "questions": {
    "cleaner": {
      "type": "choice",
      "instructions": "Which implementation is cleaner?",
      "criteria": { "a": "A, the original", "b": "B, the refactor" }
    },
    "same_total": {
      "type": "binary",
      "instructions": "Do A and B return the same total for every order?",
      "criteria": { "true": "The same total for every order", "false": "Some order gets a different total" }
    },
    "culprit": {
      "type": "extract",
      "instructions": "Copy the expression in B that changes the result.",
      "criteria": "An expression, copied as written"
    },
    "swap_risk": {
      "type": "score",
      "instructions": "Given the answers above, how risky is replacing A with B as written?",
      "criteria": ["Safe", "Needs a test", "Changes money"]
    }
  }
}

Response: 200

{
  "answers": {
    "cleaner": {
      "type": "choice",
      "choice": "b",
      "probabilities": { "a": 0.09, "b": 0.91 },
      "confidence": 0.88,
      "evidence": [
        { "text": "for (let i = 0; i < order.lines.length; i++) {", "start": 54, "end": 100 },
        { "text": "total = total + line.cents * qty;", "start": 287, "end": 320 },
        { "text": ".filter((line) => line.status === 'damaged')", "start": 398, "end": 442 },
        { "text": "(sum, line) => sum + line.cents * (line.qty || 1)", "start": 462, "end": 511 }
      ],
      "abstained": false
    },
    "same_total": {
      "type": "binary",
      "probability": 0.04,
      "confidence": 0.93,
      "evidence": [
        { "text": "if (qty === undefined || qty === null) {", "start": 221, "end": 261 },
        { "text": "line.qty || 1", "start": 497, "end": 510 }
      ],
      "abstained": false
    },
    "culprit": {
      "type": "extract",
      "value": "line.qty || 1",
      "confidence": 0.9,
      "evidence": [{ "text": "line.qty || 1", "start": 497, "end": 510 }],
      "abstained": false
    },
    "swap_risk": {
      "type": "score",
      "score": 1.84,
      "legend": { "0": "Safe", "1": "Needs a test", "2": "Changes money" },
      "probabilities": { "0": 0.02, "1": 0.12, "2": 0.86 },
      "confidence": 0.82,
      "evidence": [
        { "text": "total = total + line.cents * qty;", "start": 287, "end": 320 },
        { "text": "sum + line.cents * (line.qty || 1)", "start": 477, "end": 511 }
      ],
      "abstained": false
    }
  }
}

Illustrative response: the offsets are exact, computed from the code above; the probabilities and confidences are placeholders until the set is run.

Accuracy

[One sentence per notable result. Where Pip wins, where it doesn’t.]

Fig. 3 Accuracy
Model Invoices MuSiQue CORD DocILE Code review Average
Pip [X]%[X]%[X]%[X]%[X]%[X]%
Jev [X]%[X]%[X]%[X]%[X]%[X]%
GPT 6.0 Astra [X]%[X]%[X]%[X]%[X]%[X]%
Opus 5.5 [X]%[X]%[X]%[X]%[X]%[X]%

Speed

Wall-clock time to finish 1,000 items of the invoice workflow. Pip: the whole job in one request. Jev: questions sent in groups of [100] per request, the most its API accepts [confirm]. LLMs: one question at a time, in sequence.

Fig. 4 Speed
Pip
[X] s
Jev
[X] s
GPT 6.0 Astra
[X] min
Opus 5.5
[X] h

Cost

Total spend for the full run across all five benchmarks, at each provider’s published rates on [date].

Fig. 5 Cost
Pip
[$X]
Jev
[$X]
GPT 6.0 Astra
[$X]
Opus 5.5
[$X]

How we ran it

[Model versions and dates. Pip: one request per job. Jev: questions in groups of [100] per request. LLMs: one call per question, run in sequence. Settings: temperature, max tokens, how answers were parsed. What counted as correct.]

Limits

[Where this comparison is weak. Rate limits vary by account tier. LLMs could be prompted differently. Name any benchmark where another model beat Pip.]

Raw results on GitHub Soon Try Pip

Everything in [brackets] is a placeholder.

esc
↑↓ move↵ open⌘K