24 Sep 2026Benchmarks[Author name] · @[handle]1 min read
Pip vs three models. Five benchmarks.
We ran Pip, Jev, Opus 5.5 and GPT 6.0 Astra on the same five jobs and measured accuracy, wall-clock time and cost.
- Faster than the next fastest
- [X]×
- Benchmarks where Pip is most accurate
- [X] / 5
- Cheaper than Jev, the next cheapest
- [X]×
Speed vs accuracy
[One sentence on the headline. Averaged across all five benchmarks. Up and to the left is better.]
Not plotted yet: a point is drawn once its accuracy and speed are measured.
- Pip
- Average [X]%
- Jev
- Average [X]%
- GPT 6.0 Astra
- Average [X]%
- Opus 5.5
- Average [X]%
The benchmarks
- Invoice workflow
- Our TypeSafe-format invoice job: extract fields, check the PO match, abstain on missing values. [N] documents.
- MuSiQue
- Multi-hop questions where each answer depends on facts found in earlier steps.
- CORD
- Receipt understanding: pull line items, totals and taxes from scanned receipts.
- DocILE
- Key information and line-item extraction from business documents.
- Code review
- Our own job: two implementations of the same function, and four typed questions about them (which is cleaner, whether they return the same result, which expression changes it, how risky the swap is). Scored against [three] reviewers’ labels. [N] pairs.
Code review: a worked example
Which is cleaner?
One pair from the set: two implementations of the same refund function, and four questions about them, sent as one request. Pip answers the question you asked (B is cleaner) and the one you didn’t: B is not the same function. Every answer points at the characters it rests on, in either implementation. Hover an answer to see its spans; press it to keep them.
function refundCents(order) {
let total = 0;
for (let i = 0; i < order.lines.length; i++) {
const line = order.lines[i];
if (line.status !== 'damaged') {
continue;
}
let qty = line.qty;
if (qty === undefined || qty === null) {
qty = 1;
}
total = total + line.cents * qty;
}
return total;
} const refundCents = (order) =>
order.lines
.filter((line) => line.status === 'damaged')
.reduce(
(sum, line) => sum + line.cents * (line.qty || 1),
0,
); One request, 4 questions. Every answer points into A and B.
same_total rests on two spans, one in each implementation: A’s null check matters only because of B’s ||, and B’s || matters only because of A’s null check. Neither is evidence on its own. swap_risk is asked given the answers above, so it reads the other three in the same pass. The difference is one value: a damaged line with qty: 0 (logged, then replaced rather than refunded) is refunded once by B and not at all by A. The fix is two characters, ??. Pip doesn’t write it; it points at where it goes.
The request and the response
Request: POST /v1/decide
{
"model": "pip-latest",
"state": "// A\nfunction refundCents(order) {\n let total = 0;\n for (let i = 0; i < order.lines.length; i++) {\n const line = order.lines[i];\n if (line.status !== 'damaged') {\n continue;\n }\n let qty = line.qty;\n if (qty === undefined || qty === null) {\n qty = 1;\n }\n total = total + line.cents * qty;\n }\n return total;\n}\n\n// B\nconst refundCents = (order) =>\n order.lines\n .filter((line) => line.status === 'damaged')\n .reduce(\n (sum, line) => sum + line.cents * (line.qty || 1),\n 0,\n );",
"questions": {
"cleaner": {
"type": "choice",
"instructions": "Which implementation is cleaner?",
"criteria": { "a": "A, the original", "b": "B, the refactor" }
},
"same_total": {
"type": "binary",
"instructions": "Do A and B return the same total for every order?",
"criteria": { "true": "The same total for every order", "false": "Some order gets a different total" }
},
"culprit": {
"type": "extract",
"instructions": "Copy the expression in B that changes the result.",
"criteria": "An expression, copied as written"
},
"swap_risk": {
"type": "score",
"instructions": "Given the answers above, how risky is replacing A with B as written?",
"criteria": ["Safe", "Needs a test", "Changes money"]
}
}
} Response: 200
{
"answers": {
"cleaner": {
"type": "choice",
"choice": "b",
"probabilities": { "a": 0.09, "b": 0.91 },
"confidence": 0.88,
"evidence": [
{ "text": "for (let i = 0; i < order.lines.length; i++) {", "start": 54, "end": 100 },
{ "text": "total = total + line.cents * qty;", "start": 287, "end": 320 },
{ "text": ".filter((line) => line.status === 'damaged')", "start": 398, "end": 442 },
{ "text": "(sum, line) => sum + line.cents * (line.qty || 1)", "start": 462, "end": 511 }
],
"abstained": false
},
"same_total": {
"type": "binary",
"probability": 0.04,
"confidence": 0.93,
"evidence": [
{ "text": "if (qty === undefined || qty === null) {", "start": 221, "end": 261 },
{ "text": "line.qty || 1", "start": 497, "end": 510 }
],
"abstained": false
},
"culprit": {
"type": "extract",
"value": "line.qty || 1",
"confidence": 0.9,
"evidence": [{ "text": "line.qty || 1", "start": 497, "end": 510 }],
"abstained": false
},
"swap_risk": {
"type": "score",
"score": 1.84,
"legend": { "0": "Safe", "1": "Needs a test", "2": "Changes money" },
"probabilities": { "0": 0.02, "1": 0.12, "2": 0.86 },
"confidence": 0.82,
"evidence": [
{ "text": "total = total + line.cents * qty;", "start": 287, "end": 320 },
{ "text": "sum + line.cents * (line.qty || 1)", "start": 477, "end": 511 }
],
"abstained": false
}
}
} Illustrative response: the offsets are exact, computed from the code above; the probabilities and confidences are placeholders until the set is run.
Accuracy
[One sentence per notable result. Where Pip wins, where it doesn’t.]
| Model | Invoices | MuSiQue | CORD | DocILE | Code review | Average |
|---|---|---|---|---|---|---|
| Pip | [X]% | [X]% | [X]% | [X]% | [X]% | [X]% |
| Jev | [X]% | [X]% | [X]% | [X]% | [X]% | [X]% |
| GPT 6.0 Astra | [X]% | [X]% | [X]% | [X]% | [X]% | [X]% |
| Opus 5.5 | [X]% | [X]% | [X]% | [X]% | [X]% | [X]% |
Speed
Wall-clock time to finish 1,000 items of the invoice workflow. Pip: the whole job in one request. Jev: questions sent in groups of [100] per request, the most its API accepts [confirm]. LLMs: one question at a time, in sequence.
- Pip
- [X] s
- Jev
- [X] s
- GPT 6.0 Astra
- [X] min
- Opus 5.5
- [X] h
Cost
Total spend for the full run across all five benchmarks, at each provider’s published rates on [date].
- Pip
- [$X]
- Jev
- [$X]
- GPT 6.0 Astra
- [$X]
- Opus 5.5
- [$X]
How we ran it
[Model versions and dates. Pip: one request per job. Jev: questions in groups of [100] per request. LLMs: one call per question, run in sequence. Settings: temperature, max tokens, how answers were parsed. What counted as correct.]
Limits
[Where this comparison is weak. Rate limits vary by account tier. LLMs could be prompted differently. Name any benchmark where another model beat Pip.]
Everything in [brackets] is a placeholder.