Benchmarks 24 Sep 2026 1 min read

Pip vs three models. Four benchmarks.

We ran Pip, Jev, Opus 5.5 and GPT 6.0 Astra on the same four jobs and measured accuracy, wall-clock time and cost.

[img] [Author name] · @[handle]
[X]×
Faster than the next fastest
[X] / 4
Benchmarks where Pip is most accurate
[X]×
Cheaper than Jev, the next cheapest

Speed vs accuracy

[One sentence on the headline. Averaged across all four benchmarks. Up and to the left is better.]

Pip
Jev
GPT 6.0 Astra
Opus 5.5
Accuracy →
Faster Time per 1,000 items (log) → Slower

The benchmarks

Invoice workflow Our TypeSafe-format invoice job: extract fields, check the PO match, abstain on missing values. [N] documents.
MuSiQue Multi-hop questions where each answer depends on facts found in earlier steps.
CORD Receipt understanding: pull line items, totals and taxes from scanned receipts.
DocILE Key information and line-item extraction from business documents.

Accuracy

[One sentence per notable result. Where Pip wins, where it doesn’t.]

ModelInvoicesMuSiQueCORDDocILEAverage
Pip [X]%[X]%[X]%[X]%[X]%
Jev [X]%[X]%[X]%[X]%[X]%
GPT 6.0 Astra [X]%[X]%[X]%[X]%[X]%
Opus 5.5 [X]%[X]%[X]%[X]%[X]%

Speed

Wall-clock time to finish 1,000 items of the invoice workflow. Pip: the whole job in one request. Jev: questions sent in groups of [100] per request, the most its API accepts [confirm]. LLMs: one question at a time, in sequence.

Pip
[X] s
Jev
[X] s
GPT 6.0 Astra
[X] min
Opus 5.5
[X] h

Cost

Total spend for the full run across all four benchmarks, at each provider’s published rates on [date].

Pip [$X]
Jev [$X]
GPT 6.0 Astra [$X]
Opus 5.5 [$X]

How we ran it

[Model versions and dates. Pip: one request per job. Jev: questions in groups of [100] per request. LLMs: one call per question, run in sequence. Settings: temperature, max tokens, how answers were parsed. What counted as correct.]

Limits

[Where this comparison is weak. Rate limits vary by account tier. LLMs could be prompted differently. Name any benchmark where another model beat Pip.]

Raw results on GitHub → Try Pip
Everything in [brackets] is a placeholder.