Why do this
[Two or three sentences. The usual way to run a benchmark is one call per question. Pip reads the input and every question together, so the whole test set can go in as one request. This post shows what that looks like at the scale of MMLU.]
Setup
[One paragraph on what “one request” means here: a single HTTP call to the serverless API, with every question read together in one pass.]
The request
The state says what the test is. Each MMLU item becomes one choice question, with its four answers as the options.
POST https://api.phronic.ai/v1/decide
{
"model": "pip-latest",
"state": "This is a test. MMLU is a benchmark of multiple-choice questions across 57 subjects, from maths and law to medicine and history. Answer each question with the correct option.",
"questions": {
"q00000": {
"type": "choice",
"instructions": "What is the chemical symbol for sodium?",
"criteria": { "A": "S", "B": "Na", "C": "So", "D": "Sd" }
},
"q00001": {
"type": "choice",
"instructions": "[Question text]",
"criteria": { "A": "[…]", "B": "[…]", "C": "[…]", "D": "[…]" }
},
... 14,040 more
}
}Results
[One sentence with the headline: time, accuracy, and total tokens.]
Comparison
[Same questions, each model through its public API. Say exactly how each other model was run: batched, how many concurrent requests, which settings.]
| Model | Requests | Time | Accuracy |
|---|---|---|---|
| Pip | 1 | [X] s | [X]% |
| Jev | [N] | [Y] s | [X]% |
| Opus 5.5 | 14,042 | [Z] | [X]% |
| GPT 6.0 Astra | 14,042 | [Z] | [X]% |
Limits
[What this does and doesn’t show. MMLU questions are independent of each other, so this measures throughput and knowledge, not dependent decisions. Note any known contamination or answer-format issues.]
Reproduce it
The script, the exact request and the raw answers are on GitHub.