24 Sep 2026Benchmarks[Author name] · @[handle]1 min read
All of MMLU in one request.
All 14,042 questions, about 1.5M tokens, sent as one API request. How we ran it, how long it took, and how it compares with other leading models.
- Questions
- 14,042
- Request
- 1
- Wall-clock
- [X] s
- Accuracy
- [X]%
- Total cost
- 4.5¢
Why do this
[Two or three sentences. The usual way to run a benchmark is one call per question. Pip reads the input and every question together, so the whole test set can go in as one request. This post shows what that looks like at the scale of MMLU.]
Setup
- Dataset
- MMLU test split · 14,042 questions · 57 subjects
- Input size
- ~1.5M tokens [confirm]
- Model
- [pip-0.1]
- Cost
- $0.045 · 1.5M tokens at $30 per billion
[One paragraph on what “one request” means here: a single HTTP call to the serverless API, with every question read together in one pass.]
The request
The state says what the test is. Each MMLU item becomes one choice question, with its four answers as the options.
POST https://api.phronic.ai/v1/decide
{
"model": "pip-latest",
"state": "This is a test. MMLU is a benchmark of multiple-choice questions across 57 subjects, from maths and law to medicine and history. Answer each question with the correct option.",
"questions": {
"q00000": {
"type": "choice",
"instructions": "What is the chemical symbol for sodium?",
"criteria": { "A": "S", "B": "Na", "C": "So", "D": "Sd" }
},
"q00001": {
"type": "choice",
"instructions": "[Question text]",
"criteria": { "A": "[…]", "B": "[…]", "C": "[…]", "D": "[…]" }
},
... 14,040 more
}
}
Results
[One sentence with the headline: time, accuracy, and total tokens.]
- STEM
- [X]%
- Humanities
- [X]%
- Social sciences
- [X]%
- Other
- [X]%
Comparison
[Same questions, each model through its public API. Say exactly how each other model was run: batched, how many concurrent requests, which settings.]
| Model | Requests | Time | Accuracy |
|---|---|---|---|
| Pip | 1 | [X] s | [X]% |
| Jev | [N] | [Y] s | [X]% |
| Opus 5.5 | 14,042 | [Z] | [X]% |
| GPT 6.0 Astra | 14,042 | [Z] | [X]% |
Limits
[What this does and doesn’t show. MMLU questions are independent of each other, so this measures throughput and knowledge, not dependent decisions. Note any known contamination or answer-format issues.]
Reproduce it
The script, the exact request and the raw answers are on GitHub.
Everything in [brackets] is a placeholder.