Skip to content

🧪 Benchmark Suites & Evaluation Methodology

This document details the design, scenarios, scoring formulas, and validation methodologies employed by the 5 specialized benchmark suites in BenchRig.

1. ⚡ Speed & Latency Suite (speed)

The speed suite evaluates inference latency, maximum decoder velocity, and prompt ingestion throughput under streaming conditions.

Scenarios

  1. Raw Decode Speed (Short Prompt):
  2. Measures maximum decoding velocity when the context window is near-empty.
  3. Evaluates pure token sampling speed unconstrained by large KV-cache buffers.
  4. Throughput Generation (512 tokens):
  5. Tasks the model with generating an extended completion to measure sustained throughput over time.
  6. Reveals thermal throttling, memory bandwidth saturation, or engine queue degradation.

Metrics Measured

  • Time-to-First-Token (TTFT): High-resolution timestamp delta from HTTP socket dispatch to the arrival of the first completion byte.
  • Prefill Speed (Prompt Tokens/s): Hardware throughput while ingesting and calculating initial attention keys/values for the prompt.
  • Decode Speed (Completion Tokens/s): Sequential auto-regressive generation speed of output tokens. A ~ prefix in the report (~3000.0 t/s) marks a value produced by the 0.001 s measurement floor on a very short answer, not a real measurement; the CSV column eval_tok_sec_floored carries the same flag.

2. 💻 Coding & Algorithmic Sandbox Suite (coding)

The coding suite measures whether an LLM produces syntactically valid and deterministically functional Python code without human intervention. Today the coding suite only exercises Python; JS/TS and other multi-language scenario packs are not bundled.

Isolated Subprocess Sandbox (benchrig/core/sandbox.py)

Rather than relying on lexical matching (e.g. BLEU/ROUGE), BenchRig executes the model's generated code inside an isolated Python subprocess: 1. Strips conversational prose, backticks (```python), and trailing explanations. 2. Appends an automated test harness with strict assert statements covering edge cases. 3. Spawns an isolated subprocess.Popen in its own process group with a strict timeout. 4. Validates assertion outcomes via out-of-band JSON token verification, ignoring candidate stdout/stderr for verdict decisions.

Subprocess Isolation & Residual Risk

BenchRig provides process-group isolation and strict timeouts, not OS-level sandboxing (such as Docker, seccomp, or WebAssembly). While out-of-band JSON tokens prevent candidate output from forging verdicts or test counts, the generated code runs under the current Python interpreter and could attempt system or network calls (subprocess, socket, ctypes).

Reasoning traces and cut-off answers. A model that thinks first (<think>...</think>, e.g. DeepSeek-R1) can spend its whole token budget before writing any code. Code inside the <think> trace is ignored, and a response that stopped at the token limit without a complete code block is reported as Response truncated at num_predict=N before the code was complete (with truncated: true in the result) instead of a misleading name '...' is not defined. To give such models room, the budget (num_predict) of the coding, reasoning and polish suites is multiplied for models matching benchmark.thinking_models (default deepseek-r1, qwq, magistral, qwen3) by benchmark.thinking_token_multiplier (default 12); see Configuration. Scorecards count cut-off responses as truncated_runs and the reports flag them with ⚠. Only the suites that score an answer (coding, reasoning, polish) are checked: the speed and context suites generate a fixed number of tokens on purpose, so stopping at the limit there is not truncation (finish_reason is still recorded).

Thinking models and the think option. Ollama streams a thinking model's reasoning in a separate thinking field. BenchRig reads it: ttft_sec is the time to the first token of any kind (the latency), answer_ttft_sec the time to the first answer token, think_time_sec the difference, and thinking_chars the amount of thinking. (Earlier versions reported only the first answer token, so a thinking model showed seconds of "TTFT", or 0.0 s when it never answered.)

Whether a model thinks is set with Ollama's think option, per suite, in benchmark.think (see Configuration): by default speed and context run with thinking off so throughput and prompt-processing latency are not mixed with thinking time, and coding, reasoning and polish use the model's own default. Turning thinking off for coding measures the model without reasoning (much faster), but the larger token budget still applies to these models: in one run deepseek-r1:14b with thinking off and the base budget (512 tokens) never produced a complete code block (0 of 4 tests), while with its default thinking it passed 4 of 4 on the same scenario (4541 tokens, 14752 characters of thinking, 73 s to the first answer token, but a ttft_sec of 0.11 s). The option is sent only to models that Ollama lists as able to think; other runtimes ignore it.

How much budget do thinking models need? Measured on deepseek-r1:14b, coding suite, one run each: a multiplier of 6 cut off 1 to 2 of the 4 scenarios, 12 cut off 1, and 20 cut off none; the scenario that ran out at 12 (6144 tokens) finished in 4609 tokens at 20. The need is a long tail, hence the default of 12. Checked with --runs 3 at 12: 12 requests, none cut off, at most 4822 tokens (of a 6144 budget), coding 98.0% [94.1-100.0] and, on the biggest scenario, 3 of 4 tests in one repetition. Check truncated_runs for your models.

Algorithmic Scenarios

Scenario Algorithmic Focus Assertions Tested
Nested Dict Flattening Deep recursion, key path concatenation, delimiter handling 4
Merge Overlapping Intervals O(N log N) sorting, greedy interval consolidation, edge intervals 5
Valid Balanced Brackets Stack data structure, pair mapping, empty and malformed string handling 6
LRU Cache Implementation Doubly-linked hash map / OrderedDict, O(1) get/put operations 2

Disaggregated Task vs. Assertion Metrics

To avoid obscuring partial successes or multi-assertion scenarios where a model fails only edge cases, BenchRig disaggregates coding metrics into two distinct levels:

  • Task-Level Success (coding_tasks_passed / coding_task_count, coding_task_pass_rate): A coding scenario is marked passed only when 100% of its unit test assertions pass. The task pass rate reflects the percentage of programming problems fully solved without errors.
  • Assertion-Level Success (coding_assertions_passed / coding_assertion_count, coding_pass_rate): Counts individual unit test assertions passed across all scenarios. coding_pass_rate preserves backward compatibility and contributes to the composite score.
  • Per-Task Run Evidence: Each result record retains task_id, passed (task boolean), passed_tests, total_tests, pass_ratio, and any sandbox_error diagnostics.
  • Reporting & Exports: The Markdown report displays task pass counts (Tasks Passed: X/Y (Z%)) alongside assertion rates (Assertions: A/B (C%)), and the CSV export includes columns for both task and assertion counts and rates (coding_task_pass_rate, coding_tasks_passed, coding_task_count, coding_assertions_passed, coding_assertion_count).

3. 🧠 Reasoning & Thought Chain Suite (reasoning)

Designed to test models that generate reasoning chains (such as DeepSeek-R1 or Qwen 2.5 with reasoning prompts).

Methodology

  • <think> Parsing: The parser in benchrig/core/reasoning_parser.py inspects whether the model separates internal reasoning (<think>...</think>) from the final response.
  • Token Budget Telemetry: Tracks the proportion of tokens spent on internal exploration vs the final delivered answer.
  • Deterministic Verification: Extracts the final answer and checks exact equality against ground-truth mathematical and logical solutions.

What counts as an answer

  • The answer is what follows </think>. A response cut off inside <think> has no answer and is scored as wrong (truncated_thinking: true); text that only appears in the reasoning trace never counts, even if it contains the expected words.
  • regex checks look only at the end of the answer (the last 300 characters), after markdown, bullets, punctuation and line breaks are collapsed to spaces. A conclusion written as Box 1: Oranges, Box 2: ... on one line, on separate lines or as bullets reads the same, while words from the right answer that merely appear earlier in the reasoning do not count. The bundled patterns use bounded gaps (a few filler words) instead of .*, so they cannot be satisfied by scattered mentions.
  • LaTeX in answers is simplified before matching (\boxed{\dfrac{1}{6}} reads as 1/6), because reasoning models often answer that way.
  • Every result records answer_excerpt (the last 400 characters of the answer) so a score can be audited from results/runs/*.json without re-running the model.
  • final_answer scenarios (nine of the twelve) take the last Final answer: ... line and require it to match an accepted pattern in full; no marker means no answer. exact_or_contains matches numbers as whole numbers, so 11/60 no longer counts as 1/6 and 1114.60 no longer counts as 114.6 (both used to pass).
  • The expected answers of the new scenarios are computed by independent code in tests/test_scenarios.py, which also rejects near misses, so a wrong expectation cannot rank models wrongly.

Reasoning ground-truth comparison

The evaluator field on a reasoning scenario (added in spec.md §12.3) is an opt-in layer that runs before the scenario's check_type. Without it, the scenario behaves exactly as it always has.

evaluator Effect
(absent) Today's behavior. Whitespace and word forms are not normalized.
"numeric" Trim whitespace on both sides; if both parse as a Fraction, compare exactly. "42", "42.0", and " 42 " all match "42"; word answers like "forty-two" do not silently match a numeric "42".
"numeric_text" Same as "numeric", plus an English text-to-number conversion on both sides: zero, one, ..., nineteen, twenty, thirty, ..., ninety, hundred, thousand, million, and hyphenated forms like "forty-two", "twenty-one". Conversion is case-sensitive: "Forty-Two" does NOT match "forty-two".

When only one side parses as a number (after normalization), the numeric layer is a no-op and the underlying check_type decides. This is the invariant that keeps word-puzzle answers (expected_answer is itself a word) from being silently accepted as numeric.

Unknown values for the field are treated as None, with an evaluator.unknown structured-log event emitted on stderr so a typo does not silently disable comparison.

Scenarios

Scenario Check Expected
Multi-Step Discount & Tax Calculation contains number 114.6
Three Mislabeled Boxes Logic Puzzle regex on the conclusion Box 1: Oranges, Box 2: Apples and Oranges, Box 3: Apples
Set Theory & Probability contains number 1/6
Two Trains Meeting Time Final answer 11:20
Knights and Knaves (3 people) Final answer A is a knave, B and C are knights
Distinct-Digit Multiples of Five (4-digit numbers) Final answer 952
Linear Recurrence (a10 of a1=3, a(n+1)=2a(n)-1) Final answer 1025
Chained Percentage Changes (80, +25%, -20%, +10%) Final answer 88
Letter Counting (r in "strawberry raspberry blueberry") Final answer 8
Bat and Ball ($1.10 total, bat $1.00 more; cents for the ball) Final answer 5
Multiples of Three or Five (1 to 100) Final answer 47
Letters in One Word (r in "strawberry") Final answer 3

In one run of phi4-mini, qwen2.5-coder:7b and gemma3:12b the new scenarios split the models (Letter Counting 0 of 3, Distinct-Digit Multiples 2 of 3) where the old three mostly passed for everyone. Three easier ones (bat and ball, multiples of three or five, letters in one word) were added so that weaker models also pass some: phi4-mini on Ollama passed them 3 of 3 times each, while the hard ones (Letter Counting, Distinct-Digit Multiples) stayed at 0 of 3. Scores are still coarse with twelve scenarios; add your own for a finer ranking (Custom scenarios).

4. 📐 Context Saturation Suite (context)

Assesses how inference performance degrades as context length increases.

Methodology

  • Runs five steps, labelled 512, 1024, 2048, 4096 and 8192 (context_scaling.json). Each step sets the request's num_ctx to the label and sends a prompt of about fill_ratio (0.75) of that window, built from a repeated filler paragraph.
  • A fact is hidden in the text (needle, placed at needle_depth, by default halfway, at a sentence boundary) and the question asks for it (instruction, answer expected in expected). A step passes when the answer contains the expected number as a whole number. Each step hides a different code, so it cannot be memorised. The suite therefore measures whether the model uses its context, not only how fast it reads it; the share of steps passed is the scorecard's context_retrieval_pct.
  • On runtimes that reload the model when num_ctx changes (Ollama), an untimed request with the same num_ctx runs first, so the timed request measures prompt processing and not model load. The load time is recorded as load_time_sec.
  • Samples Time-to-First-Token (TTFT) and memory growth across each step.
  • Identifies the threshold where the model's KV-cache exceeds physical GPU VRAM and triggers spilling into system RAM.

Reading the context numbers

  • The label is the context window, not the prompt length. Prompt size is estimated from characters (about 4 per token), and real tokenizers differ, so the real token count is recorded as prompt_tokens_actual. For Phi-4-mini the five steps sent 289, 565, 1107, 2199 and 4374 tokens (about 55% of each window). Use fill_ratio in the scenario file to change the fill.
  • TTFT now grows with the prompt. Before the warm-up was added, the Ollama numbers here were dominated by model reload (about 4 s per step measured directly, about 7 s in the suite), because every step changed num_ctx. Results from earlier versions of this suite are not comparable.
  • Where a runtime cannot fit the window, the step is recorded as a failure with the engine's error; that is a finding, not a harness fault.
  • The scorecard's avg_ttft_sec still averages every suite, including the large-context steps, so compare TTFT per suite (results/runs/*.json) when the prompts differ in size.

5. 🇵🇱 Polish NLP Linguistic Suite (polish)

Evaluates linguistic precision in morphologically rich and complex languages (Polish).

Scenarios

  • Grammatical Declension: Testing correct nominal and adjectival case inflections across all 7 Polish cases (przypadki).
  • Idiomatic Nuance: Translating and interpreting idioms and culturally specific expressions.
  • Text Summarization: Preserving factual precision under strict word-count limits.

Tool-use suite (opt-in)

benchrig --suite tool_use --runs 3 measures native function-call reliability, not execution or retrieval quality. The 18 bundled cases cover tool selection, integer/enum/array arguments, correct no-call answers, two-step fixture calls, Polish prompts and retrieval-shaped requests. Function names are inert: BenchRig never reads the requested file, searches a real index, or runs a model-selected command.

A task passes when every planned turn succeeds. A call requires exactly the expected tool, schema-valid arguments, exact JSON types and no extra arguments. Object key order is ignored; array order and string contents matter. Plain-text call JSON and malformed/extra calls fail. A no-call response must also match its expected answer, so a refusal does not count as correct abstention. The first wrong turn stops the conversation; later planned turns remain failed evidence.

Reports show task success, tool selection accuracy, argument accuracy, abstention accuracy, false-call rate and structured-call rate, with numerators/denominators. Selection/argument/structured-call denominators include all planned call turns; abstention/false-call denominators include all planned no-call turns. Errors and skipped turns are counted alongside those rates. Unsupported cases are excluded from eligible denominators and counted separately. Warm-ups are excluded. A zero denominator is n/a, not a 0% score. Results apply to a model/runtime pair.

Initial transport is non-streaming: request latency is wall time, and TTFT, prefill/decode timing or throughput are unavailable unless reported by the server. Tool metrics do not feed legacy composite, timing or memory aggregates. Tool-only runs expose those legacy fields as unavailable; raw JSON preserves turn evidence, effective sampling settings and reported execution placement.