Skip to content

BenchRig

Cross-platform benchmarking and hardware profiling for local LLMs on Ollama, Microsoft Foundry Local, direct ONNX Runtime GenAI and Prism. Tailored for macOS Apple Silicon (Metal, Unified Memory) and Linux / WSL2 (NVIDIA CUDA).

Alpha software (v0.1.0)

It works and is tested, but flags, file formats and scoring may still change. See the changelog.

Try it

pipx install benchrig                 # or: pip install benchrig
benchrig --check                      # runtimes, accelerator, installed models
benchrig --models qwen2.5-coder:7b --suite coding

Reports land in results/ (LATEST_SUMMARY.md, latest.json, and one JSON file per run in results/runs/).

What you get

  • Practical workloads, not perplexity. Coding tasks are executed in a sandbox against unit-test assertions; reasoning answers are checked against ground truth (including <think> output). See Benchmark suites.
  • Several runtimes, one scorecard. Compare llama.cpp and ONNX Runtime GenAI on identical hardware, with a fair 1:1 parameter set. See Runtimes and Cross-engine benchmarking.
  • True time to first token. Measured from the stream, next to decode and prefill throughput.
  • Hardware telemetry. VRAM or unified memory, GPU utilisation, power and temperature while a model runs. See Hardware telemetry.
  • Your scenarios. Bundled suites are plain JSON and can be overridden with ./scenarios or --scenarios-dir. See Custom scenarios.

Where to go next

I want to... Read
Run my first benchmark Quickstart
Choose or configure a runtime Runtimes, Configuration
Look up a flag or a recipe CLI reference
Use an NVIDIA GPU with Foundry Local on WSL2 Foundry GPU setup, CUDA & TensorRT guide
Write my own scenarios Scenario reference, Custom scenarios
Understand or extend the code Architecture, Development
  • prism-local: local server and CLI for ONNX Runtime GenAI and Ollama; benchmarked here as the prism runtime.
  • local-coders: agent skills and MCP servers that offload coding work to local models.