smolperfbenchmark: Local LLM Benchmark Leaderboard on Jetson Orin Nano Super

Live leaderboard: smolperfbenchmark.vercel.app
Harness: github.com/YuvrajSingh-mist/smolperfbenchmark

If you are searching for how to run a local LLM, what the best local LLM models are on small hardware, or a real Jetson Orin Nano Super LLM benchmark, most “LLM benchmark” pages are aimed at someone else. Arena scores, HELM-style suites, and vendor peak-tok/s slides assume a datacenter GPU with unlimited power and cooling. That is useful for training research. It is the wrong ruler for hardware people already own.

smolperfbenchmark is a public, reproducible performance leaderboard for small open-weight models on consumer and edge boards — an edge AI systems bench, not a capability leaderboard. The first live device is the Jetson Orin Nano Super developer kit (8 GB). Raspberry Pi 5, OnePlus 10R, Xiaomi Pad 6, and Mac Mini M4 are on the roadmap.

This post is the short version: why the project exists, how the Jetson measurements are defined, what the first non-reasoning slice shows (including llama.cpp vs Ollama), and what you should not conclude from it. The full Jetson sweep write-ups are linked at the end.

Who this is for

Search traffic around this topic clusters into a few intents this project actually answers:

• Running an LLM on Jetson Orin Nano Super — tok/s, power, thermals, and which small models fit 8 GB.
• Ollama on Jetson (and Ollama vs llama.cpp) — same GGUF, both backends, so the gap is measurable.
• Local LLM benchmark / local LLM benchmark leaderboard — filterable by power mode, context, gen length, and backend.
• Local LLM hardware benchmarks — not a shopping guide; a conditioned performance index for boards you can buy.

It is not optimized for “best local LLM for coding” chat quality, LM Studio setup walkthroughs, or cloud LLM leaderboards.

Why local LLM numbers keep lying

Cloud-facing benches omit the constraints that dominate when you actually run a local LLM on a power-capped board:

1. Power is the budget. A board locked at 7–25 W does not care about theoretical FLOPs; it cares how many output tokens you get per joule (and how many tokens per second you keep under that cap).
2. Thermals are not optional. Sustained decode that forces throttling is a different system than a 30-second demo.
3. The runtime stack matters. The same GGUF behind llama.cpp (including llama.cpp server) and behind Ollama can differ enough that a “model ranking” without a backend is incomplete — which is why “llama.cpp vs Ollama” keeps showing up next to local LLM searches.
4. Prompt × generation shape matters. Prefill-heavy and decode-heavy cells tell different stories; a single cherry-picked length hides that.

Existing edge demos often report one tok/s number under conditions you will never re-run. What was missing was a shared protocol: fixed load generator, locked power modes, aligned power/thermal logs, and published artifacts.

That is the gap smolperfbenchmark fills.

What this project is (and is not)

What it is

A systems performance benchmark for small / tiny instruct models (roughly 135M–few-B) on real boards:

• Per-device harness folders (scripts, generators, reports) rather than one opaque mega-framework.
• A public leaderboard with filters for suite, backend, power mode, context, and generation length.
• Headline efficiency metric: output tokens per joule (tok/J), alongside tok/s, TTFT, ITL, power, and junction temperature.

What it is not

• Not a capability / quality bench (no MMLU, no chat Elo).
• Not a claim that Jetson rankings transfer to phones or Macs.
• Not a vendor scoreboard. Numbers are generated, not hand-authored; caveats ship with the tables.

Contributions, stated plainly:

1. A reproducible on-device sweep protocol using pinned aiperf 0.11.0 (44addf0), concurrency 1, fixed prompt×gen cells.
2. A public leaderboard that ranks configs by metrics that matter under power caps.
3. Open artifacts and device READMEs so others can re-run or extend the board list.

Measurement protocol (Jetson Orin Nano Super LLM benchmark, live today)

Hardware. NVIDIA Jetson Orin Nano Super developer kit, 8 GB (Ampere, unified LPDDR5), JetPack R36.x / CUDA 12.x — the board people mean when they search “Jetson Orin Nano Super 8GB” / “Jetson Orin Nano Super developer kit.”

Suites live on the board.

Backends. Same local GGUFs served by llama.cpp (CUDA) and/or Ollama, so “Ollama on Jetson” vs bare llama.cpp is a first-class comparison — not a vibe check.

Stabilization. Power modes via nvpmodel / jetson_clocks (7W · 15W · 25W · MAXN). Power and thermals from tegrastats (VDD_CPU_GPU_CV, junction temps).

Load generator. aiperf 0.11.0, typically 20 requests per cell, concurrency 1. Table cells show mean ± std over those requests; chart/sort defaults use aiperf p50.

Jetson tok/J (published definition). Decode-phase energy only:

tok/J = OSL / (decode_power_W × p50_decode_s)

Other boards in the repo use related but not identical formulas (documented per folder). Do not cross-compare tok/J across devices as if they shared one definition.

Results snapshot (Jetson · non-reasoning · llama.cpp · 25W)

Representative cell on the live site: context 2048, generation 256, Jetson Orin Nano Super at 25W with llama.cpp.

Figure 1. Chart view on smolperfbenchmark.vercel.app with filters locked to non-reasoning · llama.cpp · 25W · ctx 2048 · gen 256.

Figure 2. Leaderboard table for the same filters. Latency/throughput cells show mean ± std over ~20 aiperf requests; sort still uses p50.

Finding 1 — Under a 25W cap, size dominates efficiency

On this cell, SmolLM2-135M leads at roughly 165 tok/s and ~29.6 tok/J. Moving to ~1B-class instruct models (Llama-3.2-1B, Gemma3-1B) drops both throughput and tok/J by several× on the same board and backend. If your constraint is joules (or sustained heat), parameter count is not a free variable — which is exactly why small language models matter for edge boards.

Finding 2 — llama.cpp vs Ollama is not a footnote

On the full Jetson non-reasoning sweeps, llama.cpp often leads Ollama by a large margin on sub-1B transformers at matched quants (see the long-form report). Relative order still moves with power mode and cell. If your question is “Ollama Jetson Orin Nano Super vs llama.cpp,” filter the leaderboard to the same ctx/gen/power and compare backends — do not trust a single viral tok/s screenshot.

Finding 3 — Rankings are cell-conditioned; the config is the object

Changing power mode, context, or generation length reshuffles relative order. A row is (device, suite, model, quant, backend, power mode, ctx, gen). Publishing only “Model X = N tok/s” throws away the axes people need to reproduce the claim — especially for a Jetson LLM deployment where 15W vs 25W is a different product. That is why this is a filterable local LLM benchmark leaderboard, not a static table.

How to read the leaderboard without fooling yourself

1. Pick the device tab first. Only Jetson is live today; other tabs are placeholders until those runs publish.
2. Lock the filters to the cell you care about (power, ctx, gen, backend).
3. Use tok/J when energy matters; tok/s when latency matters. They often agree directionally on one board, but not always.
4. Open the column i tips on the site for the exact statistic behind each column.
5. Read the device report for full sweeps, OOMs, and methodology notes (Tiny LLM / Jetson write-up, Bonsai write-up).

Limitations (please keep these attached)

Honest scope is part of the product:

• Capability is out of scope. A faster 135M model is not “better” for your task without a quality evaluation.
• One live device so far. Cross-device conclusions wait for Pi / phone / Mac Mini publication — including any Raspberry Pi LLM claims.
• tok/J formulas differ by hardware family in the harness. Do not merge Jetson decode-phase tok/J with Pi window-average tok/J into one chart.
• Synthetic prompts. aiperf fixed-length synthetic loads are for systems measurement, not chat realism.
• Concurrency 1. Multi-user serving needs a different study.
• Quantization is part of the config. Q4_K_M vs Q8_0 changes both footprint and speed; compare like with like.

If you cite a number from the leaderboard, cite the full config and the date you looked.

Reproducing a run

git clone https://github.com/YuvrajSingh-mist/smolperfbenchmark.git
cd smolperfbenchmark
uv sync
.venv/bin/aiperf --version   # must print 0.11.0

Jetson non-reasoning (25W sweet spot), from that device folder:

bash bench-non-reasoning.sh --power-mode 1

Useful flags: --backend llamacpp|ollama|both, --only <substring>, --reqs N, --resume <dir>, --dry-run. Scripts auto-relaunch into tmux. Details live in each device README.

Why a leaderboard instead of another static table

A static blog table goes stale the week after you ship it. A filterable leaderboard makes the conditioned nature of on-device / local LLM performance visible: same model, different power mode, different story. The long-form reports still matter for narrative and failure modes; the leaderboard is the index.

The project goal is practical: help someone choose a model that fits, stays cool enough, and spends joules honestly on hardware they can buy — whether that is a Jetson Orin Nano today or a Pi / phone / Mac Mini when those suites publish.

Citation

@misc{singh2026smolperfbenchmark,
      title={smolperfbenchmark: On-Device LLM Leaderboard},
      author={Yuvraj Singh},
      year={2026},
      howpublished={\url{https://github.com/YuvrajSingh-mist/smolperfbenchmark}},
}

License: MIT.
Contact: yuvraj.mist@gmail.com

Links

• Leaderboard: smolperfbenchmark.vercel.app
• Harness: github.com/YuvrajSingh-mist/smolperfbenchmark
• Jetson non-reasoning report: smolhub.com/posts/jetson-nano-super-benchmark-non-reasoning
• Bonsai report: smolhub.com/posts/jetson-orin-nano-super-bonsai-benchmark

Figures 1–2 are screenshots of smolperfbenchmark.vercel.app (Jetson non-reasoning · llama.cpp · 25W · ctx2048 × gen256). Open the site for interactive filters.