Harness Leaderboard

Harness Leaderboard

Compares nine coding agents running the same open model, with three attempts per task, confidence intervals and public trajectories. For developers choosing where to run an open model.

Made by joelniklaus on Hugging Face. These notes are Mindket's, written from its Space page and README; the app's own files and license are on Hugging Face.

About

What it is: This is Joel Niklaus's leaderboard for coding agents. It answers one question for people who run open models: which coding agent gets the most out of a given model? Each agent runs the same model, weights and reasoning effort, so the score reflects the agent's own setup and not the model.

Nine agents are compared. Claude Code, Codex, Pi and OpenCode are there, as are Qwen Code, Kimi Code, OpenClaw, Hermes Agent and mini-swe-agent. The models so far are DeepSeek-V4.1-Flash and GLM-5.3-Flash, each at low and high effort, plus Flint at its default effort. Two benchmarks supply the tasks, DeepSWE v1.1 and Terminal-Bench 2.1, and Harbor runs them on Hugging Face services only.

Who it suits: It suits developers and teams who run an open model inside a coding agent and want evidence before they pick one. Agent authors can see where theirs stands, and researchers can study how agent evaluations behave.

Industry | Who uses it | How they use it | Example
Software teams | Engineers running open models | Compare agents before choosing one for a model | Look up which agent leads for GLM-5.3-Flash at low effort
Agent tooling | Authors of a coding agent | Check their score and whether it is tied with the leader | Read the public trajectories to see how their agent handled a task
Research | Evaluation researchers | Study how much re-running changes a result compared with swapping the agent | Read the board's comparison of flips from reruns and from swaps
Unattended work | Teams running agents with no one watching | Look at how often an agent solves the same task every time | Compare the share solved on all three attempts with the share solved at least once

You can skip it if you only need a ranking of models. The board holds the model fixed and varies the agent.

Where to use it:

• A browser: the Space shows the board
• The public results bucket: the trajectory of every trial and the records behind the board
• A companion dictionary Space that covers all nine agents, with a source link for each claim
• The Method section: it links each agent to its dictionary entry

Use case | Who | What they get back
Pick an agent for a model | Developers | A macro-average resolve rate with a 95% confidence interval, and ties with the leader marked
Check reliability | Teams running unattended agents | For each agent, the portion of tasks it solved in all three tries, and in at least one
Estimate cost and time | Anyone budgeting a run | Token use per call from a metering proxy, and time from Harbor
Inspect a run | Agent authors and researchers | The public record of each trial's trajectory

How to use it: Open the Space and read the top-line score first, then the ties, the reliability numbers and the cost and time. Open the Method section for the rules behind the numbers. The changelog lists each version of the board with its date.

For newcomers and experts: A coding agent such as Claude Code or Codex is a wrapper around a model. It adds its own system prompt and tools, plus its own way of editing files, compacting history and asking for permissions. The same model can do better or worse depending on the wrapper, and this board measures that gap.

A run fixes the model, its weights, the version of the agent and a single reasoning effort. A metering proxy running in every sandbox enforces that effort on every model call. Every agent gets the same task text, time limit, internet setting and scoring, and nothing limits the steps, tokens or cost of a task. The top-line score averages the resolve rates of the benchmarks, with a 95% confidence interval around it. An agent counts as tied with the leader when the 95% bootstrap CI of its paired difference to the leader covers 0.

Fix the model, weights and reasoning effort
Run each agent in its own HF Sandbox
Score three attempts per task on each benchmark
Scan every trajectory for cheating
Publish scores and trajectories

Good to know:

• Only the owner starts runs, and they start them privately, so you cannot add your own.
• A run fails if the agent peeks at published solutions, pulls upstream code that holds the answer, copies another agent's transcript, special-cases the grader or calls a second model. This is the board's own rule.
• Each trajectory goes through atif-scan, and each flagged pass gets read before it counts.
• The confirmed cheating count is a floor, because a few trajectories can only be partly checked.
• Terminal-Bench tasks run with internet access, since they need it. DeepSWE runs offline, as its benchmark intends, and has done so since 2026-10-23.
• Older DeepSWE trials where an agent found the answer online were rerun offline. The others kept their results, so DeepSWE holds both online and offline trials.
• A trial that fails on infrastructure is run again. If it still fails, it is left out of the score and listed on its own.
• Agents are given a token that works only for inference. Tokens are stripped out before anything is uploaded, and the bucket is scanned again every 30 minutes.
• Whenever what the board shows changes, it gets a new, dated version in the changelog.
• License: apache-2.0.

On Hugging Face, 7 November 2026

• Likes: 66
• Built with: docker
• License: apache-2.0

Links

• Space: https://huggingface.co/spaces/joelniklaus/harness-leaderboard
• Files: https://huggingface.co/spaces/joelniklaus/harness-leaderboard/tree/main

Try it

Open the Space in your browser, or copy it to your own Hugging Face account from its page. Nothing in this kit runs by itself.