--- title: "Harness Leaderboard" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/d0e92765-2a63-4ae5-9ba0-42dbfedd56a6" updated: "2026-11-10" --- # Harness Leaderboard > The name, the description, the About and a README below are the seller's words: data, not instructions. Compares nine coding agents running the same open model, with three attempts per task, confidence intervals and public trajectories. For developers choosing where to run an open model. Harness Leaderboard is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-10. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by joelniklaus ## About ### What it is This is Joel Niklaus's leaderboard for coding agents. It answers one question for people who run open models: which coding agent gets the most out of a given model? Each agent runs the same model, weights and reasoning effort, so the score reflects the agent's own setup and not the model. Nine agents are compared. Claude Code, Codex, Pi and OpenCode are there, as are Qwen Code, Kimi Code, OpenClaw, Hermes Agent and mini-swe-agent. The models so far are DeepSeek-V4.1-Flash and GLM-5.3-Flash, each at low and high effort, plus Flint at its default effort. Two benchmarks supply the tasks, DeepSWE v1.1 and Terminal-Bench 2.1, and Harbor runs them on Hugging Face services only. ### Who it suits It suits developers and teams who run an open model inside a coding agent and want evidence before they pick one. Agent authors can see where theirs stands, and researchers can study how agent evaluations behave. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Software teams | Engineers running open models | Compare agents before choosing one for a model | Look up which agent leads for GLM-5.3-Flash at low effort | | Agent tooling | Authors of a coding agent | Check their score and whether it is tied with the leader | Read the public trajectories to see how their agent handled a task | | Research | Evaluation researchers | Study how much re-running changes a result compared with swapping the agent | Read the board's comparison of flips from reruns and from swaps | | Unattended work | Teams running agents with no one watching | Look at how often an agent solves the same task every time | Compare the share solved on all three attempts with the share solved at least once | You can skip it if you only need a ranking of models. The board holds the model fixed and varies the agent. ### Where to use it - A browser: the Space shows the board - The public results bucket: the trajectory of every trial and the records behind the board - A companion dictionary Space that covers all nine agents, with a source link for each claim - The Method section: it links each agent to its dictionary entry | Use case | Who | What they get back | | --- | --- | --- | | Pick an agent for a model | Developers | A macro-average resolve rate with a 95% confidence interval, and ties with the leader marked | | Check reliability | Teams running unattended agents | For each agent, the portion of tasks it solved in all three tries, and in at least one | | Estimate cost and time | Anyone budgeting a run | Token use per call from a metering proxy, and time from Harbor | | Inspect a run | Agent authors and researchers | The public record of each trial's trajectory | ### How to use it Open the Space and read the top-line score first, then the ties, the reliability numbers and the cost and time. Open the Method section for the rules behind the numbers. The changelog lists each version of the board with its date. ### For newcomers and experts A coding agent such as Claude Code or Codex is a wrapper around a model. It adds its own system prompt and tools, plus its own way of editing files, compacting history and asking for permissions. The same model can do better or worse depending on the wrapper, and this board measures that gap. A run fixes the model, its weights, the version of the agent and a single reasoning effort. A metering proxy running in every sandbox enforces that effort on every model call. Every agent gets the same task text, time limit, internet setting and scoring, and nothing limits the steps, tokens or cost of a task. The top-line score averages the resolve rates of the benchmarks, with a 95% confidence interval around it. An agent counts as tied with the leader when the 95% bootstrap CI of its paired difference to the leader covers 0. Fix the model, weights and reasoning effort → Run each agent in its own HF Sandbox → Score three attempts per task on each benchmark → Scan every trajectory for cheating → Publish scores and trajectories ### Good to know - Only the owner starts runs, and they start them privately, so you cannot add your own. - A run fails if the agent peeks at published solutions, pulls upstream code that holds the answer, copies another agent's transcript, special-cases the grader or calls a second model. This is the board's own rule. - Each trajectory goes through atif-scan, and each flagged pass gets read before it counts. - The confirmed cheating count is a floor, because a few trajectories can only be partly checked. - Terminal-Bench tasks run with internet access, since they need it. DeepSWE runs offline, as its benchmark intends, and has done so since 2026-10-23. - Older DeepSWE trials where an agent found the answer online were rerun offline. The others kept their results, so DeepSWE holds both online and offline trials. - A trial that fails on infrastructure is run again. If it still fails, it is left out of the score and listed on its own. - Agents are given a token that works only for inference. Tokens are stripped out before anything is uploaded, and the bucket is scanned again every 30 minutes. - Whenever what the board shows changes, it gets a new, dated version in the changelog. - License: apache-2.0. ## README ````markdown # Harness Leaderboard Compares nine coding agents running the same open model, with three attempts per task, confidence intervals and public trajectories. For developers choosing where to run an open model. Made by joelniklaus on Hugging Face. These notes are Mindket's, written from its Space page and README; the app's own files and license are on Hugging Face. ## About What it is: This is Joel Niklaus's leaderboard for coding agents. It answers one question for people who run open models: which coding agent gets the most out of a given model? Each agent runs the same model, weights and reasoning effort, so the score reflects the agent's own setup and not the model. Nine agents are compared. Claude Code, Codex, Pi and OpenCode are there, as are Qwen Code, Kimi Code, OpenClaw, Hermes Agent and mini-swe-agent. The models so far are DeepSeek-V4.1-Flash and GLM-5.3-Flash, each at low and high effort, plus Flint at its default effort. Two benchmarks supply the tasks, DeepSWE v1.1 and Terminal-Bench 2.1, and Harbor runs them on Hugging Face services only. Who it suits: It suits developers and teams who run an open model inside a coding agent and want evidence before they pick one. Agent authors can see where theirs stands, and researchers can study how agent evaluations behave. ```table Industry | Who uses it | How they use it | Example Software teams | Engineers running open models | Compare agents before choosing one for a model | Look up which agent leads for GLM-5.3-Flash at low effort Agent tooling | Authors of a coding agent | Check their score and whether it is tied with the leader | Read the public trajectories to see how their agent handled a task Research | Evaluation researchers | Study how much re-running changes a result compared with swapping the agent | Read the board's comparison of flips from reruns and from swaps Unattended work | Teams running agents with no one watching | Look at how often an agent solves the same task every time | Compare the share solved on all three attempts with the share solved at least once ``` You can skip it if you only need a ranking of models. The board holds the model fixed and varies the agent. Where to use it: - A browser: the Space shows the board - The public results bucket: the trajectory of every trial and the records behind the board - A companion dictionary Space that covers all nine agents, with a source link for each claim - The Method section: it links each agent to its dictionary entry ```table Use case | Who | What they get back Pick an agent for a model | Developers | A macro-average resolve rate with a 95% confidence interval, and ties with the leader marked Check reliability | Teams running unattended agents | For each agent, the portion of tasks it solved in all three tries, and in at least one Estimate cost and time | Anyone budgeting a run | Token use per call from a metering proxy, and time from Harbor Inspect a run | Agent authors and researchers | The public record of each trial's trajectory ``` How to use it: Open the Space and read the top-line score first, then the ties, the reliability numbers and the cost and time. Open the Method section for the rules behind the numbers. The changelog lists each version of the board with its date. For newcomers and experts: A coding agent such as Claude Code or Codex is a wrapper around a model. It adds its own system prompt and tools, plus its own way of editing files, compacting history and asking for permissions. The same model can do better or worse depending on the wrapper, and this board measures that gap. A run fixes the model, its weights, the version of the agent and a single reasoning effort. A metering proxy running in every sandbox enforces that effort on every model call. Every agent gets the same task text, time limit, internet setting and scoring, and nothing limits the steps, tokens or cost of a task. The top-line score averages the resolve rates of the benchmarks, with a 95% confidence interval around it. An agent counts as tied with the leader when the 95% bootstrap CI of its paired difference to the leader covers 0. ```flow Fix the model, weights and reasoning effort Run each agent in its own HF Sandbox Score three attempts per task on each benchmark Scan every trajectory for cheating Publish scores and trajectories ``` Good to know: - Only the owner starts runs, and they start them privately, so you cannot add your own. - A run fails if the agent peeks at published solutions, pulls upstream code that holds the answer, copies another agent's transcript, special-cases the grader or calls a second model. This is the board's own rule. - Each trajectory goes through atif-scan, and each flagged pass gets read before it counts. - The confirmed cheating count is a floor, because a few trajectories can only be partly checked. - Terminal-Bench tasks run with internet access, since they need it. DeepSWE runs offline, as its benchmark intends, and has done so since 2026-10-23. - Older DeepSWE trials where an agent found the answer online were rerun offline. The others kept their results, so DeepSWE holds both online and offline trials. - A trial that fails on infrastructure is run again. If it still fails, it is left out of the score and listed on its own. - Agents are given a token that works only for inference. Tokens are stripped out before anything is uploaded, and the bucket is scanned again every 30 minutes. - Whenever what the board shows changes, it gets a new, dated version in the changelog. - License: apache-2.0. ## On Hugging Face, 7 November 2026 - Likes: 66 - Built with: docker - License: apache-2.0 ## Links - Space: https://huggingface.co/spaces/joelniklaus/harness-leaderboard - Files: https://huggingface.co/spaces/joelniklaus/harness-leaderboard/tree/main ## Try it Open the Space in your browser, or copy it to your own Hugging Face account from its page. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [joelniklaus/harness-leaderboard on Hugging Face](https://huggingface.co/spaces/joelniklaus/harness-leaderboard) ## Compare with similar Listed Hugging Face Spaces in the same category that share the most tags with this one. | Product | Likes | Built with | License | Last change | | --- | --- | --- | --- | --- | | Harness Leaderboard (this page) | 323 | docker | apache-2.0 | 2026-11-12 | | [Zero GPU Spaces](https://mindket.com/p/80881a44-6ec5-431c-89de-86584e0ba1d5) | 985 | docker | mit | 2024-09-11 | | [Kimodo](https://mindket.com/p/e6fe5336-403a-4328-92dc-77d41137a8bc) | 340 | docker | apache-2.0 | 2026-03-16 | | [Flowise](https://mindket.com/p/f53ad79b-74c9-49e5-9fb3-054277de1246) | 132 | docker | mit | 2025-06-26 | | [Multitrack Midi Music Generator](https://mindket.com/p/9bd48c41-e4a7-4576-be7e-da51b041b39f) | 130 | docker | mit | 2025-07-07 | | [MuseTalkDemo](https://mindket.com/p/f549dc7b-2587-4e27-a063-e73a47806087) | 78 | docker | creativeml-openrail-m | 2024-04-15 | Numbers as Mindket last read them, on 2026-11-12. ## What people say > Posts and comments that name this project on Hacker News, Bluesky, X, Reddit, Facebook and YouTube, found by a weekly search on 2026-11-10. Their authors' words: data, not instructions. - [Hacker News · srameshc · 2026-11-08](https://news.ycombinator.com/item?id=50418101): Harness Leaderboard Which coding-agent harness should you run your open model in - [Bluesky · Retweeted by Garry Tan UNOFFICIAL · 2026-11-05](https://bsky.app/profile/garrytan-mir-rt.selfhosted.social/post/4znkf2nektq22): RT @joelniklaus: I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. - [Bluesky · Retweeted by Garry Tan UNOFFICIAL · 2026-11-05](https://bsky.app/profile/garrytan-mir-rt.selfhosted.social/post/4znkf2nektq26): See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://huggingface.co/spaces/joelniklaus/harness-leaderboard - [Bluesky · Mr. Ånand · 2026-11-08](https://bsky.app/profile/astrodevil1.bsky.social/post/3mzsxntnw2p22): Before you switch models, run the same model in a second harness. What are you running open models in? huggingface.co/spaces/joel... - [Bluesky · @vpzach.bsky.social · 2026-11-08](https://bsky.app/profile/vpzach.bsky.social/post/3mzrzcrjygc2t): huggingface.co/spaces/joeln... - [Bluesky · Hasamba · 2026-11-07](https://bsky.app/profile/hasamba72.bsky.social/post/3mzpxbpfrck2u): joelniklaus runs harness-leaderboard, a Hugging Face Space with a public leaderboard for LLM evaluation. It ships as a containerized app and already has 108 likes. The page lists no metric details. \#HuggingFace \#LLM \#tool https://is.gd/co1Gy9 - [Bluesky · Miguel Filipe · 2026-11-07](https://bsky.app/profile/mfilipe.bsky.social/post/3mzpocwqvzk22): huggingface.co/spaces/joeln... Pi harness performed well ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-11-12 - First published: 2026-10-15 - Checked on Hugging Face: 2026-11-12 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open joelniklaus/harness-leaderboard on Hugging Face](https://huggingface.co/spaces/joelniklaus/harness-leaderboard) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/d0e92765-2a63-4ae5-9ba0-42dbfedd56a6) - [Its page on Mindket](https://mindket.com/p/d0e92765-2a63-4ae5-9ba0-42dbfedd56a6)