Methodology

Our job is to answer one practical question: given a real task, does this agent get it done, and what does it cost? Here's exactly how we do it and where it falls short.

1. Tasks

Each task is a small, self-contained repository with a written brief in the voice of real users (bug reports, a feature request). It has a handful of visible tests the agent can run and a larger hidden suite it never sees. Tasks are zero-dependency so the environment can't skew results.

2. Validation before publication

A task is published only if the harness proves it discriminates: untouched code must fail and our reference fix must pass 100% of hidden tests. This check runs automatically on every site build, and its per-test results are published on each task page.

3. Running an agent

• Fresh temporary directory, initialised as a git repo with a baseline commit.
• The agent receives TASK.md as its prompt and full permission to read, edit and run commands inside the workspace.
• Automated mode: the agent's headless CLI runs with no human input (interventions = 0 by construction). Manual-session mode (for IDE agents): an operator pastes the prompt and logs every intervention.
• 30-minute wall-clock timeout. The default model and settings are used unless the run record says otherwise.
• We aim for 3 attempts per agent per task and publish every attempt, because agents are nondeterministic. One lucky run isn't a result.

4. Scoring

• Score: percentage of hidden tests passed. Solved: 100%.
• Disqualification (score 0, shown with flags): modifying or deleting visible tests, or adding dependencies.
• Time: wall clock measured by the harness. Cost: taken from the provider's usage dashboard or the agent's own cost report, and the source is always printed next to the number. If we can't measure it, we show "—".
• Leaderboards rank by mean score, then solve rate, cost and time.

5. Verification tiers

• AgentRank-run We executed it, the full log is published, and it's eligible for ranking.
• Community-submitted Run by a builder or user with the same harness and submitted by pull request. CI re-applies the submitted diff to a clean copy of the task and re-scores it, and the PR fails if the numbers don't match. These runs are ranked on a separate community leaderboard. The score is proven; agent identity, time and cost are self-reported.

6. Limitations (read these)

• Small sample. We're launching with one task. One task tells you how agents handle this kind of job, not which agent is best overall.
• Public hidden tests. Task #1's hidden suite is public so anyone can reproduce scores. That means a future model could have seen it in training. Later tasks will keep a private holdout set alongside a public one.
• Versions drift. Agents and models update weekly. Every run records its version and model, and results are dated for that reason.
• Conflict-of-interest disclosure. Parts of this site and harness were written with help from AI coding tools, including Claude Code, which is also listed here. That's why scoring is fully automated, the code is open, and every run can be rerun by anyone.
• Not a replacement for research benchmarks. SWE-bench and Terminal-Bench evaluate at much larger scale. We focus on practical, explainable tasks and cost.

7. AgentRank Score (every agent)

Every listed agent gets a 0–100 AgentRank Score computed only from public GitHub data, so anyone can recompute it. It measures traction and upkeep, not how well the agent completes tasks. Task performance is a separate benchmark score, and only agents with real runs have one.

• 30%: Popularity: stars, percentile vs. every listed agent (log scale)
• 20%: Adoption: forks, percentile (log scale)
• 25%: Activity: how recently code was pushed
• 15%: Momentum: stars gained per day since creation, percentile
• 10%: Openness: OSI open-source licence detected

Activity points: pushed in the last 7 days = 100, 30 days = 85, 90 days = 60, 180 days = 35, 365 days = 15, older = 0. Percentiles are relative to every agent in the directory, so scores shift as the ecosystem changes. Momentum treats repos younger than 30 days as 30 days old. The code is in site/score.mjs.

8. The agent directory

The directory lists the most-starred open-source AI agent repositories on GitHub. It's built by querying the GitHub search API across agent-related topics and keywords (for example topic:ai-agent, topic:coding-agent and "autonomous agent"), with a 200-star minimum and forks and archived repos excluded. Names, makers, avatars, stars, licences and dates come straight from GitHub. Categories are assigned automatically from each repo's name, description and topics, so they can be wrong. A listing isn't an endorsement and isn't a benchmark result. Only agents with runs get scores. The list refreshes weekly. Makers can claim a listing or ask for a correction.

9. Corrections

Found an error? Open a correction. Corrections are made publicly in the git history.