Tasks /
Task #1: Fix a broken authentication flow
A small Node.js auth module ships with six user-reported bugs: broken password checks, case-sensitive emails, sessions that die instantly, logout that doesn't log out, account takeover via re-registration, and an off-by-one lockout. The agent gets the bug reports and 3 visible tests; scoring uses a 19-test hidden suite.
Reading bug reports Debugging Security fundamentals Not breaking the public API
Results
No verified runs yet, so no rankings.
We won't publish a leaderboard until real runs exist. Every score here will come from the open-source harness and link to its full diff, test output and agent log. Nothing is estimated, copied from marketing pages or invented.
Harness check (Oct 19, 2026): untouched code scores 4/19, the reference fix scores 19/19. The scorer separates broken code from fixed code. See per-test evidence.
What the agent sees
Task: fix the Notely login system
You are working in a small Node.js project (no dependencies, Node 20+). The auth module
lives in src/auth.js. Users have filed these bug reports:
- "I signed up a minute ago and I can't log in with the exact password I just set."
- "I registered as
Sam@Example.com(with a stray space) and nowsam@example.comdoesn't work." - "Right after logging in, every page says my session is invalid."
- "I clicked log out, but my old session token still works."
- "Someone was able to 'register' with my email and it replaced my password."
- "Security review: the account lockout lets an attacker try more than 5 wrong passwords in 15 minutes."
Requirements:
- Fix all six problems in
src/auth.js. - Keep the public API the same:
createAuth,AuthError, and theregister,login,verify,logoutfunctions with their current signatures and return shapes. - Emails are case-insensitive and surrounding whitespace is ignored.
- Registering an email that already exists must throw
AuthErrorwith codeEMAIL_TAKEN. - Sessions last 1 hour. After 5 failed attempts within 15 minutes, further logins for that email throw
AuthErrorwith codeLOCKED(even with the correct password) until the window passes. - Run
npm testto check your work. The visible tests do not cover everything; the evaluator runs an additional hidden test suite. - Do not add dependencies. Do not modify or delete tests.
How it's scored
- Score = hidden tests passed ÷ 19.
- Solved = all 19 hidden tests pass.
- Disqualified (score 0) if visible tests are edited or deleted, or if dependencies are added.
- Timeout: 30 minutes of wall-clock time.
- We record time, cost (with its source), human interventions, files changed, and agent version and model.
Hidden test source (public so results can be reproduced; see the limitations on the methodology page).
Harness validation
Before any agent is scored, the harness scores the untouched starting code and our reference fix. A task is published only if broken code fails and the reference passes every test. Passed Oct 19, 2026 · Node v22.23.3 · harness v1.0.0
| Hidden test | Untouched code | Reference fix |
|---|---|---|
| H01 register then login with same password succeeds | Fail | Pass |
| H02 login is case-insensitive and trims whitespace | Fail | Pass |
| H03 verify returns the normalized email | Fail | Pass |
| H04 session is valid immediately after login | Fail | Pass |
| H05 session is still valid at 59 minutes | Fail | Pass |
| H06 session expires after 1 hour | Fail | Pass |
| H07 logout invalidates the token | Fail | Pass |
| H08 logout leaves other sessions intact | Fail | Pass |
| H09 duplicate registration throws EMAIL_TAKEN and keeps the original password | Fail | Pass |
| H10 duplicate check is case-insensitive | Fail | Pass |
| H11 sixth attempt after 5 failures is LOCKED even with the right password | Fail | Pass |
| H12 four failures then the right password still works | Fail | Pass |
| H13 lockout lifts after the 15 minute window | Fail | Pass |
| H14 lockout counts across email casing | Fail | Pass |
| H15 lockout is per account | Fail | Pass |
| H16 unknown user gets INVALID_CREDENTIALS | Pass | Pass |
| H17 validation still rejects weak passwords and invalid emails | Pass | Pass |
| H18 passwords are not stored in plaintext | Pass | Pass |
| H19 unknown token verifies to null | Pass | Pass |
| Total | 4/19 | 19/19 |
The 4 tests that pass on untouched code are regression guards (validation, hashing, unknown users). They catch fixes that break working behaviour.
Reproduce
git clone https://github.com/RK-droid/agentrank.git && cd agentrank
node harness/validate.mjs # self-check: baseline fails, reference passes
node harness/run.mjs run --task auth-fix-01 --agent <slug> --cmd '<agent headless command>'