TIGER-AI-Lab/ClawBench?utm_source=trendshift.io: Trending on GitHub
More badges · featured in 37 curated lists
ClawBench: Can AI Agents Complete Everyday Online Tasks?
ClawBench is an open-source benchmark that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. V1 lives in test-cases/v1/, V2 in test-cases/v2/. It records five layers of evidence and supports request-based scoring and trajectory evaluation against human references. See the live leaderboard for scores by corpus, harness, and scoring rubric.
We asked frontier AI agents to do what people do every day -- order food, book travel, apply for jobs, write reviews, manage projects. In the historical V1 paper evaluation, the strongest evaluated agent completed about 1 in 3 tasks.
V1: 152 everyday tasks · 143 live sites | V2: 129 tasks · 63 live sites | 281 total across 163 live websites · 15 life categories
The paper reports 153 (V1) and 130 (V2); two ASPCA tasks were removed after publication, so the shipping corpus is 152 and 129.
Built by NAIL Group · Sister project: HarnessBench — fixes the base model, varies the harness · Runs on any Chrome.
中文
What are you looking for?
I want to… Start here
Run my model Quick start
See scores Live leaderboard · Results
Browse the tasks Task explorer · Datasets
Reproduce a score Reproduction workflow · Trace cookbook
Understand the benchmark Paper · FAQ
Change the framework or add tasks Source layout · Contributing
All guides and integrations: Documentation index.
Quick start
Prerequisites: Python 3.11+, uv, and a running Docker or Podman engine. See platform setup.
Installed package
uv tool install clawbench-eval clawbench
Launch from the directory where you want your configuration and results. The interactive TUI guides you through adding a model and selecting a task. Configure the default judge, deepseek-v4-pro, as well; the model setup guide explains judge selection and how to run without scoring.
Source checkout
git clone https://github.com/TIGER-AI-Lab/ClawBench.git cd ClawBench cp models/models.example.yaml models/models.yaml
Edit models/models.yaml with your model and judge API settings.
uv run clawbench
./run.sh is a shortcut for the same TUI. For a non-interactive run from the checkout:
uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf your-provider-model-id
Replace your-provider-model-id with a configured model key. The first run builds container images; the local browser's printed noVNC URL lets you watch the agent. Recordings and metadata land under test-output/.
Continue with the getting started guide for batch and human runs, harness selection, and remote browser options. The CLI reference covers non-interactive commands. Coding agents can start with AGENTS.md.
How it works
You pick a task ClawBench spins up Agent drives the Interceptor captures from V1 or V2 an isolated Docker browser: navigates, every action across everyday scenarios container + Chromium fills forms, clicks all 5 layers of data
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ "Book a pet │ ──► │ Container │ ──► │ AI Agent │ ──► │ 5 layers │ │ sitter on │ │ + Chromium │ │ browses the │ │ intercepted │ │ Rover" │ │ + Agent │ │ live site │ │ & recorded │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
Live Websites Isolated Containers Request Interceptor Five-Layer Recording
Container internals ┌─────────────────────────────────────────────────┐ │ Container (Docker / Podman) │ │ │ │ ┌──────────┐ CDP Fetch/Runtime/Page events │ │ │ Chromium ├─────────────────────────────┐ │ │ │ :9222 CDP│ │ │ │ └──────────┘ │ │ │ │ │ │ ┌──────────┐ ┌────────────────▼─┐ │ │ │ Xvfb │◄──ffmpeg──►│ FastAPI Server │ │ │ │ :99 │ x11grab │ :7878 │ │ │ └──────────┘ └──────────────────┘ │ │ │ │ │ ┌───────▼─────────┐ │ │ │ /data │ │ │ │ actions.jsonl │ │ │ │ requests.jsonl │ │ │ │ screenshots/ │ │ │ │ recording.mp4 │ │ │ └─────────────────┘ │ └─────────────────────────────────────────────────┘
Datasets
ClawBench ships three Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command.
Dataset What's in it Get it
NAIL-Group/ClawBench (mirrored at TIGER-Lab/ClawBench) Task definitions, rubrics, and metadata for V1 and V2 — what to attempt and how it's judged. hf download --repo-type dataset NAIL-Group/ClawBench
NAIL-Group/ClawBenchV1Trace One directory per V1 model run: recording.mp4, requests.jsonl, actions.jsonl, agent-messages.jsonl, interception.json, run-meta.json. hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace
TIGER-Lab/ClawBenchV2Trace Same 5-layer bundle for V2 runs. Rolling — new models added as they're evaluated. hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace
Build with the traces: See the Trace Cookbook for download recipes, data schemas, and research directions.
The trace datasets are large; use hf download --include "
🏆 Live leaderboard: claw-bench.com/leaderboard (V2 default, two-stage scoring — interception + LLM judge). Full scoring formula in eval/scoring.md. Add your run: PR to leaderboard/results.csv.
News
[2026.10.10] — 🎉 ClawBench has surpassed 1,000 GitHub stars! Thank you to everyone who has starred, used, and contributed to the benchmark.
[2026.08.20] — 🏆 Our paper has been accepted to EMNLP 2026 Findings.
[2026.08.20] - Added Kernel as a supported remote browser runtime. Thanks to @rgarcia.
[2026.08.18] — Added WebBrain as a supported harness. Thanks to @alectimison-maker.
[2026.08.16] — Released RewardHarness, our self-evolving agentic reward framework: 47.4% on EditReward-Bench from just 100 preference demos, with no reward-model training. Details →
[2026.08.03] — Added Browserbase as a remote browser runtime. Details →
[2026.07.30] — v0.8.0: Gemini-as-judge, random-click baseline harness, EdgeBench/SForge adapter, remote-browser CDP support. Details →
[2026.07.25] — 🏆 Our paper has been accepted by COLM 2026 WAB.
Earlier updates: docs/news.md · full change history: CHANGELOG.md
✨ Awesome Works using ClawBench
Authors from Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern, ByteDance Seed, and HKUST build on ClawBench — and Li Auto's Mach-Mind-4-Flash technical report evaluates on it.
😊 Google DeepMind, University of Oxford & Columbia University, The Recipe for Intelligence in Natural and Artificial Systems (DOI)
😊 Stanford, UC Berkeley, Microsoft Research & UCSB, Auditing Agent Harness Safety (Code, Project)
😊 Google, Agentic Coding Needs Proactivity, Not Just Autonomy (Google Research Blog)
😊 Harvard Kempner Institute, Massachusetts General Hospital & CUHK, NeuroClaw Technical Report (Code, Project)
😊 ETH Zürich & Handshake AI Research, Verifying Agents in Rubric-Graded Environments (Code, Workshop)
😊 University of Oxford, NUS & Peking University, OpenClaw Research: A Systematic Survey of Large Language Model Agents in Open Deployment (Project, Resources)
😊 Northwestern University, A Survey of the OpenClaw Ecosystem: From Platform Extensibility to Constraint Design (Project & Resources)
😊 UC Davis & UT Dallas, Toward Trustworthy Computer-Use Agents: Risk Propagation, Evaluation Gaps, and Human Governance (Code & Project, Resources)
9 more works 😊 ByteDance Seed & HKUST, Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context (Models)
😊 Tencent Hunyuan & Fudan University, TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
😊 Unipat AI, VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild (Code, Project)
😊 Tsinghua University & CUHK, WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation (Code, Project)
😊 NUS, HKUST, Tsinghua University & Peking University, Towards Long-Horizon Agents: A Survey (Project, Resources)
😊 HKU MMLab, UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (Code, Project)
😊 Tsinghua University & SJTU, MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop (Code, Project)
😊 Peking University & CUHK, π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows (Code, Project)
😊 SJTU, AcademiClaw: When Students Set Challenges for AI Agents (Code, Project)
If we missed your work, please open an issue or submit a pull request.
Results
ClawBench leaderboard · by corpus × harness · live at claw-bench.com
V2 (Hermes) · 8 models · ds-v4-pro judge, lenient + strict
Rank Model Harness Intercepted Reward (lenient) Reward (strict) Pass / Total
1 claude-opus-4-7 hermes 54.6% 44.6% 24.6% 58 / 130
2 gpt-5.5 hermes 45.4% 35.4% 18.5% 46 / 130
3 glm-5.1 hermes 48.5% 34.6% 17.7% 45 / 130
4 deepseek-v4-pro hermes 43.9% 33.9% 12.3% 44 / 130
5 openrouter-owl-alpha hermes 14.6% 0.0% 0.0% 0 / 130
6 z-ai/glm-4.5-air:free hermes 4.6% 2.3% 0.8% 3 / 130
7 deepseek-v4-flash:free hermes 3.1% 2.3% 0.0% 3 / 129
8 minimax-m2.5:free hermes 2.3% 1.5% 0.0% 2 / 130
Intercepted = final HTTP request matched the task's URL/method (Stage 1, deterministic). Reward (lenient) = additionally judged by deepseek/deepseek-v4-pro to fulfill the instruction under the "no contradiction → match" rubric (Stage 2). Reward (strict) = same judge, strict rubric ("ambiguous → mismatch"). Ranked by Intercepted; Reward as tiebreak. Totals reflect the corpus size at run time.
V2 (OpenClaw) · 1 model
Rank Model Harness Intercepted Reward (lenient) Reward (strict) Pass / Total
1 glm-5.1 openclaw 0.0% 0.0% 0.0% 0 / 130
V1 (Hermes) · 6 frontier models, original paper rubric
Rank Model Harness Pass Rate Pass / Total
1 claude-opus-4-6 hermes 61.4% 94 / 153
2 claude-sonnet-4-6 hermes 56.9% 87 / 153
3 claude-haiku-4-5-20251001 hermes 30.1% 46 / 153
4 gpt-5.4-2026-03-05 hermes 25.5% 39 / 153
5 gpt-5.4-mini-2026-03-17 hermes 24.8% 38 / 153
6 kimi-k2.5 hermes 17.6% 27 / 153
V1 Pass Rate is from the original paper rubric (Claude Code agentic-eval subagent comparing each run against human reference trajectories under eval/agentic_eval.md). The two-stage Reward (interception + deepseek/deepseek-v4-pro lenient judge) for V1 will appear here once V1 trace bundles are re-judged.
V1 per-category breakdown (Sonnet 4.6 vs 6-model comparison)
Rank Model Overall Daily Finance Work Dev Academic Travel Social Pets
1 Claude Sonnet 4.6 33.3 44.2 50.0 19.0 11.1 50.0 23.1 38.9 18.2
2 GLM-5 24.2 30.8 16.7 38.1 16.7 28.6 0.0 16.7 18.2
3 Gemini 3 Flash 19.0 15.4 33.3 23.8 22.2 28.6 30.8 11.1 0.0
4 Claude Haiku 4.5 18.3 15.4 22.2 19.0 27.8 21.4 7.7 16.7 18.2
5 GPT-5.4 6.5 9.6 0.0 0.0 11.1 7.1 7.7 0.0 9.1
6 Gemini 3.1 Flash Lite 3.3 1.9 0.0 0.0 5.6 14.3 0.0 0.0 9.1
Task categories · V1: 15 categories, 152 tasks
Category Tasks Example Platforms
Daily Life 21 Uber Eats, DoorDash, Instacart, Zillow, Craigslist
Entertainment & Hobbies 15 Ticketmaster, AMC Theatres, Topgolf, Crunchyroll
Creation & Initialization 13 Squarespace, Wix, Webflow, Ghost, Substack
Rating & Voting 10 Trustpilot, G2, Goodreads, RateMyProfessors
Travel 9 Booking.com, Expedia, Airbnb, TripAdvisor
Education & Learning 9 Coursera, Udemy, Khan Academy, Duolingo
Office & Secretary 9 Google Calendar, Slack, Notion, Trello
Beauty & Personal Care 9 Sephora, Ulta, Glossier
Job Search & HR 8 LinkedIn, Greenhouse, Lever, Workday
Pet & Animal Care 7 Chewy, Petco, Rover
Personal Management 6 Mint, YNAB, Todoist
Shopping & Commerce 6 Amazon, eBay, Etsy, Target
Nonprofit & Charity 6 GoFundMe, DonorsChoose
Academia & Research 5 Google Scholar, Semantic Scholar, OpenReview
Finance & Investment 4 Robinhood, Fidelity, Coinbase
Others 15 Automation, Dev & Tech, Government, Home Services, Automotive
Codex and Claude Code runs on V2, and the V1 OpenClaw aggregate, are still in flight — they land on the live leaderboard first.
Reproduce the leaderboard
Our scores are stable: two independent runs of the same model under the same judge (deepseek/deepseek-v4-pro, lenient rubric) reproduce Intercepted and Reward within ±2 pp on the V2 corpus.
There are two ways to verify this on your own machine.
Path A — Re-run the agent, then score
Confirms the full pipeline (your agent + our judge) lines up with our leaderboard row.
clawbench-batch --models deepseek/deepseek-v4-flash --cases-suite v2
--all-cases --harness hermes --no-judge --output-dir ./my-run
clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric both
Path B — Skip the run, re-judge our published traces
Confirms just the judge matches ours (cheap, no agent compute, useful for sanity-checking your judge config).
hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace
--include "batch-aligned-*/deepseek-v4-flash-free/**" --local-dir ./reproduce
clawbench-rescore ./reproduce --judge-model deepseek-v4-pro --rubric both
One-shot equivalent of Path B for any model in the leaderboard:
clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0
Pass criterion
For deepseek-v4-flash:free × hermes × v2, the published row is Intercepted 3.1% / Reward-lenient 2.3% / Reward-strict 0.0% (3 / 129). Path A or B counts as reproduced when all three metrics land within ±2 pp. Larger gaps usually mean a different judge model, a different rubric prompt, or a harness configuration drift — diff your eval_results/
ClawBench-Lite
New here? Run this first. test-cases/v1-lite/ is a 20-task curated subset of V1, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of browser-use/benchmark and gives you a credible signal at a fraction of the full-benchmark cost.
Tier distribution: flagship 9 / core 8 / wildcard 3 — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. Use the separate human-reference evaluation workflow to evaluate Lite trajectories under eval/agentic_eval.md; selecting the suite does not switch the native request judge to trajectory evaluation.
Run it with --cases-suite v1-lite, or inspect the link-backed task files in test-cases/v1-lite/.
Example walkthrough
The task walkthrough follows a V1 task from its JSON specification through the recorded browser session and human-reference assessment. It also explains why a placeholder interception rule needs trajectory evaluation.
Evaluation
The native runner scores intercepted requests with an LLM judge by default: a pass requires both an interception and a matching judge verdict. --no-judge reports interception-only results; clawbench-rescore can judge saved requests later. See the CLI reference.
The original V1 evaluation compares complete agent and human-reference trajectories under eval/agentic_eval.md. This is a separate workflow described in the evaluation guide. Compare scores using the same corpus revision, harness, and rubric.
To share rescored results, see the OpenEval / EvalPort export instructions.
CLI
./run.sh # interactive TUI
uv run clawbench-run
Command overview and commonly used flags: CLI reference. Run any command with --help for its complete option list.
V1 tasks are in test-cases/v1/ (152 tasks), V2 in test-cases/v2/ (129), Lite in test-cases/v1-lite/ (20), and converted Claw-Eval tasks in test-cases/claw-eval/ (19). All suites use test-cases/task.schema.json. For test case authoring, see CONTRIBUTING.md; for output structure and evaluation guidance, see eval/README.md.
How ClawBench compares
Benchmark Domain Environment Task count ClawBench difference
WebArena Synthetic web apps Self-hosted replicas 812 Live consumer sites, not admin UIs on hosted replicas
GAIA General assistants Closed-book text + tools 466 Browser-centric; end-to-end task execution
SWE-bench Software engineering GitHub repos 2,294 Non-code; everyday consumer workflows
BrowserGym Web agents Headless sandbox — Cloud-parity; records real user journeys
Mind2Web Web navigation Static traces 2,350 Dynamic live websites, not replayed traces
Online-Mind2Web Live web navigation Real websites 300 V1+V2: 281 tasks, with full five-layer recordings
VisualWebArena Visual web tasks Self-hosted (3 sites) 910 Real websites with full visual layer (vs 3 hosted apps)
WebVoyager Real-website nav Real websites (15) 643 Interception-graded vs LLM-judge-only, 143 sites covered
TheAgentCompany Office workflows Self-hosted (6 platforms) 175 Consumer everyday tasks instead of enterprise sandbox
ClawBench's niche: live consumer websites, everyday tasks, end-to-end recording. If you want a controlled sandbox or replayed traces, the projects above are excellent. If you want to know whether your agent can actually order food or book a flight today, this is the benchmark for that.
FAQ
See the FAQ for task coverage, account setup, recording artifacts, scoring, and live-site limitations. For installation and model configuration, use Getting started.
Contributing
We welcome contributions -- especially new test cases. If you've ever ordered groceries, booked an appointment, or filed a form online, you already know how to write one. Most PRs are a single JSON file and land in under a day.
Quick wins:
Add a new test case (~30 min, no container expertise needed)
Add a new category of 10+ tasks → co-author invitation on the next paper revision
Submit a new model to the public leaderboard
Browse good first issues
See CONTRIBUTING.md for the full guide and contributor recognition policy.
Community
Come hang out with researchers, builders, and contributors working on real-world browser agents.
Questions & bugsFastest route to a maintainer
中文社区研究者、开发者、贡献者交流
Dataset & leaderboard talkOn the Hub, next to the data
Citation
If you use ClawBench in your research, please cite:
@misc{zhang2026clawbenchaiagentscomplete, title = {ClawBench: Can AI Agents Complete Everyday Online Tasks?}, author = {Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen}, year = {2026}, eprint = {2604.08523}, archivePrefix = {arXiv}, primaryClass = {cs.CL}, url = {https://arxiv.org/abs/2604.08523} }
Contact
Questions, suggestions, or research collaboration? Reach the maintainer:
Yuxuan Zhang — reacher [at] cs.ubc.ca (UBC, NAIL Group) · Homepage ↗
For bug reports or feature requests, please open a GitHub issue — it's faster than email and gets seen by all maintainers.
Core Contributors
Yuxuan Zhang
Yubo Wang
Perry Zhu
Penghui Du
Junwen Miao
Advisors
Kelsey R. Allen
Wenhu Chen
Dongfu Jiang
Liang Chen
Support ClawBench
If ClawBench is useful for your research or product work, the single most helpful thing you can do is star the repo — it surfaces the benchmark to other AI-agent researchers and helps us justify continued dataset curation.
Open to contributions — new test cases, bug fixes, or evaluation submissions for a model we haven't scored yet. See CONTRIBUTING.md.
Star History
License & Acknowledgments
Apache 2.0 -- see LICENSE.
The converted Claw-Eval suite in test-cases/claw-eval/ is derived from claw-eval/claw-eval and the claw-eval/Claw-Eval dataset, which are released under the MIT License. Third-party package notices are in NOTICE.
Built with OpenClaw, opencode, Claude Code, the Claude in Chrome extension, OpenAI Codex CLI, browser-use, claw-code, Hermes Agent, Pi with pi-browser-harness, and WebBrain (selectable harnesses), Microsoft Playwright MCP (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), LiteLLM (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), noVNC (MPL 2.0), and websockify (LGPL 3.0).
Source: https://github.com/TIGER-AI-Lab/ClawBench?utm_source=trendshift.io
