AgentBench
AgentBench, from Tsinghua University's THUDM group, is a benchmark for evaluating large language models as agents (ICLR 2024). The current repository holds AgentBench FC, a function-calling version integrated with AgentRL, an end-to-end multi-task, multi-turn agent RL framework. The original prompt-based versions are still available as v0.1 and v0.2.
AgentBench FC ships fully containerised tasks: alfworld, a database benchmark, a knowledge-graph task, OS interaction and webshop. One Docker Compose command starts the AgentRL controller, a worker per task, a Freebase server and Redis, after you pull or build the task images and download the Freebase data. Results for many models are published on a public leaderboard.
It is aimed at researchers and teams comparing models for agent work rather than at building agents. The README warns that webshop needs about 16 GB of RAM to start and that the alfworld worker leaks memory and disk until restarted. Our data shows the last push on 2026-02-08.
Features
Why choose it
Trade-offs
Maintenance check
- Last commit
- 2026-02-08
- Commits, last 90 days
- 0
- Releases, last 90 days
- 0
- Latest release
- —
- Open issues
- 80
- Recent issues with at least one comment
- 12/28
Oldest week on the left; the latest four weeks are orange. Source: GitHub commit activity.
Status rule: archived on GitHub = Archived; last commit within 90 days = Active; 91 to 365 days = Slowing; older = Stale. All figures come from the GitHub API on the date shown.
Most discussed open issues
- #119 我该怎么解决这个问题,跑mind2web,不太清楚该如何操作这个任务,能给出一些具体的指导吗,谢谢18 comments · 2024-02-20
- #146 通过fastchat部署本地模型遇到的问题13 comments · 2024-07-04
- #63 cg和ltp的std都出现了问题:Error: Worker not responding7 comments · 2023-11-02
- #109 [Bug/Assistance] 6 comments · 2024-01-27
- #211 [Bug/Assistance]os任务进度一直为04 comments · 2026-01-20
Issue titles are quoted from GitHub. Open the link to check whether a problem still applies to the version you use.
Compatibility
Quick start
Use cases
Alternatives
Featured in these lists
Comments
No comments yet. Be the first!