AgentIndex icon
AgentIndex
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Submit Tool
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Home/
Observability/
AgentBench
AgentBench logo

AgentBench

Active·★ 3.8k·Apache-2.0·Updated 2026-02-08
★ Essential

AgentBench, from Tsinghua University's THUDM group, is a benchmark for evaluating large language models as agents (ICLR 2024). The current repository holds AgentBench FC, a function-calling version integrated with AgentRL, an end-to-end multi-task, multi-turn agent RL framework. The original prompt-based versions are still available as v0.1 and v0.2.

AgentBench FC ships fully containerised tasks: alfworld, a database benchmark, a knowledge-graph task, OS interaction and webshop. One Docker Compose command starts the AgentRL controller, a worker per task, a Freebase server and Redis, after you pull or build the task images and download the Freebase data. Results for many models are published on a public leaderboard.

It is aimed at researchers and teams comparing models for agent work rather than at building agents. The README warns that webshop needs about 16 GB of RAM to start and that the alfworld worker leaks memory and disk until restarted. Our data shows the last push on 2026-02-08.

This write-up is based on the project's README and public repository data. We have not hand-tested the tool. Data as of 2026-11-06.

License
Apache-2.0
Latest release
No GitHub releases
Last commit
2026-02-08
Contributors
13
Forks
296
Open issues and PRs
80
First published
2023-07-28

Source: GitHub, npm and PyPI public APIs.

#LLM Evaluation#Agent Benchmarking#Function Calling#Docker#Multi-task Learning
$ Install
$ pip install -r requirements.txt
↗ Visit site★ GitHub
01

Features

01Comprehensive LLM-as-Agent Evaluation across diverse environments.
02Function Calling integration for advanced agent interaction.
03Fully containerized deployment using Docker Compose for reproducibility.
04Multi-task and multi-turn interaction for realistic agent assessment.
05Extensible framework for adding new evaluation tasks.
02

Why choose it

+Well-known academic benchmark (ICLR 2024) with a public leaderboard.
+Five containerised tasks start with one Docker Compose command.
+Function-calling version matches how current models are used as agents; older versions remain available.
+Apache-2.0 licensed and integrated with the AgentRL training framework.
03

Trade-offs

!It evaluates models; it does not help you build an agent.
!Heavy setup: several Docker images, Freebase data and about 16 GB of RAM for webshop.
!The README notes that the alfworld worker leaks memory and disk until restarted.
04

Maintenance check

SlowingChecked: 2026-11-10
Last commit
2026-02-08
Commits, last 90 days
0
Releases, last 90 days
0
Latest release
—
Open issues
80
Recent issues with at least one comment
12/28
Commits per week, last 12 weeks
Commits per week, last 12 weeks: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

Oldest week on the left; the latest four weeks are orange. Source: GitHub commit activity.

Status rule: archived on GitHub = Archived; last commit within 90 days = Active; 91 to 365 days = Slowing; older = Stale. All figures come from the GitHub API on the date shown.

Most discussed open issues

  • #119 我该怎么解决这个问题,跑mind2web,不太清楚该如何操作这个任务,能给出一些具体的指导吗,谢谢18 comments · 2024-02-20
  • #146 通过fastchat部署本地模型遇到的问题13 comments · 2024-07-04
  • #63 cg和ltp的std都出现了问题:Error: Worker not responding7 comments · 2023-11-02
  • #109 [Bug/Assistance] 6 comments · 2024-01-27
  • #211 [Bug/Assistance]os任务进度一直为04 comments · 2026-01-20

Issue titles are quoted from GitHub. Open the link to check whether a problem still applies to the version you use.

05

Compatibility

Docker
Native
Verified via docs
Python
Native
Verified via docs
OpenAI API
Supported
Verified via docs
Large Language Models
Supported
Verified via docs
06

Quick start

1
$ pip install -r requirements.txt
07

Use cases

↳Systematically benchmark the performance of various LLM-based agents.
↳Develop and refine advanced LLM agent architectures and strategies.
↳Conduct academic research on the capabilities and limitations of agentic AI.
08

Alternatives

⚡Top VS:
VSmetamcp★2.8k→VSkeep★12.5k→VSopenstatus★9.3k→VSmcp-context-forge★4.9k→
metamcp logo
metamcp★ 2.8k
MCP Aggregator, Orchestrator, Middleware, Gateway in one docker
VSCompare→
Awesome-LLM-in-Social-Science logo
Awesome-LLM-in-Social-Science★ 652
Awesome papers involving LLMs in Social Science.
keep logo
keep★ 12.5k
The open-source AIOps and alert management platform
VSCompare→
openstatus logo
openstatus★ 9.3k
🫖 Status page with uptime monitoring & API monitoring as code 🫖
VSCompare→
mcp-context-forge logo
mcp-context-forge★ 4.9k
A Model Context Protocol (MCP) Gateway & Registry. Serves as a central management point for tools, resources, and prompts that can be accessed by MCP-compatible LLM applications. Converts REST API endpoints to MCP, composes virtual MCP servers with added security and observability, and converts between protocols (stdio, SSE, Streamable HTTP).
VSCompare→
archestra logo
archestra★ 4.5k
Archestra is an open-source enterprise AI platform: an internal chat assistant, LLM gateway, MCP gateway and registry, A2A gateway and agent runtime, built on SSO, RBAC, sandboxed execution, guardrails and OpenTelemetry.
VSCompare→
lmnr logo
lmnr★ 3.5k
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
VSCompare→
agentgateway logo
agentgateway★ 6.3k
Next Generation Agentic Proxy for AI Agents and MCP servers
VSCompare→
See all alternatives →

Featured in these lists

  • Best Open-Source AI Agent Observability and Tracing Tools

Comments

Log in to leave a comment

No comments yet. Be the first!

On this page
01Features02Why choose it03Trade-offs04Maintenance check05Compatibility06Quick start07Use cases08Alternatives
THUDM /
AgentBench
↗

Tsinghua's Apache-2.0 benchmark for evaluating LLMs as agents, now in a function-calling version with containerised tasks

3.8k296Apache-2.0
Last commit9mo ago
StatusActive
CategoryObservability
Contributors13
Links
Documentation↗Discussion↗Issues↗Releases↗
Cloud Infrastructure
Ad

Deploy on DigitalOcean

Host your MCP servers, Docker containers, and AI agents with $200 free credit over 60 days.

Claim $200 Free Credit
© 2026 AgentIndex.app|Built by a 10-year iOS Developer.
Best by use caseAboutContactAdvertisePrivacy PolicyTerms of UseQYSGitHubBuy me a coffee ☕

Browse by Category

MCP ServersCode AssistantWorkflow AutomationRAG / Knowledge BaseMulti-AgentBrowser AutomationLLM InfraDev ToolingObservability

Not affiliated with Anthropic, OpenAI or Microsoft.