coder_eval
Active·★ 154·Apache-2.0·Updated 2026-10-21
Coder Eval runs a real coding agent (Claude Code, Codex, Antigravity, OpenCode or Pi) in a sandbox against declarative YAML tasks and scores the files and commands it produced. It supports weighted 0.0-1.0 criteria, skill-trigger checks, A/B experiments and cost telemetry, and can gate CI on agent quality.
#agent-evaluation#agent-skills#agent-testing#anthropic#claude#claude-code#claude-code-plugins-marketplace#claude-code-skills
01
Features
01Declarative YAML tasks with pinned dependencies and success criteria
02Sandboxed execution with resource limits
03Weighted 0.0-1.0 scoring, including LLM-graded rubrics and skill_triggered checks
04A/B experiments across agents, models, tools and prompts
05Telemetry for every tool call, token count and cost
02
Compatibility
Claude Code
Default harness
Verified via docs
Codex
Harness
Verified via docs
OpenCode
Harness
Verified via docs
GitHub Actions
CI
Verified via docs
03
Quick start
1
$ uv tool install coder-eval
04
Use cases
↳Benchmark coding agents on your own domain tasks
↳Verify that a skill actually triggers in the agent you ship for
↳Fail CI builds when agent or skill quality regresses
05
Alternatives
⚡Top VS:
reticle★ 1.3k
AI agents can generate code, but still struggle to understand what they build. Reticle brings Jev-style machine-native runtime perception to web & desktop applications.
agents-best-practices★ 2.4k
Provider-neutral Agent Skill for Codex, Claude Code, and agentic harness design.
Genexus18MCP★ 38
GeneXus 18 MCP server for Claude, Cursor, and AI agents — read, edit, analyze KB objects (transactions, web panels, procedures, SDTs) over the Model Context Protocol.
Auto-claude-code-research-in-sleep★ 17.9k
ARIS ⚔️ (Auto-Research-In-Sleep) — Claude Code skills for autonomous ML research: cross-model review loops, idea discovery, and experiment automation via Codex MCP
Comments
Log in to leave a comment
No comments yet. Be the first!