AgentIndex icon
AgentIndex
coder_eval logo
coder_eval
★ 154
vs
mcp-nixos logo
mcp-nixos
★ 864

coder_eval vs mcp-nixos

coder_eval: Coder Eval runs a real coding agent (Claude Code, Codex, Antigravity, OpenCode or Pi) in a sandbox against declarative YAML tasks and scores the files and commands it produced. It supports weighted 0.0-1.0 criteria, skill-trigger checks, A/B experiments and cost telemetry, and can gate CI on agent quality.; mcp-nixos: MCP-NixOS is an MCP server that provides real-time, accurate information about NixOS packages, options, Home Manager, nix-darwin, and more. It works on any system without requiring Nix, using just two tools to query multiple sources.

01

TL;DR

coder_eval logo
coder_evalDev Tooling · ★ 154

Benchmark coding agents on your own domain tasks

Read coder_eval full review →
mcp-nixos logo
mcp-nixosDev Tooling · ★ 864

AI assistant queries NixOS package information

Read mcp-nixos full review →
02

Side-by-Side Comparison

Field
coder_eval logocoder_eval
mcp-nixos logomcp-nixos
Category
Dev Tooling
Dev Tooling
Stars
★ 154
★ 864
License
Apache-2.0
MIT
Install Command
uv tool install coder-eval
uvx mcp-nixos
Updated
1d ago
1mo ago
Open Source
Yes
Yes
Website
↗ Visit
↗ Visit
GitHub
↗ GitHub
↗ GitHub
Tags
agent-evaluation, agent-skills, agent-testing
ai-assistant, ai-integration, ai-tools
03

Features

coder_eval logocoder_eval
01Declarative YAML tasks with pinned dependencies and success criteria
02Sandboxed execution with resource limits
03Weighted 0.0-1.0 scoring, including LLM-graded rubrics and skill_triggered checks
04A/B experiments across agents, models, tools and prompts
05Telemetry for every tool call, token count and cost
mcp-nixos logomcp-nixos
01Unified query tool with multiple actions and sources
02Package version history with commit hashes
03Binary cache status checking
04Local flake inputs exploration
05Works without Nix installation
04

Use Cases

coder_eval logocoder_eval
↳Benchmark coding agents on your own domain tasks
↳Verify that a skill actually triggers in the agent you ship for
↳Fail CI builds when agent or skill quality regresses
mcp-nixos logomcp-nixos
↳AI assistant queries NixOS package information
↳Browse Home Manager or nix-darwin configuration options
↳Check if a package is available in binary cache
FAQ

FAQ

What is the difference between coder_eval and mcp-nixos?
Both coder_eval and mcp-nixos are in the Dev Tooling category. coder_eval has 154 stars, while mcp-nixos has 864 stars.
Which is better, coder_eval or mcp-nixos?
The best choice depends on your use case. Choose coder_eval if Benchmark coding agents on your own domain tasks, and mcp-nixos if AI assistant queries NixOS package information.
Is coder_eval free or open source?
Yes, coder_eval is open source on GitHub (Apache-2.0).
Is mcp-nixos free or open source?
Yes, mcp-nixos is open source on GitHub (MIT).
Cloud Infrastructure
Ad

Deploy on DigitalOcean

Host your MCP servers, Docker containers, and AI agents with $200 free credit over 60 days.

Claim $200 Free Credit
→

Related

Alternatives to coder_eval →Alternatives to mcp-nixos →coder_eval details →mcp-nixos details →