AgentIndex icon
AgentIndex
coder_eval logo
coder_eval
★ 154
vs
Auto-claude-code-research-in-sleep logo
Auto-claude-code-research-in-sleep
★ 17.9k

coder_eval vs Auto-claude-code-research-in-sleep

coder_eval: Coder Eval runs a real coding agent (Claude Code, Codex, Antigravity, OpenCode or Pi) in a sandbox against declarative YAML tasks and scores the files and commands it produced. It supports weighted 0.0-1.0 criteria, skill-trigger checks, A/B experiments and cost telemetry, and can gate CI on agent quality.; Auto-claude-code-research-in-sleep: Auto-claude-code-research-in-sleep (ARIS) is a set of custom Claude Code skills for autonomous ML research workflows. It orchestrates cross-model collaboration, with Claude Code executing research tasks and an external LLM (like GPT-5.4) critically reviewing. This system can autonomously discover ideas, run experiments, and write/refine research papers, allowing researchers to wake up to ready-to-submit results.

01

TL;DR

coder_eval logo
coder_evalDev Tooling · ★ 154

Benchmark coding agents on your own domain tasks

Read coder_eval full review →
Auto-claude-code-research-in-sleep logo
Auto-claude-code-research-in-sleepWorkflow Automation · ★ 17.9k
When to choose:
  • You use Claude Code and want an autonomous skill loop for machine learning research that ideates, executes experiments, and reviews results overnight.
  • You want a dual-model peer-review mechanism where a secondary LLM audits research ideas and code before running.
When to skip:
  • You need an autonomous web-search deep research agent for general factual reports; GPT Researcher is tailored for text reports.
  • You do not use Claude Code CLI or prefer a graphical node-based workflow builder.
Read Auto-claude-code-research-in-sleep full review →
02

Side-by-Side Comparison

Field
coder_eval logocoder_eval
Auto-claude-code-research-in-sleep logoAuto-claude-code-research-in-sleep
Category
Dev Tooling
Workflow Automation
Stars
★ 154
★ 17.9k
License
Apache-2.0
MIT
Install Command
uv tool install coder-eval
git clone https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep.git && cp -r Auto-claude-code-research-in-sleep/skills/* ~/.claude/skills/ && npm install -g @openai/codex && claude mcp add codex -s user -- codex mcp-server
Updated
1d ago
1d ago
Open Source
Yes
Yes
Website
↗ Visit
↗ Visit
GitHub
↗ GitHub
↗ GitHub
Tags
agent-evaluation, agent-skills, agent-testing
ai-research, ai-tools, aris
03

Features

coder_eval logocoder_eval
01Declarative YAML tasks with pinned dependencies and success criteria
02Sandboxed execution with resource limits
03Weighted 0.0-1.0 scoring, including LLM-graded rubrics and skill_triggered checks
04A/B experiments across agents, models, tools and prompts
05Telemetry for every tool call, token count and cost
Auto-claude-code-research-in-sleep logoAuto-claude-code-research-in-sleep
0118 composable skills for flexible workflow chaining.
02Automated idea discovery including literature survey, brainstorming, novelty check, and GPU pilot experiments.
03Autonomous multi-round review loop to iteratively improve research with experiments.
04Comprehensive paper writing pipeline from narrative to submission-ready LaTeX/PDF.
05Cross-model collaboration for adversarial review, breaking single-model blind spots.
04

Use Cases

coder_eval logocoder_eval
↳Benchmark coding agents on your own domain tasks
↳Verify that a skill actually triggers in the agent you ship for
↳Fail CI builds when agent or skill quality regresses
Auto-claude-code-research-in-sleep logoAuto-claude-code-research-in-sleep
↳Explore new research areas and discover novel ideas through literature surveys and brainstorming.
↳Automate the iterative review and refinement of research projects, including running experiments, until submission-ready.
↳Transform research narratives into submission-ready academic papers with a full writing and formatting pipeline.
05

Best For

coder_eval logocoder_eval
—
Auto-claude-code-research-in-sleep logoAuto-claude-code-research-in-sleep
Most PopularTrending
FAQ

FAQ

What is the difference between coder_eval and Auto-claude-code-research-in-sleep?
Both coder_eval and Auto-claude-code-research-in-sleep are in the Dev Tooling category. coder_eval has 154 stars, while Auto-claude-code-research-in-sleep has 17.9k stars.
Which is better, coder_eval or Auto-claude-code-research-in-sleep?
The best choice depends on your use case. Choose coder_eval if Benchmark coding agents on your own domain tasks, and Auto-claude-code-research-in-sleep if Explore new research areas and discover novel ideas through literature surveys and brainstorming..
Is coder_eval free or open source?
Yes, coder_eval is open source on GitHub (Apache-2.0).
Is Auto-claude-code-research-in-sleep free or open source?
Yes, Auto-claude-code-research-in-sleep is open source on GitHub (MIT).
Cloud Infrastructure
Ad

Deploy on DigitalOcean

Host your MCP servers, Docker containers, and AI agents with $200 free credit over 60 days.

Claim $200 Free Credit
→

Related

Alternatives to coder_eval →Alternatives to Auto-claude-code-research-in-sleep →coder_eval details →Auto-claude-code-research-in-sleep details →