agent-eval-harness
A general-purpose framework for evaluating and testing the performance of command-line agents.
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: agent-eval-harness. It is used for: A general-purpose framework for evaluating and testing the performance of command-line agents. Full Skill content: https://321skill.com/skills/agent-eval-harness/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
agent-eval-harness is a general-purpose evaluation framework specifically designed for command-line agents. It addresses the pain point developers face when measuring and comparing the performance of different agents or the same agent under different configurations, where a standardized, automated evaluation process is often lacking.
Through a unified entry command, the tool supports four core modes: run, grade, compare, and calibrate. Users simply specify the evaluation mode and relevant parameters via a JSON configuration to drive the entire evaluation workflow—from executing the original tasks to comparing final results—generating a structured evaluation report.
It is well-suited for agent developers, test engineers, and researchers who need to conduct agent performance benchmarking. Whether developing new agent skills or optimizing prompts or logic for existing agents, this tool enables objective, reproducible evaluations.
The recommended usage is to follow its advocated standardized workflow: utilize the accompanying Skills (such as trial-runner and trial-adapters) to generate evaluation suites and adapters, avoiding manual coding of evaluation orchestration to ensure consistency and maintainability.
Key Features
Compared to fragmented evaluation scripts, it offers a standardized CLI interface and a complete dataflow pipeline (run -> grade -> compare), and enforces the generation of evaluation logic through Skills, ensuring the reproducibility and consistency of the evaluation process.
Limitations
Not suitable for evaluating agents that require complex GUI interactions or operate in non-command-line environments.
FAQ
How can I view a detailed comparison report of the evaluation results?
Use the `compare` mode. It compares two scored JSONL files and generates a JSON report containing metrics like exactPassAtK.
What should I do if a specific task fails during the evaluation?
Failure information is encoded in the corresponding `trial_result` line and does not cause the entire CLI process to fail; the evaluation workflow can continue.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-eval-harness/raw/index.md to read the original Skill definition (Markdown format) for agent-eval-harness, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-eval-harness/raw/index.md