Agent Skills ·Intermediate

Skill Evals

How this repo checks that its skills actually work - before they ship and on every change after. Layout: one folder per skill, evals/<skill-name/, mirroring the skill's name. These files never ship in an install; the skill folders contain only what runs at runtime.

Complete open-source code
Step-by-step tutorial

Technical sheet

Category: Agent Skills
Difficulty: Intermediate
Models: OpenAI, Claude
Frameworks: -
Runs locally?: Needs cloud APIs
Required keys: None

License Apache-2.0 · content curated and maintained by Doutor AI.

Full tutorial

How this repo checks that its skills actually work - before they ship and on

every change after. Layout: one folder per skill, evals/<skill-name>/,

mirroring the skill's name. These files never ship in an install; the skill

folders contain only what runs at runtime.

The tier model follows

addyosmani/agent-skills

same names, same jobs - plus two tiers of our own, because skills here ship

executable code and his don't:

Tier	What it checks	Runs	Cost
1. Structural	Frontmatter, naming, name==dir, unfilled placeholders, text-only prompt dumps (`tools/skill_lint.py --strict`)	CI	Free
1b. Security (ours)	Install lures, undeclared network calls, credential access, obfuscated payloads (`tools/skill_scanner.py`)	CI	Free
2. Trigger & routing	Positive prompts clear near-miss negatives on description vocabulary; with 2+ skills, positives rank their own skill first and no two descriptions near-collide (`tools/run_trigger_evals.py`)	CI	Free
2b. Deterministic scripts (ours)	The skill's bundled scripts do what they claim - every classifier, edge case, and output shape against synthetic fixtures (`<skill>/test_*.py`)	CI	Free, ~10s
3. Behavioral	An agent following the skill satisfies its `expectations[]` - `evals.json` uses skill-creator's schema verbatim, so its `run_eval.py`, benchmarking, and eval viewer work against our files unmodified	On demand	Tokens

Running

# Tiers 1–2b, exactly what CI runs - deterministic, git + Python only
python3 agent_skills/evals/tools/skill_lint.py agent_skills/project-graveyard --strict
python3 agent_skills/evals/tools/skill_scanner.py agent_skills
python3 agent_skills/evals/tools/run_trigger_evals.py
python3 agent_skills/evals/project-graveyard/test_graveyard.py

Tier 3 is on demand and spends tokens: each skill's evals.json is in

skill-creator's schema, so run it with Anthropic's own tooling (install the

skill-creator plugin and point run_eval.py at the file), or by hand - fresh

agent session, paste each prompt, grade against expectations[]. Cases marked

lexical: false in trigger-cases.json (reasoning-triggered, e.g. necromancer

mode) are only covered here. Re-run tier 3 whenever SKILL.md behavior

changes; re-run tier 2 whenever a description changes.

Track record

Not theater: tier 1 caught a symlink-path bug that silently disabled relapse

detection on macOS, and an external reviewer running it in a clean Linux

checkout caught a filesystem-ordering bug that mis-attributed the kill-chain.

Both fixed before merge. That's the job.

Adding a skill

New skill → new evals/<skill-name>/ with at minimum a deterministic

test_<skill>.py (self-contained, tempdir fixtures, exit 0/1) and a

trigger-cases.json. CI picks up test_*.py automatically.

From the same category

Agent SkillsIntermediate

Advisor Orchestrator Worker

One model is a bottleneck. A team with one brain, twenty hands, and a board advisor is not.

Claude
Gemini

Agent SkillsIntermediate

Commit Archaeologist Agent Skill

git blame tells you who last changed a line. Commit Archaeologist reconstructs why the line exists.

Agent SkillsIntermediate

Dependency Doctor Agent Skill

Dependency Doctor inspects one dependency manifest for surface-level, direct-manifest footguns. It catches unpinned versions, standard-library shadowi…

Need a tailor-made solution?

Not sure where to start with AI in your business? Get a free assessment and a tailored plan - no commitment.

Save hours every week with AI, no hassle

Doutor AI gives you 1,859 ready-to-copy AI prompts, by industry and in plain English - plus a complete technical library for those who want to go further.

See plans