A committed slash-command is already an eval harness — swap the model, re-run, diff the outputs, and the diff is the measurement.

Principle

Prompts Are Evals

The insight

One artifact. Two jobs. No second file.

The committed slash-command directory is the prompt and the eval suite.

📄

/feature

.claude/commands/feature.md

Job 1 — Execute

The prompt the system invokes

Runs in production. Produces the planning doc, the code, the report.

• claude -p /feature ...
• → feature-plan.md

Job 2 — Evaluate

The eval harness for the agentic layer

Replayed on demand. Measures what changed when the model, mode, or tools changed.

• claude -p /feature --model=opus
• diff → isolates model delta

Treating it as only the first half leaves measurement on the table.

Why it works

Same prompt. Swap one variable. The diff isolates it.

Deterministic input + declared format + replayable invocation = an eval.

run A

/feature "add logout"

model:sonnet

thinking:low

tools:default

→ plan-A.md

(baseline)

⇅

diff

the only thing that changed is what you changed

run B

/feature "add logout"

model:opus

thinking:low

tools:default

→ plan-B.md

(upgrade under test)

Swap the model. Swap thinking mode. Swap allowed_tools. The prompt stays fixed so the diff stays clean.

What qualifies

A prompt is only an eval if the baseline is replayable.

✗ Not an eval

✗

Ad-hoc prose pasted into chat — no baseline to diff against.

✗

Prompt depends on today's date, a random ID, or session state — rerun drifts for the wrong reason.

✗

Output format undeclared — can't regression-check when the model "reorganizes" it.

✗

Committed but never re-run — model upgrades land invisibly.

✓ An eval

✓

Committed .md template — the text is under version control.

✓

Non-determinism declared as named variables substituted at invocation.

✓

Format section specifies the schema — JSON shape or markdown skeleton.

✓

Re-run on every model, thinking, or tool change — the diff is the metric.

If the prompt can't be replayed byte-for-byte, there is nothing to measure.

What breaks on violation

Four ways the eval harness silently disappears.

ad-hoc

"implement the thing I described in chat"

No replayable baseline. Every prompt is a one-off — no way to measure drift across model releases.

non-det

Non-deterministic input, undeclared

Date, random ID, or session state leaks in. Re-run produces different output for different reasons — the diff no longer isolates the model change.

no-fmt

Undeclared output format

The model "reorganizes" the output on a new version and breaks downstream parsing — you can't tell that's what happened.

dormant

Committed but never re-run

You own an eval harness you've never fired. Model upgrades happen invisibly; you find out in production.

Every violation turns measurement back into vibes.

The feedback loop

New model released → you already have a test suite.

Regression detection on the agentic layer — no separate eval file to maintain.

trigger

new model ships

→

re-run top 20

committed prompts

→

diff outputs

vs baseline

→

patch regressions

before rollout

The slash-command directory doubles as the regression suite. Upgrade becomes a measurement event, not a prayer.

The discipline

A prompt that can't be replayed cannot be measured.

Commit the template. Declare the variables. Specify the format. Then re-run it against every model, thinking mode, and tool config you care about — the diff is the only eval you need.