Five episodes of building and every improvement so far has been vibes-tuned. "The masking helps." "Sub-agents feel cleaner." Feels how, exactly? Compared to what? Tonight we close the season with the piece I should have written first if I'd been more disciplined: an evaluation harness with a pinned baseline, so the next time I tweak a system prompt I can tell within thirty minutes whether I made the agent better or worse.

The other thing tonight: prompt caching. We've been paying full input token price for the same system prompt and tool definitions on every turn since Ep.02. That was fine while we were exploring. It's absurd once we're running an eval that fires the agent 15 times back-to-back. Both live in the same commit because you cannot iterate on prompts without cheap iteration.

What we are building tonight

  1. evals/tasks/ — 15 verifiable SWE-lite style tasks, each a directory with setup.sh, task.md, and verify.sh
  2. evals/runner.ts — an mcc eval command that runs the agent against each task and records pass / fail / turns / tokens / cost
  3. Prompt caching wired into client.messages.create — cache_control on the system block and tools
  4. A baseline.json — the first commit-pinned run, so future runs report a delta, not a raw number

Why 15 tasks, and why "SWE-lite"

Fifteen is small enough to run in ~15 minutes and big enough to have signal — an 80% pass rate versus a 60% pass rate on 15 tasks is not statistical noise. Bigger than that and the eval stops being a thing you rerun after every prompt change.

"SWE-lite" means small, real, verifiable:

Here's what a single task looks like:

evals/tasks/03-fix-off-by-one/
├── setup.sh       # writes buggy code + failing test into $WORK_DIR
├── task.md        # "The test in sum_test.ts fails. Fix sum.ts so it passes."
└── verify.sh      # cd $WORK_DIR && npm test -- sum_test.ts

Fifteen of these gives you a signal channel. Fewer is anecdote.

The runner

Create evals/runner.ts:

import { readdir, readFile, mkdtemp, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import Anthropic from "@anthropic-ai/sdk";
import { TOOLS, runTool } from "../src/agent.js";

const execFileP = promisify(execFile);
const client = new Anthropic();
const MODEL = "claude-sonnet-4-5";
const MAX_TURNS = 30;

interface TaskResult {
  id: string;
  passed: boolean;
  turns: number;
  inputTokens: number;
  outputTokens: number;
  cacheReadTokens: number;
  cacheWriteTokens: number;
  costUsd: number;
  durationMs: number;
  error?: string;
}

async function runOneTask(taskDir: string, taskId: string): Promise<TaskResult> {
  const t0 = Date.now();
  const workDir = await mkdtemp(join(tmpdir(), `mcc-eval-${taskId}-`));
  process.env.WORK_DIR = workDir;
  process.env.MCC_AUTO_APPROVE = "1"; // no y/N inside eval

  try {
    await execFileP("sh", [join(taskDir, "setup.sh")], { env: process.env });
    const brief = await readFile(join(taskDir, "task.md"), "utf8");

    const history: Anthropic.MessageParam[] = [
      { role: "user", content: `Working directory: ${workDir}\n\n${brief}` },
    ];

    let totIn = 0, totOut = 0, totCacheR = 0, totCacheW = 0;
    let turns = 0;

    for (; turns < MAX_TURNS; turns++) {
      const response = await client.messages.create({
        model: MODEL,
        max_tokens: 4096,
        system: [
          {
            type: "text",
            text: "You are Mini Claude Code, running inside an eval harness. Complete the task and stop. Use tools decisively.",
            cache_control: { type: "ephemeral" },
          },
        ],
        tools: TOOLS.map((t, i) =>
          i === TOOLS.length - 1 ? { ...t, cache_control: { type: "ephemeral" } } : t
        ),
        messages: history,
      });

      totIn += response.usage.input_tokens;
      totOut += response.usage.output_tokens;
      totCacheR += response.usage.cache_read_input_tokens ?? 0;
      totCacheW += response.usage.cache_creation_input_tokens ?? 0;
      history.push({ role: "assistant", content: response.content });

      if (response.stop_reason !== "tool_use") break;

      const results = [];
      for (const b of response.content) {
        if (b.type === "tool_use") {
          const out = await runTool(b.name, b.input as Record<string, unknown>);
          results.push({ type: "tool_result" as const, tool_use_id: b.id, content: out });
        }
      }
      history.push({ role: "user", content: results });
    }

    let passed = false;
    try {
      await execFileP("sh", [join(taskDir, "verify.sh")], { env: process.env, timeout: 30_000 });
      passed = true;
    } catch { /* verify script exited non-zero */ }

    // Sonnet 4.5 pricing (USD per 1M tokens)
    const cost =
      (totIn - totCacheR - totCacheW) * 3 / 1e6 +
      totCacheW * 3.75 / 1e6 +
      totCacheR * 0.30 / 1e6 +
      totOut * 15 / 1e6;

    return {
      id: taskId, passed, turns,
      inputTokens: totIn, outputTokens: totOut,
      cacheReadTokens: totCacheR, cacheWriteTokens: totCacheW,
      costUsd: Number(cost.toFixed(4)),
      durationMs: Date.now() - t0,
    };
  } catch (e) {
    return {
      id: taskId, passed: false, turns: 0,
      inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0,
      costUsd: 0, durationMs: Date.now() - t0,
      error: e instanceof Error ? e.message : String(e),
    };
  } finally {
    await rm(workDir, { recursive: true, force: true });
  }
}

export async function runEval() {
  const tasksRoot = new URL("./tasks/", import.meta.url).pathname;
  const dirs = (await readdir(tasksRoot)).sort();
  const results: TaskResult[] = [];

  for (const d of dirs) {
    process.stdout.write(`[${d}] `);
    const r = await runOneTask(join(tasksRoot, d), d);
    results.push(r);
    process.stdout.write(`${r.passed ? "PASS" : "FAIL"}  ${r.turns}t  $${r.costUsd}\n`);
  }

  const passed = results.filter((r) => r.passed).length;
  const cost = results.reduce((s, r) => s + r.costUsd, 0);
  const cacheHitRate = results.reduce((s, r) => s + r.cacheReadTokens, 0) /
    Math.max(1, results.reduce((s, r) => s + r.inputTokens, 0));

  console.log(`\n=== ${passed}/${results.length} passed ===`);
  console.log(`Mean turns: ${(results.reduce((s, r) => s + r.turns, 0) / results.length).toFixed(1)}`);
  console.log(`Total cost: $${cost.toFixed(2)}`);
  console.log(`Cache hit rate: ${(cacheHitRate * 100).toFixed(1)}%`);

  return results;
}

Five ideas worth pausing on:

WORK_DIR env, not process.chdir. The agent's tools already respect a base path. Changing the whole Node process's cwd during a parallel batch is a foot-gun I've hit twice.

MCC_AUTO_APPROVE=1 inside eval. From Ep.03's confirmation gate. The eval cannot pause on y/N. If you don't want auto-approve, don't include apply_patch in eval-mode's tool set — but you almost always do want it, because patching is the interesting axis.

Full usage capture. input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens — all four. This is how you know whether prompt caching is actually working. First turn will show cache_creation; every subsequent turn should show cache_read. If not, the cache breakpoint is misplaced.

Cost computed, not guessed. Sonnet 4.5 pricing wired directly. Cache reads are 10× cheaper than fresh reads and 12.5× cheaper than cache writes. On a 30-turn task this compounds fast.

Verify script gets 30 seconds and no more. If verify.sh hangs you'll notice at task 3 and rage-quit. Timeout it.

Prompt caching: the two things that matter

Two cache_control: { type: "ephemeral" } markers, placed exactly here:

  1. On the last text block of system. Everything up to and including this block gets cached.
  2. On the last tool in the tools array. Same idea — the whole tools list up to that marker is cacheable.

That's it. Anthropic caches the prefix. Every subsequent turn within ~5 minutes reads instead of writes.

There are two things that will silently defeat you:

You mutated the tools array between turns. Any structural change (new tool, reordered tools, description edit) invalidates the cache. Keep the tools array literal and stable. If you need dynamic tools, split them out and put them after your cache breakpoint.

Your system prompt has a timestamp. Don't laugh; I've done this. "Current time: 2026-07-05T18:59:12Z" in the system block busts the cache every turn. Move volatile data into the user message.

The first baseline

Run once, commit baseline.json:

{
  "timestamp": "2026-07-05T19:14:22Z",
  "model": "claude-sonnet-4-5",
  "commit": "abc12345",
  "passed": 11,
  "total": 15,
  "meanTurns": 6.4,
  "totalCostUsd": 0.87,
  "cacheHitRate": 0.71,
  "byTask": [
    { "id": "01-add-null-check", "passed": true, "turns": 3, "costUsd": 0.031 },
    { "id": "02-rename-function", "passed": true, "turns": 5, "costUsd": 0.048 },
    { "id": "03-fix-off-by-one", "passed": true, "turns": 4, "costUsd": 0.041 },
    { "id": "04-extract-helper", "passed": false, "turns": 30, "costUsd": 0.152 }
  ]
}

The numbers above are my actual first run — 11/15, mean 6.4 turns, $0.87 total. Task 04 blew the turn cap because the agent kept re-reading files instead of remembering their contents; Ep.04's masking helped but it's still a soft spot.

The next time I change anything — system prompt, sub-agent budget, a new tool — I run mcc eval and compare. If pass rate drops, I revert. If mean turns rise but pass rate holds, I look at which tasks got slower. This is the loop I should have had from Episode 1.

What the 15 tasks look like

A honest split, roughly:

I intentionally under-specify a few of the hard ones. Real users under-specify. An agent that only works on well-formed briefs is not useful.

Pitfalls I hit while writing this

Cache breakpoint after a dynamic block. First draft I put cache_control on the system message string itself while inserting the current date at the top. Zero cache hits, first-run cost identical to no-caching. Move volatile data to the user turn or below the breakpoint.

Non-hermetic tasks. Task 07 in my draft depended on a globally installed jq. On my colleague's machine jq wasn't there, task always failed, baseline was wrong. setup.sh must install everything it needs, or the task must not depend on it. Verify locally on a clean shell.

verify.sh returning false positive. Task 12's verify.sh was grep -q pattern file. If the file didn't exist, grep still exited zero because I piped through || true somewhere. The agent scored a pass without doing anything. Delete every || true; let it explode.

Cost line lied. I initially added cache_read_input_tokens to input_tokens for total-in accounting. Anthropic's SDK returns them separately — input_tokens is non-cache only. Read the SDK types, not your memory of them.

What ships now, and what doesn't

Ship: this eval harness, this baseline, prompt caching, mcc eval command.

Not shipping in Ep.06: parallelism (run tasks concurrently), retry-on-transient-error, model comparison mode (Sonnet vs Haiku on the same set), a web dashboard. Each is a good weekend. None is on the critical path to "I can iterate on prompts without flying blind."

The whole series in one page

Six episodes, one agent, roughly 500 lines of TypeScript total:

| Ep | What we added | Files touched | |---|---|---| | 01 | 40-line REPL with streaming and history | agent.ts | | 02 | Three tools + tool-use loop | agent.ts | | 03 | apply_patch with dry-run and confirmation | agent.ts, patch.ts | | 04 | Observation masking + auto-continue | agent.ts | | 05 | Sub-agents with independent context | agent.ts, subagent.ts | | 06 | Eval harness + prompt caching + baseline | evals/runner.ts, evals/tasks/* |

If you built along, you now have something that looks and behaves very much like a small, well-scoped code agent. It is not Claude Code — Claude Code has years of hardening, dozens of tools, a real terminal UI, MCP, sandboxing, and a permission model I've handwaved. But it is a working agent whose every design decision you understand and can change tonight.

The most useful thing you now own is not the code. It's the eval harness. Everything you build from here — new tools, new sub-agent shapes, new masking heuristics, model swaps — passes through Ep.06 or it doesn't ship.

Quick Reference — Episode 06

| What | Where | |---|---| | Runner entry | runEval() in evals/runner.ts | | Task shape | setup.sh + task.md + verify.sh | | Turn cap | 30 (a hard task should be able to breathe) | | Auto-approve | MCC_AUTO_APPROVE=1 set by runner | | Cache breakpoints | last system text block, last tool in tools | | Metrics recorded | pass, turns, in/out/cache-r/cache-w tokens, cost, ms | | Baseline location | evals/baseline.json, committed | | CLI command | mcc eval (thin wrapper around runEval) |

Minimum-viable cached request:

await client.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 4096,
  system: [{ type: "text", text: SYSTEM, cache_control: { type: "ephemeral" } }],
  tools: TOOLS.map((t, i) =>
    i === TOOLS.length - 1 ? { ...t, cache_control: { type: "ephemeral" } } : t
  ),
  messages: history,
});

Seven rules that survive past the series:

  1. Pin a baseline before you tune anything.
  2. Fifteen tasks is the smallest number that gives you signal.
  3. Hermetic tasks or no task.
  4. verify.sh must fail loudly; strip every || true.
  5. Cache breakpoints go after the stable prefix, not on volatile blocks.
  6. Log all four usage fields; anything else is guessing.
  7. Every future change to the agent runs through the eval before it lands.

That's the arc. Season one closes here. The agent is small, honest, and — for the first time tonight — measurable. If you extend it, extend the eval alongside it. That is the whole discipline.

Thanks for building along.