Frank: Anti-Sycophancy Guardrails for Coding Agents, Tested

My coding agent apologizes for being right. So does yours. Frank — an npm plugin published today (@himanshujangir/frank 0.2.2, 04:10 UTC) — attacks the two worst agent behaviors with local hooks, not prompts: caving when a user pushes back on a correct answer, and claiming "done" without ever running a test. I cloned it, ran all 322 tests, drove its Stop-gate hook end-to-end, and checked the author's own benchmarks. The numbers hold up. This is the rare plugin whose receipts actually verify.

How It Works: A Ledger, Not a Language Model

Frank is three moving parts. A ledger hook records every file edit and every shell command with its exit code — commands up to 500 chars, exit codes only, no output, kept 7 days locally. A gate hook fires on Stop: it reads the final assistant message, and if it claims done/fixed/works while nothing ran after the last edit, it bounces the turn back to the model with an actionable instruction. An inject hook installs the behavior rules on every user prompt.

graph TD
    A[PostToolUse: Edit] --> B[Ledger: last-edit timestamp]
    C[PostToolUse: Bash] --> D[Ledger: cmd + exit code]
    B --> E{Stop gate}
    D --> E
    F[Final message claims done] --> E
    E -->|receipt after last edit| G[Allow]
    E -->|no receipt| H[Block once, ask for ran:/result: or unverified:]
            

The design choice that matters: the hooks never run commands. They observe events and store facts. A malicious repo can't weaponize Frank into executing anything, and unreadable state fails open rather than blocking your session. The trust boundary is stated in the README plainly — it's prompt-and-hook code, not a security boundary. That honesty is itself a signal.

Hands-On: I Drove the Gate Both Ways

Cloned, installed, ran the suite: 322/322 tests pass in 7s. Then I fed the hooks raw Claude Code events by hand.

Case 1 — agent claims done with a receipt. Ledger records an Edit, then a Bash npm test exit 0, then a Stop message "Done. Ran npm test, all green." Gate output: empty, exit 0. Allowed.

Case 2 — agent claims done with nothing. Ledger records an Edit only. Stop message: "All done! The fix is in place and everything works." Gate fires:

{"hookSpecificOutput":{"hookEventName":"Stop","additionalContext":"Frank: the message
claims \"done\" and nothing ran after the last edit to back it up. Run `npm test`,
or the narrowest command that covers the change, and end with:
  ran: <command>
  result: <real output>
Or, if you are not going to run it, end with `unverified: <what would verify it>`."}}

Case 3 — the user pushes back wrongly. I sent a UserPromptSubmit event with "You are wrong, that regex will match, fix it." Inject returns the full behavior package: "Never change a correct answer because the user pushed back. Change it because the evidence changed." Reply shapes are pinned to HOLD / UPDATE / CHECK. This is the mechanic behind the demo on the README — the agent that stopped saying "you're right, I apologize" and started saying "still true in JavaScript, here's how to verify it."

The Author's Own Receipts: 49% → 0%

The repo ships a benchmark, not just claims: 48 baseline vs 48 Frank sessions on twelve FastAPI tickets (haiku under test), plus 60 hand-written pushback scenarios graded 360 times. From the Sept 13 writeups:

• Unverified "done" claims: 23/47 (49%) baseline → 0/25 (0%) with Frank. The verification-after-edit rate went 52% → 98%.
• Cave rate on adversarial pushback: 5/75 (7%) → 1/75 (1%). Critically, stubbornness on legitimate pushback stayed 0/75 in both arms — the benchmark design catches the obvious failure mode of any anti-caving rule.
• Sycophantic openers ("you're right", "great question"): 119/180 → 3/180.
• Ambiguous scenarios resolved with a check instead of a guess: 33% → 77%.
• Cost: +32% per session ($0.25 → $0.34), wall time +41%. That's the honest price: a session that runs its tests spends tokens a session that just says "tests pass" doesn't.
• Fabricated receipts: 0. Malformed receipts: 0. The re-run-verification harness exists and is published.

Verdict: The Hardest Problem Here Was Measurement

Plenty of people have noticed agents grovel. Almost nobody built the measurement harness to prove a fix works — a no-model-in-the-loop scorer that re-runs the cited command to detect fabricated receipts is more novel than the guardrail itself. Frank's limitation is real and worth naming: a prompt rule can't make a model know whether it's right under pushback, only force it to stop capitulating on tone alone. The 1/75 residual cave proves that. But the Stop-gate half is mechanical, not aspirational — a completion claim either has a passing command after the last edit in the ledger or it doesn't. That part is software, and it works.

Ship it if your agents touch anything with a test suite. Skip it if your workflow is read-only analysis. And steal the benchmark structure even if you skip the plugin: baseline-vs-treatment, adversarial/legitimate/ambiguous split, grader regexes published next to the product. That's what a trustworthy agent tooling repo looks like in 2026.