The AI-Native SDLC, Audited
Anthropic's stage-by-stage playbook is the spine of this course — and the measurement literature, the DORA panel, and the playbook's own quietest sentence are the audit.
Spine text: The AI-Native SDLC playbook — Louis Claxton, Anthropic Applied AI, 21 August 2026
This is not a summary. The playbook is the spine — its plays are stated as written and clearly attributed. But it is a vendor document written by the team that sells the agent, and a course that only restates it teaches you to adopt rather than to decide.
So every module carries three kinds of block. The playbook's position is what it actually says. The audit is sourced pushback — from METR's RCT, the DORA panel, GitClear's longitudinal corpus, the automated-review literature, and in two places from the playbook contradicting itself. Verified marks numbers checked against primary sources in August 2026.
Module 5 is a simulator. Do it before you read modules 6–10. It forces you to commit to a control decision and write down why before it shows you anything, because choosing the enforcement mechanism is the single skill this whole document is really about.
Course Modules
- The claim the whole document rests onPremise
- The artifact chain — the actually portable ideaCore
- Plan and Design: old discipline, new priceStages 1–2
- Build: CLAUDE.md, skills, and one sentence you must not skimStage 3
- The Control BoardSimulator
- Test: the highest-ROI play in the documentStage 4
- Deploy: correlated blind spots and the anchored reviewerStage 5
- Governance as configuration, line by lineControls
- Measurement: why these indicators are gameableMetrics
- Adoption order — Monday, and neverSynthesis
The claim the whole document rests on
- State the playbook's premise precisely, and identify what it would take to falsify it
- Cite the strongest measured evidence against a naive reading of it
- Distinguish "build got faster" from "the system got faster" — the distinction the playbook makes and most readers drop
- Explain why the premise being partly wrong makes the playbook more useful, not less
What is actually being claimed
"Organizations have started using AI to write code at a speed unthinkable one year ago, yet the processes around the code haven't changed at the same pace. […] Build is no longer the constraint — the human-speed steps around it are."
Three consequences follow, and they are stated crisply: the bottleneck moves left and right of build; the controls "stop matching reality" because reviewing every line by hand "can't keep up once agents write most of the diff"; and governance costs rise because exceptions still route through committees that meet weekly.
Read carefully, this is a claim about queueing, not about productivity. It says one station in the line got faster and the others didn't, so work now piles up in front of the slow stations. That is a modest, testable, and largely correct claim about where inventory accumulates.
The trouble is that almost nobody reads it that way. It gets read as "AI made engineering fast, now fix your process to collect the winnings." That reading is not supported by the best available measurements, and the gap matters because the playbook's recommended remedies cost real money.
The audit: three findings the playbook does not cite
1. The direct RCT points the other way. METR randomised 16 experienced open-source developers across 246 real tasks in repositories they averaged five years of experience in. Forecast before the task: AI would cut completion time 24%. Measured result: completion time rose 19%. And after finishing — after living through the slowdown — participants still estimated AI had sped them up by about 20%. That is a roughly 40-point calibration error with the direction reversed (arXiv:2507.09089).
2. The panel data says faster and less stable, simultaneously. DORA's 2025 report found AI adoption positively correlated with throughput — a reversal from 2024 — while continuing to correlate negatively with delivery stability: more change failures, more rework, longer time to restore. DORA's own framing is that AI is an amplifier, not a fix: it magnifies whatever the organisation already was (2025 DORA report).
3. The code itself is drifting. GitClear's 2026 corpus — 623 million changes, 2023–2026 — reports refactoring line-moves down 70%, cross-file function calls (a reuse proxy) down 35%, duplicated code blocks up 81%, error-masking constructs up 47%, and two-week churn up 15% (The Maintainability Gap).
None of the three is a knockout, and a course that pretends otherwise is doing the same thing the playbook does in the other direction.
- METR is n=16 on mature repos the developers already knew intimately, using early-2025 tooling. That is close to the worst case for AI assistance and the authors say so. It is not evidence that AI slows down greenfield work or unfamiliar codebases.
- DORA is survey-based and correlational. "AI adoption correlates with instability" is also exactly what you would see if the teams with the worst delivery pain adopted AI hardest.
- GitClear measures diffs, not defects. Duplication is a maintainability proxy, not an outcome. And their own 2026 finding that heavy AI users out-produce non-users 4–10× but most of that gap pre-dated AI is a selection-effect warning aimed at their own data.
The position this course takes
The premise is right about queueing and wrong about celebration. Build did get faster. The evidence that the system got faster is weak, and the evidence that it got less stable is the most consistent signal in the field. Those two facts together do not undermine the playbook — they are the strongest possible argument for it. If AI amplifies whatever your delivery system already is, then the playbook's real subject is not speed. It is building the control system that makes the amplification survivable.
Read the whole document with that substitution and it improves considerably. Every play that tightens a feedback loop or converts a habit into an enforced constraint is load-bearing. Every play that mainly removes a human handoff is a throughput play, and throughput without the control system is precisely the DORA failure mode.
DORA 2025: 90% of technology professionals now use AI at work and 80%+ believe it raised their productivity; roughly 30% report little or no trust in AI output. Both halves of that sentence are true of the same population at the same time. Any strategy that assumes uniform trust or uniform distrust is modelling a workforce that does not exist.
- The premise is a queueing claim. Treat it as one and it is defensible; treat it as a productivity claim and the RCT contradicts it.
- Practitioners' self-reported speedup is not evidence. METR's participants were wrong about the sign, not just the size.
- The stability half of the DORA finding is the durable one and it is what the playbook is actually a remedy for.
- Substituting "control system" for "speed" as the document's subject makes it a better document.
The artifact chain — the actually portable idea
- State the artifact chain and what each link buys that a ticket does not
- Explain why "the chain of commits is the audit trail" is the document's single best sentence
- Identify what is genuinely new here versus what is a 1990s idea with the cost removed
- Choose a source-of-truth configuration when a legacy tracker already owns the record
The chain
"Each stage ends by writing one to version control (includingintent.md,spec.md,plan.md, the diff and its tests, the PR with its review findings, and the incident record) and the next stage begins by reading it. […] The chain of commits is also the audit trail: who asked for what, what the agent produced, and who approved it."
And the trigger discipline that goes with it: an accepted intent.md fires the design
pass, an approved spec.md fires plan mode, a merged PR fires the pipeline, a breached
control band in production writes the next intent.md.
Strip the AI framing and ask what this structure does. It gives you, for any change in production,
a single git log that answers: what problem was claimed, what was specified, what was
planned, what was built, what review found, who approved. Every one of those is timestamped by an
authority nobody in the process controls.
What is new, and what is not
Requirements traceability — a linked chain from stated need to design to implementation to test evidence — is the oldest idea in regulated software engineering. RUP had it. CMMI required it. DO-178C, IEC 62304 and ISO 26262 mandate versions of it today. The traceability matrix is not an Anthropic invention and the playbook does not claim it is.
What changed is price. Traceability failed in practice for thirty years for exactly
one reason: producing and maintaining the artifacts cost more than the artifacts were worth to the
people producing them, so they were written after the fact to satisfy an auditor and were therefore
fiction. An agent that drafts intent.md in ten minutes from a conversation, and a
plan.md generated before any code exists, collapse that cost.
So the correct claim is not "AI enables traceability." It is "AI made traceability cheap enough to be true rather than performed." That is a smaller claim and a much more defensible one, and it generalises: it holds whether or not any of the productivity numbers in Module 1 hold.
Cheap artifacts are cheap to fake. A plan.md generated after the diff, or an
intent.md back-filled to make a merged change look governed, produces a chain that reads
perfectly and certifies nothing — the same fiction as the old traceability matrix, now with a
convincing timestamp.
The defence is ordering, not content: the value of the chain comes entirely from
plan.md being committed before the diff exists. That is checkable
mechanically — compare commit timestamps — and it is the one property of the whole chain
worth enforcing with a hook. The playbook comes close (it suggests a hook to keep plan and
implementation synchronised) but never says the ordering itself is the control.
Source of truth when a tracker already owns the record
The playbook's sidebar on legacy systems is unglamorous and is the most immediately practical page in it. Three configurations, pick one per artifact:
| Configuration | Mechanism | When it is right |
|---|---|---|
| Repo is truth | Markdown artifacts are authoritative; Jira/ServiceNow reference commits | Engineering-led orgs. One timestamp authority, one tool. Cleanest, hardest to sell to a change board. |
| Legacy is truth | Tracker holds the record; markdown are working copies; Claude reads and writes back over MCP in-session | Regulated environments where the auditor already accepts the tracker. Costs you the single timestamp authority. |
| Linkage only | Artifacts carry the record ID; records carry the commit SHA | The honest starting point. Two sources of truth, explicitly acknowledged rather than accidentally created. |
Position: start at linkage, and be suspicious of anyone who wants to start at "repo is truth" in an organisation with a functioning change board. The migration cost is not technical — it is convincing an auditor to accept a new evidence store, and that is a twelve-month conversation you should not begin in week one of an AI programme.
- The artifact chain is the most transferable idea in the playbook and survives every criticism in Module 1.
- It is traceability with the cost removed — not a new concept, a newly affordable one.
- The only property worth enforcing mechanically is ordering: plan before diff, checkable from timestamps.
- Pick a source of truth per artifact. Linkage first; do not fight the change board in week one.
Plan and Design: old discipline, new price
- Describe the intent→spec collapse and the accountability question it raises
- Judge the "survival rate" metric the playbook proposes and find its gaming path
- Argue both sides of "the product owner reviews the spec but doesn't write it"
The collapse of requirements and design into one session
"Both phases happen in a single prompted session. Claude takes intent.md and
produces a requirements and design spec, constrained by the organization's skills, with areas of
concern flagged. […] The product owner reviews that spec, but doesn't write it."
The prompt it recommends is worth quoting because the last clause does most of the work: "Describe clearly any areas of concern, especially where you cannot satisfy contradicting policies."
That last clause is the good part. An analyst's real job was never transcription — it was noticing that the security standard and the UX standard cannot both be satisfied here, and escalating before anyone builds. Asking explicitly for the contradictions is asking the model to do the part that was actually valuable, and it is cheap to add to any prompt you already run.
The audit: reviewing is a weaker act than writing
"Reviews but doesn't write" assumes review catches what authorship would have caught. It does not, and the effect is measurable in the adjacent literature: when reviewers are handed an automated review, they concentrate on the locations the automation commented on and attend less to everything else (Automated Code Review In Practice). The generated spec sets the agenda for its own review. A concern the model did not flag is a concern the product owner is now statistically less likely to raise, because the flagged list looks like the list.
The systematic-review number on automation bias generally — erroneous automated advice followed at a 26% higher rate when automated recommendations are present — points the same direction.
This is not an argument against the play. It is an argument that the flagged-concerns list must never be the only prompt for the reviewer. The cheap countermeasure: have the product owner write down, before opening the spec, the two or three things they expect to be hard. Then read. Anything on their list that is absent from the spec's list is the finding.
The metric, and how it will be gamed
The playbook proposes survival rate as the lagging indicator for Plan: the share
of intent.md files a product owner accepts into Design rather than closes.
Think about which direction is "good." A high survival rate could mean intent is being captured well. It could equally mean the product owner has stopped filtering — which is exactly what happens when the queue triples in size and closing an item now requires writing a justification that the originator will read. A metric whose failure mode and success mode move the same direction is not a metric.
The repair is to pair it: survival rate and the playbook's other Plan indicator —
edits to intent.md made after the first spec.md commit. Rising survival with
rising post-spec churn means the filter broke. Rising survival with flat churn means it is working.
Neither number means anything alone, and the playbook lists them separately without saying so.
- Ask for contradictions explicitly — it is the highest-value clause in the design prompt and costs nothing.
- A generated flagged-concerns list anchors its own reviewer. Write your expectations before you read the spec.
- Survival rate is ambiguous alone; it only carries signal paired with post-spec churn.
Build: CLAUDE.md, skills, and one sentence you must not skim
- Distinguish CLAUDE.md, skills, hooks and managed settings by enforcement strength, not by feature
- Quote the playbook's own admission about skills and state what follows from it
- Apply the "twice = write it down" rule and know why keeping CLAUDE.md short is a technical constraint
CLAUDE.md: the cheapest play in the document
Run /init, cut the output down to what a new joiner needs on day one, commit it at the
repo root. Build, test and lint commands with an example of healthy output; the conventions that
matter; the mistakes the team keeps seeing. The working rule the playbook gives is the whole play:
when Claude makes the same mistake twice, the correction goes into CLAUDE.md.
CLAUDE.md is read in full at the start of every session. Every stale line is context spent on nothing, in a budget that the actual task also needs. A 400-line CLAUDE.md is not a thorough CLAUDE.md; it is a tax levied on every session in the repository forever. Prune it like you would prune a cache, not like you would prune a wiki.
The sentence
"A skill is a control, though an advisory one. It makes Claude likely to apply the policy while the code is written, and nothing forces a session to comply with it. A policy that must always hold needs something deterministic behind the skill, such as a hook that blocks the action or a review pass that re-checks the policy at the PR. The skill makes violations rare and the hook makes them close to impossible."
That paragraph is buried in the middle of the Build stage, and it is the most important thing in the document. It says, in a vendor's own words, that the mechanism most organisations will reach for first — write the policy as a skill — is not a control in the sense a compliance function means the word.
Take it seriously and a hierarchy falls out. This table is the spine of the simulator in Module 5.
| Mechanism | Enforcement | Defeated by | Use it for |
|---|---|---|---|
| Prompt instruction | None | The next session | One-off steering. Never policy. |
CLAUDE.md | Advisory | Context pressure; the model simply not doing it | Conventions, commands, repeated mistakes |
| Skill | Advisory | Not triggering; drifting from the real policy | Institutional knowledge applied consistently — and only that |
Hook (project settings.json) | Deterministic | An engineer editing the file; a command the matcher misses | Guardrails a team owns and can be trusted to keep |
| Managed settings (MDM/admin) | Deterministic, unfalsifiable locally | Nothing on the engineer's machine | Non-negotiable organisational policy |
| Sandbox / network policy | OS-level | Nothing in-process | Egress, credential and filesystem boundaries |
| Branch protection + code owners | Deterministic, external | Admin override (which is logged) | Separation of duties. The agent cannot approve its own work. |
| CI required check | Deterministic, external | Check being made non-required | Anything expensive enough to belong at the PR, not the keystroke |
The temptation is to write a skill for every policy, because a skill is a markdown file and a hook is a shell script that has to be maintained. That produces a control library that reads like a compliance programme and enforces nothing. It will pass an internal review, because internal reviews read documents.
The test: could the policy be violated by a session that simply did not load the skill? If yes, and the policy must always hold, you do not have a control — you have documentation with good intentions. Skills are for making the right thing likely and legible. Hooks, managed settings, sandboxing and branch protection are for making the wrong thing impossible. Most policies need both, and the playbook says so; the sentence is just easy to read past.
- Classify every control by enforcement strength before you write it. Advisory and deterministic are different categories, not different intensities.
- A skill that carries a must-always-hold policy without a deterministic backstop is a paper control.
- CLAUDE.md brevity is a context-budget constraint, not an aesthetic preference.
- Next module: you will make eight of these calls yourself, and you will not be shown the answer until you commit.
The Control Board
- Map a stated policy requirement onto the weakest sufficient enforcement mechanism
- Recognise a paper control while you are writing it, not during an audit
- Predict how a string-matching guardrail fails on an infinite command space
- Transfer the reasoning to a surface where hooks do not exist
Now make the calls yourself
Reading the enforcement table in Module 4 is not the same as choosing from it under pressure with a policy owner waiting. Below is a simulator. It gives you five requirements of the kind that actually arrive, and it will not show you any consequence until you have committed to a mechanism and written down why.
It is not a quiz — in most scenarios more than one option is defensible, and the reveal shows what every option would have done, not just yours. But it keeps a paper-control ledger. Every time you satisfy a must-always-hold requirement with an advisory mechanism, it lands there. In scenario five a regulator arrives, and the finding is scaled to what you actually accrued. That second-order effect is the part the playbook leaves out.
The Control Board
Scenario 1 of 5- None yet. Any must-hold policy you satisfy with an advisory mechanism lands here.
Transfer check — a surface the simulator never showed you
Every scenario above assumed a repository, a Claude Code session, and a hook runtime. This one deliberately removes all three, and the mechanism involved was never demonstrated to you. Answer from the principles, not from pattern-matching. Write your answers first — the reveal is below them.
intent.md and committing it through a GitHub connector. Your privacy policy says
no customer PII may enter an intent artifact. There is no repo checkout on their machine, no
CLAUDE.md, and no hook runtime. Where does the control live?Reveal — and where this is still genuinely argued
1. When you cannot control the runtime, control the destination. The artifact has
to cross a boundary you own: the connector writes to the intent repository, and that repository is
yours. So the control is a required CI check on the intent repo that scans incoming
intent.md for PII patterns and fails the PR, with branch protection meaning the
connector cannot merge directly. Two organisational controls sit behind it — the managed MCP
allowlist decides which connectors exist at all, and the plugin marketplace restriction decides which
skills those users can load. The advisory layer (a shared intent template encoded as a skill) still
earns its place: it makes the violation rare. The CI check makes it not ship.
2. Hooks. A hook is a script that runs on the machine where the agent's tools execute, immediately before or after a tool call. On a hosted chat surface there is no such machine under your control and no tool-call boundary you can interpose on. Everything else in the table has an analogue — advisory controls port as skills, deterministic controls port to the destination repo and to admin settings — but the per-action deterministic gate is gone. Recognising that a control class is absent rather than merely inconvenient is the whole skill here.
3. The residual risk is context, not artifacts. Every control named above acts on
what gets committed. None of them act on what the originator typed into the
conversation to get there. If a claims handler pastes a real customer record in while
brainstorming, your CI check on the resulting intent.md can be perfectly clean and the
PII has still left the boundary. That is closed by data-handling policy, admin retention settings and
training — not by anything in the SDLC. Write it down as accepted risk with a named owner, or it
will be discovered for you.
Where reasonable people split: some security teams argue the correct answer to question 1 is "don't put non-engineers on this surface at all" — keep intent capture inside the engineering perimeter and accept the handoff cost the playbook is trying to delete. That is a coherent position and it is the one most regulated firms will actually take in year one. It trades the entire benefit of Stage 1 for a control boundary that already has an auditor's signature on it, and whether that trade is right depends on your data classification, not on your enthusiasm.
- Match enforcement strength to the words in the requirement. "Should" takes a skill; "must always" takes something deterministic.
- String-matching guardrails fail on an infinite command space. Constrain the capability, not the phrasing.
- A verifier that runs the artifact the agent was allowed to weaken verifies nothing.
- Paper controls are invisible until an auditor asks for evidence, and then they are all visible at once.
Test: the highest-ROI play in the document
- Explain why the feedback loop directly addresses the DORA stability finding
- Set a verification target that is checkable without you
- State what an eval suite actually protects and what breaks it
"Always give Claude a way to verify its own work, whether tests, a build, or a screenshot diff. […] Claude iterates until the check passes, so what reaches the engineer has already passed it."
Prerequisites: none. Infrastructure: a test suite and a build that each run with one command. That is the entire cost.
Why this one first
Module 1 established the one durable empirical finding: throughput up, stability down. The feedback loop is the only play in the document that attacks that finding directly and has no prerequisites. Everything else — the artifact chain, review passes, hooks, the autonomous loop — either depends on something or improves a different quantity.
Three details in the play carry most of the value and are easy to skip past:
- A quantifiable target. "All tests in
test_status.pypass," or "the endpoint returns 200 with the new field." Not "make sure it works." The target has to be something the session can evaluate without asking you, or you are still the loop. - Failing test first, for bug fixes. Reproduce the bug as a test, confirm it fails for the reason you expect, commit it, then ask for the fix without editing the test. A test that existed before the fix and could not be rewritten is the only proof that the bug is gone.
- Example of healthy output in CLAUDE.md. Listing
make testis worth much less than listing what a passing run looks like. Without it, "all green" is the model's judgment; with it, it is a comparison.
Evals: the only thing that regression-tests your configuration
The eval play is framed as "the AI-native equivalent of stage-gate QA," and that framing undersells
it. Once you adopt CLAUDE.md, skills and hooks, you have shipped a body of configuration that
steers every session in the organisation and that nothing else tests. Code has tests.
Infrastructure has plans. Your .claude/ directory has a code review and hope. The eval
suite is the test suite for that directory, which is why the playbook triggers it on
paths: ['CLAUDE.md', '.claude/**']. That trigger is the point of the play; the schedule
is secondary.
20–50 tasks is thin, and the playbook half-admits it. It says cases stop discriminating as models improve and new ones must be added from ongoing monitoring. That is a maintenance commitment, not a one-off build, and it is the reason most eval suites die in month four. Budget the ongoing curation or do not start.
A pass-rate merge gate creates the Module 5 scenario-3 pressure, one level up. If a skill change drops the pass rate and the gate blocks the merge, the cheapest path to green is to weaken the eval case. Protect the eval directory with code owners drawn from a different team than the one that owns the configuration, or you have built a gate that grades its own homework.
A model swap invalidates the baseline in both directions. A higher pass rate after a model change is not evidence your configuration improved; it is evidence the cases got easier. Keep the model version in the run record and never compare across it without saying so.
The best single line in the stage: "Each production incident gets an eval, written by the team that owned the incident." That is a post-mortem action item that cannot rot into a wiki page, because it lives in a suite that runs. If you adopt one sentence from Stage 4, adopt that one.
- The feedback loop is the first thing to build: no prerequisites, and it attacks the one finding that replicates.
- A verification target the session cannot evaluate alone leaves you inside the loop.
- Evals are the regression tests for configuration. Nothing else tests
.claude/. - Different owners for the eval suite and the configuration it grades.
Deploy: correlated blind spots and the anchored reviewer
- Distinguish formal separation of duties from independence of judgment
- Cite the self-preference and anchoring evidence and say what each implies for review design
- Write a REVIEW.md that does not drown its reader
"Separation of duties is preserved, because the agent that wrote the code has no way to approve it. The review policy in REVIEW.md is applied to all PRs […] Approval comes from a human through branch protection, informed by the findings."
And the throughput claim: "All PRs get an identical set of review passes, with findings ranked by severity. Human attention moves up a level."
The formal claim is correct and important. An agent cannot dismiss branch protection, so the control that matters — a human code owner approves — genuinely survives. Uniform review coverage is also a real gain: the playbook is right that human review quality varies with reviewer load, and a pass that runs identically on every PR removes a real source of variance.
The audit: two independent failure modes stack here
Separation of duties assumes the reviewer's errors are independent of the author's. When the same model family writes the code and reviews it, they are not. The LLM-as-judge literature documents self-preference bias directly: judges systematically favour outputs from themselves or their own family, and the effect is tied to self-recognition — the model's ability to identify its own generations (Self-Preference Bias in LLM-as-a-Judge; LLM Evaluators Recognize and Favor Their Own Generations). A complementary finding: judges over-reward low-perplexity text — that is, text that looks familiar to them.
Translated to code review: the failure modes most likely to be missed are exactly the ones the generating model finds natural. Not random misses — systematically the same misses. Twenty review passes on twenty PRs by the same model is one opinion applied twenty times, and a control whose errors are perfectly correlated does not compose with itself.
What follows: where the stake justifies the cost, run the review pass on a different model family than the one that authored, and treat agreement between them as signal rather than treating a single pass as coverage. Where it does not justify the cost, be honest in the control description: this is a recall improvement on known classes, not independent assurance.
The second failure mode lands on the human the playbook is relying on. Reviewers given an automated review focus on the locations the automation commented on (Automated Code Review In Practice), and erroneous automated advice is followed at materially higher rates when automation is present. So the finding list does not merely inform the human — it redistributes their attention, away from the unflagged 90% of the diff.
Now stack the two. The model misses a class of defect because it authored in that style; the human
misses the same class because nothing flagged it. The playbook's promise that "human attention moves up
a level" is the right ambition and the mechanism actively works against it unless you design for it:
require the reviewer to state what the change is supposed to do before reading findings, and
check the diff against plan.md rather than against the finding list.
REVIEW.md, and the one play to decline
The REVIEW.md pattern is good and the nit cap is the reason. "Report at most five nits per review; summarize the rest as a count" is a load-management control, and it is the difference between a review a human reads and a review a human scrolls. The monthly tuning ritual — rate findings, exclude generated paths, exclude what CI already enforces — is the maintenance that keeps it that way.
The playbook suggests letting Claude "babysit the PR to merge" — a slash command that sweeps unresolved review comments and failing checks, pushes fixes, and repeats until the PR is green and waiting only on code-owner approval.
That is the highest-risk paragraph in the document, because "green" is now produced by the same system that produced the code, in a loop optimising for green, with the human's attention arriving only at the end and anchored by a clean board. It is not that the loop is wrong — it is that it should not be adopted before the Stage 4 protections exist (committed failing tests, a hook preventing test edits, evals owned by another team). Adopt the fix loop; adopt it last.
- Formal separation of duties holds. Independence of judgment does not, when author and reviewer share a model family.
- Cross-family review where the stake justifies it; otherwise describe the control honestly as recall, not assurance.
- The finding list reallocates human attention. Check the diff against plan.md, not against the findings.
- Cap the nits. Adopt the auto-fix-to-green loop last, or not yet.
Governance as configuration, line by line
- Explain why the sandbox block, not the permissions block, is the load-bearing part
- Identify the two keys most teams omit and what they cost
- State three risks that no line in this file addresses
The worked example
The playbook's regulated-enterprise settings block is the most concrete artifact in the document. It is worth walking, because the interesting part is the internal logic: each layer exists because the layer above it has a hole.
| Key | What it buys | The hole it closes |
|---|---|---|
permissions.deny | Secrets out of context; no tool-level network egress | — |
permissions.allow | Pre-approves the safe inner loop | Stops the deny list becoming prompt fatigue — a real control failure, since a person clicking through prompts is not approving |
disableBypassPermissionsMode + allowManagedPermissionRulesOnly | No engineer, project file or CLI flag can widen the rules | Local override of team settings |
sandbox + network.allowedDomains | OS-level egress allowlist | A tool-level deny on WebFetch does not stop a shell command reaching the network |
failIfUnavailable + allowUnsandboxedCommands:false | Refuses to start without the sandbox; no retry outside it | The sandbox silently degrading to advisory |
sandbox.credentials | Denies ~/.ssh, ~/.aws/credentials; strips named env vars | permissions.deny governs Claude's file tools — a sandboxed shell command could still read those paths |
allowManagedHooksOnly | Your approval gates are the only hooks that run | Local hooks adding to or replacing the gates |
disableSideloadFlags + strictKnownMarketplaces | Every skill, agent, hook and MCP server arrived via the approved marketplace | Anything loaded from a home directory |
allowManagedMcpServersOnly | The tool surface is a platform-owned allowlist | Ad-hoc connectors expanding what the agent can reach |
requiredMinimumVersion | Refuses to start below an assessed build | Controls enforced by a version nobody evaluated |
"Consider the above a starting point to tailor, rather than a recommendation to copy. Every deny trades against capability, and the right balance depends on the data classification of the repo."
The position: the sandbox block is the control, the permissions block is the ergonomics
Most teams that adopt this file will tune permissions carefully and treat
sandbox as advanced configuration to do later. That is backwards, and the playbook's own
annotation says why twice: the tool-level denies govern the agent's file and fetch tools, and a shell
command routes around both. permissions shapes what the agent does by default;
sandbox bounds what any process it spawns can do. Only the second one is a
boundary in the sense a security review means.
The two keys most often omitted are failIfUnavailable and
requiredMinimumVersion, and they share a failure mode: without them, the control set
degrades silently to a weaker one that still looks configured. A sandbox that could not
initialise and a build predating the key you rely on both produce a machine that reports normal and
enforces less.
1. Nothing here constrains what happens to data already in context. Every key is about acquisition and egress paths. Once a document is in the session, no setting governs how it is used in the output. That is a data-handling and retention question, answered by admin retention settings and the Compliance API, not by this file.
2. Nothing here is a control on the human. The whole file constrains the agent. The engineer steering it retains all their own access, and the playbook's own audit story depends on attribution — each non-interactive run acting under the agent's own identity so the log separates agent action from human action. That separation is what makes the log evidence; it is not enforced by any key above.
3. Nothing here proves the control held. A settings file is a statement of intent about the future. The evidence an auditor wants is the OpenTelemetry export with per-decision allow/block verdicts and timestamps, plus CI check-run history. Configure the export at the same time as the settings or you will have controls and no record of them — which, as Module 5 scenario 5 shows, audits identically to having no controls.
- Layer order matters: permissions shape defaults, the sandbox draws the boundary. Do the sandbox first.
failIfUnavailableandrequiredMinimumVersionconvert silent degradation into a loud failure. Set both.- Ship the telemetry export with the settings. Unevidenced controls audit as absent controls.
Measurement: why these indicators are gameable
- Classify the playbook's indicators and spot the structural bias in the set
- Name the veto metric and defend the choice
- Explain why self-reported speedup is disqualified as evidence
What the playbook measures
Every play ends with a leading and a lagging indicator, which is more measurement discipline than most vendor documents attempt. Collect them and a pattern appears immediately.
| Kind | Examples from the playbook | Problem |
|---|---|---|
| Elapsed-time deltas (most of the leading indicators) | Time to committed intent.md; intent→spec commit gap; time to first review; time from band breach to intent.md; time waiting at each approval gate | These measure the step that just got cheap. They will improve whether or not anything downstream improved, and they will improve most where quality dropped most. |
| Volume / rate | Concurrent sessions per engineer; changes merged per engineer per week; share of pipeline failures triaged without paging | Throughput without a stability pair. DORA's finding is precisely that these can rise while the system degrades. |
| Genuine outcome measures | Rework cycles per change; first-pass CI success; defects caught before merge vs escaping to production; repeat incidents of the same class; DORA four keys | These are the real ones and they are outnumbered roughly three to one. |
A document arguing that build got faster and the surrounding stages did not will naturally propose indicators that measure the surrounding stages getting faster. Every one of those will move. None of them tells you whether the change was good.
The playbook does list the honest ones — it names DORA explicitly under CI/CD, it names change failure rate, it names repeat incidents. They are simply distributed one per play, so no reader assembles them into a scorecard. Assemble them.
The position: one veto metric, one throughput metric, one leading indicator per stage
Change failure rate is the veto metric. Not because it is the most informative number, but because it is the one that the entire body of external evidence says will move against you: DORA finds AI adoption correlating with instability across successive report years, and GitClear's maintainability signals point the same way. If throughput rises and change failure rate holds flat, the programme is working. If throughput rises and change failure rate rises, you have bought the DORA failure mode at speed, and every elapsed-time indicator in the document will be green while you do it.
Two consequences follow, and both are unwelcome:
- Baseline before you start. Change failure rate and time-to-restore, measured for a quarter, before the first CLAUDE.md lands. Without a baseline you will be arguing from anecdote in month six, and the anecdotes will favour whoever is most enthusiastic.
- Cap the scorecard. Twenty indicators is not measurement, it is a dashboard nobody reads and a menu for whoever needs a good number this quarter. One veto, one throughput, one leading indicator for the stage you are currently changing. Retire the leading indicator when you move on.
METR's participants forecast a 24% speedup, measured 19% slower, and still believed they had been sped up after experiencing the slowdown (arXiv:2507.09089). Meanwhile DORA 2025 finds 80%+ of technology professionals believe AI increased their productivity. Both facts can be true, and together they mean one thing for your programme: developer sentiment is a legitimate metric for adoption and retention, and inadmissible as evidence of productivity. Do not let a satisfaction survey settle an argument that a delivery metric can settle.
- Most of the playbook's leading indicators measure the step that got cheap. They will all go green.
- Change failure rate holds the veto. Throughput without it is the documented failure mode.
- Baseline before the first change, or you will never be able to answer the question.
- Self-reported speedup is adoption evidence, not productivity evidence. METR settles this.
Adoption order — Monday, and never
- Order the plays by return and prerequisite, not by stage number
- Name what to defer indefinitely and defend the deferral
- State the one-sentence version of the whole document
The playbook is explicit that its stages are not an adoption order — "the plays are listed with stage; the arrows give the order to adopt them in. The two are not the same." It then leaves the ranking to you. This is the ranking, with reasons.
Monday
- The feedback loop (Stage 4). One command for build, one for test, one for lint, each with an example of healthy output in CLAUDE.md, and a stated rule that verification is part of "done." Highest return, no dependencies, and the only play that attacks the stability finding head on.
- CLAUDE.md (Stage 3).
/init, cut it to a page, commit it, and adopt the twice-is-a-rule discipline. It is the substrate every later play reads.
Do these before you baseline nothing else — but do also start the change-failure-rate baseline from Module 9 in the same week, because it takes a quarter to be worth anything.
Weeks two to six
- Run the Module 5 exercise for real. Sit down with your policy owners, list the must-always-hold policies, and classify each one against the enforcement table. This costs a half-day and it is the difference between a control programme and a document library.
- Hooks for the policies that came out of that exercise — and managed settings for the subset an engineer must not be able to switch off. Start with the sandbox block.
- plan.md, with ordering enforced. Plan mode is free; the discipline that makes it worth anything is committing the plan before the diff exists.
- Skills, but only now — after you can tell which policies have a deterministic backstop and which are advisory by design. Writing skills first is how the paper-control library gets built.
The quarter
- Artifact chain at linkage. intent.md and spec.md in a shared home, record IDs both ways, no fight with the change board yet.
- PR review passes with REVIEW.md, nit cap set, monthly tuning ritual scheduled, generated paths excluded. Cross-family review for the highest-stake repositories.
- Evals in CI, triggered on
CLAUDE.mdand.claude/**, owned by a different team than the one that owns the configuration. - The @claude fix loop, last, once the Stage 4 protections exist.
Defer, and be willing to defer indefinitely
Stage 6, the autonomous loop. A deterministic script watching a control band and invoking an agent with no person in the path is a genuinely good design, and the tiering by sigma is sound. It is also the play with the longest prerequisite chain in the document: it needs intent.md, PR review, hooks as an action boundary, and a rehearsed rollback path. An organisation that has all four already has most of the benefit. An organisation that does not will be building an autonomous loop on top of controls it has not yet tested, which is the worst possible order.
Scheduled scanning and Claude Tag on-call are lower risk and can jump the queue if you already have the PR review gate — both route findings through it rather than around it, which is exactly the property that makes them safe to adopt early.
What to be most sceptical of: any proposal that starts by rebuilding your intake process before a single feedback loop exists. That is adopting the throughput half of the document and skipping the control half, which is the failure the external evidence describes.
The one-sentence version
The playbook is a control-system document wearing a productivity document's clothes, and its own best sentence — that a skill is an advisory control and nothing forces a session to comply with it — is the test to apply to every play in it, including the ones you are most excited about.
What a practitioner does differently on Monday
- Write the three-command verification block into CLAUDE.md, with example healthy output, and make verification part of "done."
- Start the change-failure-rate baseline. Nothing else in the programme is arguable without it.
- Take your top five must-always-hold policies and write, next to each, whether its current enforcement is advisory or deterministic. Expect the answer to be uncomfortable for at least three of them.
- Before the next spec review, write down the two things you expect to be hard — then read the flagged-concerns list.
- Adopt by return and prerequisite, not by stage number. Feedback loop and CLAUDE.md first, always.
- Classify your policies before you write a single skill.
- Defer the autonomous loop until the four controls it sits on have been tested in anger.
- Baseline in week one or spend month six arguing from anecdote.
Need this for a date?
Turn this course into a ramp-up pack sized to your minutes per day, or build an interview or certification pack for the day you need it.