A project-management case study. One person owned scope, quality, and risk for an idea-to-code orchestration system — directing AI agents as both the delivery resource and, by the end, the adversarial reviewer. The system was built under the same discipline it enforces, and every claim on this page traces to a git commit or a document in the project repository.
§1 · The project
The product: Conductor — a Claude Code plugin that runs software products from idea to code through enforced quality gates. Its Definition track is live: a nine-stage document pipeline (concept → research → strategy → definition → PRD → UX → system specs → technical design → backlog) where every document is approved by a human at a closed-question gate, every approval binds to a content hash, and any post-approval edit halts the pipeline as detected drift. Seventeen document templates, five gate skills, five zero-dependency scripts, and a cross-model review harness.
The constraints: a single owner with finite session time; a deliverable whose entire value is trustworthiness — a gate system that can be talked around is worse than none — and a delivery resource (AI engineering agents) that is fast, tireless, and confidently wrong just often enough that nothing it reports can be accepted without evidence. The recursive twist: the product being built is a management system, so it was built under its own philosophy — gates, ledgers, adversarial review, evidence before claims — and every place the discipline slipped became a documented lesson in the product itself.
§2 · The management system
"Approved" means this exact text: every gate records a sha256 of the content approved. A one-byte edit afterward reads as STALE and halts the pipeline until a human re-approves — drift is caught, never absorbed.
source: scripts/validate-state.js · MANUAL §gatesApproval is a two-option question — only "Approved" opens a gate. Every approval must consume exactly one open request (pairing, never raw counts — the count form shipped once, was wrong, and is now test-banned from returning).
source: skills/advance · docs/lessons.md #1"An instruction that says check X is a missing script." The backlog contract, template citation rules, CLI behaviors, and the state schema are all enforced by the 66-test suite — including lints born directly from named failures.
source: CLAUDE.md · scripts/check-backlog.jsAn append-only table of every smoke run and review round — findings, fixes, suite counts, evidence pointers. Seven standing lessons distilled from it, one tracked through five recurrences of the same failure class.
source: docs/lessons.md, verification recordTier 1: every change wave gets an adversarial Codex review. Tier 2: an opt-in red-team at the strategy and PRD gates, before approval freezes the doc. Disagreement between model families is treated as signal, not noise.
source: scripts/codex-review.js · docs/reviews/§3 · Timeline
Click a phase to expand its record.
Brainstorm → approved spec → a 13-task implementation plan → subagent-driven build with an independent review after every task. Tagged v0.1.0-session1 with 20 tests. The whole-branch review caught a Critical defect in the plan itself: the decision log was hash-bound and appended-to at every gate — it would have tripped its own drift detector at the first gate after strategy. Fixed before first use; the management system's first save was of itself.
Three external reviews converged: a direction-drift critique ("your gates protect documents, not direction"), a Codex mechanical review ("your trust layer is prototype-grade — the validator checks shape, not the state machine"), and the responses that reconciled them. Outcome: the roadmap was re-sequenced around a "Trust Core" — executable state transitions before visibility features — and an atomic state-file contract was chosen over an event-sourced design.
First human-driven run: 28 structured findings, headlined by a live event-log asymmetry — 7 approvals against 4 requests, re-gates skipping the request half. The fix wave then shipped a stated invariant ("request count == approval count") pinned by its own regression test — and the invariant was false, diverging during every legitimate awaiting-approval window. A cross-model re-review caught it; the replacement (per-stage pairing) is the rule today.
A deliberately hostile trim (drop the optional strategy stage) with seven failure predictions pre-registered from a desk read. All seven confirmed — including a decision log that was never created (≈80 founder decisions unlogged, silently) and an undeclared dependency the staleness graph couldn't see. Ceremony cost measured honestly for the first time: 3h35m against a 60–75 minute budget. Same-day sweep: suite 35 → 46, with new lints that catch both confirmed template classes.
All six ranked fix candidates shipped across v0.1.3–0.1.5: the decision log now survives any trim, gates became closed questions, findings got a ledger, dependencies got declared. The adversarial review of that wave found the project's signature failure inside the wave: the documentation sweep ran for one commit of four and was skipped for the rest, leaving the implementer guide teaching the exact bug the wave had fixed. Separately, an independent Codex full-manifest run demonstrated the core motivation in one line: nine synthetic gates went green with zero human approvals — approval integrity is prose until the Trust Core ships.
The Codex CLI was wired in as a standing reviewer (v0.1.6) — read-only sandbox, pinned model, loud failures. Its bootstrap assignment: review its own integration diff, through the wrapper under review. It found five real defects — including a stale-report masquerade in that very wrapper (proven with a /usr/bin/true repro) and the fifth instance of the unmirrored-gate pattern. All five fixed with regression tests the same day (v0.1.7, 66 tests). The review panel validated itself by catching its own harness.
§4 · The numbers
tests passing at each pinned count · Aug 21 – Sep 22, 2026
Counts as recorded in commit messages. The long flat stretch is the critique period — design work, no code. The right-edge climb is verification converting findings into pinned behavior: 46 tests became 66 in a single day of review cycles.
47 total across 7 working days · the shape of punctuated, gated delivery
Nothing lands between sessions by design — every commit belongs to a gated working session with a review cycle. The two single-commit days are documentation waves from the critique period.
layered verification: each net catches what the others miss
Conservative counts from the verification record. The cross-model bar is the argument for the review panel: 19 findings — several invisible to same-model review by construction, including an invariant whose own test was enforcing its wrongness.
§5 · Exhibits
Excerpts from the project's real management documents, lightly condensed.
excerpt · docs/lessons.md · convention: an append-only row per smoke run or review round
| Round | Outcome | Suite |
|---|---|---|
| Session 1 final review | Plan-authored Critical caught before first use (decision-log drift bomb) | 20 |
| Smoke run 1 | 28 findings · 7/4 event asymmetry found live · 6 fixes merged | 20 → 34 |
| Codex re-review of run-1 fixes | 7 findings incl. the false count-equality invariant pinned by its own test | → 35 |
| Smoke run 2 | 7/7 pre-registered predictions confirmed · pairing verified live | 35 → 46 |
| Candidates wave + reviews | Fix candidates 1–6 shipped · sweep-skipped-itself caught · 2 code defects fixed | 46 → 57 |
| Codex full-manifest smoke | All 9 stages, 17 templates rendered · 9 synthetic gates went green — the Trust Core case, demonstrated | — |
| Reviewer bootstrap (v0.1.6–7) | Codex reviewed its own integration: 5 real defects, all fixed same-day | 57 → 66 |
condensed · docs/lessons.md · each lesson names what was paid for once and must not be paid for again
| # | Lesson | Instances | Executable form |
|---|---|---|---|
| 1 | Executable invariants over prose ones — a prose-grep test enforces its own wrongness, silently and indefinitely | 2 | pairing audit · check-backlog.js |
| 2 | Detection without persistence is indistinguishable from no detection | 5 fixes | drift/stage/finding events |
| 3 | A trimmed manifest breaks prose that names dropped stages | recurred ×4 | templates.test.js lint |
| 4 | A lesson fixed at the instance is still owed at the class | 5 | open — Trust Core commit-time check |
| 5 | What a stage reads is a dependency, whether the manifest says so or not | 2 | declared edges + citation warning |
| 6 | The pipeline produces knowledge it has no place to put — decisions, assumptions, findings | ~100 items | init-scaffolded log · findings events |
| 7 | The run depends on things no file states — conventions, registry access, "explicit approval" | 4 | closed-question gate; rest open |
Lesson 4's five instances are the register's spine: the same unmirrored-gate drift caught by a task review, a smoke run, two same-model waves, and finally a cross-model review — escalating layers, one class. It closes only when the mirror rule runs at commit time.
condensed · docs/reviews/2026-09-22-first-live-codex-tier1.md · v0.1.6 diff, reviewed by the wrapper under review
| Severity | Finding | Disposition |
|---|---|---|
| IMPORTANT | The new gate offer isn't mirrored in revise — the unmirrored-gate failure, fifth instance, in the commit wiring the reviewer in | Mirrored; lesson 4 amended |
| IMPORTANT | A pre-existing file can masquerade as a fresh review: exists+size passes when the subprocess writes nothing — reproduced with /usr/bin/true | .pending temp, promoted only after verification; regression test |
| IMPORTANT | Outcome events logged unconditionally would create a stray .pipeline/ in unmanaged repos | All review events now condition on a managed project |
| MINOR | A docs list can normalize to zero targets and launch a review of nothing | Rejected before spawning |
| MINOR | The command's advertised write contract was false | Manual corrected |
Why this exhibit: it is the review panel validating itself. The reviewer showed no deference to the harness invoking it, disclosed what the read-only sandbox prevented it from testing, and every finding survived independent verification. All five were fixed with regression tests the same day.
The class of lesson, as recorded: a fabricated cross-model review is worse than none — and a review harness must be unable to pass stale evidence as fresh.
structure · /conductor:status output · read-only; problems always print first
Two details worth noticing: problems print before progress — a clean ledger never buries a known defect — and the "Now" line refuses to name a next stage while drift is detected, because a pipeline that suggests forward motion over a known problem is lying politely.
§6 · What the record demonstrates
| Competency | Where the record shows it |
|---|---|
| Quality management | Layered verification with counted catches (smoke runs → same-model review → cross-model review → desk-read predictions); no wave ships unreviewed; 66 tests including lints born from named failures |
| Risk management | Hash-bound approvals; a pairing invariant replacing a demonstrably false one; a backlog gate that blocks on priority inversion by explicit owner decision, not default |
| Process improvement | Seven standing lessons distilled from a verification record; one failure class tracked through five instances across escalating detection layers until its executable fix was specified |
| Managing multi-vendor AI delivery | Two model families with governed roles — canonical repo vs. independent port, pinned review model and effort, accuracy-first policy for cheaper models, and a hard rule against fabricated reviews |
| Evidence-based decisions | Pre-registered predictions (7/7); a founder decision log for every gate-behavior change; ceremony cost measured (3h35m vs. a 75-minute guess) and fed back into the design |
| Communication & continuity | Four-home knowledge routing (lessons · manual · implementer guide · code); an append-only verification record; a status report that prints problems before progress |
| Honest scope control | A "what's real today" header in the product's own manual; enforcement claims graded into four levels (detection-when-invoked ≠ prevention); visibility features deliberately re-sequenced behind the trust layer |
§7 · Retrospective