Observify

User Tests · holt-ecosystem-packets
← all user tests
USER TEST PLAN

holt-ecosystem-packets

Derived from: no git repository here — this workspace is disk-only (NOTES.md: "Not a git repository —

0/49 checked · 0 pass · 0 fail

Flows

check each step and final check
F1

Workspace boots with its pinned model, effort, and permission posture

Proves an authoring session actually gets the ceiling the workspace promises before you spend a comprehensive run on the wrong model, the wrong effort, or with the deny-list quietly not holding.

Persona / precondition: a human at a fresh terminal, about to author or verify a Holt packet per the README's "Run it" section.

0/6

Run claude --version. Expect: 2.1.225 (Claude Code) — verified installed version at authoring
time; a later version is fine, this step only proves claude resolves at all.

Run grep -E '"model"|"effortLevel"|"fallbackModel"' .claude/settings.json. Expect: three lines
verbatim: "model": "claude-fable-5",, "effortLevel": "xhigh",, and a "fallbackModel": [ array
naming "claude-opus-4-8" then "claude-sonnet-4-6".

Launch claude (interactive) inside the directory. Expect: the session starts with no
SessionStart hook failed banner — the hook it runs, think-like-fable/tools/kit-remember-hook.sh,
exists on disk and is executable (verified -rwxr-xr-x, 8814 bytes).

Inside the session, run /effort with no argument. Expect (confirm live — not run in this authoring
pass):
it reports the active level as xhigh, matching the persisted effortLevel. Per Claude Code's
own settings schema, xhigh "sits between high and max," and max is session-only (not settable in
settings.json) — which is exactly why CLAUDE.md separately tells the operator to run /effort max
at the start of a big authoring run rather than trusting the settings-file pin alone.

[sensitive content omitted]
deny rule in .claude/settings.json still refuses the read, even though defaultMode is
bypassPermissions — the deny list should hold under bypass mode, not just under the ask-first default.

Final check: exit the session; ls -1 .claude/ still lists exactly agents, commands, memory,
plans, settings.json, settings.local.json, settings.local.json.bak-model-pin,
settings.local.json.bak-posture, skills — no stray artifact was dropped at the workspace's top level.

F2

The exemplar packet (Authify) is actually complete

Proves the one packet every other packet is told to match — README: "already authored as the gold-standard exemplar" — really is complete and internally consistent.

Persona / precondition: reading packets/01-authify/ cold, no session needed.

0/6

Run ls -1 packets/01-authify/. Expect: exactly nine lines — BACKLOG.md, BRANDING.md,
DEPLOYMENT.md, MOBILE.md, NOTES.md, PACKET.md, README.md, USER_FLOWS.md,
WIREFRAMES.html — the full 8-artifact set CLAUDE.md defines, plus a bonus MOBILE.md.

Run wc -l packets/01-authify/PACKET.md. Expect: 284 — comfortably over /verify-packet's own
">200 lines" floor for "full hi-fi."

Run grep -c pgTable packets/01-authify/PACKET.md. Expect: 8 — a real Drizzle schema block, not a
stub.

Run grep -n "pass3-scheme" packets/01-authify/WIREFRAMES.html. Expect: line 184 reading
<!-- pass3-scheme: Vault Steel · analog: 1Password / Doppler / IAM console -->.

Run tail -8 packets/01-authify/PACKET.md. Expect: the file ends inside a ## Honesty caveat block
naming "Assumptions made," "PROPOSED (pending confirmation)," "⚠ Verify against repo," "Blockers," and
"Deferred" — the apparatus CLAUDE.md requires at the end of every packet.

Final check: run grep -c "PROPOSED — confirm with Wolf" packets/01-authify/PACKET.mdExpect:
0 — and sed -n '239p' packets/01-authify/PACKET.mdExpect:
## 3 · Marketable Features for Prod *(LOCKED — source: build conversation 2026-07-01)*. The exemplar's
hero wedge is genuinely locked, not a derived placeholder.

F3

The exemplar wireframe honors its own engineering rules

Proves WIREFRAMES.html actually follows the per-app-identity, functional-expression, and reduced-motion rules /verify-packet is supposed to gate on — not just that a file exists.

Persona / precondition: reading packets/01-authify/WIREFRAMES.html, optionally in a browser.

0/6

Run grep -cE '^\s*:root\s*\{' packets/01-authify/WIREFRAMES.html. Expect: 1 — exactly one real
token :root{ declaration (line 11), matching /verify-packet's "no two apps share a palette" rule.

Run grep -nE '(^|[^a-zA-Z0-9_.$])function[[:space:]]*\(' packets/01-authify/WIREFRAMES.html.
Expect: no output — zero function( declarations; the "functional expressions only" rule
(arrow consts) holds.

Run grep -n "prefers-reduced-motion" packets/01-authify/WIREFRAMES.html. Expect: at least one hit,
including line 179: @media (prefers-reduced-motion: reduce){.

Run a well-formedness check:
python3 -c "import html.parser
p=html.parser.HTMLParser(); p.error=lambda m: print('ERR:',m)
p.feed(open('packets/01-authify/WIREFRAMES.html',encoding='utf-8').read())
print('done')"
.
Expect: done printed with no ERR: lines above it (verified clean on this exact file).

open packets/01-authify/WIREFRAMES.html in a real browser. Expect (confirm live — not opened in a
real browser this pass):
a scrollable phone/tablet/desktop-frame wireframe page with visible content,
not a blank page or a wall of raw HTML text.

Final check: run grep -co 'class="screen"' packets/01-authify/WIREFRAMES.htmlExpect: 8 phone
screens — and grep -co 'class="flowrow' packets/01-authify/WIREFRAMES.htmlExpect: 4, matching
USER_FLOWS.md's four flows (AUTHIFY-F1..AUTHIFY-F4) — the flow-map row count equals the flow count,
per /verify-packet's own cross-check.

F4

manifest.json and packets/ agree on shape

Proves the 29-app queue and the packets directory are the same 29 apps, ranked the same way — the structural precondition every other flow assumes.

Persona / precondition: reading manifest.json and packets/ cold.

0/4

Run python3 -c "import json; print(len(json.load(open('manifest.json'))['apps']))". Expect: 29.

Run ls -1 packets/ | wc -l. Expect: 29 — one folder per manifest app.

Run this consistency check:
python3 -c "
import json,os
d=json.load(open('manifest.json'))['apps']
ranks=[a['rank'] for a in d]
print('ranks 1..29:', sorted(ranks)==list(range(1,30)))
dirs=set(os.listdir('packets'))
print('every rank-slug has its folder:', all(f'{a[\"rank\"]:02d}-{a[\"slug\"]}' in dirs for a in d))
"
.
Expect: ranks 1..29: True then every rank-slug has its folder: True.

Final check: sed -n '31p;33p;61p' docs/decisions/locked-decisions.mdExpect: the anchor-table
header row, then row 1 | authify | ..., then row 29 | unify | ... — the anchor table's own numbering
also runs 1–29 with no gaps, agreeing with manifest.json and packets/.

F5

Live `/verify-packet` gate on the exemplar

Proves the actual gate command — not a hand-audit of its criteria — passes on the packet everything else is measured against.

Persona / precondition: inside a claude session in this workspace.

0/3

Run /verify-packet authify. Expect: PASS — every checklist box in
.claude/commands/verify-packet.md checked, no FAIL line. This is a prediction from a hand-audit of
the same criteria in F2/F3 (flag legend present, Drizzle schema present, threat model present, one
:root block, no function( declarations, screen/flow counts matching) — the live gate is what actually
confirms it. Not run in this authoring pass.

Run /verify-packet quantify (an app manifest.json currently marks "planned" — see F8). Expect (to
confirm):
also PASS — quantify's PACKET.md is 438 lines with 8 pgTable blocks and a 46KB
wireframe, the same structural signals as a passing packet, despite its "not started" manifest status.

Final check: if step 2 comes back PASS, that is itself evidence for F8's finding — a "planned" app
passing the completeness gate confirms the manifest status is the thing that's wrong, not the packet.

F6

BACKLOG.md's own verification command doesn't exist

Proves (or disproves) that a build agent following BACKLOG.md's instructions can actually verify a ticket. This is a KNOWN DEFECT — the step below exposes it; the correct behaviour is stated as the expectation, and it currently fails.

Persona / precondition: README: "Feed BACKLOG.md to a build agent." — this is the ticket source a build agent (or a human) is told to run verification commands from.

0/4

Run grep -c "packet_gate.py" packets/01-authify/BACKLOG.md. Expect: 50 — the ticket template
names python3 scripts/packet_gate.py authify --root . as required "Verification" evidence 50 times in
this one file (verified: 49 per-ticket occurrences plus 1 in the summary section).

Run find . -iname "packet_gate*" from the workspace root. CORRECT BEHAVIOUR — expect: a path to
scripts/packet_gate.py, since 51 verification-required lines across packets/01-authify/BACKLOG.md and
packets/07-clarify/BACKLOG.md depend on it existing. Currently fails: verified empty output — the
script does not exist anywhere in this workspace.

Run the exact command the ticket names: python3 scripts/packet_gate.py authify --root ..
CORRECT BEHAVIOUR — expect: the packet-completeness gate runs and prints a pass/fail verdict.
Currently fails: verified output is
python3: can't open file '.../scripts/packet_gate.py': [Errno 2] No such file or directory, exit code
2 — a build agent following this ticket's own "Verification" section hits an immediate, unfixable
failure on evidence the ticket calls mandatory.

Final check: grep -rl "packet_gate.py" --include="*.py" --include="*.sh" . from the workspace root —
Expect (if the defect were fixed): at least one script file implementing it. Currently: no output —
confirmed nowhere in the tree, including _build/ (whose gate.sh is a different tool, for already-built
v1 repos, not for grading an authored packet).

F7

Running `/packet` on a new app today would under-produce

Proves the documented authoring path actually yields the 8-file contract CLAUDE.md promises. This is a KNOWN DEFECT — exposed by comparing the command's own text against its stated contract, not by guessing.

Persona / precondition: reading .claude/commands/packet.md, .claude/skills/holt-packet/SKILL.md, CLAUDE.md, FIXES.md, and README.md side by side — no session needed.

0/4

Read the last line of .claude/commands/packet.md. CORRECT BEHAVIOUR — expect: it names all 8
artifacts. Currently fails: verified text is Write complete artifacts to
packets/<rank>-<slug>/ (PACKET.md, WIREFRAMES.html, NOTES.md).
— 3 of 8, silently omitting BRANDING.md,
USER_FLOWS.md, BACKLOG.md, DEPLOYMENT.md, and README.md.

Read the "Output contract" line of .claude/skills/holt-packet/SKILL.md. CORRECT BEHAVIOUR —
expect:
the same 8 files. Currently fails: verified text is
packets/<rank>-<slug>/PACKET.md, .../WIREFRAMES.html, .../NOTES.md. Nothing half-written. — the same
3-of-8 undercount, and the top-level README.md's own "Run it" section repeats it too: Packets are
written to packets/<rank>-<slug>/ as PACKET.md + WIREFRAMES.html + NOTES.md.

Run ls -1 .claude/skills/ and ls -1 .claude/agents/. CORRECT BEHAVIOUR — expect: a
brand-identity skill and a brand-strategist agent, both referenced by name in CLAUDE.md
("...see brand-identity" for BRANDING.md; "Subagents in .claude/agents/system-designer,
wireframe-smith, feature-strategist, brand-strategist") and in FIXES.md
("New brand-identity skill + brand-strategist agent"). Currently fails: verified ls -1
.claude/skills/
returns only digital-rain-wireframes, holt-packet, marketable-features; ls -1
.claude/agents/
returns only feature-strategist.md, system-designer.md, wireframe-smith.md
neither file exists.

Final check: despite the gap above, every one of the 29 packets on disk already has all 8 files
(confirmed in F2/F4) — so the gap is specifically in whether re-running /packet on a genuinely new app
today, exactly as documented, would reproduce that. Not run in this pass (would write new packet files,
out of scope for this authoring pass) — a tester should run /packet <a currently-unauthored slug, if one is
later added to manifest.json>
and confirm whether the agent's judgment fills the documented gap or the
output actually lands short at 3 files.

F8

Program-tracking docs disagree with the packets directory

Proves the docs a packet author is told to ground in (CLAUDE.md: "read before writing... never write from memory") actually reflect which packets exist. This is a KNOWN DEFECT.

Persona / precondition: reading manifest.json, docs/canonical/App_Registry.md, and docs/canonical/Current_Status.md against the packets/ directory.

0/3

Run this check:
python3 -c "
import json,glob,os
d=json.load(open('manifest.json'))['apps']
for a in d:
p=glob.glob(f\"packets/{a['rank']:02d}-{a['slug']}\")[0]+'/PACKET.md'
w=glob.glob(f\"packets/{a['rank']:02d}-{a['slug']}\")[0]+'/WIREFRAMES.html'
lines=sum(1 for _ in open(p,encoding='utf-8')); size=os.path.getsize(w)
if a['status'] in ('planned','idea') and lines>200 and size>20000:
print(a['slug'], a['status'], lines, size)
"
.
CORRECT BEHAVIOUR — expect: no output — manifest.json's own legend defines "planned" as "named,
not started," so nothing marked "planned" should have a full packet. Currently fails: verified
output lists 9 apps — quantify planned 438 46245, enforcify planned 417 56947,
clarify planned 465 54245, incentify planned 434 53376, bookify planned 403 49782,
workify planned 397 50473, integrify planned 393 55513, alertify planned 414 47543,
botify planned 375 49954 — each already has a fully authored packet and should read "staged"
("packet/wireframe authored") per the manifest's own legend.

Run grep -n '| \*\*Enforcify\*\*\|| \*\*Quantify\*\*\|| \*\*Clarify\*\*' docs/canonical/App_Registry.md.
CORRECT BEHAVIOUR — expect: a status glyph reflecting an authored packet. Currently fails:
verified all three rows carry ("not started" per that doc's own legend), and
docs/canonical/Current_Status.md's per-app table states outright: **Enforcify** | ⚪ | Policy gate /
enforcement decisions. Planned as the first deep module of the Agent Control Plane product packet
(next on the roadmap).
— directly contradicted by packets/06-enforcify/'s existing 417-line
PACKET.md and 57KB WIREFRAMES.html.

Final check: holt-packet/SKILL.md's own Phase 6 says a passing verify should flip the status
"planned→staged" — so the 9 apps in step 1 are packets that plausibly passed a gate at some point without
that update landing. Re-running F5's step 2 (/verify-packet quantify) live is the fastest way to confirm
whether today's gate would still say PASS, which would settle that the manifest — not the packet — is stale.

F9

FIXES.md's outstanding-work claim is stale

Proves whether the one "what's left to do" document in this workspace still describes reality. This is a KNOWN DEFECT.

Persona / precondition: reading FIXES.md (dated in its own header "repair pass (2026-07-02)") against the packets it describes as unfinished.

0/4

Read FIXES.md's "Still to do" section. Expect: it claims packets 02–29 are "usable first drafts"
whose "PACKET/BRANDING prose still carries the name-split anchors (e.g. Lockify 'lock record', purpose
pasted as the hero wedge)."

Run grep -rli "lock record" packets/*/PACKET.md packets/*/BRANDING.md. CORRECT BEHAVIOUR — expect:
no hits, if the claim in step 1 is now false. Currently: verified no output — zero packets contain
the name-split defect FIXES.md says is still outstanding.

Run this check for pasted-purpose hero wedges:
python3 -c "
import json,glob
d=json.load(open('manifest.json'))['apps']
hits=0
for a in d:
p=glob.glob(f\"packets/{a['rank']:02d}-{a['slug']}\")[0]+'/PACKET.md'
if a['purpose'].strip() in open(p,encoding='utf-8').read(): hits+=1
print('pasted-purpose hits:', hits)
"
.
Expect: pasted-purpose hits: 0 — the other defect FIXES.md flags as outstanding is also gone.

Final check: compare packets/*/PACKET.md mtimes to FIXES.md's own mtime (stat -f "%Sm" FIXES.md
vs. stat -f "%Sm" packets/*/PACKET.md). Expect: several packets (e.g. 07-clarify, 08-incentify,
15-indexify, 29-unify) were modified after FIXES.md — consistent with them having been fixed in a
later pass that never came back to update FIXES.md's "Still to do" list or CLAUDE.md's "not the
first-draft prose in the other folders" framing.

F10

The anchor table covers all 29 apps with no leaked name-splitting

Proves the root-cause fix locked-decisions.md claims to be — the per-app object/verb table — is actually complete, since every packet's hero wedge depends on it.

Persona / precondition: reading docs/decisions/locked-decisions.md.

0/4

Run grep -c "^| [0-9]" docs/decisions/locked-decisions.md. Expect: 29 numbered rows.

Run sed -n '33p' and sed -n '61p' on the file. Expect: row 1 is authify (credential . broker
(mint/validate/revoke) . is the minter; Au . Credential dashboard + one-tap revoke
) and row 29 is
unify (message / channel . chat . A, Au . (KILL candidate - WHISTLE covers notifications) team chat).

Run grep -c "🔒 Anti-slot-filling rules" — actually: grep -n "Anti-slot-filling rules"
docs/decisions/locked-decisions.md
. Expect: one hit, heading the 6 numbered rules (derive
object+verb from purpose, never paste the purpose string, hero wedge is crafted not restated, no legend
dumps, one version string, functional expressions only) that F9 confirmed are actually being followed.

Final check: the anchor table's 29 rows, manifest.json's 29 apps, and packets/'s 29 folders (F4) all
agree — this is the one tracking surface in the workspace that is not stale.

F11

The exemplar's deployment gate is a real Coolify launch checklist

Proves DEPLOYMENT.md is the "Coolify launch gates: web/api/worker/db/Redis services, envs, backup, restore, rollback, and the bare-domain migration off *.poc" CLAUDE.md promises, not a placeholder.

Persona / precondition: reading packets/01-authify/DEPLOYMENT.md.

0/5

Run grep -c "^## " packets/01-authify/DEPLOYMENT.md. Expect: 13 sections.

Run grep -n "^## " packets/01-authify/DEPLOYMENT.md. Expect: headings including Required
Services
, Environment Variables, Coolify Setup, Backups And Restore Drill, DNS And TLS,
Deployment Gates, Rollback Plan, and Launch Checklist — the full backup/restore/rollback set
CLAUDE.md requires.

Run grep -c Redis packets/01-authify/DEPLOYMENT.md. Expect: 5 — the Redis/BullMQ service is
named as its own required service, not folded silently into "API."

Run grep -n -i poc packets/01-authify/DEPLOYMENT.md. Expect: one hit noting *.poc.thewolf.tech
"currently points at production infrastructure per canonical docs" — the bare-domain migration note
CLAUDE.md calls for.

Final check: run grep -n "GET /healthz\|/healthz" packets/01-authify/DEPLOYMENT.mdExpect: the
Deployment Gates table names a concrete health-check command
(curl -sS https://authify-api.thewolf.tech/healthz) with a concrete expected response shape, not just an
unverifiable "confirm it's up."

Not covered and why

  • Actually running /packet <new-slug> end-to-end (F7's final check): would write real packet files

for a currently-unqueued app — out of scope for a read-only authoring pass, and there is no 30th app in
manifest.json today to try it against cleanly.

  • Feeding a packet's BACKLOG.md to a real build agent (README's own stated end use): happens in a

different project (e.g. ~/Projects/HoltEcosystem/authify-api-v1), outside this workspace's boundary.

  • The live Holt apps' actual HTTPS status (authify.thewolf.tech etc.): docs/canonical/Current_Status.md

itself flags these as "⚠ confirm live URLs" — this plan stays inside the packet workspace's own claims
rather than reaching across the network to a different project's infrastructure.

  • Full 29-way parity sweep: F2/F3/F6 sample the exemplar; F8/F9 sample a handful of the other 28. A

complete per-app pass (every packet's own packet_gate.py references, every pgTable/threat-model/
screen-count check) is mechanical but long — worth scripting once F5's live gate is confirmed.

  • /packet-all's long autonomous run: by design a multi-hour Workflows run; too slow for a manual

test pass. /verify-packet (F5) and reading its own worklist logic (.claude/commands/packet-all.md
step 2: "apps ... whose packet has no passing packet yet") stand in for exercising it directly.

  • chatgptbackup.zip / claude-chat-backup.zip (314MB / 76MB, sitting at the workspace root): unrelated

to the packet program; not part of any packet's claims, so not tested here.

Results

FlowVerdictFailures → tickets
F1 session boot & posture
F2 exemplar completeness
F3 exemplar wireframe rules
F4 manifest ↔ packets shape
F5 live /verify-packet gate
F6 packet_gate.py missing
F7 /packet under-produces
F8 manifest/canonical docs stale
F9 FIXES.md stale
F10 anchor table complete
F11 deployment gate complete
updated just nownext 3m 00s