Observify

User Tests · book-it
← all user tests
USER TEST PLAN

book-it

Derived from: no git repository — confirmed via git rev-parse --show-toplevel failing on

0/48 checked · 0 pass · 0 fail

Flows

check each step and final check
F1

Catalog listing: the safe read-only entry point

Proves the tool boots, reads the reMarkable store without asking for anything, and never writes — this is the command the skill's own docs say to "start here, always."

Persona / precondition: at ~/Projects/kit/think-like-fable, no arguments, nothing else running.

0/5

Run python3 tools/bookit/bookit.py with no lens argument. Expect: exit code 0, and the
first line reads the shape Catalog: <N> metadata records; <M> visible documents; <F> folders;
<O> orphaned payloads.
— as of this writing (2026-08-07) that line reads exactly Catalog: 1283
metadata records; 1206 visible documents; 76 folders; 0 orphaned payloads.
The library syncs
from a live device, so match the sentence's shape and punctuation; the counts themselves will
drift.

Check the next two lines. Expect: a File types: epub=<n>, notebook=<n>, pdf=<n> line and a
Sources: <none>=<n>, ... line, both non-empty.

Scroll to the Proposed lenses: block. Expect: it opens with medium — curated web
articles plus ror.web device clips
, lists one <folder> — folder lens (<n> documents) line per
real reMarkable folder including book-it — folder lens (4 documents), and closes with the
two composite lines company:<workspace-path> — company/{CHARTER,CONTRACTS,operating-system,roles,NOTES}.md
and board:<observify-project-slug> — regenerable Observify kanban board edition.

Confirm nothing was written. Expect: ls -la ~/Projects/book-it/work/ shows no new file and
no changed mtime versus before step 1 — this invocation only reads.

Final check: re-run the same command a second time back to back. Expect: byte-identical
output apart from anything that changed on the real device in between (this is a live read, not a
cache).

F2

CLI surface matches what the code actually accepts

Proves --help is not stale — the fastest way a CLI tool rots is a flag that used to work.

Persona / precondition: same shell, same directory.

0/4

Run python3 tools/bookit/bookit.py --help. Expect: exit 0 and a usage: line naming
exactly six long flags in this order: [--dry-run] [--yes] [--deliver] [--deliver-pi] [--include
PREFIXES] [--cover PATH] [--notes PATH]
, plus the optional positional [lens].

Check the lens help text. Expect: it reads exactly medium, board:<slug>, a library
folder, company:<workspace-path>, or backup
.

Run python3 tools/bookit/bookit.py --bogus-flag. Expect: argparse's own refusal — exit
code 2 and a stderr line containing unrecognized arguments: --bogus-flag — never a Python
traceback.

Final check: every flag named in --help is one this plan exercises somewhere below (--yes
in F5/F6, --dry-run in F6/F7, --deliver/--deliver-pi in F6) — cross-check after finishing the
whole plan.

F3

Build toolchain preflight: six hard prerequisites

Proves the operational floor the skill's docs claim is real, not aspirational — an infra tool is only as trustworthy as its stated prerequisites.

Persona / precondition: same shell; none of these commands touch book-it itself.

0/6

Run command -v xelatex pdftoppm pdfinfo pandoc epubcheck. Expect: all five resolve; on
this machine, verified: /Library/TeX/texbin/xelatex, /opt/homebrew/bin/pdftoppm,
/opt/homebrew/bin/pdfinfo, /opt/homebrew/bin/pandoc, /opt/homebrew/bin/epubcheck.

Run epubcheck --version. Expect: contains EPUBCheck v5.3.0 (the skill's stated hard
minimum is v5.x).

Run pandoc --version | head -1. Expect: starts with pandoc 3. — verified today: pandoc
3.9.0.2
.

Run ~/.local/bin/rmapi version. Expect: prints a version string — verified today:
v0.0.34. This does not require rmapi to be authenticated.

Run python3 -c "import pygments; print(pygments.__version__)". Expect: prints a version —
verified today: 2.20.0.

Final check: none of steps 1–5 required network access or modified any file — re-running them
is always safe.

F4

Board lens prerequisite: the local Observify API

Proves the board: lens's one network dependency is up, gives the right remedy when it is not, and that book-it's own project is itself a trackable board.

Persona / precondition: same shell; the local Observify server should be up on :8787 (verify first with lsof -nP -iTCP:8787 -sTCP:LISTEN).

0/4

Run curl -s "http://127.0.0.1:8787/api/kanban?project=book-it". Expect: JSON with
"reachable": true and a projects entry whose "id" is "book-it" and whose "path" is
/Users/wolf/Projects/book-it.

Run python3 tools/bookit/bookit.py board: (colon, empty slug). Expect: exit 2, stderr
exactly book-it: board lens needs an Observify project slug.

Run python3 tools/bookit/bookit.py board:zzz-nonexistent-project-xyz. Expect: exit 2,
stderr containing Observify board 'zzz-nonexistent-project-xyz' is unavailable (HTTP 404).
followed by the remedy line launchctl kickstart -k gui/$UID/observify.runtimecards.8787.

Final check: neither step 2 nor step 3 wrote anything under ~/Projects/book-it/work/ — both
fail before reaching any proposal-writing code (verified: board_manifest() raises before
_run_board_lens() ever calls save_proposal()).

F5

KNOWN DEFECT: every real lens crashes before doing any work

Proves — or rather, currently disproves — the single most basic promise of the tool: that giving it a real lens name does something. This is the headline finding of this plan.

Persona / precondition: same shell; ~/Projects/book-it/ledger.yaml present (it is: 3 recorded editions from earlier today).

0/7

Run python3 tools/bookit/bookit.py medium (no --yes — this is meant to be the safe,
read-only "write me a proposal" step the skill's own docs say is side-effect-free). CORRECT
BEHAVIOUR:
a PROPOSAL written: .../PROPOSAL-medium.md line plus chapter/appendix/flag counts.
CURRENTLY FAILS: exit 2, stderr exactly book-it: ledger is not valid book-it YAML/JSON:
/Users/wolf/Projects/book-it/ledger.yaml
— verified today, reproducibly.

Run python3 tools/bookit/bookit.py book-it (the folder lens matching book-it's own real
4-document reMarkable folder). CORRECT BEHAVIOUR: a proposal naming those 4 documents.
CURRENTLY FAILS: the identical book-it: ledger is not valid book-it YAML/JSON: ... line —
verified today.

Run python3 tools/bookit/bookit.py board:book-it (book-it's own project board, confirmed
reachable in F4.1). CORRECT BEHAVIOUR: a board proposal with card/column counts.
CURRENTLY FAILS: the identical ledger error — verified today. Note what this masks: the
crash happens in _run_board_lens() immediately AFTER a successful board_manifest() call, so
this also proves the board-fetch half of the pipeline is fine — only the ledger read breaks it.

Run python3 tools/bookit/bookit.py this-folder-does-not-exist-zzz (a typo'd, nonexistent
folder). CORRECT BEHAVIOUR: the folder-lens's own validation error, no library documents
found for folder lens: this-folder-does-not-exist-zzz
. CURRENTLY FAILS: the SAME ledger
error fires first — select_manifest() (where that validation lives) is never even reached,
because read_ledger() is called before it on every non-board lens.

Confirm none of steps 1–4 wrote or touched a file. Expect: stat -f "%Sm" ~/Projects/book-it/work/PROPOSAL-medium.md
shows its pre-existing timestamp, unchanged by step 1; ls ~/Projects/book-it/work/PROPOSAL-book-it.md
~/Projects/book-it/work/PROPOSAL-board-book-it.md
both report no such file. The crash is loud
and inert, not silently corrupting — small mercy, still a full outage of every real lens.

Read the root cause directly: tools/bookit/ledger.py's read_ledger() calls json.load() on
ledger.yaml, and write_ledger() calls json.dump() (its own comment: "Write JSON syntax,
which is valid YAML"). Confirm the live file is NOT that JSON-flavoured subset —
python3 -c "import json; json.load(open('/Users/wolf/Projects/book-it/ledger.yaml'))".
Expect: a traceback ending json.decoder.JSONDecodeError: Expecting value: line 1 column 1
(char 0)
— the file on disk is genuine block-style YAML (- approval: '...', folded multi-line
scalars) that json.load cannot parse at all.

Final check: this defect blocks EVERY lens that reaches read_ledger() before erroring for its
own reason — medium, every folder lens, company:, and board:<any-slug-that-would-otherwise-
resolve>
. The only lenses left standing are the ones that error out earlier for an unrelated
reason (F4's bad/empty board slugs) or never touch the ledger at all (F1's no-arg listing, F6/F7's
backup).

F6

Structural human gates hold even under F5's outage

Proves the safety rails that don't depend on the ledger are still intact — a full outage in one subsystem should not silently loosen an unrelated guard.

Persona / precondition: same shell; continues directly from F5 (ledger still unreadable).

0/5

Run python3 tools/bookit/bookit.py medium --yes --deliver. Expect: exit 2, stderr
exactly book-it: structural human gate: --yes cannot be combined with --deliver or
--deliver-pi
. This check runs BEFORE the ledger is ever touched, so it is unaffected by F5.

Run python3 tools/bookit/bookit.py medium --yes --deliver-pi. Expect: the identical
refusal text as step 1.

Run python3 tools/bookit/bookit.py backup --yes. Expect: exit 2, stderr exactly
book-it: backup accepts only --dry-run.

Run python3 tools/bookit/bookit.py backup --deliver. Expect: the identical refusal text as
step 3 (backup rejects --deliver the same way it rejects --yes).

Final check: all four refusals happened with no network call and no file write — confirm with
lsof -i -a -p $$ showing no open connections during the run (or simply: none of these commands
took more than a fraction of a second).

F7

Whole-library backup, dry-run first

Proves the one destructive-shaped lane (a full rsync of the reMarkable store to the Pi) previews its exact commands and opens nothing when asked to.

Persona / precondition: same shell. Do not run bare backup (without --dry-run) as part of this plan — that opens a real SSH connection to wolf@10.0.0.11 and starts a real rsync; that is a host-mutation action outside a test plan's mandate (Article IV) and is not scripted here.

0/3

Run python3 tools/bookit/bookit.py backup --dry-run. Expect: exit 0, first line exactly
DRY RUN — no connection to the Pi was opened., followed by three shell command lines in this
order: an ssh wolf@10.0.0.11 'mkdir -p /srv/nvme/remarkable-backup' line, an rsync -a
--link-dest=/srv/nvme/remarkable-backup/latest '<store path>/' wolf@10.0.0.11:/srv/nvme/remarkable-backup/<UTC-stamp>/

line, and an ssh wolf@10.0.0.11 'ln -sfn <UTC-stamp> /srv/nvme/remarkable-backup/latest' line
— the <UTC-stamp> is today's real timestamp in YYYYMMDDTHHMMSSZ shape and will differ every
run; everything else is byte-exact.

Run it again immediately. Expect: the same three command shapes with a NEW, later
<UTC-stamp> — proving the stamp is computed fresh each invocation, not cached.

Final check: confirm no connection was actually opened — ssh wolf@10.0.0.11 true was never
invoked by either run (there is no way to prove a negative from here; the honest claim is that the
source code's dry_run branch returns before the subprocess.run calls, verified by reading
tools/bookit/pi.py, not by an external observation).

F8

Ledger ground truth vs the files it claims to have delivered

Proves — or disproves — that delivered: true in the ledger means the artifact is actually where the ledger says it is. This is a second, smaller finding on top of F5.

Persona / precondition: same shell; read-only inspection of ~/Projects/book-it/ledger.yaml and ~/Projects/book-it/work/.

0/5

Run grep -n "^ lens:" ~/Projects/book-it/ledger.yaml. Expect: exactly 3 matches: two
lens: medium lines and one lens: board-holt line.

For the board-holt edition (dated 2026-08-07T18:25:03Z), find its output.path and
output.sha256, then run shasum -a 256 on that path. Expect: the hash matches the ledger's
recorded a349e8ef3eb4c410 f76b375374e4e6a2 27439d5a3dd99760 450d658c9d9c0655 (one 64-character hash, shown in four 16-character groups) exactly, and the file
exists — this edition's record is currently honest.

For the FIRST medium edition (dated 2026-08-07T14:10:00Z, output.path ending
medium-20260807.epub), check whether that file exists at its recorded path. Expect (and
verified true):
it does NOT — that entry carries its own withdrawn: field explaining why
(436 epubcheck errors ... removed from device; superseded by a rebuild), so this one is
self-documenting.

For the SECOND medium edition (dated 2026-08-07T18:20:42Z, delivered: true, output.path
ending medium-20260807-182040-669263.epub, sha256
79c9ad929a9f7124 82e8b53c5897f4f0 2742f0789691b351 ba6d2c4d2672ac74 (one 64-character hash, shown in four 16-character groups)), check the same two things.
CORRECT BEHAVIOUR: either the file exists at that path, or the entry carries its own
withdrawn: note like its sibling above. CURRENTLY FAILS: the file is NOT at that path —
it exists only as ~/Projects/book-it/work/rejected/medium-20260807-182040-669263.epub.REJECTED-436-errors
(confirmed same sha256), and the ledger entry has no withdrawn: field and still reads
delivered: true. Run shasum -a 256
~/Projects/book-it/work/rejected/medium-20260807-182040-669263.epub.REJECTED-436-errors
to
confirm the hash match yourself.

Final check: this gap means a --deliver attempt against lens medium today (were F5 not
already blocking it first) would resolve _latest_output("medium") to a ledger row claiming
success for a file that has been quarantined — the honest fix is a withdrawn: note on that
second entry, matching the first.

F9

Cover-art convention: book-it has no cover of its own yet

Proves the "never a missing-file crash" promise for a project with no art prepared — verified from the code path, since actually building the cover into a PDF is blocked by F5.

Persona / precondition: read-only inspection; no build is run (F5 blocks it anyway).

0/4

Run ls -d ~/Projects/book-it/covers/book-it 2>&1 and ls ~/Projects/book-it/covers/book-it-cover.png 2>&1.
Expect: both report "No such file or directory" — confirmed today: book-it has no
covers/book-it/ folder and no legacy book-it-cover.png, unlike agentify, holt, kit,
medium, pi, and terminus3, which all have one.

Read tools/bookit/render_pdf.py's art_project() and resolve_art(). Expect: for lens
board-book-it (the internal name a board:book-it build would carry), art_project() strips
the board- prefix to get project name book-it, then resolve_art() finds no folder and no
legacy file, leaves art.cover as None, and returns cleanly — no exception raised.

Read bookit.py's _art_line(). Expect: when art.cover is None it returns exactly no
art placed for this project — typographic cover
— this is the line a real build would print,
once F5 is fixed and a board build can actually run.

Final check: this is a code-path verification, not an executed build — mark it explicitly
UNVERIFIED-AT-RUNTIME until F5 is fixed and board:book-it --yes can be attempted for real.

F10

This very file: the two consumers that discover it

Proves the reason this document exists at all: one authoring pass lights up both a board-book chapter and Observify's /testit surface.

Persona / precondition: the local Observify server up on :8787; this file just written to ~/Projects/book-it/USER_TESTING.md.

0/5

Before this file existed, run curl -s http://127.0.0.1:8787/api/testit. Expect (verified
today, baseline):
a "plans" array of 14 projects, none with "slug": "book-it".

Now that this file exists, run the same curl -s http://127.0.0.1:8787/api/testit. Expect:
the array now includes an entry with "slug": "book-it", "project": "book-it", and a
"derived_from" string that starts with this file's exact provenance line, plus a
"progress" object whose "total" matches this file's real item count.

Run curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/testit/book-it. Expect:
200 — the rendered checklist page for this exact plan, with no restart of the server needed;
_testit_discover_plans() re-scans ~/Projects on every request.

Separately, read tools/bookit/board.py's _manual_docs(). Expect: it globs up to three
directories deep for *.md and casefolds each filename, matching this file because name ==
"user_testing.md"
after casefold — so a future board:book-it --yes build (once F5 is fixed)
would fold this exact file into the "Dogfood and walkthrough docs" chapter automatically, with
no book-it code change required. This half is code-verified only; F5 currently blocks running
the build to see the chapter land for real.

Final check: both consumers found the SAME file at the SAME path
(~/Projects/book-it/USER_TESTING.md) through two independent discovery mechanisms — Observify's
own-process rescan (step 2/3) and book-it's file-glob (step 4) — proving the "one authoring pass
lights up both" claim rather than assuming it.

Not covered and why

  • A real (non---dry-run) backup: opens a live SSH connection to the Pi and writes into

/srv/nvme/remarkable-backup/ — a host-mutation action outside a read-only test plan's mandate
(Article IV). F7 covers everything reachable without opening that connection.

  • Any --yes build (medium, a folder lens, or board:<slug>), and everything downstream of one

(the P0–P5/G1–G4 gates, --include, --notes, --cover, --deliver, --deliver-pi): all
categorically blocked today by F5's ledger defect. Once fixed, re-run this plan's F5 as a
regression check, then extend it with a real medium --yes (against a scratch BOOKIT_HOME,
never the live ledger) through to a --dry-run delivery.

  • The Founder's device confirmation (previews opened, the book visible on the shelf): the

skill's own doctrine calls this a categorically human step (L201/L202) — no tester script can
discharge it.

  • The $book-it harness-neutral invocation inside a live kit/Codex session: this plan was run

from a bare interactive shell, not a dispatched agent session, so that specific invocation
surface is untested here.

  • company:<workspace-path>: not separately walked — it shares the exact same read_ledger()

call path as medium and the folder lenses, so F5's finding applies to it without a dedicated
repro.

Results

FlowVerdictFailures → tickets
F1 catalog listing
F2 CLI surface
F3 toolchain preflight
F4 board prerequisite
F5 ledger defect (KNOWN DEFECT)
F6 structural gates
F7 backup dry-run
F8 ledger ground truth
F9 cover-art convention
F10 dual discovery
updated just nownext 3m 00s