Observify

User Tests · agentify
← all user tests
USER TEST PLAN

agentify

Derived from: commit b5fe60e (dev), 2026-07-25, against the production deployment at

16/66 checked · 16 pass · 0 fail

Flows

check each step and final check
F1

Sign in, wrong passphrase first

Proves the identity boundary: deny-by-default, no account enumeration, then a clean entry.

Persona / precondition: allowlisted user with a set passphrase, signed out, fresh tab on https://agentify.thewolf.tech.

5/5

Load the site. Expect: brief Loading…, then the sign-in card: heading ▸ Sign in to
Agentify
, fields Email, Passphrase, Invite code (optional).
`

Enter your email, a wrong passphrase, click Sign in. Expect: error text exactly
invalid credentials — no hint whether the email exists. `

Enter a non-existent email and any passphrase, click Sign in. Expect: the same
invalid credentials — byte-identical to step 2. `

Enter your email with the correct passphrase, click Sign in. Expect: the session
console at #/: header shows the Agentify wordmark, your email, and your role pill
(god/orchestration/trial), tabs SESSION / MARKETPLACE, buttons Tour and Sign out.
`

Final check: reloading the page keeps you signed in (session cookie survives a refresh).
`
---

F2

First-visit product tour

Proves a stranger's first two minutes: the tour auto-opens, demos safely, and exits cleanly.

Persona / precondition: a browser profile that has never visited the site (or clear the localStorage key agentify.tour.seen.v1), signed in.

5/5

Arrive at #/ for the first time. Expect: the tour opens by itself: panel titled
Product tour, step counter 1 / 17, a Demo mode badge in the header, and a banner
Scripted product walkthrough with a Demo data badge. `

Click Next through at least 5 steps. Expect: the step counter advances, each step
highlights a different area (Welcome, Access, Billing, Credentials, Kit Atlas, …)
and the demo pane changes with it. Nothing you click in the demo spends money or touches a
real box (the demo console says so explicitly). `

Press Esc. Expect: the tour closes; the Demo mode badge disappears; the real
console is back. `

Click Tour in the header, then click Skip tour. Expect: it reopens on demand and
dismisses again. `

Final check: reload — the tour does not auto-open a second time.
`
---

F3

Start a free-trial session (blank)

Proves the core promise: a real cloud session with no key, no account, no spend.

Persona / precondition: signed in; no session currently active.

5/5

On #/, find Start a session. In the kit picker, leave Blank session (no kit) selected
(it shows selected). Expect: Model reads @cf/openai/gpt-oss-20b; Billing reads
Free trial — no key needed (runs on agentify's Cloudflare allowance); Storage reads
Ephemeral — fresh box, zip in/out. `

Check the trial readout under Billing. Expect: Shared demo pool: N / 10,000 Neurons
remaining
plus your own allowance — real numbers, not a loading spinner stuck forever.
`

Click Start session. Expect: the confirm bar: This provisions a cloud box and
reserves shared demo allowance. Continue?
with Confirm & start / Cancel.
`

Click Confirm & start. Expect: within ~15s a session card appears: 8-char id, model,
ephemeral, status pill turning ready (green), ticker ending terminal live, telemetry
ttyd attached, and the Codex boot box in the terminal (OpenAI Codex, the model, a
/data/workspaces/sess-… directory). No red error banner. `

Final check: the panes below the terminal render their honest empty states: Fleet shows
No fleet reported yet…, Live agents shows one of its three empty-state messages, Workspace
shows workspace empty — import files to get started. `
---

F4

Real work in the session (base kit)

Proves the terminal is a working agent, the base kit is present, and no internal tooling vocabulary leaks.

Persona / precondition: F3's session live.

1/5

Click into the terminal and type ls ~/skills ~/.agents/skills, press Enter. Expect: both
directories list exactly: begin continue dogfood ground handoff test-it.
`

Type $begin and press Enter. Expect: within ~90s (whole turns render at once on the
trial model) the agent responds with a short interview — 3-5 questions about your goal — in
plain product language. It must NOT mention any internal kit name, ask where a kit root is,
or reference directories like kernel/. `

Answer the questions with a small project (e.g. "a CLI that counts words in a file; done =
it runs on a sample"). Expect: the agent creates files (a PLAN.md, source files) and
RUNS shell commands visibly — no sandbox errors (bwrap…), no approval prompts, no
invalid_prompt / 429 red JSON. Slow is fine; silent death is a fail.
`

When it reports the slice built, check the Workspace pane. Expect: the file count is
> 0 and PLAN.md appears in the listing. `

Final check: ask it run the project and show me the output — it executes and shows real
program output that matches what was built. `
---

F5

/test-it inside the session

Proves the newest base skill: the agent writes a user-testing plan for what it just built.

Persona / precondition: F4 completed in the same session.

0/3

Type $test-it and press Enter. Expect: the agent produces USER_TESTING.md in the
workspace: flows derived from the actual project, numbered steps, an Expect: per step,
pass/fail record lines, and a "not covered" section. `

Check the Workspace pane. Expect: USER_TESTING.md is listed with a non-zero size.
`

Final check: read one flow — you (a human) could execute it cold without asking the agent
anything. `
---

F6

Marketplace kit → seeded session

Proves the marketplace-to-session bridge: browse, pick, and the kit's intake seeds the box.

Persona / precondition: signed in; end any active session first (F7 step 4 shows how).

0/6

Click MARKETPLACE. Expect: title Marketplace, stat tiles Kits / Verticals /
Tags with non-zero numbers, vertical filter chips starting with all, and kit cards.
`

Type a term in Search kits by title, vertical, tags… that matches a kit you can see.
Expect: the result count line (N of M kits) drops and only matching cards remain.
`

Click a kit card. Expect: the kit page: ← Marketplace back link, How it runs diagram,
Skills list, Intake cards, a /begin prompt block with a copy button, and the button
Use this kit →. `

Click Use this kit →. Expect: back on #/, the wizard's kit picker has this kit
pre-selected (accent border, selected), and the detail box says This kit's /begin intake
will be seeded into your workspace as START-HERE.md.
`

Click Start session, then Confirm & start. Expect: the session boots (as F3.4)
and START-HERE.md appears in the Workspace pane listing. `

Final check: in the terminal, cat START-HERE.md shows the kit's intake text matching the
kit page's /begin prompt. `
---

F7

Session lifecycle: reset, wipe, end, single-active

Proves the lifecycle controls and the one-session-at-a-time rule.

Persona / precondition: a live session with at least one file in the workspace (F4 or F6).

0/5

Click Reset. Expect: a browser confirm: Recreate the box and keep your workspace?
This spends against your cap.
Accept it. The card shows Resetting…, then returns to
ready — and the Workspace pane still lists your files. `

Click Wipe. Expect: confirm Erase the workspace and start clean? This spends against
your cap.
Accept. After Wiping…ready, the workspace is workspace empty — import
files to get started
(files gone). `

Open the site in a second tab and try to start another session (Start session → Confirm &
start). Expect: A session is already active. Refreshing… and the tab shows the existing
session instead of a duplicate. `

Click End session. Expect: after Ending…, back on the wizard with the note
Session ended. Accrued $X.XX against your cap. (a number, even $0.00).
`

Final check: the idle console aside shows sessions none active and Awaiting commission.
Start a session to bring a cloud box online.
`
---

F8

Workspace export and import

Proves work leaves and enters the ephemeral box intact.

Persona / precondition: a live session with 2+ files (re-run F4 briefly, or import in step 3).

0/4

In Workspace, click Export .zip. Expect: the browser downloads
agentify-<sessionId>.zip; opening it shows the same files the pane lists.
`

Click Wipe (accept the confirm). Expect: workspace empty — import files to get
started
. `

Click Import files and choose the zip from step 1 (or drag it onto the listing box —
it should light up on dragover). Expect: Importing…, then a green imported N files
and the file rows return with sizes. `

Final check: in the terminal, ls shows the restored files and one of them opens with
cat showing its original content. `
---

F9

Vault a credential, run vaulted billing

Proves the encrypted vault and the bring-your-own-key path.

Persona / precondition: signed in, no active session; a real API key you're willing to store (e.g. Anthropic or OpenAI).

0/6

Scroll to Connect model account at the bottom of #/. Expect: status Not connected,
a Credential type select listing providers (OpenAI API, Anthropic Claude, xAI Grok,
Google Gemini, Cloudflare Workers AI, Local (Ollama), …).
`

Pick a provider, paste the key into Secret, click Connect. Expect: the status turns
green: Connected — <provider label>, GitHub no, updated just now. The key itself is never
redisplayed anywhere. `

In the wizard set Billing to Vaulted key — bill my connected credential. Expect: the
Model dropdown now shows that provider's model family, with the helper Using the connected
<kind> provider…
. `

Start + confirm a session, and in the terminal ask the model something trivial. Expect:
a normal boot and a real answer (billed to your key, not the trial pool).
`

End the session; click Disconnect in the credentials panel. Expect: status returns to
Not connected. `

Final check: with nothing connected, choosing Vaulted key billing and starting a session
is refused with Connect the required credential, then start a session.
`
⚠ Known gotchas for this flow (found 2026-07-25, real):

  • Use an OpenAI key with full permissions. A restricted/read-only key reaches OpenAI fine but

the session dies with 401 … Missing scopes: api.responses.write. Create the key with All
permissions (or Restricted + Responses→Write). The same error also appears if your org/project
role is Reader rather than Writer/Owner.

  • Only OpenAI and Azure are realistically usable today. Every vaulted provider is configured

with wire_api="responses", but Anthropic, Gemini, OpenRouter, and Ollama serve
chat-completions, not /responses — Codex removed wire_api="chat", so those families cannot
work without a translating bridge (Cloudflare has one; the others do not). ⚠ Reasoned from the
providers' APIs, not yet tested per-provider — worth a dedicated pass.
---

F10

Native Codex login

Proves the primary power-user mode: your own subscription inside the terminal.

Persona / precondition: signed in, no active session; an OpenAI/ChatGPT account.

0/9

Set Billing to Native login — sign in inside the terminal, bills my subscription.
Expect: the How native Codex login works guide expands (steps 01/02/03, a codex login
snippet with a Copy button) and the Model list changes to the OpenAI family
(gpt-5-codex, gpt-5, o4-mini). `

Start + confirm the session, choose 1. Sign in with ChatGPT, press Enter, and open the
URL Codex prints. Approve it. Expect: your browser lands on a localhost:1455 page saying
it can't be reached — this is expected, and the address bar holds your sign-in code.
`

Copy that whole address. Above the terminal, paste it into Signing in? Finish it here
and click Finish sign-in. Expect: Signed in — restarting your terminal… followed by
Signed in. Your terminal is ready. `

Look at the terminal — it should have relaunched by itself. Expect: a working Codex prompt,
signed in, with no sign-in menu and no manual Reset needed.

⚠ If you paste a URL from an EARLIER sign-in attempt, the expected result is the refusal
That link did not complete the sign-in — it is probably from an earlier attempt…, NOT a
success message. Worth testing deliberately: start a sign-in, cancel it with
esc, start a
second, and paste the FIRST one's URL.

Type a prompt (e.g. say hi). Expect: a real answer on gpt-5-codex, billed to your
subscription — no 429, no invalid_prompt. `

[sensitive content omitted]
Expect: a refusal, not a fetch — the relay accepts only loopback callback URLs.
`

Two-user check (needs Jan). With your native session live and signed in, have Jan start a
native session, begin a sign-in there, and paste HIS callback URL into HIS session.
Expect: Jan's sign-in either completes in his own session or is refused — it must never
land in yours. Then check your terminal: it must still be signed in as you.
`

Wrong-session paste. Paste a callback URL for a port your session does not own (e.g. any
unrelated local port). Expect: no sign-in from this session is listening on that port
the relay proves ownership rather than trusting the session id you named.
`

Final check: end the session, start a fresh trial session: it must NOT inherit your
native login (each session's auth is isolated — a fresh box knows nothing of your account).
`
---

F11

Trial guardrails (conditional)

Proves the money guard refuses instead of overspending. Only executable when the shared pool is actually low/exhausted — do not force it; check it on a busy day.

0/1

With the daily pool exhausted, click Confirm & start on a trial session. Expect: the
refusal The free daily pool is used up. It resets at 00:00 UTC. — and no session appears.
`
⚠ Known gotcha (observed 2026-07-25): the "resets at 00:00 UTC" copy is optimistic —
Cloudflare's allocation was still exhausted at 00:46 UTC. The refusal is honest; the reset time
in the message is not something agentify can guarantee. Consider softening the copy or deriving
the reset from upstream.
---

F12

God-kit overlay (god account only)

Proves the role gate and the overlay superset.

Persona / precondition: signed in as the god account; no active session.

0/3

Look at the wizard as the member account first (sign out/in as needed). Expect: no
Add the god kit checkbox exists for a non-god role. `

As god: check Add the god kit, start + confirm a trial session. Expect: session boots;
ls ~/skills | wc -l in the terminal prints 40 (the full set), and the six base skills are
among them. `

Final check: $backlog (a god-overlay skill) is recognized by codex as a command.
`
---

F13

Phone + notifications (real handset)

Proves the mobile surface — the known gap: it has only ever been tested in a synthetic viewport, never on the owner's device.

Persona / precondition: your actual phone, on the live site, signed in.

0/5

Load the site and sign in on the phone. Expect: no horizontal scroll anywhere; the tour
(if it opens) is a full-screen single-column dialog labeled Demo first · step details below.
`

Start a trial session and type into the terminal. Expect: the keyboard doesn't cover the
input; the session card and panes stack single-column and stay usable.
`

In Mobile notifications, click Enable notifications and accept the permission prompt.
Expect: chip turns enabled, message Notifications enabled on this device.
`

Click Send test. Expect: Test sent to 1 device(s). and a real push arrives on the
phone. `

Final check: end the session from the phone (End session reachable and working).
`
---

F14

Error paths sweep

Proves the product fails loudly and honestly at its edges.

0/4

GitHub storage without a token:[redacted] set Storage to GitHub — persistent workspace via
GitHub
, enter any repo URL, start + confirm.
Expect:** refusal Connect the required
credential, then start a session.
with the hint about the Connect model account panel.
`

Trial signup while closed: signed out, enter an email and click Start a free trial.
Expect: self-serve signup is not open yet. Enter an invite code above.
`

Marketplace search with gibberish: search zzzzzz. Expect: No kits match your
search.
— not an empty white area. `

Diff pane without git: in a session whose workspace has no repo, expand Code review.
Expect: workspace is not a git repository — run git init or clone a repo in the
terminal
. `
---

Not covered and why

  • GitHub persistent storage end-to-end (clone→work→push): needs a vaulted GitHub token and a

sacrificial repo. Planned, not scripted here.

  • Keep storage across teardown: needs two sessions separated by a deliberate teardown;

semantics should be scripted once the first pass of this document is green.

  • Multi-user concurrency (two people, two sessions, isolation between them): needs a second

tester (Jan) — sessions must never see each other's workspaces or auth.

  • Trial overage / midnight reset: crossing 00:00 UTC and CF billing edges — observed

passively, not forced.

  • Observify pane live mode: server-side URL is deliberately unconfigured; only the disabled

empty-state is testable (F3 final check).

  • Slack / Telegram notifications: operator tokens unset in prod.
  • Operator CLI (agentify setpass, deploy lanes): server-side operations, not user flows.
  • iOS push specifics: F13 covers one handset; iOS PWA push behaves differently and needs its

own pass if an iPhone is the daily driver.

Results

FlowVerdictFailures → tickets
F1 sign in
F2 tour
F3 trial session
F4 real work
F5 /test-it
F6 marketplace seed
F7 lifecycle
F8 export/import
F9 vault
F10 native login
F11 trial guard
F12 god overlay
F13 phone
F14 error paths
updated just nownext 3m 00s