Observify

User Tests · conductify
← all user tests
USER TEST PLAN

conductify

Derived from: commit a7c3974 (dev), 2026-08-07, against the local dev instance only —

0/60 checked · 0 pass · 0 fail

Flows

check each step and final check
F1

Create an account

Proves the identity boundary opens with a real signup, not just a login form.

Persona / precondition: dev servers running, signed out, fresh tab on http://localhost:5173.

0/6

Load the site. Expect: it redirects to /signin and renders the heading Conductify above
a panel titled Create your account, with Email and Password fields and a button
Create account.

Type an email with no @ (e.g. nope) and any password, click Create account. Expect:
the server rejects it and the form shows the alert Invalid email address.

Fix the email, enter a 5-character password, submit. Expect: the alert
Password must be at least 6 characters.

Use a brand-new email and a 6+ character password, submit. Expect: you land on
/settings/provider (Choose your provider) — signup redirects straight into provider setup,
not the dashboard.

Click Need an account? Sign up is now Already have an account? Sign in — go back to
/signin (sign out first via the topbar Sign out button) and sign up again with the exact
same email/password. Expect: the alert Email already in use.

Final check: on the provider screen from step 4, click Continue without configuring
anything. Expect: it accepts the skip and lands on /dashboard showing Dashboard — proving
provider setup is optional, not a hard gate, at signup time.

F2

Sign in: shared wrong-credentials error

Proves deny-by-default: a wrong password and an unknown email read identically.

Persona / precondition: signed out, on /signin in Sign in mode (click the toggle if it still reads Create your account).

0/4

Enter seed-grounded@example.test with a deliberately wrong password, click Start session.
Expect: the alert Invalid email or password — no hint that the email exists.

Enter a made-up email (e.g. nobody-here@example.test) with any password. Expect: the
byte-identical alert Invalid email or password.

Enter seed-grounded@example.test with the fixture password
conductify-seed-fixture-not-for-signin. Expect: you land on /dashboard, topbar shows
your email seed-grounded@example.test and a Sign out button.

Final check: the Dashboard's Orchestrations list is not empty — it shows one row whose name
is derived from the prompt A habit tracker for people who want gentle streaks, not guilt trips.
and whose Status column reads Complete.

F3

Choose your provider

Proves both provider lanes save, validate honestly, and never echo a secret back.

Persona / precondition: signed in as any account; open Help is closed; navigate to /settings/provider (from the Dashboard, this is reachable once you add a provider chip click — if none is visible yet, sign up a fresh throwaway account per F1 to land here directly).

0/6

Select the BYOK Anthropic radio. Leave the key field empty and note the submit button is
disabled — Expect: the button (Save and validate) is disabled while the key field is
empty.

Type an obviously-fake key (e.g. sk-ant-not-a-real-key) and click Save and validate.
Expect: the key field clears itself immediately, and the panel shows a validation result —
either an error alert or Provider validated. — never the key you typed, anywhere on screen.

Click Replace key, then switch the radio to Free / local endpoint. Expect: the
Endpoint dropdown defaults to Ollama (local, no key) with Base URL prefilled
http://localhost:11434/v1 and a recommendation note ★ Best for orchestration here:
gpt-oss:20b
.

Change the endpoint dropdown to Groq — inference cloud (free tier, not xAI). Expect: Base
URL updates to https://api.groq.com/openai/v1 and Model updates to
llama-3.3-70b-versatile.

Click Save (leave the optional API key blank). Expect: the success line
Provider saved. Free tiers only need you signed in on the platform — no validation required.
— no validation call is forced for a free/local endpoint.

Final check: click Continue. Expect: you land back wherever you came from (Dashboard
for a signed-in return visit), and reopening /settings/provider shows the Groq endpoint you just
saved still selected — the choice persisted server-side, not just in the form.

F4

First-run guide opens over the sign-in screen (known defect)

Proves the first-run guide is scoped to a signed-in session, not just a browser profile.

Persona / precondition: a completely fresh browser profile for localhost:5173 (private/ incognito window, or clear cookies + localStorage for the origin) — signed out, never dismissed the guide before.

0/4

Load http://localhost:5173/ for the first time on this profile. Expect (correct
behaviour):
since nobody has signed in yet, the screen shows only the sign-in form —
Create your account — with no other dialog. Currently fails: AppLayout in
web/src/App.tsx calls GET /api/onboarding and opens OnboardingGuide (role="dialog"
aria-label="Conductify guide"
, Step 1 of 5 / Describe your project) unconditionally on
mount, with no check on state.status === "signed-in" or on the current route — so the modal
opens directly on top of the sign-in form before an account exists.

With the guide open, try clicking directly on the Email field visible behind the dimmed
backdrop, and try pressing Escape. Expect (correct behaviour, matching the app's own
dialog convention — e.g. DashboardDialog's scrim closes onMouseDown and its Escape
handler is wired): either the backdrop or Escape dismisses the guide so the field becomes
usable.
Currently fails: OnboardingGuide.tsx's .guide-scrim has no click/mousedown
handler and the component has no keydown listener at all — neither backdrop clicks nor Escape
do anything; only its own
Skip or Got it** buttons close it.

Click Skip. Expect: the dialog closes and the sign-in form is now interactive — this
part passes.

Final check: reload the page on the same profile, still signed out. Expect: the guide does
NOT reopen (confirming POST /api/onboarding/seen really did persist seen=true for this browser
profile) — which is exactly what shows the gating is profile-scoped rather than auth-scoped, the
root cause of step 1.

F5

Ground a new project from the Dashboard

Proves the empty-state → prompt → questionnaire hand-off for a brand-new orchestration.

Persona / precondition: signed in as a fresh account with zero orchestrations (sign up a new one per F1, skip provider setup).

0/5

On /dashboard, with no rows yet. Expect: the panel shows the heading
Start with the project you are about to build and a button Create orchestration.

Click Create orchestration. Expect: you land on /new showing From prompt to
performance.
and the lede Paste a project prompt and we'll turn it into a short
questionnaire — pick, customize, or defer each answer — then summarize what we learned.

Click Start grounding with the prompt textarea still empty. Expect: the inline alert
Tell us what you're building first. — no request is sent.

Type A trip-planning app for small friend groups. and click Start grounding. Expect:
the button reads Grounding… while it runs, then you land on the questionnaire at
/projects/<id> showing the section heading scope and the question text
Who is this project for? with three candidate buttons Just me, My team, The public.

Final check: the status line above the question reads 0 of 3 answered — the fixed mock
questionnaire always has exactly 3 questions.

F6

Answer the questionnaire: pick, write-in, defer, and review

Proves every answer path and the review step before grounding is final.

Persona / precondition: F5's fresh questionnaire on screen, question 1 of 3 showing.

0/6

Click My team. Expect: the button gets is-selected styling and you immediately
advance to question 2, section scope, text What's the single most important outcome? with
candidates Save time, Make money, Learn something; the status line reads 1 of 3
answered
.

Ignore the candidates: type Ship a v1 in two weeks into Or write your own answer and
click Use this answer. Expect: it advances to question 3, section constraints, text
Any hard deadline? with the note This helps us size the first milestone. and candidates
No deadline, Within a month; status line reads 2 of 3 answered.

Click Not sure yet — skip for now (this question is non-blocking). Expect: you land on
the review screen, heading Review your answers, listing all 3 questions with their answers —
the deadline question shows Deferred; status line before this screen would have read 2 of 3
answered (1 deferred)
.

Click Change next to Who is this project for?. Expect: it reopens question 1 with
My team still shown as selected (is-selected / aria-pressed="true").

Press the Escape key. Expect: you return to the review screen without changing the saved
answer.

Final check: click Continue to grounded summary. Expect: you land on the summary
screen, heading Grounded summary, showing the alert ⚠ 1 item deferred.

F7

Engineer the orchestration plan

Proves grounding turns into a real, deterministically-fixtured plan under mock mode.

Persona / precondition: F6's grounded summary on screen (or sign in as seed-grounded@ example.test, whose project is already at this stage — skip straight to step 2).

0/7

On the summary screen, click Engineer orchestration. Expect: the button reads
Engineering… while it runs, then you land on the plan screen, heading Orchestration plan
v1
.

Read the four stat tiles at the top. Expect (mock fixture is fixed regardless of your
prompt):
3 units, 4 phases, 2 guardrails, 5 commands.

Open the Units panel. Expect: exactly three cards named api-core (family claude,
tier flagship), web-ui (family chatgpt, tier standard), e2e-watch (family local,
tier lightweight).

Open the Phases panel. Expect: four rows in order — pilot, fan-out, watch,
done-audit — the last one described as Cross-check every ledger claim against git truth
before minting done
.

Open the Guardrails panel. Expect: exactly two rows: G-GATE (Nothing lands red) and
G-HONEST-LLM (Mock output always flagged; never presented as real).

Open the Commands panel and expand the Launch api-core card. Expect: the command well
shows exactly claude -p "Build the REST API and persistence layer", and a Firing sequence
reveals Resolve executorclaude · flagship.

Final check: click Copy on the Launch web-ui card, then paste anywhere. Expect: the
clipboard contains exactly codex exec "Build the React screens consuming the API" and the button
briefly reads Copied.

F8

Export the plan: markdown, commands, and the runnable kit

Proves every export artifact is real and reflects the plan on screen.

Persona / precondition: F7's plan on screen (conductor's desk visible on the right).

0/5

Click Export markdown in the desk's Export panel. Expect: a file
orchestration-plan-v1.md downloads; opening it shows the same units/phases/guardrails you
just read on screen.

Click Export commands. Expect: a file orchestration-commands-v1.sh downloads
containing all 5 commands, including the exact kickoff line echo "Conductify plan v1: 3
units, phases pilot -> fan-out -> watch -> done-audit"
.

Click Export orchestration kit. Expect: the button reads Exporting orchestration
kit…
then Exported orchestration kit, and a .zip downloads.

In a terminal, unzip the downloaded kit and run ./conductify status. Expect: the script
is executable and prints a status readout without erroring (no command not found, no
traceback) — per app/RUNBOOK.md, the fixed verb surface is exactly status, start,
pause, continue, stop, watch, audit, next.

Final check: run ./conductify next. Expect: it names the next not-started unit from
among api-core, web-ui, e2e-watch — not an error, not a blank line.

F9

Generate and export packets

Proves the applicability heuristic and that every generated packet is honestly mock-flagged.

Persona / precondition: F7's plan on screen, whose grounded answer to "Who is this project for?" was My team (not "Just me") — this makes all three packet kinds applicable, matching server/src/packets/applicability.ts's rule.

0/6

Scroll the desk to Packets. Expect: three cards, all offered (none show a "not
offered" line): Branding, Wireframes, Tickets.

Expand Wireframes and click Generate. Expect: the summary line reads 2 screens
once generation finishes, and a Mock mode badge is visible on the card.

Open the generated wireframes (click the card). Expect: screens named exactly Home and
Recipe detail, with Home containing regions Header, Recipe grid, Quick add.

Expand Branding and click Generate. Expect: the summary line reads 3 colors · 3
taglines
; opening it shows the palette Sage #7A9B76, Clay #C97B5C, Parchment
#F4EFE6 and the tagline Cook what you have.

Expand Tickets, click Generate, then click Export markdown. Expect: a file
tickets-packet.md downloads containing ticket id PKT-T001 titled Recipe schema +
persistence
.

Final check: click Regenerate on the Branding card. Expect: the button briefly reads
Regenerating…, then a Generated confirmation appears, and the card auto-collapses a moment
later — regenerating never duplicates the card or leaves two open at once.

F10

Dashboard: edit the base prompt and delete an orchestration

Proves the row-menu actions and their confirmation dialogs.

Persona / precondition: signed in with at least one orchestration row (any prior flow's project, or seed-grounded@example.test's seeded row).

0/6

On /dashboard, click the row's overflow (three-dot) menu button. Expect: a popover with menu items
Open, Edit base prompt, Delete.

Click Edit base prompt. Expect: a dialog titled Edit base prompt with the warning
Changing this re-grounds the project — the current score and packets will be replaced. and a
textarea pre-filled with the current prompt.

Clear the textarea entirely and click Save and re-ground. Expect: the inline error
The base prompt cannot be empty. Enter the project you want to re-ground. — no request is
sent.

Click Cancel, reopen the menu, click Delete. Expect: a dialog Delete <name>? with
the body This removes the orchestration, its score, and its packets. Billed generations
already recorded in usage are not deleted.

Click Delete orchestration. Expect: the button reads Deleting…, the dialog closes,
and the row is gone from the list.

Final check: reload /dashboard. Expect: the deleted row does not reappear — the deletion
was persisted server-side (DELETE /api/projects/:id204), not just removed from local state.

F11

Ownership and error-path sweep

Proves the API refuses honestly at its edges and the applicability heuristic really discriminates by audience, not by a hard-coded packet list.

Persona / precondition: a terminal with curl; a second, personal-only project (see step 4).

0/5

Signed out, in a terminal: curl -s -o /dev/null -w "%{http_code}\n"
http://localhost:3001/api/me
. Expect: 200 with body {"user":null} (an unauthenticated
identity check is not itself an error).

Still signed out: curl -s -w "\n%{http_code}\n" http://localhost:3001/api/orchestrations.
Expect: body {"error":"Sign in required. Create an account or sign in to access
projects, packets, and usage."}
and status 401.

Signed in, in the browser, edit the URL to /projects/does-not-exist-12345. Expect: a
brief loading state, then a redirect to /dashboard — no stack trace, no blank screen.

Create one more fresh project (per F5), but this time answer Just me to Who is this
project for?
, answer the rest however, and engineer its plan. Open Packets. Expect:
the line Branding not offered — The grounded answers don't describe an audience beyond the
builder.
while Wireframes and Tickets remain generate-able on the same project.

Final check: on that same personal-only project, generate the Tickets packet. Expect: it
succeeds normally (2 tickets) — the audience heuristic gates Branding specifically, it does not
degrade the rest of the packets pipeline.

Not covered and why

  • Real Anthropic generation (a genuine, non-mock questionnaire/plan/packet call): needs a real

[sensitive content omitted]
section already carries verbatim real-call evidence (model, token counts, a persisted
llm_calls row) — re-run that, not this document, for a fresh real-call proof.

  • A genuinely reachable local Ollama / free-tier endpoint: app/DOGFOOD.md's auth self-use

pass probed 127.0.0.1:11434 and found it unavailable in that environment too; F3 above only
proves the save path, not a real completion through the endpoint.

  • Production deployment: no host, no Coolify app, and no git remote exist yet

(specs/deploy/BACKLOG.md's non-delegable human gates are still open) — everything here targets
localhost only.

  • Multi-user isolation (two different accounts never seeing each other's projects): the

ownership guard is covered by server/src/ownership.guard.test.ts in the gate; this document
only exercises it from one account's perspective (F11).

  • The output_truncated ceiling path for an oversized Tickets packet: only reachable with a

real, very large model response; mock mode's fixed 2-ticket payload can never trigger it.

  • Mobile/responsive layout: no viewport or touch-target pass was done this round.

Results

FlowVerdictFailures → tickets
F1 sign up
F2 sign in
F3 choose provider
F4 first-run guide on sign-in (defect)
F5 ground new project
F6 questionnaire
F7 engineer plan
F8 export plan/kit
F9 packets
F10 dashboard edit/delete
F11 error paths
updated just nownext 3m 00s