# S10 — Agent / AI infrastructure

Re-surveyed **2026-08-07** (snapshot 2026-08-08 02:02 UTC; previous pass 2026-07-28 13:06 UTC). Every declared scan ran live at full fidelity: no sudo, no degrade rung, no `auth.json`/`ntfy.env`/`*token*` file opened, and no transcript store read — bulk stores were counted and sized only. `kill -0` was used as the POSIX read-only liveness probe; it delivers no signal.

**17 inventory items:** 8 observer watchers · 2 observer daemons (new kind) · 4 agent-CLI homes · 2 ollama models · 1 chatcode component. Unknowns: **1**.

## The headline: the fleet is alive again

On 2026-07-28 this surface recorded **all nine watchers dead**, every row carrying `broken`. At this scan **all eight are live**, and each is confirmed *twice* — the pid-file lease plus a `ps -p <pid> -o command=` match against the expected script. That double check matters: a bare `kill -0` cannot tell a live watcher from an unrelated process that inherited a recycled pid. The fleet has been up continuously since **2026-08-05 08:51:51** (the router alone restarted 2026-08-06 20:03:05, after a recalibration).

## What changed on the roster

| Watcher | Status | Why |
| --- | --- | --- |
| `claude`, `codex`, `grok`, `agy`, `inbox`, `router` | live, unchanged roles | Local Python watchers under `~/.observer-watch`, all pid 572xx from the 2026-08-05 arming |
| `pi` | live | Still `remote-loop.sh` → `wolf@wolfplex.local`. **Now the only remote watcher.** |
| `macb`, `macb-codex` | **retired** | Both pid files survive only as `pids/macb.pid.bak-restart` and `pids/macb-codex.pid.bak-restart`, stamped 2026-07-28 19:56 — the Mac B estate was removed and its two watchers went with it |
| `export` | **new** | Runs `~/Projects/Kit/scripts/session_export.py watch --interval 1800 --hours 6` — a session-transcript export pass every 30 minutes over a rolling 6-hour window |

Net: **9 → 8** pid-tracked watchers. `export` is the only watcher whose script lives *outside* `~/.observer-watch`, so its code belongs to S11 while its pid lease belongs here.

## Two new always-on daemons — and a gate blind spot

Two processes now run detached from `~/.observer-watch` that did not exist at the last pass:

- **`memory-sampler.py`** — live since 2026-08-06 18:28 (27h+). Samples RAM per agentic session tree every 5 minutes (claude, codex, ollama, llama-server, observify, the Docker VM, chrome, and the observer fleet as one group), appending to `memory-ledger.tsv` with tree RSS, root footprint, and a per-cycle `system` row carrying swap and free percentage.
- **`ceiling-checker.py`** — restarted 2026-08-07 21:45 after its ceilings were recalibrated from 305 ledger cycles / 27h. Polls the ledger ~every 60s, applies per-kind ceilings, and **alerts only** — it never kills, signals, or throttles; eviction is explicitly Phase 2 and not built. Rules need the condition on consecutive cycles (4 for claude/codex rows, 2 elsewhere) and fire on transition only.

Together they are a textbook Article IX pairing: a probe that can return false, plus a proven path to the human (Pi ntfy primary, labelled `osascript` fallback). The recalibration itself was driven by that discipline — the old `free_pct < 15` rule was true in **0 of 305 cycles**, a probe that could not fire, and it was darkening a whole ANDed rule.

**Both were ungated — flagged rather than quietly patched, and now closed by conductor ratification (2026-08-07).** They hold their pidfiles at the *root* of `~/.observer-watch`, not under `pids/`, so this surface's parity probe (`for f in ~/.observer-watch/pids/*.pid`) could not see them, and the `observer-daemon` kind sat outside every `kind_in` scope. Their rows were evidence-backed and *uncontradictable* — the worst combination a census can produce, because it reads exactly like a verified row.

The conductor's ruling, recorded in the spec: *a census that cannot see the machine's own always-on watchers is exactly the blind spot the surface exists to prevent — these daemons are guards, and an ungated guard is indistinguishable from one that never ran.* The probe moved to meet the daemons' own pidfile convention rather than the daemons being narrowed to fit the probe. What landed in `specs/s10-agent-infra.yaml`:

```yaml
parity:
  - name: observer-daemon-count
    kind_in: [observer-daemon]
    labels_cmd: |
      for f in ~/.observer-watch/*.pid; do basename "$f" .pid; done
liveness:
  - name: observer-daemons-alive
    scope_kinds: [observer-daemon]
    cmd: |
      for f in ~/.observer-watch/*.pid; do
        name=$(basename "$f" .pid); pid=$(cat "$f" 2>/dev/null)
        [ -n "$pid" ] || continue
        kill -0 "$pid" 2>/dev/null || continue
        ps -p "$pid" -o command= 2>/dev/null | grep -q "/.observer-watch/$name.py" && echo "$name"
      done
```

The liveness probe is deliberately **stronger than the drafted version**. A bare `kill -0` proves only that *some* process owns the pid the lease file recorded — on a machine spawning and reaping processes constantly, a recycled pid inherited by anything else would let a dead guard report itself alive, which is a probe that cannot return false. The `ps -p` command-line match against `~/.observer-watch/<name>.py` is what closes that, and it is cheap here only because a daemon's script path is derivable from its label. The watcher kind cannot borrow the trick — `claude` runs `claude-all-watch.py`, `pi` runs `remote-loop.sh`, `export` runs `session_export.py` — so that check stays a scan-time discipline in `field_derivation.live_now` there, and the spec now says why the two probes differ in strength instead of leaving it as an unexplained inconsistency.

Also landed: `observer-daemon` added to `field_derivation.kind` and to the `id`/`label`/`paths`/`trigger`/`live_now`/`purpose`/`kill` derivations (purpose now *requires* stating whether an alerting daemon is ADVISORY or ACTIVE — "a memory guard" that turns out to kill processes is a different machine than one that pushes a notification); a `scan_commands.observer_root_daemon_pids` block; the root-pidfile glob added to `scope.exhaustive`; `*.env` added to the never-open hard rule for `ntfy.env`; and a re-evaluation of `transient_services` for the new kind — **not** granted, because a stale root `.pid` whose process is gone is a real failure to surface, not a transient exemption.

## The skills mirror has drifted for the first time

`~/.agents/skills` is a synced *copy* of `~/.claude/skills`, not a symlink, and on 2026-07-25 the two were identical. They are no longer:

```
diff <(ls -1 ~/.claude/skills) <(ls -1 ~/.agents/skills)
21d20  < graphify
45d43  < update-board
```

47 skills on the Claude side, 45 on the agents side; both misses are one-directional. `graphify` landed 2026-07-31 alongside the `uv tool` install S8 records; `update-board` is the second. Whatever syncs these trees either has not run since those skills landed or does not cover them. **Investigate the sync mechanism — do not hand-copy the two directories**, or the next drift will be silent again. This is precisely the condition the spec's `agents_skills_bridge` probe was written to catch, and it caught it.

## What it costs

The router's own ledger (`observer-router.py --report`, real numbers from the models' response bodies):

| Metric | Now | At authoring |
| --- | ---: | ---: |
| Total real spend | **$1.384220** | ~$0.85 |
| Local prompt tokens | 38,048,491 | — |
| Local completion tokens | 6,114,008 | — |
| Haiku tokens (cloud fallback) | 68,009 | — |

The local brain is carrying roughly **99.8%** of the token volume; the entire cloud slice is 68k tokens for $1.38 lifetime. The `RECONCILE: MISMATCH` between `journal.jsonl` and `state.json` is the expected cache lag the spec documents (state is a periodically-recomputed view of an append-only journal), not a defect.

The *real* cost is memory, not dollars. `gpt-oss:20b` is now pinned resident with **`UNTIL = Forever`** rather than a timed idle window, so `llama-server` (pid 33153) holds ~13 GB of unified memory continuously. That single change is why the ceiling-checker carries a dedicated `llama-server` rule with a 15 GB panic threshold. The second always-on cost is unchanged: `dev.chatcode.gateway` runs under `caffeinate -i -s`, so the Mac cannot sleep while it is loaded.

## Plaintext credentials on this surface

| Where | What | How it was established |
| --- | --- | --- |
| `~/Library/LaunchAgents/dev.chatcode.gateway.plist` | `GATEWAY_AUTH_TOKEN` | Redacted plist projection; value replaced with `<redacted>` before the output left the terminal |
| `~/.codex/auth.json` | file-level | Mode `0600`, 4577 bytes — **never opened**; filename + permissions is sufficient evidence |
| `~/.grok/auth.json` | file-level | Mode `0600`, 1757 bytes — **never opened** |
| `~/.observer-watch/ntfy.env` | **new this pass** | Mode `0600`, 222 bytes — the ceiling-checker's push-channel credentials, **never opened** |

`GATEWAY_SSH_KEYS_FILE` was also redacted. It is in fact a filesystem path rather than a credential, but it matches the secret-like key pattern on "KEYS" and the rule is redact-on-match — redacting a path costs nothing, second-guessing the rule costs everything.

## What chatcode actually is

`dev.chatcode.gateway` is a WebSocket relay dialing `wss://cp.chatcode.dev/gw/connect` with gateway id `gw-baIv-3rxP0IWEl1W1ueAa`, a 5s health interval, a 50-session ceiling, `RunAtLoad` + `KeepAlive`, wrapped in `caffeinate -i -s`, logging to `~/Library/Logs/chatcode-gateway.{log,err.log}`. It is the bridge that lets an external control plane drive terminal sessions on this Mac — which is exactly why it holds both a bearer token and an SSH authorized-keys reference. Stopping it would sever remote session control; it would not affect any local watcher. **`dev.chatcode.maintenance` remains absent** — only the gateway plist exists, unchanged from 2026-07-28, so this kind stays at one row.

## The one remaining unknown

`nomic-embed-text:latest` (274 MB) has never been resident in `ollama ps` across three surveys. This pass narrowed the question rather than closing it: a bounded search across `~/.observer-watch/*.py` and every `~/.claude/skills/*/SKILL.md` found **exactly one mention on the machine** — `hire/SKILL.md` line 60, where it appears in a model-registry table as a *candidate* for a "Librarian / RAG" seat. A roster entry is not a caller. No process, script, or config invokes it. It stays flagged `unknown` (the honest verdict is "no consumer found", not "no consumer exists") and it is a clean **prune candidate**: `ollama rm nomic-embed-text:latest` reclaims 274 MB once Wolf confirms no planned RAG use.

## Dispositions

- `observer-claude`, `observer-codex`, `observer-grok`, `observer-agy`, `observer-inbox`, `observer-router`, `observer-pi`, `observer-export` — **keep**; all live, all double-verified, `broken` flags cleared.
- `observer-memory-sampler`, `observer-ceiling-checker` — **keep**; both live and self-documenting. The gate gap is **closed**: `parity-observer-daemon-count` and `liveness-observer-daemons-alive` now cover them, so they are no longer unwatched watchers.
- `claude-code` — investigate; the `frontend-design` plugin's literal `unknown` cache-version directory with no `bin/` is unchanged since 2026-07-25 and keeps the `broken` flag. Note the clean-environment PATH in S8 does *not* carry the dead `.../unknown/bin` entry — it appears only in caller-inherited PATHs, which is why S8 records it as prose.
- `agents-skills-bridge` — **investigate**; first recorded mirror drift. Fix the sync, not the symptom.
- `codex` — keep; watch the growth. 260 project sections and a 780 MB `logs_2.sqlite` (up from 336 MB) make it the heaviest CLI home on disk.
- `grok` — keep; pinned at `grok-0.2.117-macos-aarch64`, the resolution target for S8's `grok`/`agent` symlinks.
- `gpt-oss-20b` — keep; but the `UNTIL = Forever` pin is a deliberate ~13 GB standing memory reservation and should be a conscious choice, not a default.
- `nomic-embed-text-latest` — **prune candidate**; no consumer found on the machine.
- `chatcode-gateway` — keep; security-sensitive by design. Credential-bearing, externally bridged, and sleep-blocking — the three properties that earn the flag.