Phase 1 — The VLM path (VSS gaming-GPU workstream)¶

For agentic workers: execute task-by-task, checkbox syntax; every step closes on an executed command + result in the ledger, never on written code. Milestone review is the built-in code review. Ledger rows for this phase are 1.x in docs/plans/2026-09-23-vss-gaming-gpu-ledger.md.

Status: ACTIVE — M0 closed 2026-09-25 (PR #6678 merged to main as a0105b63, every CI tier green; ledger "P0 — phase close"). Tasks 1.1-1.2 closed 2026-09-25 (ledger Phase 1 table rows 1.1-1.2); execution continues at 1.3.

AMENDED 2026-09-25: spec rev 6 (owner ruling, ledger F11). Specialists are resident, and their results reach the VLM.

• Task 4 (1.4) is rescoped. Keep --model-control-mode=none and serve a static vlm model repository (YOLO26, re-ID, and optionally the threat detector). Explicit mode, real load/unload in model_management.py (its 501s stay), the model_preload/model_unload contract flip and the specialist wake-on-open hook are dropped. 7f7113ed (TritonClient.load_model/unload_model) and the uncommitted ai/gateway/adapters/lifecycle.py were built for explicit mode. Revert and discard them unless something already depends on them.
• New Task 3b (1.3b): the specialist stage. Today nothing fills specialist_outputs: vlm_analyzer defaults it to empty, and AssessInput has no such field. 3b wires face match, plate read, re-ID and the optional threat detector into specialist_outputs, adds the optional field to AssessInput, and gives synthetic items verdict-changing specialist cases.
• The AssessInput freeze (environment facts, below) is reopened by the owner for exactly this one optional field. It defaults to empty, so the 421 stored items stay valid.

The reason: a fixed GPU footprint for S1 and S4, no specialist cold starts, and a predictable neighbour for RT-VLM's fixed vLLM reservation in Phase 4. Context comes from what reaches the prompt, not from how models load.

AMENDED 2026-09-25, later the same day: specialist model picks for 3b (owner rulings, ledger F12). Host-side research verified candidates on their model cards and repos. The full shortlist is docs/vss-integration/14-specialist-model-research.md.

• Licenses are not a selection criterion (owner ruling). Pick on accuracy, size, latency and ONNX fit. AGPL, GPL, non-commercial and research-only weights and datasets are all acceptable.
• The face gap is closed with option A, under conditions: one embedding space, four outcomes, a quality gate. Task 3b's face box below now names the models.
• Threat: the evidence argues against including the current detector. 3b still makes and ledgers the call.
• Re-ID: prove which vector space the vlm path uses end to end before calling it ready.
• YOLOE-26 is approved for a future open-vocabulary slot. It is a spec change (rev 7), not part of Phase 1.

Goal: land spec §"Phase 1: The VLM path" — the vlm_assess contract (1.1), the ai-vlm service + vlm compose profile (1.2), key_frame_selector + vlm_client + vlm_analyzer with §6 invariants, failure ladder, degradation wiring and wake-on-open (1.3), residency control (1.4), PIPELINE_MODE defaulting to vlm (1.5), the frontend (1.6) — closing on M1: vlm mode runs end to end on the A5500 with Qwen3-VL-4B as the smoke model, and every CI tier is green (spec §8:425). The A5500 bring-up (1.7) is owner-run.

Spec/authority: docs/superpowers/specs/2026-09-23-vss-gaming-gpu-profile-design.md rev 6 (this plan was drafted against rev 5; rev 6 changed §2's Triton row and specialist stage, §5's snapshot, §6's cold starts, and §8's 1.3b/1.4, so re-read those sections) — §2 (unit table :96-102), §3 (contract :113-166), §4 (data mapping :168-212), §6 (invariants/failure ladder/cold start :297-340), §7 (test layers :342-354), §8 Phase 1 + machine table (:414-425, :362) + A5500 vlm-mode checklist (:482-500). Owner ruling F10 (ledger :110-124): legacy unsupported, S2/S3 fixed bars, wire doctrine. Ledger E1-E13, G0/P0 rows, F1-F10 are the environment-of-record.

Machine split (E10, spec §8:362): all of Phase 1 except 1.7 runs in this sandbox — code, every test tier, and live GPU serving through agent-gpu (honest --vram ≈14 for a VLM server, agent-gpu rm when done). Integration requires the same-session docker compose -f docker-compose.test.yml up -d health (E11) or the tier didn't run. 1.7 and the Triton-explicit-mode live evidence (F3: no arm64/sbsa tritonserver base exists here) are A5500/owner-run [O]; repo-side code for both is proven here against fakes. Rev 6: there is no explicit mode. The A5500 evidence becomes "the gateway serves the static vlm repository, and no retired model loads".

Tech stack: unchanged from P0, plus: ai-vlm:sm103 (llama.cpp b7972, sm_103 aarch64) serving Qwen3-VL-4B for probe/serve evidence; contract generator scripts/gen-ai-contract.py; FakeProvider conformance via httpx.ASGITransport.

Sequencing (binding)¶

1. M0-before-everything: no Phase-1 code until PR #6678 merges. This plan is drafted in parallel; its decisions are reviewable now.
2. 1.1-first: the contract is the spine — every later task consumes vlm_assess/VlmVerdict/goldens. After 1.1, tasks 1.3 (backend), 1.4 (residency) and 1.6 (frontend) may parallelize (brief :37-39).
3. 1.2 before 1.3's live evidence: the chat-shape enforcement probe needs a served ai-vlm; 1.3's unit/contract work runs against FakeProvider meanwhile.
4. 1.3-before-1.5: the shipped default flips to vlm only once the vlm path exists end-to-end (a default with no path behind it is a broken default).
5. 1.5 carries its own test-wire obligation in the same slice (ledger item 14): the integration tier runs the SHIPPED default, so flipping it means the tier runs vlm_analyzer — vlm-mode fixtures land with the flip, legacy stays pinned only per-subject via legacy_wire. A flip that re-widens a wire pin is rejected.
6. 1.3b before M1 (rev 6): M1's end-to-end proof includes specialist context in the prompt.
7. 1.7 gates M1's hardware half [O]: bring-up, S1/S4, and the empirical wake-check (spec §6:335: health probes may not wake a sleeping llama.cpp — the sandbox proves the request-side at 1.3, the A5500 run proves wake-through-sleep).
8. Phase 2 may run before M1 closes (owner ruling 2026-09-25, ledger F13). Finish this plan's agent-doable work first: the rest of 1.3b, 1.5, 1.6, 1.7's repo side, and the M1 code review and report draft. Then, rather than idling on an owner gate (the 1.7 A5500 run, the PR that M1's CI needs, or any stop-and-ask), write the Phase 2 plan and execute it: the 2.1 replay harness, then 2.2's bake-off runs on the GB300. Switching to Phase 2 mid-plan is also allowed while a Phase 1 question waits on the owner. The limits are in F13.

Global constraints¶

• GPU only through agent-gpu; honest --vram; agent-gpu rm when done; never touch the rootful dgx-inference containers.
• D10: real-camera imagery/labels/snapshots never enter git; stored rows reference images by path; aggregate metrics only in docs.
• Legacy is UNSUPPORTED (F10): vlm_analyzer is a SIBLING of nemotron_analyzer, not an edit; no new legacy work; its tests keep passing; nothing deleted until R8 (build empty states). PIPELINE_MODE may accept legacy but nothing may wire it as a fallback or rollback target.
• Wire doctrine (ledger item 14): integration tier runs the shipped default; never re-widen a wire pin.
• Model licenses are not a selection criterion (owner ruling, ledger F12). Flag a license only when it restricts the use itself, for example KPR's Hippocratic License (surveillance clauses), and let the owner decide.
• Schema, golden payload and shape snapshot are GENERATED from VlmVerdict — never hand-written (spec §3:124-127; E5 is the cautionary precedent). Goldens emit only for object-rooted schemas — both vlm_assess sides get an object root.
• backend/ai_contract never imports ai.* and never imports the backend.api.schemas eager aggregate — VlmVerdict lives in a standalone module the generator can exec-import.
• New service ports go into .env.example FIRST (AGENTS.md:150): AI_VLM_PORT=8098 [V 2026-09-25: absent from .env.example; ports block ends at 8097; grep for AI_VLM_PORT hits only the spec].
• Probe discipline: enforcement probe budget n_predict/max_tokens = 400 (64 truncates and FAKE-IGNORES — S-2/P0.3 trap, "the serve is the gate"); re-run the probe on any engine/model swap; CI CLI and runtime gate share build_probe_schema/_probe_completion (scripts/vlm_probes/enforcement.py) so they cannot probe different grammars.
• S5 semantics: a failed/unparseable verdict is verification_failed + NULL score/level — never a default 50 (the pinned test_ai_degradation.py:246-254 shape is legacy semantics vlm mode must not reproduce). rejected ⇒ score clamped ≤ SeverityService low_max AND never notifies.
• TDD: failing test first, per site. Gates: uv run pytest backend/tests/unit/ -n auto; contracts -n0 --timeout=30, never xfail/skip in contracts/ai_providers; integration same-session compose-up; cd frontend && npm ci && npm run typecheck when frontend changes; scripts/gen-ai-contract.py --check whenever ai_contract is touched (1.1 and 1.4 will touch it).
• Doc-vs-repo conflicts → ledger row or dated errata, never a silent spec edit; keep [V][C][E][?][O][A]; run-count claims are [V] only with a same-session run.
• Every commit: uv tool run pre-commit run --files … + message check via DIRECT uv tool run conventional-pre-commit + Co-Authored-By trailer; zero hook churn before committing. Pushes, PRs and merges only on the owner's go-ahead.
• STOP AND ASK (unchanged): VLM pick at M2; go-live sign-off 3.1; S2_MAX/S3_MIN (owner sets before 2.2 — this plan encodes no numbers for them); any change to a D# or S#; anything touching dgx-inference; any Brev spend; any agent-gpu refusal; reasoning-LLM R10; doc-12 roadmap items; pushes/PRs/merges. Plus two plan-flagged ones: any widening of SECURITY_RELEVANT_CLASSES (notification_filter.py:22, frozen), and any change to the fire/smoke override semantics (below, Task 3) beyond "keep working unchanged".

Known environment facts carried forward (all [V] in the ledger, re-verified at plan time)¶

• Greenfield [V 2026-09-25]: pipeline_mode — 0 hits in backend/config/compose/.env.example; ai-vlm/vlm profile — 0 hits in all compose files; backend/services/vlm_client.py/vlm_analyzer.py/key_frame_selector.py — do not exist; VLM_OPS/VlmVerdict — 0 hits in the generator and backend/ai_contract; ProviderId.OPENAI_VLM/RTVI_VLM — 0 hits.
• Already shipped (do not rebuild): P0.25 null-safety (websocket.py:257-262, event_broadcaster.py:326-337, notification_filter.py:65-69/:113-132); P0.4 event_verifications (models/event_verification.py) + verification REST/WS field (schemas/event_verification.py:75 exclude_if absent-not-null; test_p04_verification_field.py pin); P0.3 constrained verdict + shared probe seam (nemotron_analyzer.py:326-370, :751+). event_verifications needs only create_all — no migration.
• Serving: sanctioned images ai-vlm:sm103 (b7972) + ai-vlm:sm103-b11090 (F8); recipe agent-gpu run --image ai-vlm:sm103 --vram 14 --port 8080 --mount models:/models --mount out:/out --env MODEL_PATH=… --env MMPROJ_PATH=… (G0.2/F1; libgomp from out/libs/gomp; models/ mode 755 for USER llama). Chat-completions multimodal json_schema already proven ENFORCED at b7972 with images (scripts/vlm_probes/s2_multimodal_schema.py).
• Wire shape asymmetry: the shipped P0.3 legacy probe is /completion with TOP-LEVEL json_schema; vlm_assess is POST /v1/chat/completions with response_format={"type":"json_schema"} + up to 4 base64 image_url parts (spec §3:141-145). The vlm probe needs its own nonce-const against the chat shape.
• Batch→event seam [V]: AnalysisQueueWorker._process_analysis_item calls analyzer.analyze_batch at pipeline_workers.py:1049; constructor default NemotronAnalyzer at :799; manager analyzer param :1511/:1600-1607; standalone factory create_analysis_worker :2155 and api/dependencies.py:962 bypass container.py:503 — all three seams need the same 1.5 branch. Batch opens in batch_aggregator.add_detection (:445, new-batch branch :605-629) — the wake-on-open hook.
• Contract landmines [V]: POST /v1/chat/completions is already llm_chat_completion's path (operations.py:215-228) and the fake mounts one route per op.path (fake/app.py:88-89) — vlm_assess must carry a distinct registered path; MATRIX_SLOTS is pinned to exactly 4 keys (test_conformance_ops.py:400-411) — new ProviderIds reuse a column, they don't mint a fifth; a new ProviderId without a PROVIDER_SLOT entry KeyErrors at registration (provider.py:137-143); the parity checker yields CLAIM-GW404 (CI red) until .github/ai-parity-baseline.json carries the new id.
• notification_filter.should_notify has NO production caller — the live notification decision is alert_engine.py:451 (already NULL-safe). DegradationManager is instantiated nowhere in prod (only read by system.py:4116-4145); ServiceHealthMonitor's list is hardcoded [yolo26, nemotron] (main.py:1024-1038). 1.3 wires one machinery and ledgeres the call.
• Eval store at $AGENT_GPU_DIR/out/eval-store/eval.sqlite: 421 items (benign 139 / incident 282), runs=0/results=0 [V re-verified read-only 2026-09-25]. AssessInput is FROZEN (backend/evaluation/assess_input.py, extra="forbid") — Phase 1 honors it or re-freeze is an owner call; the replay harness itself is Phase 2.1. Owner call made (F11): reopened for one optional specialist_outputs field, default empty. No other change.
• Frontend null-lie sites [V]: EventCard.tsx:156,232,282,410, EventDetailModal.tsx:95,520,600-602,1135, EventListView.tsx:21,298,362, MobileEventCard.tsx:30,88,180, EventTimeline.tsx || 0 sites, EventFilters.tsx:19-37 (no verdict filter), EnrichmentPanel.tsx:461,503+; ActivityFeed's interim chip (:166-224, comment "design is 1.6's") is the pattern to generalize.

File structure¶

Task 1: 1.1 — the contract (spine; spec §3:113-166)¶

• [x] TDD red first: a contract test that VLM_OPS yields a vlm_assess op whose response JSON Schema equals VlmVerdict.model_json_schema() (drift doctrine), and goldens regenerate. Decisions, made here and executed there: VlmVerdict home = backend/services/vlm_verdict.py (standalone; backend/api/schemas/__init__.py is a 700-line eager aggregate banned inside ai_contract; event_verification.VerificationCriterion is REUSED/aliased, not duplicated). Registered path = /vlm/chat/completions — the real wire is /v1/chat/completions (llama.cpp), but llm_chat_completion owns that op.path and the fake would double-mount; the evidence column records the real-engine spelling and vlm_client speaks it. Matrix column = per_model_server with required={"vlm_assess"} (the 4-key MATRIX_SLOTS pin forbids a fifth; spec's "own slot" is naming, not a key).
• [x] ProviderIds: OPENAI_VLM, RTVI_VLM minted with PROVIDER_SLOT entries + required={"vlm_assess"}; update every provider-keyed literal the spec doesn't mention (EXPECTED_DEPLOYED, SIBLING_SENTINELS, test_conformance_numeric.py:1035 loop, test_ai_provider.py:64-78 sets). Do NOT add vlm_assess to CLIENT_OP_MAP/CLIENT_GLOBS (the two-way AST map test would red, and _bound_or_reject would install a NOT-WIRED sentinel).
• [x] FakeProvider: vlm_assess returns a deterministic verdict keyed on an image-ref hash (deliberate payload read — today the handler ignores body except model_name); plus the §7 fault-injection knobs the ladder needs (timeout / schema-invalid body / 5xx). Fake-availability column fake: True is mandatory or registration fails.
• [x] Parity + counts: run scripts/check-ai-provider-parity.py; add the new id to .github/ai-parity-baseline.json so the ratchet reads DECLARED, not CLAIM-GW404. Bump AGENTS.md counts 37/43/43 → 38/45/45 (path-validated validator).
• [x] Gates: regenerate, scripts/gen-ai-contract.py --check clean; contracts tier -n0 --timeout=30 green (no xfail/skip — the whole tier must already know the new op); unit -n auto; ledger row 1.1 [V].

Task 2: 1.2 — the ai-vlm service + vlm profile (spec §2:98)¶

• [x] .env.example FIRST: AI_VLM_PORT=8098 into the AI SERVICE PORTS block (env-first rule AGENTS.md:150); VLM_MODEL_PATH/VLM_MMPROJ_PATH follow prod.yml:158's ${VAR:-default} style, not ghcr's hardcoded literal.
• [x] docker-compose.prod.yml: ai-vlm with profiles: [vlm]; llama-server flags per §2 (--jinja, --sleep-idle-seconds, --alias, 2 slots, ctx for 4 images + ~6K text + output — values chosen and ledgered, mirroring ghcr.yml:277-285's CTX_SIZE/PARALLEL budget comment). No hard backend depends_on (a profiled service breaks default up); graceful degradation via 1.3's wiring — ledger the call. docker-compose.ghcr.yml stays OUT of Phase 1 (the A5500 builds from context with CUDA_ARCHITECTURES=86, per the spec checklist); ledger that scope call.
• [x] ai/vlm/Dockerfile: add the §2 flags + 2-slot/ctx values; parameterize arch so sm_103 aarch64 and sm_86 x86 build from ONE Dockerfile via build-arg (no per-host fork).
• [x] GB300 dev path: ai-vlm served via agent-gpu (existing recipe) or the gb300 overlay — never by editing prod compose. Live serve check: container up, /health ok, probe s2_multimodal_schema-shaped check passes at --vram 14; agent-gpu rm after. Ledger row 1.2 [V] with serve output.

Task 3: 1.3 — selector, client, analyzer, §6 ladder (spec §2:99-101, §6:297-340)¶

• [x] key_frame_selector — Hypothesis property tests first: PURE, ingest-agnostic over image refs; 1-4 stills; best detection per camera/class + most recent; refs are paths under settings.foscam_base_path (Detection.file_path/thumbnail_path), never bytes.
• [x] vlm_client — chat-shape request (≤4 base64 image parts + structured context); enforcement probe on the chat shape with its own nonce-const, once per endpoint, caching only ENFORCED (reuse build_probe_schema/_probe_completion — one grammar for CI and runtime); CircuitBreaker-wrapped; "accepts-but-ignores schema must read as NOT enforced" regression (E5 class) pinned.
• [x] vlm_analyzer — SIBLING module; pure AssessInput builder honoring the frozen shape (production from DB, replay from store — ONE code path, what makes 2.1 possible); verdict→Event per §4: score after §6 invariants, level via SeverityService (model never emits a level), llm_prompt/interactions store the prompt with images REFERENCED BY PATH; one event_verifications row per event; broadcast payload MUST include the verification key (P0.4 renderer on the analyzer's own tx, not a re-query); replay never broadcasts.
• [x] §6 invariants as table-driven tests: rejected ⇒ clamped ≤ low_max AND never notifies even when a camera risk_threshold sits inside the low band; uncertain keeps its score + needs-review, notifies normally; VLM never originates an event — runs only on detector-closed batches; the fire/smoke bypass and fire override keep working UNCHANGED (non-VLM evidence paths; any change there = stop-and-ask).
• [x] Failure ladder via FakeProvider fault injection: transport error ⇒ retry ONCE at temp 0 within S4 budget; still bad ⇒ verification_failed + NULL score/level; notification via the shipped detector-only rule; repeated failures ⇒ breaker opens ⇒ ai-vlm unhealthy ⇒ DEGRADED surfaced via health endpoint + Prometheus. Decide and ledger which degradation machinery hosts this (spec-named DegradationManager vs test-banked AIFallbackService; prod instantiates neither today) and land the rejected-gate in alert_engine where the live decision actually happens.
• [x] Wake-on-open: fire-and-forget from add_detection's new-batch branch, POST /v1/chat/completions max_tokens:1, EXACTLY once per batch open (pin the count); never under the camera lock; never a health probe. Sandbox [V]: against an agent-gpu-served ai-vlm, idle-sleep the server, open a batch-shaped wake, observe wake + exactly one request; the through-sleep proof repeats at M1 [O]. [DONE 2026-09-26: pins + live [V] — the broker was briefly unreadable (ledger open-issue 15, closed); executed live: sleep observed server-side, real wake_ai_vlm() from sleep → one request + one wake, live assess returned a strict-schema verdict, and the traversal guard refused a path outside the capture root before any read. Live evidence appended to open-issue 15.]
• [x] Gates + integration (same-session compose-up): NULL-score events ride the live pipe db→WS→frontend-type→notification without crash or lie; legacy tier untouched (its tests still green — sibling, not edit). Ledger row 1.3 [V]. (Live-services reading: the sandbox's Postgres :5433 + Redis :6380 stand in for compose-up — same live-pipe semantics; ledgered with row 1.3.)

Task 3b: 1.3b — the specialist stage (spec rev 6: §2 specialist-stage row, §5 snapshot, §7; owner ruling F11)¶

• [x] Find the existing specialists first; don't rebuild them: (inventory + the threat call ledgered in row 1.3b [V])

• face detection and recognition against known household members (model_zoo yolo11-face, plus the face-recognition/household services);

• the plate read (fast-alpr), and matching it against household vehicles;
• re-ID (the Triton reid model, and the track or re-ID service that consumes it);
• optionally the threat detector (Triton threat).

Ledger each [V] with path:line. Ledger the threat-detector call (include it or not, with the reason). It is optional under D3 rev 6.

• [x] Face leg (F12 ruling: option A, with conditions). The inventory found a gallery and matcher (face_recognition_service.py:401, face_identity.py:114-151) but no server-side embedding extractor. Build it as a model_zoo loader behind an interface: an aligned 112×112 crop goes in, and an L2-normalized 512-d vector comes out. Test it on a deterministic fake. Absent weights mean unavailable. (Shipped d3f60079 + 3a8e2184 — the loader, the two hash-pinned models.yml rows, the deterministic fake AND a real-bytes proof. The ruling's enrollment half of "one embedding space" shipped 2026-09-26 in 752ad1e0 — server-side loader at both enrollment endpoints, model_id on FaceEmbedding + FaceDetectionEvent (sentinel default, mismatch ⇒ re-enroll), client-vector endpoint retired 410 — evidence in ledger row 1.3b.)

• [x] Models (research doc 14 §1). Detect with SCRFD-10G-KPS (scrfd_10g_bnkps.onnx), which gives 5 landmarks, then align with the ArcFace 5-point template. Embed with w600k_r50.onnx (IJB-C 97.25). Both come from InsightFace buffalo_l. Load only those two ONNX files, with onnxruntime on CPU. Do not use the insightface package's FaceAnalysis: it loads all five pack models, which is where the old "~1.5 GB VRAM" figure came from. Pin both files by SHA-256 in their models.yml rows. The vlm path uses SCRFD in place of yolo11-face, which gives boxes only (no landmarks), so its crops can't be aligned. The old detector stays until R8.

• One embedding space. Enrollment computes gallery vectors server-side with the same loader; a client-computed vector can't be trusted to share our space. Record the model id on every stored face embedding. If the probe's and the gallery's model ids differ, the result is unavailable (re-enroll), never a score. Ledger what happens to the vector-enrollment path.
• [x] Four outcomes: a household match (name + score), unknown (good crop, no match), not identifiable (below the quality gate), and unavailable. The quality gate is a minimum face size plus the SCRFD score, both from config. Never emit unknown for a crop that fails the gate. It pushes the VLM toward alarm, so it drives S2 false positives.
• [x] Match thresholds are config-driven and provisional: they get calibrated later on the owner's gallery and night footage. Ledger two items for later, without building them: AdaFace IR101/IR18 as the small/night-face challenger in a bake-off, and CR-FIQA as a face-quality model. (Both ledgered in row 1.3b, neither built.)

• [x] The face leg runs on CPU, so it stays out of the VRAM budget. Ledger its S4 cost per face. (Median 100 ms detect + 68 ms embed per face on an idle CPU, row 1.3b.)

• [x] Plate leg: fast-alpr as found. (Shipped in 3a8e2184 through fast_alpr_loader + HouseholdMatcher.match_vehicle; the [alpr] extra is absent in this sandbox, so the leg answers unavailable there and that arm is pinned. The v2 swap could not be checked here for the same reason — ledgered and skipped per this box's own clause.) Optional, only if cheap: fast_alpr_loader.py:32 pins cct-xs-v1-global-model, and cct-xs-v2-global-model scores US plate_acc 0.929 vs 0.905 (the author's validation splits, which differ). Check that the installed fast-plate-ocr accepts v2 names before switching; otherwise ledger it and skip.

• [x] Re-ID leg: prove one vector space end to end before calling it ready. (Ledgered in row 1.3b: the vlm path's compared vectors come from the CLIP-768 reid_service.generate_embedding store, the resident Triton reid model emits OSNet-512, the /clip router is retired in vlm residency — cross-space cosine refused rather than scored; the leg ships confidence-honest, the quality gap and the rev-7 swap candidates are ledgered as named.) The inventory cites a CLIP-768 producer (reid_service.py:430), but backend/api/schemas/household_matcher.py:15,31 and enrichment_data.py:255 document 512-d OSNet. Ledger which model produces the vectors the vlm path compares, and against what. If it is a general CLIP/SigLIP image embedder rather than a trained re-ID model, ledger that as a quality gap: a 2026 comparison of 11 models measured zero-shot SigLIP2 re-ID at mAP 2.8-14.2%. Wire the specialist anyway, with confidence-honest output, and ledger the follow-up swap for rev 7. Candidates are in research doc 14 §3: PersonViT ViT-S/16 at MSMT17 R1 89.2, CLIP-ReID ViT-B/16 at 88.7. Also note in the ledger that the incumbent OSNet's "MSMT17 73.3%" is mislabeled: that is the multi-source MS+D+C→Market Rank-1. CLOSED FOR REAL 2026-09-27 (ledger item 22, owner ruling "full swap"): the parenthetical above describes the pre-swap state. OSNet-AIN x1.0 512-d is now the one person-vector space end to end — sha256-pinned weights, one derived model_id shared by all three producers, untrusted rows never scored, CROSS_SPACE_REASON deleted, the leg's refusal replaced by a real computation, and the 73.3% mislabel corrected in models.yml itself.

• [x] Threat call, with evidence (doc 14 §2). (Call made: NOT included — evidence and the rev-7 YOLOE-26 hint path ledgered in row 1.3b; GATEWAY_ENABLE_THREAT stays false and the slot renders the explicit unavailable line.) The current Subh775/Threat-Detection-YOLOv8n card reports identical 83.0% recall for every class, on an unnamed Roboflow set, and calls itself "research and educational purposes". Published CCTV weapon results are poor even for trained detectors: handguns are 16-47 px, AP50 is 57.4 on real footage, and it drops to 3.7 across datasets. The recommendation is not to include it: leave GATEWAY_ENABLE_THREAT=false. Weapon evidence returns in rev 7 as hints from YOLOE-26, gated on overlap with a hand or arm. Ledger your call; overrule the recommendation only with contrary evidence.

• [x] TDD: in vlm_analyzer's production path, run the specialists on the key frames before vlm_assess. (Red-first; TestSpecialistStageWiring pins that the stage runs over exactly the frames the VLM sees, that its texts ride the assess request's specialist_outputs, that a stage exception degrades to all-unavailable keys while the verdict still lands and broadcasts — against the scripted fake client and the FakeProvider-backed tiers.) Record one short text result per specialist in specialist_outputs, for example face: 1 unknown adult, plate: ABC123, not a household vehicle or reid: seen at driveway 2x in 20 min. A failing or absent specialist records unavailable and never blocks or fails the verdict. Pin all of this with the FakeProvider.

• [x] AssessInput gains an optional specialist_outputs: dict[str, str] that defaults to empty. This is the owner-reopened freeze: only this field, and the 421 stored items must still load (pin a round-trip of a pre-rev-6 item). The prompt builder takes the outputs from the snapshot. Replay never re-runs specialists. (Shipped 6737c750 + 3a8e2184: field added, test_pre_rev6_item_still_loads pins the round-trip, vlm_client.prompt_text renders them, and the analyzer's replay arm reads stored texts with the re-run pinned off.)
• [x] Synthetic items: add verdict-changing specialist cases. The same scene should be benign with a known face or a household plate, and suspicious with an unknown face or an unknown plate at night. Keep them labeled so the Phase 2 bake-off can measure whether the VLM actually uses the context. Report whether the S-3 uncertain hedging changes when specialist context is present. F12: add a case pinning that a tiny or night face yields not identifiable, never unknown. (Shipped: five specialist-context-cases sets — known-face/household-plate benign vs unknown-face/unknown-plate suspicious, same shape, differing only in the specialist line, plus the night 24-px not identifiable pin. The loader renders each set through the SHIPPED classifier/builders, so the F12 clause is a corpus property; knob-flip pin included. S-3 hedging: labeled for the 2.2 bake-off to measure; no measured answer yet — deferred, ledgered. Corpus 408→413.)
• [x] Budget S4: time the specialist stage per batch against a live ai-vlm, and ledger it. The specialists are resident, so there's no load latency, only inference. (Ledgered in row 1.3b: ai-vlm:sm103-v12 served via agent-gpu at --vram 14, face legs registered through the deploy zoo path (sha256-verified). SHIPPED collect_specialist_outputs per batch, median of 5: 1-pick batches 0.041–0.100 s; the 4-key-frame ceiling batch 0.412 s median / 0.514 s max; one live assess through the shipped client 3.50 s wall (confirmed, risk 0); worst-case stage+assess 4.01 s vs the 30 s p95 bar. The formal S4 p95-with-cold-starts close rides 24 GB-class hardware per F13.)
• [x] Gates (unit, contracts, same-session integration). Ledger row 1.3b [V]. (Row 1.3b is CLOSED [V] in the ledger. Tiers re-run on d1a44c62 for the close report, not inherited (the two heads after it are docs-only): unit 30,533 passed with the sanctioned systemctl env-fail plus one timing-flaky liveness pin (0 reproductions in 4 repeats, 103/103 on a loaded re-run of its own file — named, not counted away); contracts 646 with zero xfail/skip in ai_providers; integration 4,089 passed / 107 skipped / 2 xfailed / 0 failed against PG :5433 + Redis :6380 connect-verified in the same session; gen-ai-contract.py --check current.)

Task 4: 1.4 — residency control (spec §2:102, §6:336-341; F3 splits the evidence)¶

AMENDED 2026-09-25 (spec rev 6, F11): the explicit-mode design below is dropped. The two new boxes directly below replace it; the original boxes are kept, struck through, as history.

• [x] Rev 6: llama.cpp idle sleep (with the 1.2 Dockerfile change) and the ai-vlm wake-on-open (already shipped in 1.3) stay. Evidence: idle-sleep proven [V]. There is no Triton specialist wake hook. (Idle-sleep + real-function wake proven live in open-issue 15's closure, 2026-09-26; wake-on-open unit-pinned in 1.3. No wake hook was ever built here — the explicit-mode artifacts were reverted per ruling 1.)
• [x] Rev 6: static vlm model repository. The gateway loads only YOLO26, re-ID and (if 3b includes it) threat, with --model-control-mode=none unchanged. Retired models are excluded by the repository, not by load control. Pick the mechanism, for example a vlm repository path or a filtered copy selected by env, and ledger it. Pin it with a test that the vlm repository contains exactly the chosen models. Leave model_management.py's 501s and the contract's model_preload/model_unload availability untouched. Revert 7f7113ed and discard the uncommitted ai/gateway/adapters/lifecycle.py unless something already depends on them (ledger which). Live evidence on the A5500 [O] is that no retired model appears in Triton's model index. (Mechanism: ai/gateway/residency.py named sets + entrypoint step 0d move-not-delete prune, env-selected; vlm set pinned exactly ("yolo26","reid") + opt-in threat; ledger row 1.4.)
• [ ] ~~Sandbox-provable half: llama.cpp idle-sleep flags land with 1.2's Dockerfile change; the wake-on-open Triton specialist hook gets unit pins against a fake gateway load API.~~ Dropped (rev 6).
• [ ] ~~Triton explicit-mode half~~ Dropped (rev 6). (code here, evidence on the A5500 [O]): entrypoint.sh:131 --model-control-mode=none → explicit with the vlm load set (YOLO26 + on-demand specialists ONLY — rev 5 deleted the legacy load set); model_management.py:682-760 501 handlers become real load/unload (wake-on-open for specialists requires a gateway that actually loads); flip model_preload/model_unload availability gateway:False→True, regenerate the contract, flip the 501-asserting tests to real-behavior tests. F3 stands: no arm64/sbsa tritonserver exists here, so live evidence is 1.7/1.7-adjacent [O]; ledger both halves distinctly.

Task 5: 1.5 — PIPELINE_MODE, default vlm (spec §2:76-79)¶

• [x] pipeline_mode: str = "vlm" + validator (nemotron_* block precedent); legacy parses (code stays until R8) but is LOUD at startup and wired nowhere as fallback. (Validator raises on unknown values; the legacy WARNING names the unsupported path — backend/core/config.py, pinned by test_pipeline_factory.py.)
• [x] Branch the analyzer at ALL THREE seams (constructor :799, create_analysis_worker :2155, api/dependencies.py:962) — plus container.py:503 — via one factory, so no path silently runs a stale analyzer. (backend/services/pipeline_factory.py; a 5th construction site the plan did not enumerate — batch_aggregator._process_fast_path's lazy builder — is rewired too; source-scan test + git grep prove zero direct prod constructions. Ledger row 1.5.)
• [x] Test-wire obligation IN THIS SLICE (binding rule 5): integration conftest gains a vlm-mode fake (fake vlm_assess endpoint passing the chat-shape probe, mirroring mock_llm_enforces_grammar); tests whose subject IS legacy pin legacy_wire per-subject; full tier re-run compared to the banked baseline — failure identities, not counts, are the evidence. (Session-autouse mock_vlm_enforces_grammar patches VlmClient._probe_enforcement — the one seam every consumer crosses, since the analyzer owns its client's construction; legacy_wire untouched and still per-subject.)
• [x] Gates; ledger row 1.5 [V] naming the tier identity diff. (Unit identity set identical to the banked baseline's sole env-fail; contracts 646; integration + typecheck + the honest first-run flake/wire-break note in the row.)

Task 6: 1.6 — the frontend (spec §4:196-201)¶

• [x] Typecheck-red first: generated types regenerate from the 1.1/1.3 surface (openapi/api.ts, scripts/generate-ws-types.py); components typed risk_score: number become nullable-aware — every || 0 coalescing site in EventTimeline/EventListView is a null-lie and gets a verdict-aware branch instead. (Shipped bcf1e253: resolveRiskLevel (server level wins → score → NULL = NO level, never 0) + riskSortKey/compareRiskSortKey (unverified ranks worst-first; the Infinity−Infinity=NaN comparator collapse fixed) in src/utils/risk.ts; every named-surface || 0 became a verdict-aware branch across 7 components + severityColors. The generated surfaces already carried the 1.1/1.3 additions — generate-ws-types.py --check and gen-ai-contract.py --check green, no regeneration needed.)
• [x] Shared verdict badge (generalize the ActivityFeed chip + RiskBadge): confirmed/rejected/uncertain/verification_failed + unverified; rejected stays listed but de-emphasized; event detail gains scene description, criteria checklist, reviewed frames (key_frame refs → thumbnails); EventFilters gains a verdict filter; retired-enrichment panels show "not analyzed in VLM mode" empty states (R8: empty states, not deletions). (Shipped 677a40f6 + 1c575e21 + ca6d734c: one-owner common/VerdictBadge keyed on the VERDICT — the interim ActivityFeed chip keyed on risk_score===null was blind twice over; the filter shipped end-to-end (backend EXISTS/'none'/422 prerequisite + FilterChips section + ?verdict= round-trip; the plan names EventFilters.tsx — dead code, zero non-self imports — FilterChips is the live surface, ledgered); EventVerificationSection renders scene/criteria/reviewed-frames thumbnails; retired enrichment reads "not analyzed in VLM mode", gated on the event's own verification row as the data-driven vlm-mode signal — no new pipeline-mode plumbing.)
• [x] vitest for new components + npm run typecheck; ledger row 1.6 [V]. (Events folder 1266 passed + the 1.6 six-file suite 307 + full-tier re-run on the final head, all EXIT=0; typecheck EXIT=0; ledger row 1.6 + the item-21 CI correction.)

Task 7: 1.7 — A5500 bring-up, vlm mode (owner-run [O])¶

• [x] Repo-side half DONE 2026-09-27 (ledger 1.7): scripts/a5500_precheck.py llm_model/ai_llm_mount checks → ai-vlm model/mount check (TDD); dated handout (successor to 2026-09-25-a5500-bringup-checklist.md) restating spec :482-500: GPU_*=0, CUDA_ARCHITECTURES=86 check-before-build, no legacy LLM deployed, Qwen3-VL-4B smoke serve + enforcement probe BEFORE any event reaches it, TMPDIR/__pycache__ traps, /platform-healthcheck + root AGENTS.md infra checklist. → 2026-09-27-a5500-vlm-bringup-checklist.md; the predecessor carries a SUPERSEDED banner. Real-tree verdict: 2 FAIL (gpu_assignment, cuda_arch — the FAIL text now names ai-vlm), and the VLM serving pair PASSes — the A5500's gap is GPU assignment and arch, not weights. Two compose findings the check now says out loud: docker-compose.ghcr.yml has no ai-vlm at all, and prod compose's ai-llm is unprofiled so a --profile vlm up starts it next to ai-vlm.
• [ ] Owner-run: execute; ledger [O] rows (commands + results). M1's wake-through-sleep check happens here (spec §6:335). Minimum end-to-end proof agreed with the owner in the handout: synthetic still → detector-closed batch → vlm verdict → event + verification row → badge → notification decision.

Task 8: M1 exit + close¶

• [x] Ledger 1.x rows each closed with command + result + commit + machine; [O] vs [V] honest; no unobserved passes. (Each of 1.1–1.7 now names its commit(s) — the missing hashes were pulled from git log at close time, not written from memory at task time; hardware-dependent halves stay [O] and say so.)
• [x] M1 assertion: (1) DONE — every CI tier green, re-measured on every head since the first green. d1a44c62 reported 67 success / 0 failure / 1 neutral / 7 skipped with all four previously-red gates green (Collection Sanity after the registry re-key 64cb82ba; the coverage trio after a re-run that confirmed the earlier rc=3 xdist worker crash was transient). Four heads followed (8ad70693 is the code-bearing one — the evaluation-harness fixes): a64ce655 (code-carrying head) and 219fa5f8 (head at this row's close) each re-report the SAME identity 67 / 0 / 1 / 7 from the paginated check-runs API after every job concluded (neutral = Trivy, its identity on every head; captures /tmp/ci_a64ce655_final.tsv, /tmp/ci_219fa5f8_final.tsv, 75 rows each), and at the PR level gh pr checks 6681 reads 67 pass / 8 skipping / 0 fail, MERGEABLE / CLEAN — so #6681's CI line is green on the current head, merge still owner-gated. One red on an intermediate head (41b4780e: Test Coverage Summary + CI Gate failure, 19 cancelled) is a self-cancellation artifact of pushing two heads 33 s apart — the repo's workflows set cancel-in-progress: true, and the summary's own line is X coverage gate job result = cancelled; the run's conclusion is cancelled, so nothing failed, verified by log. The enumeration of following heads retired here — it rots on the next push; what holds is the re-measurement. (2) A5500 vlm-mode end-to-end [O, owner-run]; (3) wake-request verified through sleep [O, owner-run]. S1/S4 measured on the A5500 at/after 1.7 — not claimed here.
• [x] Built-in code review across the diff; milestone report (proven with S#s / failed & why / next = Phase 2, already under way under F13 / decisions owed). (Closed at the fixes landing, 2026-09-27. Three passes over the range: the seven findings at 13f2d65e (§2 of the draft); the fan-out over the Phase 2 harness — ledger item 29, eight-for-eight of those agents' criticals fabricated, checked against the tree and rejected; and the full-file pass the fan-out skipped — ledger item 30: eleven findings, nine real at their file:line, all fixed red-first at 8ad70693, one measurement (S3's midpoint-vs-declared-minimum floor) queued as the owner's call, the bake-off report carrying its dated erratum. The review's residual is not code: two decisions — item 30's floor derivation and item 28's probe budget, whose 400-token cliff the review's follow-up had only ever measured on build A, and which the same-day B ladder (n=8/cell, shipped probe body: 0/8 echo at 400, 8/8 at 700) closed as a SHARED class — 700 clears it on both builds, input to item 28, no new question.) Report: 2026-09-27-vss-m1-close-report.md, on the confirmed CI/integration verdicts.)

Definition of done¶

• 1.1 shipped: vlm_assess + VlmVerdict-derived schema/goldens, FakeProvider deterministic-by-image-hash + fault injection, conformance green with the new ProviderIds registered; counts and parity baseline updated.
• 1.2 shipped: ai-vlm profiled service, env-first port, one Dockerfile for both arches; live GB300 serve + probe evidence in the ledger.
• 1.3 shipped: selector/client/analyzer with §6 invariants, ladder, degradation, wake-on-open — all with executed sandbox evidence; zero default-score sites on the vlm path.
• 1.3b shipped (rev 6, F12): the specialist stage feeds specialist_outputs, AssessInput round-trips it (and pre-rev-6 items still load), verdict-changing synthetic cases exist, and the stage's S4 cost is ledgered. The face leg runs SCRFD + w600k_r50 on CPU with hash-pinned weights, one embedding space and four outcomes. The re-ID vector space is ledgered end to end, and the threat call is ledgered with its evidence.
• 1.4 shipped (rev 6): idle sleep proven [V], and the static vlm model repository is pinned by test, with live evidence on the A5500 [O]. No explicit mode, no contract flip. ~~Triton explicit-mode code + contract flip landed, live evidence deferred to the A5500 [O] with F3 quoted (no silent N/A).~~
• 1.5 shipped: default vlm, all three seams branched, integration tier re-based on the new shipped default the same slice it flips.
• 1.6 shipped: badges/filters/frames/empty states + null-safe rendering, typecheck + vitest green.
• 1.7 handout dated and repo-side precheck updated; execution is the owner's.
• Every CI tier green on the M1 head.

Not done here (honest gaps): S1-S4 measurements (A5500), S2/S3 against the owner's bars (bars set before 2.2), any claim about the chosen production VLM (M2 is the owner's).

Out of scope¶

Replay harness vlm_replay.py (2.1) and the bake-off (2.2): they belong to the Phase 2 plan, which F13 lets start before M1 (sequencing rule 8). Any Brev spend (2.3); S2_MAX/S3_MIN numbers; ghcr-compose ai-vlm variant (ledgered scope call in 1.2); R8 deletions (legacy code stays); home-box go-live (3.1); RT-VLM (Phase 4); doc-12 postponed-roadmap items.