Skip to content

Home Security Intelligence - Roadmap

Audience: Agents/engineers continuing work after the MVP phases in docs/plans/.

This document captures ideas beyond the MVP and provides enough context to help another agent:

  • understand why an idea matters,
  • identify where it would land in the codebase,
  • estimate complexity / prerequisites,
  • and avoid "nice idea, unclear next step" traps.

Context (What MVP already establishes)

The MVP architecture is already oriented around event-based reasoning:

  • Camera uploads arrive via FTP into /export/foscam/{camera_name}/
  • A file watcher detects new media
  • YOLO26 produces per-frame detections
  • Redis batching groups detections into an "event window"
  • Nemotron produces a risk score + summary + reasoning
  • Results persist in PostgreSQL and are surfaced via APIs/WebSockets to the dashboard

This makes the system naturally extensible in three directions:

  1. Better signals (more/cleaner context into the model and into heuristics)
  2. Better delivery (alerts, workflows, investigation UX, reliability)
  3. Better understanding (search, patterns, entities, continuity)

Guiding principles (to keep scope sane)

If you're an agent picking up roadmap work, treat these as constraints:

  • Local-first: default to single-machine, PostgreSQL + Redis, minimal ops burden.
  • Privacy-aware: cameras are sensitive; prefer opt-in for identity features (faces/plates).
  • Event-centric: keep "event" as the unit of human attention; avoid raw-frame firehose UX.
  • Measurable wins: prioritize changes that reduce false positives, increase recall, or improve time-to-action.
  • Operational robustness: a home system must recover from restarts, partial outages, and storage constraints.

Implemented Features (Post-MVP Complete)

The following features originally planned for post-MVP have been implemented:

1) Alerting & Escalation - IMPLEMENTED

Location: backend/services/alert_engine.py, backend/services/notification.py, backend/models/alert.py

What's implemented:

  • Alert rules engine with multiple condition types:
  • Risk threshold (e.g., alert on risk_score >= 70)
  • Object-based rules (person, vehicle, animal)
  • Camera selection (specific cameras or all)
  • Time range/schedule support
  • Zone-based conditions
  • Minimum confidence thresholds
  • Alert deduplication with cooldown/suppression logic (backend/services/alert_dedup.py)
  • Notification channels:
  • Email via SMTP
  • Webhooks (HTTP POST with SSRF protection)
  • Push notifications (stubbed)
  • Severity taxonomy: LOW, MEDIUM, HIGH, CRITICAL
  • Frontend: AlertRulesSettings component, NotificationSettings

2) Spatial Intelligence & Zones - IMPLEMENTED

Location: backend/models/zone.py, backend/services/zone_service.py, backend/api/routes/zones.py

What's implemented:

  • Per-camera zones with polygon/rectangle support
  • Zone types: entry_point, driveway, sidewalk, yard, other
  • Normalized coordinates (0-1 range) stored as JSONB
  • Spatial heuristics:
  • Ray casting algorithm for point-in-polygon testing
  • Dwell time tracking
  • Line crossing detection
  • Approach vector calculation (direction, speed, ETA)
  • Zone context in alerts: Rules can filter by zone_ids

3) Entity Continuity (ReID-lite) - IMPLEMENTED

Location: backend/services/reid_service.py, backend/services/osnet_loader.py

What's implemented (re-ID producer swapped 2026-09-26, ledger item 20 — OSNet-AIN x1.0 is the one person-vector space; the CLIP /clip router no longer computes person vectors):

  • OSNet-AIN x1.0 embeddings (512-dimensional person vectors) from the resident zoo handle, SHA-256-pinned weights, model_id provenance on every stored vector
  • Cross-camera entity matching with configurable similarity threshold (default: 0.7, OSNet-space, provisional pending calibration)
  • Redis storage for embeddings with 24-hour TTL, partitioned by model_id
  • Concurrency-based rate limiting to prevent resource exhaustion
  • Entity attributes tracking (clothing, color from vision extraction)
  • Vehicles: no embedding producer ships in resident mode — vehicle identity rides license-plate match (a vehicle re-ID model is a named follow-up)

4) Pattern-of-Life / Anomaly Detection - IMPLEMENTED

Location: backend/services/baseline.py, backend/models/baseline.py

What's implemented:

  • Baseline activity modeling with exponential moving average and configurable decay
  • Per-camera activity rates by hour and day-of-week
  • Class-specific frequencies (e.g., "vehicles after midnight are rare")
  • Rolling 30-day window for baseline calculations
  • Anomaly scoring with configurable threshold (default: 2.0 standard deviations)
  • Minimum sample requirements for reliable detection

5) Search & Investigations - IMPLEMENTED

Location: backend/services/search.py, backend/api/routes/events.py

What's implemented:

  • PostgreSQL full-text search with tsvector/tsquery
  • Search across: summary, reasoning, object types, camera names
  • Phrase search using double quotes
  • Boolean operators: AND, OR, NOT
  • Filtering: time range, camera IDs, severity levels, object types, reviewed status
  • Relevance-ranked results with scores

6) Better Media Handling (Clips) - IMPLEMENTED

Location: backend/services/clip_generator.py, backend/services/video_processor.py

What's implemented:

  • Event clip generation from video files using ffmpeg
  • Image sequence to video conversion (MP4/GIF)
  • Configurable pre/post roll (0-300 seconds)
  • Secure path validation preventing command injection
  • Output formats: mp4, gif with H.264 codec

7) Reliability & Operations - IMPLEMENTED

Location: backend/services/circuit_breaker.py, backend/services/retry_handler.py, backend/core/redis.py

What's implemented:

  • Circuit breaker pattern with CLOSED/OPEN/HALF_OPEN states
  • Prometheus metrics for circuit breaker monitoring
  • Retry handler with exponential backoff and jitter
  • Dead-letter queue (DLQ) for failed jobs with inspection API
  • Health endpoints at /api/system/health/*
  • Pipeline latency metrics and observability

8) Security Hardening - IMPLEMENTED

Location: backend/api/middleware/auth.py, backend/api/middleware/rate_limit.py, backend/models/audit.py

What's implemented:

  • API key authentication (opt-in via API_KEY_ENABLED)
  • SHA-256 hashed key storage
  • WebSocket authentication via query parameter or Sec-WebSocket-Protocol
  • Rate limiting using Redis sliding window algorithm
  • Audit logging with comprehensive action types:
  • Event actions (reviewed, dismissed)
  • Settings changes
  • Authentication (login/logout)
  • API key management
  • Security events (rate limit exceeded, file magic rejected)
  • SSRF protection for webhooks

Future Enhancements (Not Yet Implemented)

Natural language "chat with your security history" (RAG)

Why it matters:

  • Query: "Did any unknown vehicles park in the driveway this week?"
  • Requires: event/detection index + retrieval + summarization

Implementation ideas:

  • Embed event summaries using the existing CLIP service (ai-gateway /clip router)
  • Vector similarity search for relevant events
  • LLM summarization of retrieved context

Complexity: Medium-High (requires RAG infrastructure)

NIM / Standardized Inference Deployment

Why it matters:

  • Replace ad-hoc llama.cpp process management with production inference service
  • Better scaling, consistency, and model versioning

Implementation ideas:

  • NVIDIA NIM containers for standardized inference
  • Model registry for version management
  • Blue-green deployment for model updates

Complexity: Medium (deployment infrastructure)

Digital Twin Reconstruction (USD / Omniverse)

Why it matters:

  • Generate structured 3D event reconstructions for replay/forensics
  • Very cool demo; likely not a practical priority early

Prerequisites:

  • Accurate depth estimation (Depth Anything V2 already in model zoo)
  • Camera calibration data
  • 3D scene understanding

Complexity: High (requires significant new infrastructure)

Face Recognition / License Plates (Privacy-Sensitive)

Why it matters:

  • High user value for identifying known vs unknown visitors
  • Requires careful privacy handling

Current state:

  • Detection models already in the model zoo (backend/services/model_zoo.py): yolo11-face (200 MB VRAM), yolo11-license-plate (300 MB VRAM, legacy), fast-alpr (28 MB, end-to-end plate detection + OCR that replaces YOLO11+PaddleOCR), and paddleocr (100 MB VRAM, legacy text recognition)
  • Read APIs and UI already exist: /api/face-events/* (backend/api/routes/face_recognition.py), /api/plate-reads/* (backend/api/routes/plate_reads.py), and the frontend face-recognition/ and entities/ components
  • No opt-in consent flow exists yet — nothing in the tree gates these features behind user consent

What's needed:

  • Opt-in consent flow and UI (the remaining work)
  • Data retention policies specific to biometric/plate data (global retention is RETENTION_DAYS=30)

Complexity: Medium (models and read paths exist; privacy-aware UX is the gap)

Advanced Behavior Analysis

Why it matters:

  • Detect complex behaviors like loitering patterns, tailgating, repeated visits
  • Goes beyond single-event analysis

Implementation ideas:

  • Temporal graph of entity movements
  • Pattern matching on behavior sequences
  • Long-term entity history

Complexity: High (requires temporal reasoning infrastructure)

Home Automation Integration - MOSTLY IMPLEMENTED

Why it matters:

  • Trigger smart home actions on security events
  • Integration with Home Assistant, MQTT, etc.

What's already implemented:

  • MQTT event publisher (backend/services/mqtt_publisher.py): events, alerts, detections, zone crossings and health published to hsi/* topics, with QoS and retain options and Prometheus metrics
  • Home Assistant MQTT Discovery (backend/services/ha_discovery.py): auto-configures cameras, alerts and events in Home Assistant (homeassistant/{component}/hsi_{id}/config)
  • MQTT command handler (backend/services/mqtt_command_handler.py) and Frigate integration (backend/services/frigate_integration.py)
  • MQTT configuration API at /api/mqtt-config (GET/PUT config, status, test, reconnect/disconnect — backend/api/routes/mqtt_config.py)

Remaining ideas:

  • Custom action rules that trigger outbound smart-home actions on event types (inbound integration exists; user-defined outbound actions do not)

Complexity: Low (the MQTT transport and HA discovery are done; only user-defined action rules remain)

How to Pick "What Next" (A Pragmatic Rubric)

If you're choosing roadmap work, prioritize items that:

  1. Reduce false positives (already done: zones, anomaly scoring, dedupe)
  2. Reduce time-to-action (already done: alerts, notifications, search)
  3. Increase usability of history (already done: search, timeline)
  4. Reduce operational friction (already done: model downloads, setup, reliability)

The remaining items are truly "nice to have" enhancements. Focus on:

  1. User-requested features - If users are asking for something specific
  2. Privacy-safe wins - Home automation is lower risk than face recognition
  3. Demo value - RAG/chat interface is impressive but complex

Avoid jumping into "big bets" until core stability is proven in production use.