Observability Hub¶

Comprehensive observability stack providing structured logging, distributed tracing, metrics collection, and alerting for the home security intelligence system.

Overview¶

The observability infrastructure provides full visibility into system operation through four pillars: structured logging with JSON output and trace correlation, Prometheus metrics for quantitative monitoring, distributed tracing via OpenTelemetry for request flow analysis, and Grafana dashboards with Alertmanager for visualization and alerting.

All components are designed for correlation. Log entries include trace IDs and span IDs enabling navigation from logs to traces. Metrics include labels that map to trace attributes. Grafana datasources are configured with derived fields and trace-to-metrics queries for seamless navigation between observability data types.

The stack runs entirely in containers alongside the application services, with Prometheus scraping metrics endpoints, Jaeger collecting traces, and Grafana providing unified visualization. Alertmanager routes alerts based on severity and component, with inhibition rules preventing alert storms.

Note: Loki log aggregation is planned but not currently deployed. The docker-compose.prod.yml does not include a Loki service. Log references to Loki in the architecture diagrams represent the intended future state.

Documents¶

Architecture Diagram¶

graph TD
    subgraph "Application Layer"
        BE[Backend Service<br/>backend/api/main.py]
        AI[AI Services<br/>ai-yolo26, ai-llm, ai-florence]
    end

    subgraph "Instrumentation"
        LOG[Structured Logging<br/>backend/core/logging.py:842-912]
        MET[Prometheus Metrics<br/>backend/core/metrics.py:107-500]
        TRC[OpenTelemetry Tracing<br/>backend/core/telemetry.py:135-268]
    end

    subgraph "Collection Layer"
        PROM[Prometheus<br/>monitoring/prometheus.yml]
        LOKI[Loki<br/>Log Aggregation]
        JAEG[Jaeger<br/>Trace Collection]
    end

    subgraph "Visualization & Alerting"
        GRAF[Grafana Dashboards<br/>monitoring/grafana/dashboards/]
        AM[Alertmanager<br/>monitoring/alertmanager.yml]
    end

    BE --> LOG
    BE --> MET
    BE --> TRC
    AI --> MET

    LOG --> LOKI
    MET --> PROM
    TRC --> JAEG

    PROM --> GRAF
    LOKI --> GRAF
    JAEG --> GRAF
    PROM --> AM

    AM --> |Webhooks| BE

Quick Reference¶

Key Concepts¶

Trace Correlation¶

Every log entry includes trace_id and span_id fields when OpenTelemetry is active. This enables navigation from logs to the corresponding distributed trace in Jaeger:

trace_id=abc123def456... span_id=789xyz...

Grafana's Loki datasource is configured with derived fields to extract trace IDs and link directly to Jaeger (monitoring/grafana/provisioning/datasources/prometheus.yml:198-214).

Metric Cardinality Control¶

All metric labels are sanitized through allowlists to prevent cardinality explosion (backend/core/sanitization.py). Camera IDs, error types, object classes, and risk levels are validated before being used as label values.

Multi-Window Alerting¶

Alerts use multi-window burn rate calculations for SLO monitoring. Fast burns (14.4x over 1h) trigger critical alerts, while slow burns (3x over 1d) trigger warnings (monitoring/prometheus-rules.yml:141-169).

Configuration¶

Monitoring Stack Components¶

The monitoring stack provides comprehensive observability through integrated components that share data and enable cross-correlation between metrics, logs, and traces.

Datasources¶

Related Hubs¶

• Resilience Patterns - Circuit breakers and error monitoring
• System Overview - Architecture context
• AI Orchestration - AI service metrics
• Testing - Observability testing patterns