Observability Hub¶

Comprehensive observability stack providing structured logging, distributed tracing, metrics collection, and alerting for the home security intelligence system.

Overview¶

The observability infrastructure provides full visibility into system operation through four pillars: structured logging with JSON output and trace correlation, Prometheus metrics for quantitative monitoring, distributed tracing via OpenTelemetry for request flow analysis, and Grafana dashboards with Alertmanager for visualization and alerting.

All components are designed for correlation. Log entries include trace IDs and span IDs enabling navigation from logs to traces. Metrics include labels that map to trace attributes. Grafana datasources are configured with derived fields and trace-to-metrics queries for seamless navigation between observability data types.

The stack runs entirely in containers alongside the application services: Prometheus scrapes metrics endpoints, Alloy receives OTLP traces and forwards them to Tempo, Loki stores logs, and Grafana provides unified visualization. Alertmanager routes alerts based on severity and component, with inhibition rules preventing alert storms (monitoring/alertmanager.yml:45, 118). Alloy also scrapes container logs and ships them to Loki (monitoring/alloy/config.alloy:50-140).

Documents¶

Architecture Diagram¶

graph TD
    subgraph "Application Layer"
        BE[Backend Service<br/>backend/main.py]
        AI[AI Containers<br/>ai-gateway :8090, ai-llm :8091]
    end

    subgraph "Instrumentation"
        LOG[Structured Logging<br/>backend/core/logging.py]
        MET[Prometheus Metrics<br/>backend/core/metrics.py]
        TRC[OpenTelemetry Tracing<br/>backend/core/telemetry.py]
    end

    subgraph "Collection Layer"
        PROM[Prometheus<br/>monitoring/prometheus.yml]
        LOKI[Loki<br/>log aggregation]
        ALLOY[Alloy<br/>OTLP collector :4317]
        TEMPO[Tempo<br/>trace storage :3200]
    end

    subgraph "Visualization & Alerting"
        GRAF[Grafana Dashboards<br/>monitoring/grafana/dashboards/]
        AM[Alertmanager<br/>monitoring/alertmanager.yml]
    end

    BE --> LOG
    BE --> MET
    BE --> TRC
    AI --> MET

    LOG --> |container logs| ALLOY
    ALLOY --> |loki.write| LOKI
    MET --> PROM
    TRC --> |OTLP gRPC| ALLOY
    ALLOY --> |OTLP| TEMPO

    PROM --> GRAF
    LOKI --> GRAF
    TEMPO --> GRAF
    PROM --> AM

    AM --> |Webhooks| BE

Quick Reference¶

Key Concepts¶

Trace Correlation¶

Every log entry includes trace_id and span_id fields when OpenTelemetry is active. This enables navigation from logs to the corresponding distributed trace in Tempo:

trace_id=abc123def456... span_id=789xyz...

Grafana's Loki datasource is configured with derived fields to extract trace IDs and link directly to Tempo (monitoring/grafana/provisioning/datasources/prometheus.yml:225-231).

Metric Cardinality Control¶

All metric labels are sanitized through allowlists to prevent cardinality explosion (backend/core/sanitization.py). Camera IDs, error types, object classes, and risk levels are validated before being used as label values.

Multi-Window Alerting¶

Alerts use multi-window burn rate calculations for SLO monitoring. Fast burns (14.4x over 1h) trigger critical alerts, while slow burns (3x over 1d) trigger warnings (monitoring/prometheus-rules.yml:141-169).

Configuration¶

Monitoring Stack Components¶

The monitoring stack provides comprehensive observability through integrated components that share data and enable cross-correlation between metrics, logs, and traces.

Datasources¶

Related Hubs¶

• Resilience Patterns - Circuit breakers and error monitoring
• System Overview - Architecture context
• AI Orchestration - AI service metrics
• Testing - Observability testing patterns