Alertmanager¶

Alert routing, notification channels, and inhibition rules for the home security intelligence system.

Key Files:

• monitoring/alertmanager.yml (238 lines) - Alertmanager configuration
• monitoring/alerting-rules.yml (1241 lines) - Core alert rule definitions
• monitoring/gpu-alerts.yml, monitoring/ai-pipeline-alerts.yml, monitoring/prometheus_rules.yml - Additional rule groups (all loaded via rule_files in monitoring/prometheus.yml:26-33)
• monitoring/prometheus-rules.yml (175 lines) - Recording rules for SLIs, error budgets, burn rates
• monitoring/grafana/provisioning/alerting/log-alerts.yml - Grafana log-based alerts

Overview¶

Prometheus evaluates the rule files above and sends firing alerts to Alertmanager, which groups them, applies inhibition rules, and forwards them to receivers. Alertmanager itself runs as the alertmanager compose service and mounts monitoring/alertmanager.yml (docker-compose.prod.yml:1055).

In the shipped configuration every receiver delivers through the same webhook: http://backend:8000/api/webhooks/alerts. The receivers exist so that routing, batching, and repeat intervals can differ per class of alert; Slack, email, and PagerDuty are present only as commented examples to enable (monitoring/alertmanager.yml:184-199). Severity and component labels decide which receiver handles an alert.

Note: monitoring/alertmanager.yml is not envsubst-templated. Webhook URLs use container service names on the compose network (fixed internal ports), independent of .env port variables (monitoring/alertmanager.yml:4-9).

Architecture¶

graph TD
    subgraph "Alert Sources"
        PROM[Prometheus rule files<br/>alerting-rules, gpu-alerts,<br/>ai-pipeline-alerts, prometheus_rules]
        GRAF[Grafana log alerts<br/>provisioning/alerting/log-alerts.yml]
    end

    subgraph "Alertmanager"
        ROUTE[Route tree<br/>alertmanager.yml:45-116]
        GROUP[Grouping &amp; batching]
        INHIB[Inhibition rules<br/>alertmanager.yml:118-174]
    end

    subgraph "Receivers alertmanager.yml:176-238"
        CRIT[critical-receiver]
        SLO[slo-receiver]
        PIPE[pipeline-receiver]
        INFRA[infrastructure-receiver]
        GPU[gpu-receiver]
        WARN[warning-receiver]
        INFO[info-receiver]
    end

    BE[Backend webhook<br/>POST /api/webhooks/alerts]

    PROM --> ROUTE
    GRAF -.Grafana-managed.-> GRAF
    ROUTE --> GROUP --> INHIB
    INHIB --> CRIT --> BE
    INHIB --> SLO --> BE
    INHIB --> PIPE --> BE
    INHIB --> INFRA --> BE
    INHIB --> GPU --> BE
    INHIB --> WARN --> BE
    INHIB --> INFO --> BE

Alert Configuration¶

Global Settings¶

monitoring/alertmanager.yml:10-38 sets resolve_timeout: 5m. SMTP and Slack settings are shipped commented out — uncomment smtp_* plus an email_configs block, or set slack_api_url plus a slack_configs block on a receiver, to enable those channels. Notification templates load from /etc/alertmanager/templates/*.tmpl (monitoring/alertmanager.yml:41-42).

Route Configuration¶

The route tree (monitoring/alertmanager.yml:45-116) groups by ['alertname', 'component', 'severity'] and defaults to default-receiver with group_wait: 30s, group_interval: 5m, repeat_interval: 4h. Child routes, evaluated in order:

The critical route sets continue: true, so a critical alert is also evaluated against the component routes below it.

Receivers¶

All eight receivers (monitoring/alertmanager.yml:176-238) post to http://backend:8000/api/webhooks/alerts. Only info-receiver sets send_resolved: false. critical-receiver carries commented slack_configs and email_configs examples for production use; there is no PagerDuty receiver.

Inhibition Rules¶

monitoring/alertmanager.yml:118-174 defines eight suppressions, all keyed on the HSI* alert names that actually exist in the rule files:

HSISlowDetection, HSISlowAnalysis, and HSIExtremeLatency are referenced by inhibition rules but their alert groups are commented out in monitoring/alerting-rules.yml:274-291 because the recording rules they depend on need hsi_stage_duration_seconds_bucket, which the backend does not yet emit. Those three inhibitions are therefore inert until the latency groups are re-enabled.

Alert Rule Definitions¶

Core Service Health¶

monitoring/alerting-rules.yml:6-115. These are availability alerts driven by blackbox probes and the json-exporter health gauges, not by up{job=...}:

GPU Alerts¶

Two files contribute GPU alerts. From monitoring/alerting-rules.yml:116-236 (backend- and DCGM-derived metrics):

From monitoring/gpu-alerts.yml (raw DCGM exporter metrics): GPUMemoryNearFull, GPUMemoryHigh, GPUHighTemperature (DCGM_FI_DEV_GPU_TEMP > 85), GPUTemperatureElevated (> 75), GPUUtilizationSaturated, GPUUnderutilizedMemoryBound, GPUMemoryBandwidthSaturated, GPUHighPowerUsage, GPUClockSpeedDegraded, DCGMExporterDown, NoGPUMetrics.

Queue and Error-Rate Alerts¶

Queues (monitoring/alerting-rules.yml:238-272):

Error rates (monitoring/alerting-rules.yml:292-317), driven by an API success-rate recording rule rather than a pipeline error counter:

SLO Burn Rate Alerts¶

Multi-window burn rate on API availability (monitoring/alerting-rules.yml:329-363):

Latency burn-rate alerts are commented out pending the same missing histogram metrics (monitoring/alerting-rules.yml:364-372).

AI Pipeline and Worker Alerts¶

monitoring/ai-pipeline-alerts.yml covers the enrichment pipeline, LLM behaviour, and scoring: GPUInferenceFailures, GPUMemoryHigh, GPUMemoryCritical, EnrichmentPipelineTimeout, EnrichmentPipelineTimeoutCritical, EnrichmentModelErrorRate, EnrichmentModelErrorCritical, EnrichmentQualityDegraded, PromptTruncationHigh, PromptContextUtilizationHigh, LLMInferenceLatencyHigh, LLMInferenceLatencyCritical, CoalescingMergeRateLow, CoalescingMergeRateHigh, RiskScoreCalibrationDrift, RiskScoreAllCritical, RiskScoreAllLow, CLIPServiceDown, FlorenceServiceDown, CLIPAnomalyErrorsHigh.

monitoring/alerting-rules.yml also defines Prometheus self-monitoring alerts (Prometheus*, lines 415-690), worker alerts (HSIWorkerFailed, HSIWorkerNotRunning, HSIWorkerConsecutiveFailures, lines 695-748), plus circuit-breaker, profiling, websocket, cache, batch, and system groups.

Recording Rules for Alerts¶

Pre-computed SLI metrics and burn rates (monitoring/prometheus-rules.yml:119-175):

# Error budget remaining, API availability target 99.5%
- record: hsi:error_budget:api_availability_remaining
  expr: 1 - ((1 - hsi:api_availability:ratio_rate30d) / (1 - 0.995))

# Burn rates against the same target
- record: hsi:burn_rate:api_availability_1h
  expr: (1 - hsi:api_availability:ratio_rate1h) / (1 - 0.995)
- record: hsi:burn_rate:api_availability_6h
  expr: (1 - hsi:api_availability:ratio_rate6h) / (1 - 0.995)
- record: hsi:burn_rate:api_availability_1d
  expr: (1 - hsi:api_availability:ratio_rate1d) / (1 - 0.995)

Detection- and analysis-latency burn rates are defined in the same group (monitoring/prometheus-rules.yml:133-137, 158-164) against a 95% within-SLO target.

Grafana Log-Based Alerts¶

Grafana evaluates LogQL rules against Loki independently of Prometheus (monitoring/grafana/provisioning/alerting/log-alerts.yml):

These live in Grafana's alerting engine (folder "HSI Alerts", evaluated every minute), so they do not pass through Prometheus; notification policy is Grafana's own.

Alert Labels¶

Standard labels for routing and filtering, as used across the rule files:

Alert Annotations¶

Standard annotations for context:

Testing Alerts¶

Verify alert rules:

# Rule syntax (promtool ships with the prom image)
podman exec prometheus promtool check rules /etc/prometheus/alerting-rules.yml

# Query Prometheus for active alerts
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts'

# Check Alertmanager status
curl -s http://localhost:9093/api/v2/status | jq

Silences¶

Create temporary silences for maintenance:

curl -X POST http://localhost:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HSIGPUMemoryElevated", "isRegex": false}
    ],
    "startsAt": "2026-09-22T10:00:00Z",
    "endsAt": "2026-09-22T12:00:00Z",
    "createdBy": "admin",
    "comment": "GPU maintenance window"
  }'

Related Documents¶

• Prometheus Metrics - Metric definitions used in alerts
• Grafana Dashboards - Alert visualization
• Structured Logging - Log-based alerting