Monitoring Guide¶

Monitor system health, GPU performance, service status, and failed job queues.

Table of Contents¶

• Overview
• Health Check Endpoints
• Service Level Objectives
• GPU Monitoring
• Service Health Monitoring
• Circuit Breaker Integration
• Dead Letter Queue Monitoring
• Prometheus and Alerting
• Grafana Dashboards
• Troubleshooting

Overview¶

The monitoring system provides real-time visibility into three domains:

1. GPU Monitoring - Utilization, memory, temperature, power
2. Service Health - AI services, Redis, PostgreSQL connectivity
3. Dead Letter Queue - Failed job inspection and recovery

Architecture¶

Event Sources           Collection          Storage           Delivery
--------------         -----------         ---------         ---------
GPU Hardware     -->   GPUMonitor     -->  PostgreSQL  -->   WebSocket
AI Services      -->   HealthMonitor  -->  Memory      -->   REST API
Redis Queues     -->   DLQ Stats      -->  Redis       -->   Dashboard

Health Check Endpoints¶

Health Check Hierarchy¶

The system provides three levels of health checks, each serving a different purpose:

flowchart TD
    subgraph "Health Check Hierarchy"
        direction TB
        L["/health<br/>Liveness Probe"] --> R["/api/system/health/ready<br/>Readiness Probe"]
        R --> F["/api/system/health/full<br/>Full Health Check"]
    end

    L --> LC{Process Running?}
    LC -->|Yes| LOK["200 OK<br/>status: alive"]
    LC -->|No| LFAIL["No Response<br/>Container restart"]

    R --> RC{DB + Redis OK?}
    RC -->|Yes| ROK["200 OK<br/>ready: true"]
    RC -->|No| RFAIL["503 Unavailable<br/>ready: false"]

    F --> FC{All Services OK?}
    FC -->|All Healthy| FOK["200 OK<br/>status: healthy"]
    FC -->|Non-Critical Failing| FDEG["200 OK<br/>status: degraded"]
    FC -->|Critical Failing| FFAIL["503 Unavailable<br/>status: unhealthy"]

    style L fill:#e3f2fd
    style R fill:#e8f5e9
    style F fill:#fff3e0
    style LOK fill:#c8e6c9
    style ROK fill:#c8e6c9
    style FOK fill:#c8e6c9
    style FDEG fill:#fff9c4
    style LFAIL fill:#ffcdd2
    style RFAIL fill:#ffcdd2
    style FFAIL fill:#ffcdd2

Liveness Probe¶

Endpoint: GET /health

Simple check that returns "alive" if the process is running.

curl http://localhost:8000/health

{
  "status": "alive"
}

Use Cases: Kubernetes liveness probe, Docker HEALTHCHECK, process monitoring

Readiness Probe¶

Endpoint: GET /api/system/health/ready

Checks if the system is ready to accept traffic: database, Redis, AI services, and background workers (ReadinessResponse in backend/api/schemas/system.py).

curl http://localhost:8000/api/system/health/ready

{
  "ready": true,
  "status": "ready",
  "services": {
    "database": {
      "status": "healthy",
      "message": "Database connected",
      "details": {}
    },
    "redis": {
      "status": "healthy",
      "message": "Redis connected",
      "details": { "redis_version": "7.4.0" }
    },
    "ai": {
      "status": "healthy",
      "message": "AI services reachable",
      "details": {}
    }
  },
  "workers": []
}

status is one of ready, degraded, not_ready.

HTTP Status:

• 200 OK - System is ready
• 503 Service Unavailable - System is not ready

Full Health Check¶

Endpoint: GET /api/system/health/full

Comprehensive health check including all services and circuit breakers.

curl http://localhost:8000/api/system/health/full

{
  "status": "healthy",
  "ready": true,
  "message": "All systems operational",
  "postgres": {
    "name": "postgres",
    "status": "healthy",
    "message": "Database operational"
  },
  "redis": {
    "name": "redis",
    "status": "healthy",
    "message": "Redis connected",
    "details": {
      "redis_version": "7.4.0"
    }
  },
  "ai_services": [
    {
      "name": "yolo26",
      "display_name": "YOLO26 Object Detection",
      "status": "healthy",
      "url": "http://ai-gateway:8090/yolo26",
      "response_time_ms": 45.2,
      "circuit_state": "closed",
      "last_check": "2026-01-08T10:30:00Z"
    }
  ],
  "circuit_breakers": {
    "total": 5,
    "open": 0,
    "half_open": 0,
    "closed": 5,
    "breakers": {
      "yolo26": "closed",
      "nemotron": "closed"
    }
  },
  "workers": [
    {
      "name": "file_watcher",
      "running": true,
      "critical": true
    }
  ],
  "timestamp": "2026-01-08T10:30:00Z",
  "version": "0.1.0"
}

Status Values¶

Service Level Objectives¶

Critical Services¶

Critical services must be healthy for the system to be considered ready.

Non-Critical Services¶

Non-critical services can fail without blocking system readiness.

SLO Metrics¶

[!NOTE] > hsi_stage_duration_seconds is defined in backend/core/metrics.py but nothing in the backend observes it yet, so its _bucket series never appear. The latency_alerts group (HSISlowDetection, HSISlowAnalysis, HSIExtremeLatency) is therefore commented out in monitoring/alerting-rules.yml, and the latency recording rules evaluate empty. See SLO Definitions.

Error Budget Policy¶

GPU Monitoring¶

Metrics Collected¶

Configuration¶

Polling Interval Guidance¶

API Endpoints¶

# Current GPU stats
curl http://localhost:8000/api/system/gpu

# GPU history
curl "http://localhost:8000/api/system/gpu/history?since=2025-12-30T09:45:00Z&limit=300"

WebSocket Updates¶

GPU stats are delivered via /ws/system stream:

{
  "type": "system_status",
  "data": {
    "gpu": {
      "gpu_name": "NVIDIA RTX A5500",
      "utilization": 45.0,
      "memory_used": 8192,
      "memory_total": 24576,
      "temperature": 62.0,
      "power_usage": 125.5
    }
  },
  "timestamp": "2025-12-30T10:15:00.000Z"
}

Service Health Monitoring¶

Monitored Services¶

Service Status Values¶

Exponential Backoff¶

Recovery attempts use exponential backoff:

WebSocket Status Updates¶

{
  "type": "service_status",
  "data": {
    "service": "yolo26",
    "status": "unhealthy",
    "message": "Health check failed"
  },
  "timestamp": "2025-12-30T10:15:00Z"
}

Circuit Breaker Integration¶

Circuit Breaker State Diagram¶

The circuit breaker pattern protects the system from cascading failures when AI services become unavailable.

stateDiagram-v2
    [*] --> Closed

    Closed --> Open : failure_count >= 5
    Closed --> Closed : success / reset count

    Open --> HalfOpen : recovery_timeout (30s)

    HalfOpen --> Closed : success_count >= 2
    HalfOpen --> Open : any failure

    note right of Closed
        Normal operation
        Requests pass through
        Count failures
    end note

    note right of Open
        Service failing
        Requests fail immediately
        Wait for recovery timeout
    end note

    note right of HalfOpen
        Testing recovery
        Allow max 3 test requests
        Success closes, failure reopens
    end note

Circuit Breaker States¶

Configuration¶

CircuitBreakerConfig(
    failure_threshold=5,      # Failures before opening
    recovery_timeout=30.0,    # Seconds before half-open
    half_open_max_calls=3,    # Test calls in half-open
    success_threshold=2,      # Successes to close
)

WebSocket Broadcasting¶

{
  "type": "circuit_breaker_update",
  "data": {
    "timestamp": "2026-01-08T10:30:00Z",
    "summary": {
      "total": 5,
      "open": 1,
      "half_open": 0,
      "closed": 4
    },
    "breakers": {
      "yolo26": "open",
      "nemotron": "closed"
    }
  }
}

Dead Letter Queue Monitoring¶

DLQ Queues¶

API Endpoints¶

# Get DLQ counts
curl http://localhost:8000/api/dlq/stats

# List jobs in DLQ
curl "http://localhost:8000/api/dlq/jobs/dlq:detection_queue?start=0&limit=50"

# Requeue all jobs
curl -X POST "http://localhost:8000/api/dlq/requeue-all/dlq:detection_queue"

# Clear DLQ
curl -X DELETE "http://localhost:8000/api/dlq/dlq:detection_queue"

DLQ Stats Response¶

{
  "detection_queue_count": 3,
  "analysis_queue_count": 1,
  "total_count": 4
}

Job Failure Structure¶

{
  "original_job": {
    "file_path": "/export/foscam/front_door/image001.jpg",
    "camera_id": "front_door"
  },
  "error": "Connection timeout to YOLO26",
  "attempt_count": 3,
  "first_failed_at": "2025-12-30T10:00:00Z",
  "last_failed_at": "2025-12-30T10:01:30Z",
  "queue_name": "detection_queue"
}

DLQ Dashboard¶

Access the DLQ Monitor in Settings in the web interface:

• Badge showing total failed job count
• Expandable panels for each queue
• Job details with error messages
• Requeue All / Clear All buttons
• Auto-refresh every 30 seconds

Prometheus and Alerting¶

Enable Monitoring Stack¶

Prometheus, Alertmanager, Grafana, Loki, Tempo and the exporters are default compose services — no --profile flag:

podman compose -f docker-compose.prod.yml up -d

# Access Alertmanager (binds 127.0.0.1)
open http://localhost:9093

Prometheus Metrics¶

Backend metrics are exposed at /api/metrics with an hsi_ prefix (backend/core/metrics.py):

# Circuit breaker state (0=closed, 1=open, 2=half_open)
hsi_circuit_breaker_state{service="yolo26"} 0

# Health check latency (json-exporter gauges feed the HSI* health alerts)
hsi_health_check_latency_seconds{component="postgres"} 0.002

# System health gauge (1 healthy / 0.5 degraded / 0 unhealthy)
hsi_system_healthy 1

Pre-Configured Alerts¶

See Prometheus Alerting for the full measured tables.

Critical Alerts¶

Warning Alerts¶

Alerting Rules Example¶

groups:
  - name: health_alerts
    rules:
      - alert: CircuitBreakerOpen
        expr: hsi_circuit_breaker_state == 1
        for: 60s
        labels:
          severity: warning
        annotations:
          summary: 'Circuit breaker {{ $labels.service }} is open'

Grafana Dashboards¶

Access Grafana¶

Grafana is served from the /grafana/ sub-path on the host port ${GRAFANA_PORT:-3002}:

open http://localhost:3002/grafana/
# Credentials: GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASSWORD:-admin}

Available Dashboards¶

Provisioned from monitoring/grafana/dashboards/:

• consolidated - single overview dashboard
• api-health, ai-services, ai-service-health - service and API status
• hsi-gpu-metrics - utilization, memory, temperature trends
• enrichment-pipeline, clip-florence-intelligence, nemotron-prompt-analytics, scene-ocr, video-analytics, analytics - AI pipeline detail
• hsi-profiling, hsi-request-profiling - profiling regressions
• logs, tracing - Loki and Tempo views

SLO Dashboard Panels¶

1. SLO Compliance Gauges - Current compliance for each SLO
2. Error Budget Remaining - Time-based visualization
3. Burn Rate Trends - Multi-window burn rate graphs
4. Historical SLI Trends - 30-day rolling SLI values

Troubleshooting¶

GPU Stats Not Updating¶

1. Verify pynvml installation:

python -c "import pynvml; pynvml.nvmlInit(); print('OK')"

1. Check NVIDIA driver:

nvidia-smi

1. Review GPU monitor logs:

   grep "GPU" data/logs/security.log
   

Health Checks Failing¶

1. Test service endpoints directly:

curl http://localhost:8095/health  # YOLO26
curl http://localhost:8091/health  # Nemotron

1. Check service logs in their respective terminals

2. Verify network connectivity (especially in Docker)

DLQ Growing¶

1. Check error messages in DLQ jobs
2. Verify AI service availability
3. Review retry configuration
4. Check for resource exhaustion (memory, disk)

Built-in Alert Conditions¶

API Reference¶

System Health¶

GPU Monitoring¶

DLQ Management¶

Storage¶

See Also¶

• Operator Hub - Main operator documentation
• Deployment Guide - Service setup
• Administration Guide - Configuration
• SLO Definitions - Detailed SLO specifications
• Prometheus Alerting - Alert configuration