Monitoring Guide¶

Monitor system health, GPU performance, service status, and failed job queues.

Table of Contents¶

• Overview
• Health Check Endpoints
• Service Level Objectives
• GPU Monitoring
• Service Health Monitoring
• Circuit Breaker Integration
• Dead Letter Queue Monitoring
• Prometheus and Alerting
• Grafana Dashboards
• Troubleshooting

Overview¶

The monitoring system provides real-time visibility into three domains:

1. GPU Monitoring - Utilization, memory, temperature, power
2. Service Health - AI services, Redis, PostgreSQL connectivity
3. Dead Letter Queue - Failed job inspection and recovery

Architecture¶

Event Sources           Collection          Storage           Delivery
--------------         -----------         ---------         ---------
GPU Hardware     -->   GPUMonitor     -->  PostgreSQL  -->   WebSocket
AI Services      -->   HealthMonitor  -->  Memory      -->   REST API
Redis Queues     -->   DLQ Stats      -->  Redis       -->   Dashboard

Health Check Endpoints¶

Health Check Hierarchy¶

The system provides three levels of health checks, each serving a different purpose:

flowchart TD
    subgraph "Health Check Hierarchy"
        direction TB
        L["/health<br/>Liveness Probe"] --> R["/api/system/health/ready<br/>Readiness Probe"]
        R --> F["/api/system/health/full<br/>Full Health Check"]
    end

    L --> LC{Process Running?}
    LC -->|Yes| LOK["200 OK<br/>status: alive"]
    LC -->|No| LFAIL["No Response<br/>Container restart"]

    R --> RC{DB + Redis OK?}
    RC -->|Yes| ROK["200 OK<br/>ready: true"]
    RC -->|No| RFAIL["503 Unavailable<br/>ready: false"]

    F --> FC{All Services OK?}
    FC -->|All Healthy| FOK["200 OK<br/>status: healthy"]
    FC -->|Non-Critical Failing| FDEG["200 OK<br/>status: degraded"]
    FC -->|Critical Failing| FFAIL["503 Unavailable<br/>status: unhealthy"]

    style L fill:#e3f2fd
    style R fill:#e8f5e9
    style F fill:#fff3e0
    style LOK fill:#c8e6c9
    style ROK fill:#c8e6c9
    style FOK fill:#c8e6c9
    style FDEG fill:#fff9c4
    style LFAIL fill:#ffcdd2
    style RFAIL fill:#ffcdd2
    style FFAIL fill:#ffcdd2

Liveness Probe¶

Endpoint: GET /health

Simple check that returns "alive" if the process is running.

curl http://localhost:8000/health

{
  "status": "alive"
}

Use Cases: Kubernetes liveness probe, Docker HEALTHCHECK, process monitoring

Readiness Probe¶

Endpoint: GET /api/system/health/ready

Checks if system is ready to accept traffic (infrastructure only).

curl http://localhost:8000/api/system/health/ready

{
  "ready": true,
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 2.5
    },
    "redis": {
      "status": "healthy",
      "latency_ms": 1.2
    }
  }
}

HTTP Status:

• 200 OK - System is ready
• 503 Service Unavailable - System is not ready

Full Health Check¶

Endpoint: GET /api/system/health/full

Comprehensive health check including all services and circuit breakers.

curl http://localhost:8000/api/system/health/full

{
  "status": "healthy",
  "ready": true,
  "message": "All systems operational",
  "postgres": {
    "name": "postgres",
    "status": "healthy",
    "message": "Database operational"
  },
  "redis": {
    "name": "redis",
    "status": "healthy",
    "message": "Redis connected",
    "details": {
      "redis_version": "7.4.0"
    }
  },
  "ai_services": [
    {
      "name": "yolo26",
      "display_name": "YOLO26 Object Detection",
      "status": "healthy",
      "url": "http://ai-yolo26:8095",
      "response_time_ms": 45.2,
      "circuit_state": "closed",
      "last_check": "2026-01-08T10:30:00Z"
    }
  ],
  "circuit_breakers": {
    "total": 5,
    "open": 0,
    "half_open": 0,
    "closed": 5,
    "breakers": {
      "yolo26": "closed",
      "nemotron": "closed"
    }
  },
  "workers": [
    {
      "name": "file_watcher",
      "running": true,
      "critical": true
    }
  ],
  "timestamp": "2026-01-08T10:30:00Z",
  "version": "0.1.0"
}

Status Values¶

Service Level Objectives¶

Critical Services¶

Critical services must be healthy for the system to be considered ready.

Non-Critical Services¶

Non-critical services can fail without blocking system readiness.

SLO Metrics¶

Error Budget Policy¶

GPU Monitoring¶

Metrics Collected¶

Configuration¶

Polling Interval Guidance¶

API Endpoints¶

# Current GPU stats
curl http://localhost:8000/api/system/gpu

# GPU history
curl "http://localhost:8000/api/system/gpu/history?since=2025-12-30T09:45:00Z&limit=300"

WebSocket Updates¶

GPU stats are delivered via /ws/system stream:

{
  "type": "system_status",
  "data": {
    "gpu": {
      "gpu_name": "NVIDIA RTX A5500",
      "utilization": 45.0,
      "memory_used": 8192,
      "memory_total": 24576,
      "temperature": 62.0,
      "power_usage": 125.5
    }
  },
  "timestamp": "2025-12-30T10:15:00.000Z"
}

Service Health Monitoring¶

Monitored Services¶

Service Status Values¶

Exponential Backoff¶

Recovery attempts use exponential backoff:

WebSocket Status Updates¶

{
  "type": "service_status",
  "data": {
    "service": "yolo26",
    "status": "unhealthy",
    "message": "Health check failed"
  },
  "timestamp": "2025-12-30T10:15:00Z"
}

Circuit Breaker Integration¶

Circuit Breaker State Diagram¶

The circuit breaker pattern protects the system from cascading failures when AI services become unavailable.

stateDiagram-v2
    [*] --> Closed

    Closed --> Open : failure_count >= 5
    Closed --> Closed : success / reset count

    Open --> HalfOpen : recovery_timeout (30s)

    HalfOpen --> Closed : success_count >= 2
    HalfOpen --> Open : any failure

    note right of Closed
        Normal operation
        Requests pass through
        Count failures
    end note

    note right of Open
        Service failing
        Requests fail immediately
        Wait for recovery timeout
    end note

    note right of HalfOpen
        Testing recovery
        Allow max 3 test requests
        Success closes, failure reopens
    end note

Circuit Breaker States¶

Configuration¶

CircuitBreakerConfig(
    failure_threshold=5,      # Failures before opening
    recovery_timeout=30.0,    # Seconds before half-open
    half_open_max_calls=3,    # Test calls in half-open
    success_threshold=2,      # Successes to close
)

WebSocket Broadcasting¶

{
  "type": "circuit_breaker_update",
  "data": {
    "timestamp": "2026-01-08T10:30:00Z",
    "summary": {
      "total": 5,
      "open": 1,
      "half_open": 0,
      "closed": 4
    },
    "breakers": {
      "yolo26": "open",
      "nemotron": "closed"
    }
  }
}

Dead Letter Queue Monitoring¶

DLQ Queues¶

API Endpoints¶

# Get DLQ counts
curl http://localhost:8000/api/dlq/stats

# List jobs in DLQ
curl "http://localhost:8000/api/dlq/jobs/dlq:detection_queue?start=0&limit=50"

# Requeue all jobs
curl -X POST "http://localhost:8000/api/dlq/requeue-all/dlq:detection_queue"

# Clear DLQ
curl -X DELETE "http://localhost:8000/api/dlq/dlq:detection_queue"

DLQ Stats Response¶

{
  "detection_queue_count": 3,
  "analysis_queue_count": 1,
  "total_count": 4
}

Job Failure Structure¶

{
  "original_job": {
    "file_path": "/export/foscam/front_door/image001.jpg",
    "camera_id": "front_door"
  },
  "error": "Connection timeout to YOLO26",
  "attempt_count": 3,
  "first_failed_at": "2025-12-30T10:00:00Z",
  "last_failed_at": "2025-12-30T10:01:30Z",
  "queue_name": "detection_queue"
}

DLQ Dashboard¶

Access the DLQ Monitor in Settings in the web interface:

• Badge showing total failed job count
• Expandable panels for each queue
• Job details with error messages
• Requeue All / Clear All buttons
• Auto-refresh every 30 seconds

Prometheus and Alerting¶

Enable Monitoring Stack¶

docker compose --profile monitoring -f docker-compose.prod.yml up -d

# Access Alertmanager
open http://localhost:9093

Prometheus Metrics¶

# Circuit breaker state (0=closed, 1=open, 2=half_open)
circuit_breaker_state{service="yolo26"} 0

# Health check latency
health_check_latency_seconds{service="postgres"} 0.002

# Service availability
service_available{service="yolo26"} 1

Pre-Configured Alerts¶

Critical Alerts¶

Warning Alerts¶

Alerting Rules Example¶

groups:
  - name: health_alerts
    rules:
      - alert: CriticalServiceUnhealthy
        expr: service_available{critical="true"} == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: 'Critical service {{ $labels.service }} is unhealthy'

      - alert: CircuitBreakerOpen
        expr: circuit_breaker_state > 0
        for: 60s
        labels:
          severity: warning
        annotations:
          summary: 'Circuit breaker {{ $labels.service }} is not closed'

Grafana Dashboards¶

Access Grafana¶

open http://localhost:3002
# Default credentials from GF_ADMIN_PASSWORD in .env

Available Dashboards¶

• System Overview - Service health, circuit breakers
• GPU Metrics - Utilization, memory, temperature trends
• SLO Dashboard - Compliance gauges, error budget
• AI Pipeline - Detection/analysis latency, queue depths

SLO Dashboard Panels¶

1. SLO Compliance Gauges - Current compliance for each SLO
2. Error Budget Remaining - Time-based visualization
3. Burn Rate Trends - Multi-window burn rate graphs
4. Historical SLI Trends - 30-day rolling SLI values

Troubleshooting¶

GPU Stats Not Updating¶

1. Verify pynvml installation:

python -c "import pynvml; pynvml.nvmlInit(); print('OK')"

1. Check NVIDIA driver:

nvidia-smi

1. Review GPU monitor logs:

   grep "GPU" data/logs/security.log
   

Health Checks Failing¶

1. Test service endpoints directly:

curl http://localhost:8095/health  # YOLO26
curl http://localhost:8091/health  # Nemotron

1. Check service logs in their respective terminals

2. Verify network connectivity (especially in Docker)

DLQ Growing¶

1. Check error messages in DLQ jobs
2. Verify AI service availability
3. Review retry configuration
4. Check for resource exhaustion (memory, disk)

Built-in Alert Conditions¶

API Reference¶

System Health¶

GPU Monitoring¶

DLQ Management¶

Storage¶

See Also¶

• Operator Hub - Main operator documentation
• Deployment Guide - Service setup
• Administration Guide - Configuration
• SLO Definitions - Detailed SLO specifications
• Prometheus Alerting - Alert configuration