HSI Metrics Coverage This document maps all HSI (Home Security Intelligence) Prometheus metrics to their corresponding Grafana dashboard panels.
Overview HSI exposes Prometheus metrics for comprehensive system observability. Metrics follow these naming conventions:
Prefix: hsi_ (Home Security Intelligence) Counters end with _total Histograms/durations end with _seconds Gauges use descriptive names without suffix Metrics are defined in /backend/core/metrics.py (Prometheus) and /backend/core/otel_metrics.py (OpenTelemetry).
Pipeline Stage Metrics Metric Type Labels Dashboard Panel Description hsi_stage_duration_secondsHistogram stageConsolidated Pipeline Stage Latency, Stage Duration P50/P95/P99 Duration of pipeline stages (detect, batch, analyze) hsi_detect_latency_p95_msGauge - Consolidated Detection Latency Pre-calculated P95 detection latency in milliseconds
Stage Labels: detect, batch, analyze, watch
Detection Metrics Metric Type Labels Dashboard Panel Description hsi_detections_processed_totalCounter - Consolidated Detection Rate, Event/Detection Rate Total detections processed by YOLO26 hsi_detections_by_class_totalCounter object_classConsolidated Detections by Class Detections by object class (person, car, etc.) hsi_detection_confidenceHistogram - Consolidated Average Confidence Detection confidence scores (0.0-1.0) hsi_detections_filtered_low_confidence_totalCounter - Consolidated Filtered Detections Detections filtered due to low confidence hsi_detection_queue_depthGauge - Consolidated Detection Queue Depth Images waiting in detection queue
Event Metrics Metric Type Labels Dashboard Panel Description hsi_events_created_totalCounter - Consolidated Event Rate Total security events created hsi_events_by_camera_totalCounter camera_id, camera_nameConsolidated Events by Camera Events per camera hsi_events_reviewed_totalCounter - Consolidated Events Reviewed Events marked as reviewed hsi_events_acknowledged_totalCounter camera_name, risk_levelConsolidated Events Acknowledged Events acknowledged by users
Risk Analysis Metrics Metric Type Labels Dashboard Panel Description hsi_risk_scoreHistogram - Consolidated, Analytics Average Risk Score, Risk Score Trend Risk score distribution (0-100) hsi_events_by_risk_level_totalCounter levelConsolidated, Analytics Events by Risk Level, High/Critical Alerts Events by risk level (low, medium, high, critical) hsi_prompt_template_used_totalCounter templateConsolidated Prompt Templates Used Prompt template usage (basic, enriched, vision)
Risk Levels: low, medium, high, critical
AI Service Request Metrics Metric Type Labels Dashboard Panel Description hsi_ai_request_duration_secondsHistogram serviceConsolidated YOLO26/Nemotron/Florence Latency Duration of AI service requests hsi_yolo26_inference_secondsHistogram - AI Services Inference Latency Percentiles YOLO26 detection with optimized buckets hsi_nemotron_inference_secondsHistogram - Consolidated Nemotron Inference Latency Nemotron LLM with optimized buckets hsi_florence_inference_secondsHistogram - AI Services Florence Latency Florence vision-language with optimized buckets hsi_florence_task_totalCounter taskConsolidated Florence Tasks Florence task invocations (caption, ocr, detect, dense_caption)
Services: yolo26, nemotron, florence, clip
Enrichment Model Metrics Metric Type Labels Dashboard Panel Description hsi_enrichment_model_calls_totalCounter modelConsolidated Enrichment Model Calls Enrichment model invocations hsi_enrichment_model_duration_secondsHistogram modelConsolidated Enrichment Model Latency Model inference duration hsi_enrichment_model_errors_totalCounter modelConsolidated Enrichment Errors Enrichment model errors hsi_enrichment_success_rateGauge modelConsolidated Success Rate Success rate per model (0.0-1.0) hsi_enrichment_partial_batches_totalCounter - - - Batches with partial enrichment hsi_enrichment_failures_totalCounter model- - Enrichment model failures hsi_enrichment_batch_status_totalCounter status- - Batch status (full, partial, failed, skipped) hsi_enrichment_retry_totalCounter endpoint- - Retry attempts by endpoint
Models: brisque, brisque-quality, weather, weather-classification, depth, vehicle-damage-detection, and others
Queue and Pipeline Metrics Metric Type Labels Dashboard Panel Description hsi_analysis_queue_depthGauge - Consolidated Analysis Queue Depth Batches waiting in analysis queue hsi_dlq_depthGauge queue_name- - Dead letter queue depth hsi_queue_overflow_totalCounter queue_name, policyConsolidated Queue Overflow Queue overflow events hsi_queue_items_moved_to_dlq_totalCounter queue_nameConsolidated Items Moved to DLQ Items moved to dead-letter queue hsi_queue_items_dropped_totalCounter queue_nameConsolidated Items Dropped Items dropped (drop_oldest policy) hsi_queue_items_rejected_totalCounter queue_nameConsolidated Items Rejected Items rejected (reject policy) hsi_pipeline_errors_totalCounter error_typeConsolidated Pipeline Errors Pipeline errors by type
Worker Metrics Metric Type Labels Dashboard Panel Description hsi_worker_restarts_totalCounter worker_nameConsolidated Worker Restarts Total worker restarts hsi_worker_crashes_totalCounter worker_nameConsolidated Worker Crashes Total worker crashes hsi_worker_max_restarts_exceeded_totalCounter worker_name- - Times max restart limit exceeded hsi_worker_statusGauge worker_name- - Worker status (0=stopped, 1=running, 2=crashed, 3=restarting, 4=failed) hsi_worker_active_countGauge - Consolidated Active Workers Workers currently active hsi_worker_busy_countGauge - Consolidated Busy Workers Workers processing tasks hsi_worker_idle_countGauge - Consolidated Idle Workers Workers idle hsi_pipeline_worker_restarts_totalCounter worker_name, reason_categoryConsolidated Pipeline Worker Restarts Pipeline worker restarts by reason hsi_pipeline_worker_restart_duration_secondsHistogram worker_name- - Restart operation duration hsi_pipeline_worker_stateGauge worker_nameConsolidated Pipeline Worker State Worker state (0=stopped, 1=running, 2=restarting, 3=failed) hsi_pipeline_worker_consecutive_failuresGauge worker_nameConsolidated Consecutive Failures Consecutive worker failures hsi_pipeline_worker_uptime_secondsGauge worker_nameConsolidated Worker Uptime Uptime since last successful start
LLM Token and Cost Metrics Metric Type Labels Dashboard Panel Description hsi_nemotron_tokens_input_totalCounter camera_idConsolidated Input Tokens Total input tokens sent to Nemotron hsi_nemotron_tokens_output_totalCounter camera_idConsolidated Output Tokens Total output tokens received hsi_nemotron_tokens_per_secondGauge - Consolidated Token Throughput Current token throughput hsi_nemotron_token_cost_usd_totalCounter camera_idConsolidated Token Cost Estimated cost for token usage hsi_llm_context_utilizationHistogram - - - Context window utilization (0.0-1.0+) hsi_llm_context_utilization_ratioGauge modelConsolidated Context Utilization Current context utilization ratio hsi_prompts_truncated_totalCounter - Consolidated Prompts Truncated Prompts requiring truncation hsi_prompts_high_utilization_totalCounter - Consolidated High Utilization Prompts Prompts exceeding utilization threshold
Cost Tracking Metrics Metric Type Labels Dashboard Panel Description hsi_gpu_seconds_totalCounter model- - GPU time consumed by AI models hsi_estimated_cost_usd_totalCounter service- - Estimated cost based on cloud equivalents hsi_event_analysis_cost_usd_totalCounter camera_id- - Cost per event analysis hsi_daily_cost_usdGauge - Consolidated Daily Cost Current daily estimated cost hsi_monthly_cost_usdGauge - Consolidated Monthly Cost Current monthly estimated cost hsi_budget_utilization_ratioGauge periodConsolidated Budget Utilization Budget utilization ratio (daily, monthly) hsi_budget_exceeded_totalCounter period- - Times budget threshold exceeded hsi_cost_per_detection_usdGauge - Consolidated Cost per Detection Average cost per detection hsi_cost_per_event_usdGauge - Consolidated Cost per Event Average cost per security event
Cache Metrics Metric Type Labels Dashboard Panel Description hsi_cache_hits_totalCounter cache_type- - Cache hits hsi_cache_misses_totalCounter cache_type- - Cache misses hsi_cache_invalidations_totalCounter cache_type, reason- - Cache invalidations hsi_cache_stale_hits_totalCounter cache_type- - Stale cache hits (SWR) hsi_cache_background_refresh_totalCounter cache_type, status- - Background refresh operations
Cache Types: event_stats, cameras, system, events
Redis Pool Metrics Metric Type Labels Dashboard Panel Description hsi_redis_pool_sizeGauge pool_type- - Redis connection pool size hsi_redis_pool_availableGauge pool_type- - Available pool connections hsi_redis_pool_in_useGauge pool_type- - Connections in use
Video Analytics - Object Tracking Metrics Metric Type Labels Dashboard Panel Description hsi_tracks_created_totalCounter camera_idVideo Analytics Tracks Created (1h) Object tracks created hsi_tracks_lost_totalCounter camera_id, reasonVideo Analytics Tracks Lost (1h), Track Loss Reasons Tracks lost (timeout, out_of_frame, occlusion) hsi_tracks_reidentified_totalCounter camera_idVideo Analytics Tracks Reidentified (1h) Tracks reidentified after being lost hsi_track_duration_secondsHistogram camera_id, entity_typeVideo Analytics Track Duration P95 Duration from creation to loss hsi_track_active_countGauge camera_idVideo Analytics Active Tracks Current active tracks
Track Loss Reasons: timeout, out_of_frame, occlusion
Video Analytics - Zone Monitoring Metrics Metric Type Labels Dashboard Panel Description hsi_zone_crossings_totalCounter zone_id, direction, entity_typeVideo Analytics Zone Entries, Zone Crossings Over Time Zone boundary crossings (enter, exit) hsi_zone_intrusions_totalCounter zone_id, severityVideo Analytics Zone Intrusions (1h) Zone intrusion alerts hsi_zone_occupancyGauge zone_idVideo Analytics Current Zone Occupancy Current entities in zone hsi_zone_dwell_time_secondsHistogram zone_idVideo Analytics Zone Dwell Time P95 Time spent in zone
Zone Directions: enter, exit Severity Levels: low, medium, high
Video Analytics - Loitering Detection Metrics Metric Type Labels Dashboard Panel Description hsi_loitering_alerts_totalCounter camera_id, zone_idVideo Analytics Loitering Alerts (1h), Loitering Alerts Over Time Loitering alerts generated hsi_loitering_dwell_time_secondsHistogram camera_idVideo Analytics Median Loitering Duration Loitering dwell time
Video Analytics - Action Recognition Metrics Metric Type Labels Dashboard Panel Description hsi_action_detections_totalCounter action_type, camera_idVideo Analytics Actions Recognized (1h), Actions by Type (1h) Action detections recorded by record_action_detection; the series was defined inside backend/services/action_recognition_service.py (not backend/core/metrics.py), so the 2026-09-23 archive of that chain (full X-CLIP removal, incl. the POST /api/action-events analyze endpoint) removed the definition itself — the series no longer exists until a ST-GCN++-era feeder defines and emits it hsi_action_confidenceHistogram action_typeVideo Analytics Median Confidence, Confidence by Action Type Action confidence scores (same status as hsi_action_detections_total: defined in the archived service, so the series no longer exists) hsi_enrichment_model_duration_secondsHistogram modelVideo Analytics Inference Latency P95 Action-inference latency from the Triton stgcn_action skeleton path ({model="stgcn"})
Action Types: walking, loitering, fighting, and others
Retired: hsi_action_recognition_total / _confidence / _duration_seconds (the backend/core/metrics.py video-analytics family) and the enrichment_action_recognition_* families (removed from ai/enrichment/metrics.py with the NEM-5563 xclip retirement) are never observed outside tests. The labelled members export nothing; the unlabelled _duration_seconds histogram does scrape as zero-valued buckets (prometheus_client emits unlabelled histograms unconditionally), so panels against it would read a flat 0, never real latency — no panel may query any of them for signal.
Video Analytics - Face Recognition Metrics Metric Type Labels Dashboard Panel Description hsi_face_detections_totalCounter camera_id, match_statusVideo Analytics Faces Detected, Known/Unknown Faces, Face Detections Over Time Total faces detected hsi_face_quality_scoreHistogram - Video Analytics Median Face Quality, Face Quality Score Distribution Face quality scores hsi_face_embeddings_generated_totalCounter match_statusVideo Analytics, AI Known Faces (1h), Unknown Faces (1h), Known vs Unknown Faces Face embeddings generated (NEM-4143) hsi_face_recognition_confidenceHistogram - AI Services Recognition Confidence Distribution Face recognition confidence scores (NEM-4143) hsi_face_matches_totalCounter person_id- - Face matches against known persons
Match Status: known, unknown
GPU Metrics Metric Type Labels Dashboard Panel Description hsi_gpu_utilizationGauge - Consolidated GPU Utilization GPU compute utilization % hsi_gpu_temperatureGauge - Consolidated GPU Temperature GPU temperature (Celsius) hsi_gpu_memory_used_mbGauge - Consolidated GPU Memory Used GPU memory used (MB) hsi_gpu_memory_total_mbGauge - Consolidated GPU Memory Total Total GPU memory (MB) hsi_gpu_sm_clock_mhzGauge - Consolidated SM Clock Current SM clock speed hsi_gpu_sm_clock_max_mhzGauge - Consolidated SM Clock Max Max SM clock speed hsi_gpu_memory_clock_mhzGauge - Consolidated Memory Clock Current memory clock hsi_gpu_memory_clock_max_mhzGauge - Consolidated Memory Clock Max Max memory clock hsi_gpu_power_limit_wattsGauge - Consolidated Power Limit GPU power limit (Watts) hsi_gpu_throttle_reasonsGauge - Consolidated Throttle Reasons GPU throttling reasons hsi_gpu_pstateGauge - Consolidated P-State GPU performance state hsi_gpu_fan_speedGauge - Consolidated Fan Speed GPU fan speed % hsi_gpu_compute_processesGauge - Consolidated Compute Processes Active compute processes hsi_gpu_pcie_tx_throughput_kbsGauge - Consolidated PCIe TX Throughput PCIe transmit throughput hsi_gpu_pcie_rx_throughput_kbsGauge - Consolidated PCIe RX Throughput PCIe receive throughput hsi_gpu_encoder_utilizationGauge - Consolidated Encoder Utilization Video encoder utilization hsi_gpu_decoder_utilizationGauge - Consolidated Decoder Utilization Video decoder utilization hsi_gpu_memory_bandwidth_utilizationGauge - Consolidated Memory Bandwidth Util Memory bandwidth utilization hsi_gpu_temp_slowdown_thresholdGauge - Consolidated Temp Slowdown Threshold Temperature slowdown threshold hsi_gpu_pcie_link_genGauge - Consolidated PCIe Link Gen PCIe link generation hsi_gpu_pcie_link_widthGauge - Consolidated PCIe Link Width PCIe link width hsi_gpu_pcie_replay_counterCounter - Consolidated PCIe Replay Counter PCIe replay error counter hsi_gpu_bar1_used_mbGauge - Consolidated BAR1 Memory Used BAR1 memory aperture used
System Metrics Metric Type Labels Dashboard Panel Description hsi_uptime_secondsGauge - Consolidated System Uptime System uptime in seconds hsi_total_camerasGauge - Consolidated Total Cameras Number of configured cameras hsi_total_eventsGauge - Consolidated Total Events Total events in database
Circuit Breaker Metrics Metric Type Labels Dashboard Panel Description hsi_circuit_breaker_stateGauge breakerConsolidated Circuit Breaker State Breaker state (0=closed, 1=open, 2=half_open) hsi_circuit_breaker_trips_totalCounter breakerConsolidated Circuit Breaker Trips Total breaker trips
Real User Monitoring (RUM) Metrics Metric Type Labels Dashboard Panel Description hsi_rum_page_load_time_secondsHistogram - Consolidated Page Load Time P75 Browser page load time hsi_rum_fcp_secondsHistogram - Consolidated First Contentful Paint P75 First Contentful Paint (FCP) hsi_rum_lcp_secondsHistogram - Consolidated Largest Contentful Paint P75 Largest Contentful Paint (LCP) hsi_rum_cls_bucketHistogram - Consolidated Cumulative Layout Shift P75 Cumulative Layout Shift (CLS)
OpenTelemetry Metrics (otel_metrics.py) These metrics use OpenTelemetry instrumentation and are available when OTEL_METRICS_ENABLED=True.
Circuit Breaker OTel Metrics Metric Type Labels Description circuit_breaker.stateGauge breakerCurrent state (0=closed, 1=open, 2=half_open) circuit_breaker.transitionsCounter breaker, from_state, to_stateState transitions circuit_breaker.failuresCounter breakerFailures recorded circuit_breaker.successesCounter breakerSuccesses recorded circuit_breaker.rejectedCounter breakerCalls rejected
AI Model Latency OTel Histograms Metric Type Labels Description ai.detection.latencyHistogram model.version, batch.size, gpu.idYOLO26 inference latency (ms) ai.nemotron.latencyHistogram model.version, batch.size, gpu.id, tokens.generatedNemotron LLM latency (ms) ai.florence.latencyHistogram model.version, batch.size, gpu.id, task.typeFlorence latency (ms) ai.pipeline.latencyHistogram camera.id, pipeline.stage, detection.countTotal pipeline latency (ms) ai.batch.processing_timeHistogram batch.size, camera.count, batch.idBatch processing time (ms)
Metrics Not Yet Visualized The following metrics exist in the codebase but do not have dedicated dashboard panels:
Metric Reason hsi_dlq_depthAlert-based monitoring only hsi_enrichment_partial_batches_totalDetailed operational metric hsi_enrichment_failures_totalUses hsi_enrichment_model_errors_total instead hsi_enrichment_batch_status_totalDetailed operational metric hsi_enrichment_retry_totalDetailed operational metric hsi_worker_max_restarts_exceeded_totalAlert-based monitoring only hsi_worker_statusUses hsi_pipeline_worker_state instead hsi_pipeline_worker_restart_duration_secondsDetailed operational metric hsi_gpu_seconds_totalCost calculation intermediate hsi_estimated_cost_usd_totalCost calculation intermediate hsi_event_analysis_cost_usd_totalCost calculation intermediate hsi_budget_exceeded_totalAlert-based monitoring only hsi_cache_*Detailed operational metrics hsi_redis_pool_*Detailed operational metrics hsi_face_embeddings_generated_totalNow dashboarded (NEM-4143) - Known/Unknown faces hsi_face_matches_totalDetailed operational metric All OpenTelemetry metrics Exported to OTel backend (Tempo/Jaeger)
Dashboard Reference Dashboard UID Purpose Primary Metrics Consolidated consolidatedMain operations dashboard Pipeline, AI services, GPU, workers, costs AI Services ai-servicesAI gateway / Triton detection monitoring hsi_ai_* / hsi_yolo26_* (backend/core/metrics.py, gateway-served) HSI Analytics hsi-analyticsEvents and risk analysis Backend API data (JSON datasource) Video Analytics video-analyticsTracking, zones, faces, actions hsi_tracks_*, hsi_zone_*, hsi_face_*, hsi_action_*, hsi_loitering_* HSI Profiling hsi-profilingContinuous profiling (Pyroscope) CPU and memory flame graphs HSI System Logs hsi-logsCentralized logging (Loki) Log volume, error patterns HSI Distributed Tracing hsi-tracingRequest tracing (Jaeger) Trace search and analysis
Adding New Metrics When adding new metrics:
Define the metric in /backend/core/metrics.py following naming conventions Add recording method to MetricsService class Call the method from relevant service code Add to appropriate Grafana dashboard(s) Update this documentation Example metric definition:
MY_NEW_METRIC = Counter (
"hsi_my_new_metric_total" ,
"Description of what this metric tracks" ,
labelnames = [ "label1" , "label2" ],
registry = _registry ,
)