Profiling (Pyroscope)¶

Continuous profiling dashboard for analyzing CPU and memory usage across all services.

What You're Looking At¶

The Profiling page provides continuous profiling capabilities powered by Pyroscope and visualized through Grafana. This page helps you identify performance bottlenecks, memory leaks, and CPU hotspots in the AI pipeline services.

Layout Overview¶

+----------------------------------------------------------+
|  HEADER: Flame Icon | "Profiling" | Action Buttons        |
+----------------------------------------------------------+
|                                                          |
|  +----------------------------------------------------+  |
|  |                                                    |  |
|  |           GRAFANA DASHBOARD EMBED                  |  |
|  |                                                    |  |
|  |  +------------+ +------------+ +------------+      |  |
|  |  | Service    | | Profile    | | Time       |      |  |
|  |  | Selector   | | Type       | | Range      |      |  |
|  |  +------------+ +------------+ +------------+      |  |
|  |                                                    |  |
|  |  +------------------------------------------+      |  |
|  |  |                                          |      |  |
|  |  |           FLAME GRAPH                    |      |  |
|  |  |                                          |      |  |
|  |  +------------------------------------------+      |  |
|  |                                                    |  |
|  +----------------------------------------------------+  |
|                                                          |
+----------------------------------------------------------+

The page embeds the HSI Profiling dashboard from Grafana, which provides:

• Service Selector - Choose which service to profile (backend, YOLO26, Nemotron)
• Profile Type - Switch between CPU and memory profiles
• Time Range - Select the time period to analyze
• Flame Graph - Visual representation of where time/memory is spent

Key Components¶

Header Controls¶

Flame Graph¶

The flame graph is the primary visualization for understanding where time or memory is being consumed:

• Width - Represents the proportion of time/memory used by that function
• Depth - Shows the call stack hierarchy (callers on top, callees below)
• Color - Different colors represent different packages or modules
• Hover - Shows detailed information about that function call

Reading the Flame Graph:

Profile Types¶

Service Selection¶

The dashboard can profile these services:

Understanding Profiling Data¶

CPU Profiling¶

CPU profiles show where processing time is spent. Look for:

1. Wide flames at the bottom - Functions doing actual computation
2. Unexpected wide bars - Code paths consuming more CPU than expected
3. Third-party libraries - External code that may need optimization or caching

Common CPU Hotspots in AI Pipelines:

Memory Profiling¶

Memory profiles show allocation patterns. Look for:

1. Growing allocations - Potential memory leaks
2. Large single allocations - May cause GC pressure
3. Frequent small allocations - May benefit from pooling

Common Memory Patterns:

Correlation with Tracing¶

Profiling data can be correlated with distributed traces:

1. Find a slow trace in the Tracing page
2. Note the time range of the slow operation
3. Open Profiling and select the same time range
4. Identify which code paths consumed the most resources during that period

This helps pinpoint exactly why a specific request was slow.

Settings & Configuration¶

Grafana URL¶

The Grafana URL is automatically configured from the backend. If the embedded dashboard fails to load, verify:

1. Grafana is running and accessible
2. The grafana_url config setting is correct
3. Network connectivity between frontend and Grafana

Pyroscope Data Source¶

Pyroscope must be configured as a data source in Grafana:

# Grafana provisioning (monitoring/grafana/provisioning/datasources/prometheus.yml)
- name: Pyroscope
  type: pyroscope
  url: http://pyroscope:4040
  access: proxy

Retention¶

Profiling data retention is configured in Pyroscope:

Troubleshooting¶

Dashboard Shows "No Data"¶

1. Check Pyroscope is running: docker ps | grep pyroscope
2. Verify services are instrumented: Check that services have Pyroscope SDK configured
3. Check time range: Ensure the selected time range has profiling data
4. Verify datasource: Confirm Pyroscope is configured in Grafana

Flame Graph is Empty¶

1. Select a different service from the dropdown
2. Expand the time range
3. Check if the service was active during the selected period
4. Verify the profile type is appropriate for the service

High Memory Usage in Profiling¶

Continuous profiling has minimal overhead (typically 1-3%), but:

1. Reduce sampling frequency if needed
2. Limit the number of profiled services
3. Reduce retention period for older data

"Failed to load configuration" Error¶

The frontend couldn't fetch the Grafana URL from the backend:

1. Verify the backend is running
2. Check network connectivity
3. The dashboard will use /grafana as a fallback

Technical Deep Dive¶

Architecture¶

flowchart LR
    subgraph Services["Profiled Services"]
        B[Backend]
        R[YOLO26]
        N[Nemotron]
    end

    subgraph Collection["Data Collection"]
        A[Alloy]
        P[Pyroscope]
    end

    subgraph Visualization["Visualization"]
        G[Grafana]
        F[Frontend]
    end

    B -->|profiles| A
    R -->|profiles| A
    N -->|profiles| A
    A -->|push| P
    P -->|query| G
    G -->|iframe| F

    style Services fill:#e0f2fe
    style Collection fill:#fef3c7
    style Visualization fill:#dcfce7

Related Code¶

Frontend:

• Pyroscope Page: frontend/src/components/pyroscope/PyroscopePage.tsx
• Grafana URL Utility: frontend/src/utils/grafanaUrl.ts

Backend:

• Profiling Configuration: backend/core/config.py

Infrastructure:

• Pyroscope Container: docker-compose.prod.yml (pyroscope service)
• Grafana Dashboard: monitoring/grafana/dashboards/hsi-profiling.json
• Alloy Configuration: monitoring/alloy/config.alloy

Data Flow¶

1. Services are instrumented with Pyroscope SDK
2. Profile data is pushed to Grafana Alloy
3. Alloy forwards profiles to Pyroscope
4. Grafana queries Pyroscope for visualization
5. Frontend embeds Grafana dashboard in iframe

Quick Reference¶

When to Use Profiling¶

Common Actions¶