NVIDIA Technology Inventory¶

Generated: 2026-02-19 | Updated: 2026-09-22 (full-tree docs scan) Audit: 10 research agents, 5 validation agents, 5 implementation agents, ~1.5M tokens

Comprehensive audit of every NVIDIA-specific technology, library, and version number used throughout the nemotron-v3-home-security-intelligence codebase.

The 2026-02-19 "Improvements Applied" and "Documentation Fixes" sections below are a historical record of that audit's own changes; their line numbers describe the tree as it stood then. The 2026-09-22 pass re-verified the live tables (versions, line references, container images, CI workflows) against the current tree.

Table of Contents¶

• Hardware
• Container Base Images
• NVIDIA Software Products
• NVIDIA CUDA Python Packages
• Nemotron LLM
• llama.cpp (GGML CUDA Backend)
• NVIDIA Triton Inference Server
• NVIDIA TensorRT
• ONNX Runtime GPU
• CUDA Infrastructure Modules
• GPU Passthrough & Container Toolkit
• GPU Monitoring & Observability
• Prometheus GPU Metrics
• GPU Alert Rules
• NVIDIA Inference API
• CI/CD GPU Integration
• Stale References

Hardware¶

• Host NVIDIA Driver: 580.119.02 (minimum required: 580 for CUDA 13.x)
• Host CUDA Version: 13.0
• Both GPUs are Ampere sm_86 -- the RTX A400 was previously misidentified as Turing sm_75 in legacy docs

Container Base Images¶

Container availability: only ai-gateway, ai-llm and the profiled ai-llm-vllm / dcgm-exporter appear in docker-compose.prod.yml. The ai-clip, ai-florence, ai-enrichment and ai-enrichment-light images still build from the Dockerfiles below but are in no compose file -- their capabilities run inside ai-gateway (Triton routers) or inside the backend (backend/services/model_zoo.py). The ai-yolo26 standalone GPU image was retired fully on 2026-09-23 (owner ruling -- Triton on ai-gateway serves yolo26); its Dockerfile and companion build files now live in archive/ai-yolo26-image/ and it is gone from the deploy build matrices.

NVIDIA Software Products¶

NVIDIA CUDA Python Packages¶

All packages are PyTorch transitive dependencies from the CUDA torch wheel. Platform-conditional: x86_64 Linux only.

Note (2026-09-21): The versions below were captured while torch resolved from the CUDA wheel index. uv.lock now pins torch 2.14.0+cpu from download.pytorch.org/whl/cpu, which pulls none of these packages, and requirements-audit.txt is no longer committed (CI regenerates it with uv export --no-hashes; measured 2026-09-21: the current export contains zero nvidia-*-cu12/triton entries). Re-measure this table on a CUDA-configured host before treating any row as current. The one package still in the lock is nvidia-ml-py, at 13.610.43.

Note (re-measured 2026-09-21): The backend uses CPU-only PyTorch (2.14.0+cpu from download.pytorch.org/whl/cpu, uv.lock torch block). Since the CPU-wheel pin, the nvidia-*-cu12 packages appear in neither uv.lock nor the audit export — earlier revisions of this page listed them "in the audit export" from a CUDA-index era. They live inside the AI containers, which use CUDA PyTorch from their base images.

Nemotron LLM¶

Production Model¶

• Model: NVIDIA Nemotron-3-Nano-30B-A3B
• Architecture: Hybrid Mamba-Transformer MoE (52 layers: 23 Mamba SSM, 23 MoE FFN, 6 GQA Attention)
• Parameters: 30B total / 3.5B active
• Quantization: Q4_K_M (GGUF)
• File: Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf (~15 GB on disk per models.yml size_mb: 15073; the "~9.5 GB" figure still circulating in this page's quant table was a pre-release estimate)
• VRAM: ~14.7 GB
• HuggingFace: unsloth/Nemotron-3-Nano-30B-A3B-GGUF (source recorded in models.yml; download via setup_lib/model_downloader.py)

Development Model¶

• Model: Nemotron Mini 4B Instruct
• File: nemotron-mini-4b-instruct-q4_k_m.gguf
• VRAM: ~3 GB

Available GGUF Quantizations¶

NVIDIA-Provided Quantization Formats (HuggingFace)¶

Note: All vLLM-compatible formats failed on RTX A5500: NVFP4 requires H100/A100, FP8 exceeds 24 GB, AWQ has vLLM Mamba-2 bug, BF16 requires 60 GB.

llama.cpp (GGML CUDA Backend)¶

Build Configuration¶

Runtime Flags¶

llama-server \
    --model ${MODEL_PATH} \
    --host 0.0.0.0 --port ${PORT} \
    --n-gpu-layers ${GPU_LAYERS} \
    --ctx-size ${CTX_SIZE} \
    --parallel ${PARALLEL} \
    --threads ${THREADS} --threads-batch ${THREADS} \
    --batch-size ${BATCH_SIZE} \
    --ubatch-size ${UBATCH_SIZE} \
    --cache-type-k ${CACHE_TYPE_K} \
    --cache-type-v ${CACHE_TYPE_V} \
    --cont-batching \
    --metrics \
    --cache-reuse 256 \
    --mlock \
    --flash-attn on \
    ${MOE_ARGS}   # --override-tensor <pattern>=CPU (when set)

Runtime Environment Variables¶

Remaining Optimization Opportunity¶

NVIDIA Triton Inference Server¶

Server Configuration¶

Triton Client¶

Model Repository (14 models)¶

The effective device for each model is set at container startup: ai/gateway/entrypoint.sh runs ai/gateway/patch_triton_configs.py, which rewrites each config.pbtxt instance_group from the triton_kind field in models.yml at the repo root (models.yml is the single source of truth; the table below shows the resulting deployed kinds). The shipped config.pbtxt files in ai/triton/model_repository/ currently match these defaults.

Summary: 12 GPU models, 2 CPU models, one instance group each. 12 use ONNX Runtime, 1 uses the onnxruntime_onnx platform (stgcn_action), 1 uses the Python backend (florence2). No models use TensorRT backend in Triton. clip_text stays on CPU because its INT8 export contains MatMulInteger ops the CUDA EP cannot run; stgcn_action is an intentionally-CPU skeleton model. xclip_action (the second former Python-backend model) was retired from ai/triton/model_repository/ with NEM-5563 — X-CLIP is deprecated in favour of ST-GCN++ per models.yml (enabled: false) — and its config/model.py now ship under archive/triton-model-repository/.

The compose health comments (docker-compose.prod.yml line 337 and docker-compose.ghcr.yml line 235, "Triton loads 14 models") were corrected to 14 in the gateway-consolidation follow-up; xclip_action's retirement (NEM-5563) is why the count never went 15 -> 14 the way the comment's math once implied.

NVIDIA TensorRT¶

Version¶

• TensorRT 10.14.x (bundled in the nvcr.io/nvidia/tensorrt:26.08-py3 / tritonserver:26.01-py3 containers)
• Python package: tensorrt>=10.0.0, tensorrt-cu12>=10.0.0

Current Status¶

TensorRT is NOT active in Triton -- all 14 Triton models use ONNX Runtime or Python backend. TensorRT paths exist in:

1. CLIP standalone server (dev-only) -- disabled by default (CLIP_USE_TENSORRT=false)
2. Export pipeline -- ai/gateway/export/export_all.sh can generate TensorRT engines (the former standalone YOLO26 server that loaded yolo26m_fp16.engine retired 2026-09-23 with its image; ai/yolo26/build_engine.py stays for scripts/prebuild-tensorrt-engines.sh)

Engine Files (Generated at Runtime)¶

Conversion Strategy¶

Each export script tries trtexec first (pre-installed in NVIDIA containers), then falls back to the tensorrt Python API. Default workspace: 1 GB (tuned for 4 GB RTX A400).

TensorRT Infrastructure Code¶

TensorRT Environment Variables¶

Key Limitation¶

TensorrtExecutionProvider (ONNX Runtime's TensorRT EP) is never used in this codebase. All TensorRT usage goes through:

1. The tensorrt Python package directly
2. Ultralytics model.export(format="engine")
3. trtexec subprocess

ONNX Runtime GPU¶

CUDAExecutionProvider Usage¶

All GPU-accelerated ONNX models use CUDAExecutionProvider, not TensorrtExecutionProvider:

providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]

Referenced in 6+ export scripts under ai/gateway/export/:

• export_depth.py, export_vehicle.py, export_pet.py
• export_reid.py, export_fashion_clip.py, export_demographics.py

Packages¶

CUDA Infrastructure Modules¶

Custom Python modules in ai/ for GPU optimization:

CUDA Environment Variables¶

GPU Passthrough & Container Toolkit¶

CDI (Container Device Interface)¶

The project uses Podman CDI for rootless GPU access:

# ai-llm: single GPU passthrough
devices:
  - nvidia.com/gpu=${GPU_LLM:-0}

# ai-gateway: ALL GPUs passed, CUDA_VISIBLE_DEVICES restricts
devices:
  - nvidia.com/gpu=all
environment:
  - CUDA_VISIBLE_DEVICES=${GPU_AI_SERVICES:-1}

Critical CDI workaround: Using nvidia.com/gpu=N only creates /dev/nvidiaN. If N>0, CUDA fails (expects /dev/nvidia0). Fix: use nvidia.com/gpu=all + CUDA_VISIBLE_DEVICES=N.

GPU Assignment Variables (.env)¶

NVIDIA Container Toolkit Detection¶

setup_lib/nvidia_detect.py and setup_lib/nvidia_toolkit.py handle:

• GPU detection via nvidia-smi
• Driver version validation (minimum 580 for CUDA 13.x)
• Container Toolkit detection (nvidia-ctk)
• CDI spec generation: nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
• Per-distro install commands (Fedora/Debian/Ubuntu/Arch)

Docker Compose GPU Deploy Blocks¶

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          device_ids: ['${GPU_LLM:-0}']
          capabilities: [gpu]

Persistent CUDA Cache Volumes¶

GPU Monitoring & Observability¶

NVML / nvidia-ml-py (Primary)¶

Package: nvidia-ml-py>=12.560.30,<14.0.0 (resolved: 13.610.43 per uv.lock)

Installed in all AI containers and the backend. Used by:

nvidia-smi (Fallback)¶

Used as secondary data source when pynvml is unavailable:

nvidia-smi --query-gpu=temperature.gpu,power.draw,utilization.gpu,memory.used,memory.total,name \
  --format=csv,noheader,nounits

DCGM Exporter¶

DCGM Counter Fields:

Triton Native Metrics¶

Scraped from ai-gateway:8002/metrics:

• nv_gpu_utilization, nv_gpu_memory_used_bytes
• nv_inference_request_success, nv_inference_request_failure
• nv_inference_queue_duration_us, nv_inference_compute_infer_duration_us

Prometheus GPU Metrics¶

Scrape Jobs (monitoring/prometheus.yml)¶

HSI GPU Metrics (from backend API via json-exporter)¶

AI Service GPU Metrics¶

The yolo26_* gauges (ai/yolo26/metrics.py) and enrichment_vram_* gauges (ai/enrichment/model_manager.py) are still defined in code, but the standalone ai-yolo26/ai-florence/ai-clip/ai-enrichment/ai-enrichment-light containers that exposed them via /metrics are no longer deployed -- those Prometheus jobs were removed (see the "Removed targets" note in monitoring/prometheus.yml, which says Triton's native nv_* metrics are the replacement). Treat the rows above as legacy unless a scrape target for them exists.

Grafana Dashboard¶

monitoring/grafana/dashboards/hsi-gpu-metrics.json -- GPU dashboard with panels for utilization, VRAM, temperature, power, clocks, PCIe throughput, and memory bandwidth.

GPU Alert Rules¶

DCGM-Based Alerts (monitoring/gpu-alerts.yml)¶

AI Pipeline GPU Alerts (monitoring/ai-pipeline-alerts.yml)¶

GPUOOMCritical (keyed on rate(ai_gpu_oom_total[5m]) > 0) was deleted -- post-consolidation nothing in the gateway topology exports ai_gpu_oom_total, so the alert could never fire (see the deletion comment at monitoring/ai-pipeline-alerts.yml:33). GPUInferenceFailures above is the Triton-backed replacement, scraped from ai-gateway:8002/metrics.

Backend Performance Thresholds (backend/services/performance_collector.py)¶

NVIDIA Inference API¶

Used for media generation (not AI inference pipeline):

The endpoint literal also appears in scripts/synthetic/media_generator.py. The shell helpers point users to https://build.nvidia.com for keys; inference-api.nvidia.com is the host those generated calls target.

Requires NVIDIA_API_KEY or NVAPIKEY environment variable.

CI/CD GPU Integration¶

GitHub Actions Workflows¶

No GPU CI today (2026-09-22): gpu-tests.yml has been removed, and nightly.yml's extended-benchmarks job (self-hosted gpu, rtx-a5500) was removed on 2026-09-15 while the GPU runner is offline -- its in-file note says to restore it from git history when the box returns. backend/tests/gpu/ and scripts/setup-gpu-runner.sh remain, so the test suite and runner registration still exist locally; they just have no scheduled job.

GPU Runner Setup (scripts/setup-gpu-runner.sh)¶

Tests GPU with multiple CUDA images: nvidia/cuda:12.0.0, 12.2.0, 11.8.0 on Ubuntu 22.04. Installs nvidia-container-toolkit and registers runner with labels self-hosted,linux,gpu,rtx-a5500.

Why ONNX Runtime Was Chosen Over TensorRT¶

TensorRT was the original design for 5 of the 15 Triton models. The decision to use ONNX Runtime instead was not made upfront -- it evolved through practical hardware constraints discovered during implementation.

Decision Timeline¶

Root Causes (4 Blocking Issues)¶

1. A400 workspace memory constraint (primary blocker)

The trtexec builder for CLIP and similar large models needs workspace memory that competes with the total 4GB VRAM on the RTX A400. Building TensorRT engines on a 4GB GPU with Triton + other models loaded causes OOM. Discovered during Phase 2 of migration, not anticipated in the design phase.

Source: commit 8c358978 -- "CLIP/Fashion-CLIP: switch from TensorRT to ONNX Runtime (4GB A400 too small for trtexec workspace)."

2. INT8 ConvInteger operator incompatibility (NEM-5547)

onnxruntime.quantization inserts ConvInteger nodes into INT8 graphs. ONNX Runtime's CUDAExecutionProvider does not support ConvInteger -- it is a CPU-only operator. INT8 ONNX on GPU either fails to load or silently falls back to CPU. TensorRT's native INT8 path supports this, but returns to the workspace OOM problem.

Source: commit 3710a98c -- "yolo26: Use FP32 ONNX on GPU (INT8 ConvInteger ops unsupported by CUDA EP)." Preserved in current config.pbtxt line 7.

3. Ultralytics .engine metadata wrapper incompatibility

Pose and threat models exported via Ultralytics as .engine files have a metadata wrapper incompatible with Triton's tensorrt backend, which expects raw engine files without the Ultralytics envelope.

Source: commit 3710a98c -- "pose/threat: Use ONNX Runtime on GPU (not TensorRT -- Ultralytics .engine format has metadata wrapper incompatible with Triton's tensorrt backend)."

4. Florence-2 and X-CLIP cannot be exported to TensorRT/ONNX

Florence-2's autoregressive decoder and X-CLIP's custom cross-frame attention prevent static graph export. Both require the Python backend.

Source: ai/triton/model_repository/florence2/config.pbtxt line 3, archive/triton-model-repository/xclip_action/config.pbtxt line 3 (xclip_action retired from the live repo with NEM-5563).

Performance Comparison¶

The evaluation docs noted "near-TensorRT performance" from ONNX Runtime CUDA EP, making it an acceptable tradeoff given the A400's 4GB constraint. The standalone YOLO26 server on GPU 0 (A5500 24GB) continues to use native TensorRT where workspace is not a constraint.

Future Path: TensorRT Execution Provider (Research Complete 2026-02-19)¶

Research confirmed that TensorrtExecutionProvider can be enabled as a config-only change in Triton's ONNX Runtime backend. The Triton 26.01 container already ships with TensorRT linked into the ORT backend. No model re-export needed.

Recommended rollout (Wave 1 -- static-batch, lowest risk):

Add this block to threat/config.pbtxt, pose/config.pbtxt, and yolo26/config.pbtxt:

optimization {
  execution_accelerators {
    gpu_execution_accelerator : [
      {
        name : "tensorrt"
        parameters { key: "precision_mode"           value: "FP16" }
        parameters { key: "max_workspace_size_bytes"  value: "268435456" }
        parameters { key: "trt_engine_cache_enable"   value: "1" }
        parameters { key: "trt_engine_cache_path"     value: "/models/cache/trt_engines" }
      }
    ]
  }
}

Wave 2 (dynamic-batch, needs shape profiles): pet, reid, clip -- add trt_min_shapes/trt_opt_shapes/trt_max_shapes parameters.

Skip: depth (hardcoded Reshape in DPT head), florence2 (Python backend — the list formerly also named xclip_action, retired 2026-09-23; its successor stgcn_action is CPU-only and covered by the next clause), all CPU models.

Expected gains: 1.5-2x inference speedup for YOLO models, ~2x for classification models. First startup slow (~5-15 min per model for TRT compilation), cached on subsequent runs.

Rollback: Remove the optimization block. ONNX files unchanged, zero-risk revert.

Other hardware paths remain viable:

• Upgrading GPU 1 to >=8GB VRAM would unblock native trtexec workspace
• Pre-building engines outside Triton (one-shot container) could produce raw .plan files

Stale References¶

Fixed (2026-02-19)¶

Historical record — several rows have drifted again since (e.g. ai/clip/AGENTS.md line 73 still says tensorrt:26.01-py3 while ai/clip/Dockerfile now builds from 26.08-py3; the nemotron base images moved 13.1.1 -> 13.3.1; the ai/yolo26/Dockerfile cited below was retired 2026-09-23 and moved to archive/ai-yolo26-image/Dockerfile). See the live tables above for current values.

The following stale references were identified during the audit and corrected:

Remaining (In Docs/Plans Only)¶

VRAM Budget Summary¶

Estimated per-model footprint (from the 2026-02 audit; vehicle, fashion_clip and the two demographics ViTs were promoted to GPU later and are not itemized here -- add ~1.5 GB + ~0.5 GB + 2 x ~0.2 GB on GPU 1 for them).

The "Enrichment on-demand budget" (VRAM_BUDGET_GB, default 6.8 in ai/enrichment/model.py) governs the LRU manager inside the standalone ai-enrichment container, which is no longer deployed. The backend's in-process model_zoo has no budget -- it loads on demand and unloads when the last reference is released.

Improvements Applied (2026-02-19)¶

Changes implemented as a result of this audit:

Performance Optimizations¶

INT8 Quantization Infrastructure¶

Documentation Fixes (7 Stale References)¶

Research Completed (Pending Implementation)¶