AI Services Configuration¶

Configure environment variables for AI inference services.

Time to read: ~12 min Prerequisites: AI Installation

Environment Variables¶

Set these in your shell profile (~/.bashrc or ~/.zshrc) or in a .env file at the project root.

Startup Script Variables¶

There is no scripts/start-ai.sh — the host-run startup scripts are ai/start_detector.sh, ai/start_llm.sh and ai/start_nemotron.sh (each derives its own paths from its location). These read:

Log files: ai/start_nemotron.sh writes to /tmp/nemotron.log; ai/start_llm.sh and ai/start_detector.sh run in the foreground and log to the terminal. In compose, read logs with podman compose -f docker-compose.prod.yml logs ai-gateway / … logs ai-llm.

YOLO26 Detection Server (standalone, host-run)¶

Configuration for ai/yolo26/model.py when you run it directly on the host. Production detection does not use this server — YOLO26 runs as an ONNX Runtime model inside the ai-gateway Triton container (/yolo26 router); the TensorRT and torch.compile variables below apply only to the standalone host-run server. The server uses TensorRT-optimized engines for efficient GPU inference.

Core Configuration¶

TensorRT Engine Paths¶

TensorRT engines are GPU-architecture specific. Use the appropriate engine for your deployment:

Host-run engines are conventionally stored under ${AI_MODELS_PATH:-/export/ai_models}/model-zoo/yolo26/exports/. The gateway path is different: ai/triton/model_repository/yolo26 runs an FP32 ONNX model on the ONNX Runtime CUDA execution provider (NEM-5551 — INT8 ONNX was incompatible with the CUDA provider), so no TensorRT engine is involved in the compose stack.

TensorRT Version Compatibility¶

TensorRT engines are version-specific. The server automatically handles version mismatches:

When the TensorRT runtime version differs from the engine version:

1. The server detects the version mismatch error
2. If YOLO26_AUTO_REBUILD=true, it rebuilds the engine from the .pt model
3. If rebuild fails or is disabled, it falls back to PyTorch inference

torch.compile Optimization (PyTorch fallback only)¶

When using PyTorch models (not TensorRT engines), torch.compile provides 15-30% speedup:

Compilation modes:

• default - Balanced optimization
• reduce-overhead - Faster compilation, good speedup (recommended)
• max-autotune - Best performance, slower compilation

Note: torch.compile is automatically skipped for TensorRT engines since they are already graph-optimized.

Exporting TensorRT Engines¶

The standalone yolo26 GPU image was retired 2026-09-23; host-side engine building now runs through the prebuild script (or the gateway's Triton export pipeline ai/gateway/export/export_all.sh for the ONNX path Triton serves):

# Build the FP16 engine for the local GPU (ultralytics .pt -> TensorRT .engine)
./scripts/prebuild-tensorrt-engines.sh yolo26

INT8 calibration requirements (for the archived INT8 export CLI, kept at archive/ai-yolo26-image/export_tensorrt.py):

• 100-500 representative images from your deployment environment
• Cover various lighting conditions and camera angles
• Include all security-relevant object classes

For detailed (retired-era) export options, see archive/ai-yolo26-image/README.md.

NVIDIA Nemotron LLM Server¶

Host-run scripts: ai/start_llm.sh (dev, Mini 4B) and ai/start_nemotron.sh (Nano 30B, also called by the backend's ServiceHealthMonitor for auto-recovery). In compose the ai-llm container runs llama.cpp instead and reads LLM_MODEL_PATH, GPU_LAYERS, CTX_SIZE, PARALLEL (see below).

Compose ai-llm variables (from .env.example / compose defaults):

Model Zoo state machine showing model lifecycle transitions: loading, loaded, unloading, and error states.

Model Options:

For comprehensive NVIDIA Nemotron documentation, see /ai/nemotron/AGENTS.md.

Backend Configuration¶

These configure how the backend connects to AI services (backend/core/config.py):

Setting Up Environment Variables¶

The backend reads settings from the process environment first, then data/runtime.env, then the .env file at the project root (backend/core/config.py via pydantic-settings). Copy the template once and edit it:

cp .env.example .env
chmod 600 .env
# edit values with your editor

Host-run AI scripts (ai/start_*.sh) inherit variables from your shell, so export them there if you run the services outside compose.

Container Networking¶

When AI services run in containers, use appropriate host resolution:

For a decision table and copy/paste .env snippets for every deployment mode, use: Deployment Modes & AI Networking.

(The gateway host port is 127.0.0.1:${AI_GATEWAY_PORT:-8090} by default — expose it via a reverse proxy or SSH tunnel before pointing a remote backend at a host IP.)

Production compose DNS (recommended)¶

When running docker-compose.prod.yml, the backend reaches AI services by compose DNS — these are the values compose itself sets, no .env entries needed:

USE_AI_GATEWAY=true
AI_GATEWAY_URL=http://ai-gateway:8090
YOLO26_URL=http://ai-gateway:8090/yolo26
NEMOTRON_URL=http://ai-llm:8091
FLORENCE_URL=http://ai-gateway:8090/florence
CLIP_URL=http://ai-gateway:8090/clip
ENRICHMENT_URL=http://ai-gateway:8090/enrichment
ENRICHMENT_LIGHT_URL=http://ai-gateway:8090/enrich-lt

Example .env for macOS with Docker:

YOLO26_URL=http://host.docker.internal:8090/yolo26
NEMOTRON_URL=http://host.docker.internal:8091

Example .env for macOS with Podman:

export AI_HOST=host.containers.internal
YOLO26_URL=http://${AI_HOST}:8090/yolo26
NEMOTRON_URL=http://${AI_HOST}:8091

Example .env for Linux:

# Get your host IP
export AI_HOST=$(hostname -I | awk '{print $1}')
YOLO26_URL=http://${AI_HOST}:8090/yolo26
NEMOTRON_URL=http://${AI_HOST}:8091

Detection Settings¶

Fine-tune object detection behavior:

[!WARNING] > The fast path is disabled by design. config.py ships fast_path_confidence_threshold = 2.0 (an impossible value) and an empty FAST_PATH_OBJECT_TYPES, because the fast path bypasses enrichment and Nemotron then scores on partial data. Do not lower these unless enrichment is added to the fast path.

Confidence threshold trade-offs:

• Lower (0.3-0.5): More detections, more false positives
• Higher (0.6-0.8): Fewer detections, fewer false positives

Timeout Settings¶

Control connection and read timeouts:

[!NOTE] Older docs said YOLO26_READ_TIMEOUT defaults to 60.0; backend/core/config.py currently defaults to 30.0.

Systemd Services¶

There are no systemd units for the AI services in this repository — the supported production path is docker-compose.prod.yml (Podman). The only systemd units that ship are monitoring extras (monitoring/cadvisor/cadvisor.service, monitoring/dcgm/dcgm-exporter.service, both rootful).

Complete Example .env¶

# AI gateway (production compose sets these itself — needed only when the backend
# runs on the host against AI on the host)
USE_AI_GATEWAY=true
AI_GATEWAY_URL=http://localhost:8090
YOLO26_URL=http://localhost:8090/yolo26
NEMOTRON_URL=http://localhost:8091
FLORENCE_URL=http://localhost:8090/florence
CLIP_URL=http://localhost:8090/clip
ENRICHMENT_URL=http://localhost:8090/enrichment
ENRICHMENT_LIGHT_URL=http://localhost:8090/enrich-lt

# Enrichment tier routing (light = /enrich-lt, heavy = /enrichment)
ENRICHMENT_POSE_SERVICE=light
ENRICHMENT_THREAT_SERVICE=light
ENRICHMENT_REID_SERVICE=light
ENRICHMENT_PET_SERVICE=light
ENRICHMENT_DEPTH_SERVICE=light

# ai-llm container (llama.cpp) — CTX_SIZE is for the LLM container only; the
# Python backend's validator rejects values above 131072
LLM_MODEL_PATH=/models/Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf
GPU_LAYERS=auto
CTX_SIZE=262144
PARALLEL=8

# Standalone host-run YOLO26 server only (not used by the compose gateway)
# YOLO26_MODEL_PATH=${AI_MODELS_PATH:-/export/ai_models}/model-zoo/yolo26/exports/yolo26m_fp16.engine
# TORCH_COMPILE_ENABLED=true

# Detection tuning
DETECTION_CONFIDENCE_THRESHOLD=0.5

# Timeouts
AI_CONNECT_TIMEOUT=10.0
YOLO26_READ_TIMEOUT=30.0
NEMOTRON_READ_TIMEOUT=120.0

Next Steps¶

• AI Services - Start and verify services
• AI Troubleshooting - Common issues and solutions
• AI Performance - Performance tuning

See Also¶

• Environment Variable Reference - Complete configuration reference
• Risk Levels Reference - Severity threshold configuration
• AI TLS - Secure communications setup

Back to Operator Hub