AI Models Reference¶

Canonical reference for all AI models used in the Home Security Intelligence pipeline. This document provides specifications, HuggingFace links, VRAM requirements, and configuration details for each model.

Target Audiences: Developers, Operators, ML Engineers

Deployment topology: In production (docker-compose.prod.yml) the models are served by two AI services: ai-gateway — Triton + FastAPI on port 8090 with routers /yolo26 /clip /florence /enrichment /enrich-lt (ai/gateway/main.py) — and ai-llm (llama.cpp) on port 8091. Heavy enrichment models are loaded in-process by the backend from models.yml via backend/services/model_zoo.py. The standalone containers (ai-yolo26 :8095, ai-florence :8092, ai-clip :8093, ai-enrichment :8094, ai-enrichment-light :8096) are built from ai/*/Dockerfile but are not in the production compose file; their ports are the local-development defaults in backend/core/config.py.

Quick Reference¶

Summary Table¶

Service column: ai-llm = llama.cpp container (:8091) · gateway = Triton via ai-gateway (:8090) · backend = loaded in-process by backend/services/model_zoo.py from models.yml.

Core Models¶

Nemotron-3-Nano-30B-A3B (Production LLM)¶

The production model for AI-driven risk reasoning and security analysis. Uses NVIDIA's state-of-the-art reasoning model with massive context capability.

Purpose in Pipeline:

• Analyzes batches of object detections from YOLO26
• Generates risk scores (0-100) and natural language summaries
• Considers zone analysis, baseline comparison, and cross-camera correlation
• Processes enrichment data (clothing, vehicles, behavior, scene descriptions)

Why a large context matters:

• Analyze all detections across extended time windows (hours of activity)
• Include rich historical baselines ("Is this normal for 3am on Tuesday?")
• Correlate activity across multiple cameras in a single prompt
• Process detailed enrichment data from the model zoo

Environment Variables:

Nemotron Mini 4B Instruct (Development LLM)¶

A smaller, faster model for development and resource-constrained environments.

Status: Not part of the managed download set — models.yml has no entry for it. It is referenced as an optional vLLM test model (docker-compose.prod.yml ai-llm-vllm comments: nvidia/Nemotron-Mini-4B-Instruct) and survives in code as a metrics label default (backend/core/otel_metrics.py: model_version="nemotron-mini-4b-instruct"). To use it, download the GGUF manually and point LLM_MODEL_PATH at it.

Use Cases:

• Local development without high-end GPU
• Testing prompt templates and integration flows
• CI/CD pipeline testing (faster iteration)

Qwen3-VL-8B-Instruct (serving VLM, vlm mode)¶

The vision-language model that grades events in the shipped vlm pipeline mode. llama.cpp serves it with an mmproj projector file — without the projector the serve is text-only and every vlm_assess degrades silently, so the two files are one identity and are configured as a pair.

¹ The ~11 GB figure is the bake-off's broker-actual reading on a GB300 (vram_actual_mib 11216, 9376 projected), indicative only — S1's bar is measured on 24 GB-class hardware with the whole stack resident and has not been taken for this model. Weights alone are 5.03 GB + 0.75 GB. The KV pool is a function of architecture and ctx/slot settings, not of weight size: 4B and 8B share block_count 36 / head_count 32, so the 8B buys no KV relief.

The ai-llm rows above (Nemotron-3-Nano-30B-A3B, Nemotron Mini 4B) belong to the legacy serving path, which is unsupported (spec rev 5, ledger F10) — in vlm mode ai-llm is not running at all. Their figures are kept for the code that still exists, not as a current deployment target.

YOLO26 (Object Detection)¶

Real-time object detection using CNN architecture optimized for speed with TensorRT FP16 inference.

Model Variants:

FPS provenance: the only A5500 TensorRT FP16 run recorded in this repo is a single 5.76 ms / 174 FPS standalone benchmark (2026-01-26, commit a5e47aa2, variant not recorded -- see the decision timeline in NVIDIA Technology Inventory). The 223/206/174 row values are vendor-published-style figures that cannot be re-derived from the current .pt-based benchmark script; current measured numbers for the ONNX path live in docs/benchmarks/yolo26-benchmarks.md and are roughly an order of magnitude lower.

Security-Relevant Classes (9 total):

SECURITY_CLASSES = {
    "person", "car", "truck", "dog", "cat",
    "bird", "bicycle", "motorcycle", "bus"
}

Purpose in Pipeline:

• First stage of the AI pipeline
• Processes incoming camera images
• Outputs bounding boxes, class labels, and confidence scores
• Filters detections to security-relevant classes only

Environment Variables:

Enrichment Models¶

These models provide additional context for detected objects. In production they are served through ai-gateway (Triton models, reached via :8090/enrichment and :8090/enrich-lt) and/or loaded in-process by the backend from models.yml via backend/services/model_zoo.py (ai/enrichment standalone on :8094 and ai-enrichment-light :8096 exist as buildable dev services but are not in the production compose).

Florence-2 (Dense Captioning)¶

Vision-language model for extracting detailed visual attributes from security camera images. Production runs Florence-2-Base as a Triton Python-backend model in ai-gateway (models.yml: florence-2-base, required). Florence-2-Large remains a model-zoo entry (florence-2-large) that is enabled: false in models.yml; the standalone ai/florence server (dev, port 8092) also defaults to Base (FLORENCE_MODEL_PATH=/models/florence-2-base).

Supported Prompts:

Purpose in Pipeline:

• Scene understanding and captioning
• License plate and sign text extraction (OCR)
• Detailed attribute extraction for Nemotron risk analysis

Environment Variables:

CLIP ViT-L (Vision-Language)¶

Status: Superseded in production. The Triton gateway clip/clip_text models and the backend embedding service now use SigLIP 2 Base (models.yml: siglip2-base-patch16-224, ONNX from onnx-community/siglip2-base-patch16-224-ONNX, same 768-dim embedding output — backend/services/clip_loader.py). This ViT-L entry documents the standalone ai/clip server, which still loads it when CLIP_MODEL_PATH points at a CLIP checkpoint.

Generates 768-dimensional embeddings for scene anomaly detection and zero-shot classification.

Use Cases:

1. Scene Anomaly Detection: Compare current frame embedding against baseline to detect unusual changes
2. Zero-shot Classification: Classify images against text labels without retraining

Person re-identification no longer rides here: it moved to OSNet-AIN x1.0 (below) with the full swap (ledger item 20), and the /clip router is left to the scene-baseline and fashion consumers that still use it.

Environment Variables:

SigLIP 2 Base (Production Embeddings)¶

The embedding model actually deployed in production (Triton clip + clip_text models, and the backend siglip2-base-patch16-224 model-zoo entry).

The text tower runs on CPU because its INT8 quantization uses MatMulInteger ops that the ONNX Runtime CUDA execution provider does not support (models.yml triton_models comment).

FashionSigLIP (Clothing Classification)¶

Zero-shot clothing classifier for identifying suspicious clothing patterns (hoodies, face coverings) and service uniforms. Uses Marqo FashionSigLIP for 57% improved accuracy over FashionCLIP.

Performance Improvement over FashionCLIP:

• Text-to-Image MRR: 0.239 vs 0.165 (45% improvement)
• Text-to-Image Recall@1: 0.121 vs 0.077 (57% improvement)
• Text-to-Image Recall@10: 0.340 vs 0.249 (37% improvement)

Security-Focused Clothing Prompts:

SECURITY_CLOTHING_PROMPTS = [
    "person wearing dark hoodie",
    "person wearing face mask",
    "person wearing ski mask or balaclava",
    "delivery uniform", "Amazon delivery vest",
    "FedEx uniform", "UPS uniform", "USPS postal worker uniform",
    "casual clothing", "business attire or suit", ...
]

Purpose in Pipeline:

• Identify suspicious clothing (dark hoodies, face coverings)
• Detect service workers (delivery uniforms) for lower risk scoring
• Provide clothing attributes to Nemotron for context-aware analysis

Vehicle Classifier¶

Classifies vehicle types from cropped detection images.

models.yml (single source of truth) pins AventIQ-AI/ResNet-50-Vehicle-Segment-classification; the legacy ai/download_models.sh script still references the older lxyuan/vit-base-patch16-224-vehicle-segment-classification ViT checkpoint for its manual download path. The Triton vehicle model is the exported ONNX ResNet-50.

Vehicle Classes (11 total):

VEHICLE_SEGMENT_CLASSES = [
    "articulated_truck", "background", "bicycle", "bus", "car",
    "motorcycle", "non_motorized_vehicle", "pedestrian",
    "pickup_truck", "single_unit_truck", "work_van"
]

Purpose in Pipeline:

• Distinguish between personal vehicles and commercial vehicles
• Identify delivery vehicles (work vans, trucks) for context
• Provide vehicle type to Nemotron for risk assessment

Pet Classifier¶

Classifies detected animals as cats or dogs (household pets).

Purpose in Pipeline:

• Identify household pets to reduce false positives
• Distinguish resident pets from wildlife
• Filter pet detections from security alerts

Depth Anything V2 Tiny¶

Monocular depth estimation for understanding spatial relationships in camera images.

Production uses the Tiny variant (5.8 M params, 518×518 input; models.yml: depth-anything-v2-tiny, downloaded from depth-anything/Depth-Anything-V2-Small-hf per that entry's hf_repo — ai/download_models.sh clones the Tiny repo into model-zoo/depth-anything-v2-tiny). The standalone enrichment service reads DEPTH_MODEL_PATH=/models/depth-anything-v2-tiny.

Purpose in Pipeline:

• Estimate distance to detected objects
• Understand spatial relationships (near entry point, far from camera)
• Provide proximity context to Nemotron ("person approaching front door")

Environment Variables:

ViTPose+ Small (Pose Estimation)¶

Human pose estimation for posture analysis and security-relevant behavior detection.

COCO Keypoints (17):

COCO_KEYPOINT_NAMES = [
    "nose", "left_eye", "right_eye", "left_ear", "right_ear",
    "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
    "left_wrist", "right_wrist", "left_hip", "right_hip",
    "left_knee", "right_knee", "left_ankle", "right_ankle"
]

Posture Classifications:

Security Alerts:

YOLO11 License Plate Detection¶

Detects license plates on vehicles for OCR text extraction.

Purpose in Pipeline:

• Detect license plate regions on vehicle detections
• Extract plate crops for OCR processing
• Provide plate locations for downstream text extraction

YOLO11 Face Detection¶

Detects faces on person detections for demographic analysis and re-identification.

Purpose in Pipeline:

• Detect face regions on person detections
• Extract face crops for age/gender classification
• Enable face-based re-identification across cameras

PaddleOCR¶

Optical Character Recognition for extracting text from license plates and signs.

Purpose in Pipeline:

• Extract text from detected license plates
• Read text on signs and packages (delivery identification)
• Provide textual context for Nemotron analysis

Note: Optional dependency. OCR features disabled if PaddlePaddle not installed.

YOLO-World-S (Open-Vocabulary Detection)¶

Zero-shot object detection using text prompts for security-relevant objects not in COCO.

Purpose in Pipeline:

• Detect objects not in COCO dataset (knives, packages, tools)
• Enable text-prompt-based detection for custom security scenarios
• Zero-shot detection without model retraining

Security Prompts:

SECURITY_PROMPTS = [
    "knife", "gun", "weapon", "package", "box",
    "backpack", "suitcase", "crowbar", "flashlight"
]

Violence Detection¶

Binary classification for detecting violent content in video frames.

Purpose in Pipeline:

• Detect violent activity when 2+ persons detected
• Trigger high-priority alerts for physical altercations
• Provide violence context for Nemotron risk analysis

Output:

{
  "is_violent": true,
  "confidence": 0.94,
  "label": "violence"
}

Weather Classification¶

Classifies weather conditions for environmental context in risk assessment.

Weather Classes (5):

WEATHER_CLASSES = [
    "cloudy/overcast",
    "foggy/hazy",
    "rain/storm",
    "snow/frosty",
    "sun/clear"
]

Purpose in Pipeline:

• Provide environmental context for risk calibration
• Adjust visibility expectations (foggy = reduced detection confidence)
• Runs once per batch on full frame (not per detection)

SegFormer B2 Clothes¶

Semantic segmentation of clothing and body parts for person description and re-identification.

Clothing Categories (18):

CLOTHING_CATEGORIES = [
    "Background", "Hat", "Hair", "Sunglasses", "Upper-clothes",
    "Skirt", "Pants", "Dress", "Belt", "Left-shoe", "Right-shoe",
    "Face", "Left-leg", "Right-leg", "Left-arm", "Right-arm",
    "Bag", "Scarf"
]

Purpose in Pipeline:

• Enable clothing-based person matching across cameras
• Detect suspicious attire (masks, gloves, all-black)
• Provide detailed person descriptions for Nemotron

X-CLIP Base (Temporal Action Recognition — Deprecated)¶

Status: deprecated / disabled. models.yml marks xclip-base as "DEPRECATED: replaced by stgcn-plus-plus" with enabled: false and vram_mb: 0. Skeleton-based ST-GCN++ (below) now handles action recognition. The Triton xclip_action model (Python backend, CPU) has been retired from ai/triton/model_repository/ — its config and model.py now live at archive/triton-model-repository/xclip_action/ — and is absent from the gateway's ALL_MODELS and adapters.

Video-based action classification using multiple frames for temporal understanding.

Purpose in Pipeline:

• Classify security-relevant actions from video sequences
• Detect loitering, approaching door, running away
• Analyze behavior patterns over multiple frames

Security Actions:

SECURITY_ACTIONS = [
    "loitering", "approaching door", "running away",
    "suspicious behavior", "fighting", "falling",
    "walking normally", "standing still"
]

ST-GCN++ (Skeleton Action Recognition)¶

Replaces X-CLIP as the production action-recognition model: ~14 MB ONNX export of pyskl ST-GCN++ trained on NTU RGB+D 60 (60 action classes), consuming COCO 17-keypoint tracks from the pose models instead of raw video frames.

Purpose in Pipeline:

• Classify security-relevant actions from skeleton sequences (loitering, falling, fighting)
• Far cheaper than X-CLIP: no frame buffering, CPU-only inference

Smoke/Fire Detection (YOLOv8n)¶

Critical-priority model: models.yml sets priority: critical, preload: true, never_evict: true — it is the only model the VRAM evictor will not unload.

FastALPR (End-to-End License Plate OCR)¶

models.yml describes PaddleOCR as "superseded by fast-alpr for license plates"; PaddleOCR remains enabled for general sign/package text.

BRISQUE Image Quality Assessment¶

No-reference image quality metric for detecting camera tampering or motion blur.

Purpose in Pipeline:

• Detect camera obstruction or tampering (sudden quality drop)
• Identify motion blur (fast movement detection)
• Monitor general quality degradation (noise, artifacts)

Output:

{
  "brisque_score": 23.5,
  "quality_label": "good",
  "is_degraded": false
}

Score Interpretation:

• 0-20: Excellent quality
• 20-40: Good quality
• 40-60: Fair quality
• 60+: Poor quality (potential tampering)

Vehicle Damage Detection¶

Segmentation model for detecting various types of vehicle damage.

Damage Classes (6):

DAMAGE_CLASSES = [
    "crack",
    "dent",
    "glass_shatter",
    "lamp_broken",
    "scratch",
    "tire_flat",
]

Purpose in Pipeline:

• Detect suspicious vehicle damage (glass_shatter + lamp_broken at night = break-in)
• Monitor for vandalism or accidents in parking areas
• Provide damage context for security incidents

OSNet-AIN x1.0 (Person Re-identification)¶

Lightweight model for generating person embeddings for cross-camera tracking.

Purpose in Pipeline:

• Generate 512-dimensional embeddings for person tracking — the one person re-ID vector space. CLIP/SigLIP no longer produces person vectors (full swap, ledger item 20: "ignore previous architecture. we do not have to support backwards compatability.")
• Match individuals across multiple cameras
• Enable temporal tracking of persons throughout property

backend/services/osnet_loader.py loads the osnet-ain-x1-0 row of models.yml, verifies the weights sha256 before loading, and stamps every vector with the model_id above. A stored row that predates provenance decodes to the sentinel legacy-unknown-provenance and is never scored — it reads as "unavailable (re-enroll)", so re-enroll rather than backfill. When the weights are not resident, reid_service.generate_embedding() raises ReIDUnavailableError (the enrollment route answers 503 naming the cause) — never a zero-vector stub. Vehicle identity is separate: no vehicle embedding producer ships in resident mode, so it rides license-plate match; vehicle-specific re-ID is a named follow-up.

Output:

{
  "embedding": [0.123, -0.456, "..."],
  "embedding_dimension": 512,
  "model_id": "osnet-ain-x1-0@osnet_ain_x1_0_msmt17@8a07e8da3894"
}

Stored under Redis key entity_embeddings:{model_id}:{date} — the model_id partition is the guard that keeps two spaces from ever being compared. reid_similarity_threshold now defaults to 0.7 (OSNet-space value; the CLIP-tuned 0.85 would drop every legitimate match), still provisional pending calibration against real household galleries.

Threat Detection YOLOv8n¶

Weapon and threat object detection for high-priority security alerts.

Threat Classes:

THREAT_CLASSES = [
    "knife", "gun", "rifle", "pistol", "bat", "crowbar"
]

Purpose in Pipeline:

• Detect weapons on full frame when suspicious activity detected
• Trigger immediate critical-priority alerts
• Triton threat model runs at instance priority 1 (high) — the never-evicted model is Smoke/Fire YOLOv8n (models.yml: never_evict: true), not this one

ViT Age Classifier¶

Age range estimation from face or person crops.

Age Groups (6, per backend/services/age_classifier_loader.py):

AGE_GROUPS = [
    "child",        # 0-12
    "teenager",     # 13-19
    "young_adult",  # 20-35
    "adult",        # 36-50
    "middle_aged",  # 51-65
    "senior",       # 65+
]

The loader also maps a second, finer label scheme some checkpoints emit ("0-2": infant, "3-9": child, "10-19": teenager, "20-29": young_adult, "30-39"/"40-49": adult, "50-59": middle_aged, "60-69"/"70+": senior) onto these display groups.

Purpose in Pipeline:

• Provide demographic context for person descriptions
• Combined with gender for comprehensive person profiles
• Support security analysis (child alone, unusual age for time)

ViT Gender Classifier¶

Gender classification from face or person crops.

Output:

{
  "gender": "male",
  "confidence": 0.94
}

Purpose in Pipeline:

• Complete demographic profile for person descriptions
• Support cross-camera person matching
• Provide gender context for security reports

YOLOv8n Pose (Pose Estimation)¶

Pose estimation model serving the ai-gateway Triton pose model; feeds the 17-keypoint skeletons that ST-GCN++ consumes.

Purpose in Pipeline:

• Fast pose detection on the gateway GPU (the default Triton pose path)
• Same 17 COCO keypoint output format as ViTPose+ (the higher-accuracy, backend-side alternative)
• Provides the skeleton sequences for ST-GCN++ action recognition

VRAM Requirements Summary¶

Production Configuration (Reference Hardware: RTX A5500 24 GB + RTX A400 4 GB)¶

The gateway VRAM split is documented in NVIDIA Technology Inventory (VRAM Budget Summary).

Backend Model-Zoo VRAM (On-Demand, from models.yml)¶

backend/services/model_zoo.py loads these lazily and unloads them right after each use (async context manager with reference counting, CUDA cache cleared on unload), so typically only one or a few models are resident at a time rather than a budget-capped working set (23 enabled entries, ~11.7 GB total vram_mb if all were somehow resident). smoke-fire-yolov8n is the only never_evict model; the vram_mb/priority fields drive reporting and preload decisions. (Priority-ordered LRU eviction under a VRAM budget is the sibling manager in ai/enrichment/model_manager.py, which belongs to the undeployed standalone container.)

Development Configuration (Minimal)¶

Hardware Recommendations¶

Model Download¶

The primary path is setup.py deploy → setup_lib/model_downloader.py, which downloads everything declared in models.yml (repo-root file). The standalone script covers the same first-generation set manually:

# Download all models to default path (/export/ai_models)
./ai/download_models.sh

# Download to custom path
AI_MODELS_PATH=./models ./ai/download_models.sh

Download Directory Structure¶

models.yml is the single source of truth for what setup.py deploy (setup_lib/model_downloader.py) fetches and where. ai/download_models.sh is the legacy manual path (it still clones some first-generation artifacts such as model-zoo/florence-2-large, model-zoo/clip-vit-l and model-zoo/fashion-clip that the current production set replaced with florence-2-base, the SigLIP 2 ONNX export and the open_clip hf-hub FashionSigLIP cache).

${AI_MODELS_PATH}/
├── nemotron/
│   └── nemotron-3-nano-30b-a3b-q4km/
│       └── Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf
└── model-zoo/
    ├── yolo26/                       # yolo26n/s/m .pt (Ultralytics releases)
    ├── florence-2-base/              # production Triton model
    ├── siglip2-base-patch16-224/     # ONNX embeddings (clip / clip_text)
    ├── vehicle-segment-classification/
    ├── pet-classifier/
    ├── depth-anything-v2-tiny/
    ├── osnet-ain-x1-0/
    ├── yolov8n-pose/
    ├── threat-detection-yolov8n/
    ├── vit-age-classifier/           # + vit-gender-classifier, stgcn-plus-plus,
    │                                 #   yolo11-face-detection, yolo11-license-plate,
    │                                 #   smoke-fire-yolov8n, yolo-world-s,
    │                                 #   vitpose-small, segformer-b2-clothes,
    │                                 #   vehicle-damage-detection, weather-classification,
    │                                 #   violence-detection, xclip-base (disabled) ...
    └── (fashion-clip downloads into ~/.cache/huggingface/hub via download_method: hf_cache)

Manual Downloads¶

For manual downloads or air-gapped environments (rows marked script are what ai/download_models.sh clones; production model sources are the hf_repo values in models.yml):

Environment Variables Reference¶

Model Paths¶

Backend model-zoo paths resolve as MODEL_ZOO_PATH (default /models/model-zoo, the container mount of ${AI_MODELS_PATH}/model-zoo) + the local_path from models.yml.

Service Configuration¶

Architecture Overview¶

flowchart TD
    CAM[Camera Images] --> BE[backend]

    subgraph Gateway["ai-gateway :8090 (Triton + FastAPI)"]
        YOLO["/yolo26<br/>YOLO26m"]
        FLO["/florence<br/>Florence-2-Base"]
        CLIP["/clip<br/>SigLIP 2"]
        ENR["/enrichment + /enrich-lt<br/>vehicle, pet, pose, threat, reid, depth, ..."]
    end

    subgraph Zoo["backend model_zoo (in-process, models.yml)"]
        HEAVY["smoke/fire, vitpose, segformer,<br/>violence, weather, damage, OCR ...<br/>(load on demand,<br/>unload after use)"]
    end

    subgraph Analysis["Analysis Layer"]
        NEM["ai-llm :8091<br/>Nemotron LLM<br/>Risk Analysis & Scoring"]
    end

    BE -->|images| YOLO
    BE -->|detections| ENR
    BE -->|frames| FLO
    BE -->|crops| CLIP
    BE --> HEAVY
    BE -->|enriched batch| NEM
    NEM --> OUT[Risk Events]

    style YOLO fill:#22C55E,color:#fff
    style ENR fill:#3B82F6,color:#fff
    style FLO fill:#3B82F6,color:#fff
    style CLIP fill:#3B82F6,color:#fff
    style NEM fill:#A855F7,color:#fff

Pipeline Flow¶

1. YOLO26 (gateway /yolo26): Detects objects in camera images (30-50ms)
2. Enrichment (gateway /enrichment, /enrich-lt + backend model_zoo): Classifies detections (vehicle type, clothing, pet, depth, pose, threats)
3. Florence-2 (gateway /florence): Generates scene captions and OCR text (optional)
4. SigLIP 2 (gateway /clip): Scene-baseline and fashion-similarity embeddings (optional) — person re-ID vectors come from OSNet-AIN x1.0 (osnet-ain-x1-0, 512-dim), not from /clip
5. Nemotron (ai-llm :8091): Analyzes enriched detections and generates risk scores (2-5s)

Related Documentation¶

• AI Pipeline Architecture
• Enrichment Service Documentation
• Nemotron LLM Configuration
• YOLO26 Detection Server
• Risk Levels Configuration
• GPU Troubleshooting

Back to Reference Hub | AI Troubleshooting | Environment Variables