AI Models Reference¶

Canonical reference for all AI models used in the Home Security Intelligence pipeline. This document provides specifications, HuggingFace links, VRAM requirements, and configuration details for each model.

Target Audiences: Developers, Operators, ML Engineers

Quick Reference¶

Summary Table¶

Core Models¶

Nemotron-3-Nano-30B-A3B (Production LLM)¶

The production model for AI-driven risk reasoning and security analysis. Uses NVIDIA's state-of-the-art reasoning model with massive context capability.

Purpose in Pipeline:

• Analyzes batches of object detections from YOLO26
• Generates risk scores (0-100) and natural language summaries
• Considers zone analysis, baseline comparison, and cross-camera correlation
• Processes enrichment data (clothing, vehicles, behavior, scene descriptions)

Why 128K Context Matters:

• Analyze all detections across extended time windows (hours of activity)
• Include rich historical baselines ("Is this normal for 3am on Tuesday?")
• Correlate activity across multiple cameras in a single prompt
• Process detailed enrichment data from the model zoo

Environment Variables:

Nemotron Mini 4B Instruct (Development LLM)¶

A smaller, faster model for development and resource-constrained environments.

Use Cases:

• Local development without high-end GPU
• Testing prompt templates and integration flows
• CI/CD pipeline testing (faster iteration)

YOLO26 (Object Detection)¶

Real-time object detection using CNN architecture optimized for speed with TensorRT FP16 inference.

Model Variants:

Security-Relevant Classes (9 total):

SECURITY_CLASSES = {
    "person", "car", "truck", "dog", "cat",
    "bird", "bicycle", "motorcycle", "bus"
}

Purpose in Pipeline:

• First stage of the AI pipeline
• Processes incoming camera images
• Outputs bounding boxes, class labels, and confidence scores
• Filters detections to security-relevant classes only

Environment Variables:

Enrichment Models¶

These models run in the Enrichment service (port 8094) and provide additional context for detected objects.

Florence-2-Large (Dense Captioning)¶

Vision-language model for extracting detailed visual attributes from security camera images.

Supported Prompts:

Purpose in Pipeline:

• Scene understanding and captioning
• License plate and sign text extraction (OCR)
• Detailed attribute extraction for Nemotron risk analysis

Environment Variables:

CLIP ViT-L (Vision-Language)¶

Generates 768-dimensional embeddings for entity re-identification and scene anomaly detection.

Use Cases:

1. Entity Re-identification: Track the same person or vehicle across multiple cameras using embedding similarity
2. Scene Anomaly Detection: Compare current frame embedding against baseline to detect unusual changes
3. Zero-shot Classification: Classify images against text labels without retraining

Environment Variables:

FashionSigLIP (Clothing Classification)¶

Zero-shot clothing classifier for identifying suspicious clothing patterns (hoodies, face coverings) and service uniforms. Uses Marqo FashionSigLIP for 57% improved accuracy over FashionCLIP.

Performance Improvement over FashionCLIP:

• Text-to-Image MRR: 0.239 vs 0.165 (45% improvement)
• Text-to-Image Recall@1: 0.121 vs 0.077 (57% improvement)
• Text-to-Image Recall@10: 0.340 vs 0.249 (37% improvement)

Security-Focused Clothing Prompts:

SECURITY_CLOTHING_PROMPTS = [
    "person wearing dark hoodie",
    "person wearing face mask",
    "person wearing ski mask or balaclava",
    "delivery uniform", "Amazon delivery vest",
    "FedEx uniform", "UPS uniform", "USPS postal worker uniform",
    "casual clothing", "business attire or suit", ...
]

Purpose in Pipeline:

• Identify suspicious clothing (dark hoodies, face coverings)
• Detect service workers (delivery uniforms) for lower risk scoring
• Provide clothing attributes to Nemotron for context-aware analysis

Vehicle Classifier¶

Classifies vehicle types from cropped detection images.

Vehicle Classes (11 total):

VEHICLE_SEGMENT_CLASSES = [
    "articulated_truck", "background", "bicycle", "bus", "car",
    "motorcycle", "non_motorized_vehicle", "pedestrian",
    "pickup_truck", "single_unit_truck", "work_van"
]

Purpose in Pipeline:

• Distinguish between personal vehicles and commercial vehicles
• Identify delivery vehicles (work vans, trucks) for context
• Provide vehicle type to Nemotron for risk assessment

Pet Classifier¶

Classifies detected animals as cats or dogs (household pets).

Purpose in Pipeline:

• Identify household pets to reduce false positives
• Distinguish resident pets from wildlife
• Filter pet detections from security alerts

Depth Anything V2¶

Monocular depth estimation for understanding spatial relationships in camera images.

Purpose in Pipeline:

• Estimate distance to detected objects
• Understand spatial relationships (near entry point, far from camera)
• Provide proximity context to Nemotron ("person approaching front door")

Environment Variables:

ViTPose+ Small (Pose Estimation)¶

Human pose estimation for posture analysis and security-relevant behavior detection.

COCO Keypoints (17):

COCO_KEYPOINT_NAMES = [
    "nose", "left_eye", "right_eye", "left_ear", "right_ear",
    "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
    "left_wrist", "right_wrist", "left_hip", "right_hip",
    "left_knee", "right_knee", "left_ankle", "right_ankle"
]

Posture Classifications:

Security Alerts:

YOLO11 License Plate Detection¶

Detects license plates on vehicles for OCR text extraction.

Purpose in Pipeline:

• Detect license plate regions on vehicle detections
• Extract plate crops for OCR processing
• Provide plate locations for downstream text extraction

YOLO11 Face Detection¶

Detects faces on person detections for demographic analysis and re-identification.

Purpose in Pipeline:

• Detect face regions on person detections
• Extract face crops for age/gender classification
• Enable face-based re-identification across cameras

PaddleOCR¶

Optical Character Recognition for extracting text from license plates and signs.

Purpose in Pipeline:

• Extract text from detected license plates
• Read text on signs and packages (delivery identification)
• Provide textual context for Nemotron analysis

Note: Optional dependency. OCR features disabled if PaddlePaddle not installed.

YOLO-World-S (Open-Vocabulary Detection)¶

Zero-shot object detection using text prompts for security-relevant objects not in COCO.

Purpose in Pipeline:

• Detect objects not in COCO dataset (knives, packages, tools)
• Enable text-prompt-based detection for custom security scenarios
• Zero-shot detection without model retraining

Security Prompts:

SECURITY_PROMPTS = [
    "knife", "gun", "weapon", "package", "box",
    "backpack", "suitcase", "crowbar", "flashlight"
]

Violence Detection¶

Binary classification for detecting violent content in video frames.

Purpose in Pipeline:

• Detect violent activity when 2+ persons detected
• Trigger high-priority alerts for physical altercations
• Provide violence context for Nemotron risk analysis

Output:

{
  "is_violent": true,
  "confidence": 0.94,
  "label": "violence"
}

Weather Classification¶

Classifies weather conditions for environmental context in risk assessment.

Weather Classes (5):

WEATHER_CLASSES = [
    "cloudy/overcast",
    "foggy/hazy",
    "rain/storm",
    "snow/frosty",
    "sun/clear"
]

Purpose in Pipeline:

• Provide environmental context for risk calibration
• Adjust visibility expectations (foggy = reduced detection confidence)
• Runs once per batch on full frame (not per detection)

SegFormer B2 Clothes¶

Semantic segmentation of clothing and body parts for person description and re-identification.

Clothing Categories (18):

CLOTHING_CATEGORIES = [
    "Background", "Hat", "Hair", "Sunglasses", "Upper-clothes",
    "Skirt", "Pants", "Dress", "Belt", "Left-shoe", "Right-shoe",
    "Face", "Left-leg", "Right-leg", "Left-arm", "Right-arm",
    "Bag", "Scarf"
]

Purpose in Pipeline:

• Enable clothing-based person matching across cameras
• Detect suspicious attire (masks, gloves, all-black)
• Provide detailed person descriptions for Nemotron

X-CLIP Base (Temporal Action Recognition)¶

Video-based action classification using multiple frames for temporal understanding.

Purpose in Pipeline:

• Classify security-relevant actions from video sequences
• Detect loitering, approaching door, running away
• Analyze behavior patterns over multiple frames

Security Actions:

SECURITY_ACTIONS = [
    "loitering", "approaching door", "running away",
    "suspicious behavior", "fighting", "falling",
    "walking normally", "standing still"
]

BRISQUE Image Quality Assessment¶

No-reference image quality metric for detecting camera tampering or motion blur.

Purpose in Pipeline:

• Detect camera obstruction or tampering (sudden quality drop)
• Identify motion blur (fast movement detection)
• Monitor general quality degradation (noise, artifacts)

Output:

{
  "brisque_score": 23.5,
  "quality_label": "good",
  "is_degraded": false
}

Score Interpretation:

• 0-20: Excellent quality
• 20-40: Good quality
• 40-60: Fair quality
• 60+: Poor quality (potential tampering)

Vehicle Damage Detection¶

Segmentation model for detecting various types of vehicle damage.

Damage Classes (6):

DAMAGE_CLASSES = [
    "cracks",
    "dents",
    "glass_shatter",
    "lamp_broken",
    "scratches",
    "tire_flat"
]

Purpose in Pipeline:

• Detect suspicious vehicle damage (glass_shatter + lamp_broken at night = break-in)
• Monitor for vandalism or accidents in parking areas
• Provide damage context for security incidents

OSNet-x0-25 (Person Re-identification)¶

Lightweight model for generating person embeddings for cross-camera tracking.

Purpose in Pipeline:

• Generate 512-dimensional embeddings for person tracking
• Match individuals across multiple cameras
• Enable temporal tracking of persons throughout property

Output:

{
  "embedding": [0.123, -0.456, ...],
  "embedding_dimension": 512,
  "model": "osnet_x0_25"
}

Threat Detection YOLOv8n¶

Weapon and threat object detection for high-priority security alerts.

Threat Classes:

THREAT_CLASSES = [
    "knife", "gun", "pistol", "rifle",
    "bat", "hammer", "crowbar"
]

Purpose in Pipeline:

• Detect weapons on full frame when suspicious activity detected
• Trigger immediate critical-priority alerts
• Highest priority model (never evicted from memory)

ViT Age Classifier¶

Age range estimation from face or person crops.

Age Range Brackets (7):

AGE_RANGES = [
    "0-12",   # child
    "13-17",  # teenager
    "18-24",  # young adult
    "25-35",  # adult
    "36-50",  # middle-aged
    "51-65",  # mature adult
    "65+"     # senior
]

Purpose in Pipeline:

• Provide demographic context for person descriptions
• Combined with gender for comprehensive person profiles
• Support security analysis (child alone, unusual age for time)

ViT Gender Classifier¶

Gender classification from face or person crops.

Output:

{
  "gender": "male",
  "confidence": 0.94
}

Purpose in Pipeline:

• Complete demographic profile for person descriptions
• Support cross-camera person matching
• Provide gender context for security reports

YOLOv8n Pose (Alternative Pose Estimation)¶

Alternative pose estimation model for backup or faster inference scenarios.

Purpose in Pipeline:

• Backup to ViTPose+ for pose detection
• Faster inference when ViTPose unavailable
• Same 17 COCO keypoint output format

VRAM Requirements Summary¶

Production Configuration (All Services)¶

Enrichment Service Model VRAM (On-Demand)¶

Models are loaded on-demand with LRU eviction. Maximum concurrent VRAM: ~6.8 GB.

Development Configuration (Minimal)¶

Hardware Recommendations¶

Model Download¶

All models are downloaded via the ai/download_models.sh script:

# Download all models to default path (/export/ai_models)
./ai/download_models.sh

# Download to custom path
AI_MODELS_PATH=./models ./ai/download_models.sh

Download Directory Structure¶

${AI_MODELS_PATH}/
├── nemotron/
│   └── nemotron-3-nano-30b-a3b-q4km/
│       └── Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf
└── model-zoo/
    ├── florence-2-large/
    ├── clip-vit-l/
    ├── fashion-clip/
    ├── vehicle-segment-classification/
    ├── pet-classifier/
    └── depth-anything-v2-small/

Manual Downloads¶

For manual downloads or air-gapped environments:

Environment Variables Reference¶

Model Paths¶

Service Configuration¶

Architecture Overview¶

flowchart TD
    CAM[Camera Images] --> YOLO

    subgraph Detection["Detection Layer"]
        YOLO["YOLO26<br/>:8095"]
    end

    subgraph Enrichment["Enrichment Layer"]
        ENR["Enrichment<br/>:8094"]
        FLO["Florence-2<br/>:8092"]
    end

    YOLO -->|Detections| ENR
    ENR -->|Classifications| FLO

    subgraph Analysis["Analysis Layer"]
        NEM["Nemotron LLM<br/>:8091<br/>Risk Analysis & Scoring"]
    end

    YOLO -->|Detections| NEM
    ENR -->|Classifications| NEM
    FLO -->|Captions| NEM

    NEM --> OUT[Risk Events]

    style YOLO fill:#22C55E,color:#fff
    style ENR fill:#3B82F6,color:#fff
    style FLO fill:#3B82F6,color:#fff
    style NEM fill:#A855F7,color:#fff

Pipeline Flow¶

1. YOLO26: Detects objects in camera images (30-50ms)
2. Enrichment: Classifies detections (vehicle type, clothing, pet, depth, pose)
3. Florence-2: Generates scene captions and OCR text (optional)
4. CLIP: Entity re-identification across cameras (optional)
5. Nemotron: Analyzes enriched detections and generates risk scores (2-5s)

Related Documentation¶

• AI Pipeline Architecture
• Enrichment Service Documentation
• Nemotron LLM Configuration
• YOLO26 Detection Server
• Risk Levels Configuration
• GPU Troubleshooting

Back to Reference Hub | AI Troubleshooting | Environment Variables