Skip to content

Benchmark Documentation

Performance benchmarks and measurements for the Home Security Intelligence AI pipeline.

Status: historical. These are point-in-time results on an NVIDIA RTX A5500, last refreshed in early 2026. The model names in the tables reflect the model zoo of that date, not the live manifest — models.yml at the repo root is the current source of truth (e.g. CLIP-era entries were replaced by siglip2-base-patch16-224, osnet-ain-x1-0 supersedes the old osnet variant, depth-anything-v2-tiny replaced the small variant). Re-run the scripts below to regenerate results before citing any number here as current.

Current Benchmarks

Benchmark Description Last Updated
Model Zoo Benchmark Performance metrics for all Model Zoo AI models 2026-01-01
YOLO26 Benchmarks YOLO26 detection model performance benchmarks 2026-01-26
YOLO26 Export Formats YOLO26 model export format comparison 2026-01-26

Model Zoo Benchmark Summary

The Model Zoo benchmark tests AI models for:

  • Load time - Time to load model into GPU memory
  • VRAM usage - GPU memory consumed
  • Inference time - Time to process a single image
  • VRAM recovery - Whether memory is properly released after unload

Success Criteria

Criteria Target
Max VRAM per model <1500MB
Max load time <5s
VRAM recovered Yes

Last Recorded Results (2026-01-01 run)

Model Status VRAM Inference
fashion-clip ERROR N/A N/A
weather-classification OK 444MB 274ms
pet-classifier OK 2MB N/A

Running Benchmarks

Benchmarks are generated by Python scripts:

# Run Model Zoo benchmark (requires GPU)
uv run python scripts/benchmark_model_zoo.py

# Benchmark specific models
uv run python scripts/benchmark_model_zoo.py --models yolo11-license-plate,clip-vit-l

# Results are written to docs/benchmarks/model-zoo-benchmark.md

Refresh Schedule

Refresh benchmarks when:

  • New models added to the Model Zoo
  • Model updates change versions or configurations
  • Hardware changes when deploying to new GPU
  • Quarterly at minimum for ongoing validation

Refresh Procedure

# 1. Verify GPU access
nvidia-smi

# 2. Download any missing models
uv run python scripts/download-model-zoo.py

# 3. Run benchmark
uv run python scripts/benchmark_model_zoo.py

# 4. Review and commit
cat docs/benchmarks/model-zoo-benchmark.md
git add docs/benchmarks/model-zoo-benchmark.md
git commit -m "docs: refresh model zoo benchmark results"

Adding New Benchmarks

When creating new benchmark files:

  1. Use date-prefixed filenames if version-specific: YYYY-MM-DD-benchmark-name.md
  2. Include GPU/hardware information
  3. Document success criteria
  4. Include both summary and detailed results
  5. Update this README with the new entry

Back to Documentation Index