Decision: Python 3.14 Adoption¶

Date: 2026-01-21 Status: Implemented

Implementation Status (Updated 2026-01-28): Python 3.14 is now the primary runtime. The pyproject.toml specifies requires-python = ">=3.14" and all tooling (ruff, mypy) is configured for Python 3.14. Phase 1 is complete.

Context¶

The Home Security Intelligence project requires high-performance processing for:

1. Real-time AI inference: YOLO26 object detection on video streams
2. Concurrent request handling: WebSocket connections from multiple cameras
3. Batch processing: 90-second time windows with parallel analysis
4. Background tasks: Risk scoring with Nemotron LLM

Python's Global Interpreter Lock (GIL) has historically limited CPU-bound parallel performance. With Python 3.14's experimental free-threading support (PEP 703), we can potentially achieve significant performance improvements for our multi-threaded workloads.

Current Limitations¶

Evaluation Criteria¶

1. Performance improvement for CPU-bound parallel workloads
2. Compatibility with existing codebase
3. Library ecosystem support
4. Production stability
5. Migration effort

Decision Summary¶

Adopt Python 3.14 with optional free-threading support. The project will:

1. Target Python 3.14 as the primary runtime
2. Use the free-threaded build (3.14t) for production workloads
3. Maintain backward compatibility with Python 3.12+ for development
4. Adopt UUID7 for new ID generation
5. Leverage Zstd compression where beneficial

Options Evaluated¶

Option 1: Stay on Python 3.12/3.13 (Status Quo)¶

Option 2: Python 3.14 (Standard Build)¶

Option 3: Python 3.14t (Free-Threaded Build)¶

Option 4: Alternative Languages (Rust/Go for hot paths)¶

Rationale¶

We selected Option 3 (Python 3.14t) with graceful degradation to Option 2 for the following reasons:

1. Performance Benchmarks¶

Our benchmark suite shows significant improvements:

2. Architecture Alignment¶

Our existing architecture already uses ThreadPoolExecutor extensively:

# Current pattern (limited by GIL)
with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(process_frame, frames))

With free-threading, this pattern immediately benefits without code changes.

3. Gradual Adoption Path¶

We can adopt incrementally:

1. Phase 1: Development on 3.14, production on 3.12
2. Phase 2: Production testing on 3.14 (standard)
3. Phase 3: Production deployment on 3.14t
4. Phase 4: Optimize for free-threading patterns

4. Feature Benefits Beyond Performance¶

5. Risk Mitigation¶

• Feature detection patterns ensure backward compatibility
• Comprehensive benchmark suite validates performance claims
• Gradual rollout allows reverting if issues arise

Consequences¶

Positive¶

1. 3-7x speedup for CPU-bound parallel workloads
2. Simpler code: Use ThreadPoolExecutor instead of ProcessPoolExecutor
3. Lower memory: Shared objects between threads
4. Better IDs: UUID7 improves database performance
5. Future-proof: Positioned for Python's parallel evolution

Negative¶

1. Library compatibility: Some C extensions need updates
2. Thread safety: Need to audit shared mutable state
3. Debugging complexity: New class of concurrency bugs possible
4. Learning curve: Team needs to understand GIL removal implications

Neutral¶

1. Docker images: Need to build with 3.14t base
2. CI/CD updates: Test matrix needs 3.14t variant
3. Monitoring: Need to add thread contention metrics

Implementation Plan¶

Phase 1: Foundation (Week 1)¶

• [x] Create benchmark suite (scripts/benchmark_py314.py)
• [x] Document features (docs/development/python-3.14-features.md)
• [x] Create this ADR
• [x] Update pyproject.toml for Python 3.14 compatibility

Phase 2: Compatibility (Week 2)¶

• [ ] Audit ThreadPoolExecutor usage for thread safety
• [ ] Update Dockerfile for 3.14t base image
• [ ] Test all dependencies with 3.14t
• [ ] Update CI to include 3.14t test matrix

Phase 3: Optimization (Week 3-4)¶

• [ ] Identify hotspots benefiting from free-threading
• [ ] Refactor ProcessPoolExecutor to ThreadPoolExecutor where beneficial
• [ ] Add thread contention monitoring
• [ ] Performance regression testing

Phase 4: Production (Week 5+)¶

• [ ] Staged rollout to production
• [ ] Monitor performance metrics
• [ ] Document lessons learned

Feature Detection Patterns¶

Runtime Detection¶

import sys

def supports_free_threading() -> bool:
    """Check if running in free-threaded mode."""
    if hasattr(sys, '_is_gil_enabled'):
        return not sys._is_gil_enabled()
    return False

def supports_uuid7() -> bool:
    """Check if UUID7 is available."""
    import uuid
    return hasattr(uuid, 'uuid7')

def supports_zstd() -> bool:
    """Check if Zstd compression is available."""
    try:
        import compression.zstd
        return True
    except ImportError:
        return False

Graceful Degradation¶

# UUID generation with fallback
import uuid
import sys

def generate_id() -> uuid.UUID:
    if sys.version_info >= (3, 14) and hasattr(uuid, 'uuid7'):
        return uuid.uuid7()
    return uuid.uuid4()

# Parallel execution with fallback
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def get_cpu_executor(max_workers: int):
    if supports_free_threading():
        return ThreadPoolExecutor(max_workers=max_workers)
    return ProcessPoolExecutor(max_workers=max_workers)

Monitoring Requirements¶

Metrics to Track¶

1. Thread contention rate: Lock wait times
2. Memory usage: Per-worker memory consumption
3. CPU utilization: Should increase with free-threading
4. Request latency: P50, P95, P99 for API endpoints
5. Throughput: Frames processed per second

Alerting Thresholds¶

Rollback Plan¶

If issues arise with Python 3.14t:

1. Immediate: Switch Dockerfile to 3.14 (standard build)
2. Short-term: Revert to Python 3.12/3.13
3. Long-term: Re-evaluate when 3.14t stabilizes

Related Documentation¶

• Python 3.14 Features Guide
• Benchmark Script
• PEP 703 - Making the Global Interpreter Lock Optional
• PEP 778 - Add UUID version 7

Decision Approval¶

Back to Decision Records