Skip to content

Connection Troubleshooting

Solving network, container, and connectivity problems.

Time to read: ~6 min Prerequisites: None

Service Not Reachable

Symptoms

  • Error: Connection refused
  • Health check fails
  • curl returns "connection refused"

Diagnosis

# Check if service is listening (production compose binds these to 127.0.0.1)
ss -tlnp | grep 8000   # Backend
ss -tlnp | grep 8090   # AI Gateway (Triton: YOLO26, Florence, CLIP, enrichment)
ss -tlnp | grep 8091   # Nemotron (llama.cpp)

# Check container status
docker compose -f docker-compose.prod.yml ps

# Check container logs
docker compose -f docker-compose.prod.yml logs backend

Solutions

1. Start the service:

# All services
docker compose -f docker-compose.prod.yml up -d

# AI services only
docker compose -f docker-compose.prod.yml up -d ai-gateway ai-llm

2. Check port conflicts:

# Find what's using the port
sudo lsof -i :8000

# Kill conflicting process
sudo kill <PID>

3. Verify network configuration:

For Docker:

  • Services on same network can use service names (postgres, redis)
  • External access uses localhost:PORT

For production compose, all AI models run inside ai-gateway (Triton) and the LLM runs in ai-llm — there are no standalone ai-yolo26/ai-florence/ai-clip/ ai-enrichment containers. The backend reaches them by compose DNS:

  • ai-gateway:8090 (path-prefixed routers: /yolo26, /florence, /clip, /enrichment, /enrich-lt — wired as YOLO26_URL=http://ai-gateway:8090/yolo26, etc.)
  • ai-llm:8091

For native development:

  • All services use localhost
  • Verify each service is bound to correct address

Redis Not Available

Symptoms

  • Error: Connection refused to Redis
  • Health check shows Redis unhealthy
  • Queue operations fail

Diagnosis

# Check Redis is running
docker compose -f docker-compose.prod.yml ps redis

# Test Redis connection
redis-cli ping

# Check Redis logs
docker compose -f docker-compose.prod.yml logs redis

Solutions

1. Start Redis:

docker compose -f docker-compose.prod.yml up -d redis

2. Check REDIS_URL:

# Local development
REDIS_URL=redis://localhost:6379/0

# Docker network
REDIS_URL=redis://redis:6379/0

3. Check Redis memory:

redis-cli info memory

If Redis is out of memory, increase maxmemory or clear old data.

Container Crashes

Symptoms

  • Container exits immediately after starting
  • Container in "restarting" state
  • Exit code 1 or 137

Diagnosis

# Check container status and exit codes
docker compose -f docker-compose.prod.yml ps -a

# Check container logs
docker compose -f docker-compose.prod.yml logs backend

# Check recent events
docker events --since="10m" --filter="type=container"

Solutions

Exit code 1 (application error):

Check logs for:

  • Missing environment variables
  • Database connection failures
  • Configuration errors

Exit code 137 (OOM killed):

# Check if OOM killed
dmesg | grep -i "killed process"

# Increase container memory limit
# In docker-compose.yml:
services:
  backend:
    deploy:
      resources:
        limits:
          memory: 2G

Exit code 139 (segfault):

Usually indicates:

  • Incompatible library versions
  • GPU driver issues
  • Corrupted container image

Try: docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d

File Watcher Issues

Symptoms

  • Images uploaded but not processed
  • No detections created from new images
  • Pipeline status shows file watcher not running
  • Pipeline status shows watch_mode: polling-fallback (the kernel refused the native watch at startup; watch_fallback_reason names the errno)

Diagnosis

# Check pipeline status
curl http://localhost:8000/api/system/pipeline | jq .file_watcher

# Check camera folder permissions
ls -la /export/foscam/

# Check for inotify limits (Linux)
cat /proc/sys/fs/inotify/max_user_watches

# SELinux hosts: enforcing + a camera root that is not container_file_t = watch denied
getenforce; ls -Zd /export/foscam

Solutions

1. Verify folder path:

Ensure FOSCAM_BASE_PATH matches where cameras upload images.

2. Enable polling for Docker:

If using Docker Desktop on macOS/Windows:

FILE_WATCHER_POLLING=true
FILE_WATCHER_POLLING_INTERVAL=1.0

3. Increase inotify watches (Linux):

# Temporary
echo 524288 | sudo tee /proc/sys/fs/inotify/max_user_watches

# Permanent
echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

4. Check file permissions:

Backend needs read access to camera folders.

5. SELinux denies the watch (Fedora/RHEL, enforcing):

A container_t backend can read a usr_t camera root but not inotify-watch it (avc: denied { watch watch_reads } in the audit log), so the watcher falls back to polling with watch_fallback_reason: EACCES. The tracked compose files mount /cameras with :z, which makes podman relabel the root to container_file_t; a custom compose file needs the same option. Or relabel the root once:

sudo semanage fcontext -a -t container_file_t '/export/foscam(/.*)?'
sudo restorecon -R /export/foscam

setup.py deploy checks this before starting containers and reads watch_mode after the backend is healthy. Both checks only warn and never relabel anything.

WebSocket Issues

Symptoms

  • Dashboard shows stale data
  • "WebSocket connection failed"
  • Frequent reconnections

Diagnosis

# Test WebSocket connection
websocat ws://localhost:8000/ws/events

# Check for proxy issues
curl -v http://localhost:8000/ws/events

# Check WebSocket rate limiting
curl http://localhost:8000/api/system/telemetry

Solutions

1. Check idle timeout:

Default is 300 seconds. Client should send periodic pings:

setInterval(() => ws.send('{"type":"ping"}'), 30000);

2. Check proxy configuration:

If behind nginx/reverse proxy:

location /ws/ {
    proxy_pass http://backend:8000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;
}

3. Check rate limits:

WebSocket connection attempts are limited to 100/minute per client IP by default (allowed range 1-400). Adjust:

RATE_LIMIT_WEBSOCKET_CONNECTIONS_PER_MINUTE=150

CORS Errors

Symptoms

  • Browser console: "CORS policy blocked"
  • API works in curl but not browser
  • OPTIONS requests fail

Diagnosis

# Check CORS headers
curl -v -X OPTIONS -H "Origin: http://localhost:3000" \
  http://localhost:8000/api/events

Solutions

1. Add origin to allowed list:

CORS_ORIGINS=["http://localhost:3000", "http://localhost:5173", "https://your-domain.com"]

2. For development, allow all (not for production):

CORS_ORIGINS=["*"]

3. Check frontend URLs:

Ensure VITE_API_BASE_URL matches actual backend URL:

# Frontend expects backend at:
VITE_API_BASE_URL=http://localhost:8000

# Backend must allow this origin in CORS_ORIGINS

Timeouts

Symptoms

  • Health checks timeout
  • Requests hang then fail
  • "Request timeout" errors

Diagnosis

# Check service response time
time curl http://localhost:8000/health

# Check AI service response time (gateway aggregate + LLM)
time curl http://localhost:8090/health
time curl http://localhost:8091/health

Solutions

1. Increase timeout settings:

AI_CONNECT_TIMEOUT=30.0       # Default 10.0 (max 60)
AI_HEALTH_TIMEOUT=10.0        # Default 5.0 (max 30)
YOLO26_READ_TIMEOUT=120.0     # Default 30.0 (max 120)
NEMOTRON_READ_TIMEOUT=300.0   # Default 120.0 (max 600)

2. Check service load:

AI services may be overloaded. Check queue depths:

curl http://localhost:8000/api/system/telemetry | jq .queues

3. Check network latency:

If AI services are remote:

ping ai-server-hostname

Next Steps

See Also

Back to Operator Hub