Deploy AI-powered video analytics for workplace safety

Copy link

Monitor workplace safety and operational compliance in real time using AI object detection, multimodal analysis, and conversational insights.

Table of Contents

Copy link

• Detailed description
  • See it in action
  • Architecture diagrams
  • Video upload workflow
  • Application workflow
  • Components
  • Storage strategy
• Requirements
  • Minimum hardware expectations
  • Minimum software requirements
  • Required user permissions
• Configuration
• Deploy
  • Prerequisites
  • Supported model-serving profiles
  • Installation steps
  • Deployment workflow
  • Helm values
  • Delete
  • Local development (Podman Compose)
  • Local development (no containers)
• Training a custom model
• API endpoints
  • Example request
• Reference
• Tags

Detailed description

Copy link

Organizations face challenges maintaining workplace safety and operational compliance across facilities with multiple video feeds. Manual monitoring is resource-intensive and reactive, often missing critical events or patterns that indicate safety risks or operational inefficiencies.

This AI quickstart provides a complete multimodal monitoring solution that combines computer vision and large language models (LLMs) to analyze video streams in real time. The application uses trained object-detection models to identify objects, people, and events in live RTSP feeds or uploaded video files. Users can monitor multiple video sources through an interactive dashboard, view detection overlays, and ask questions about what the AI has observed using a conversational chat interface.

Key capabilities:

• Real-time analysis: Process live RTSP streams or MP4 files with per-source configuration and model selection
• Persistent storage: Upload and manage video sources backed by MinIO object storage and PostgreSQL database
• Visual monitoring: View detection results, overlays, and automated safety summaries through a React dashboard
• Conversational insights: Query the system using an OpenAI-compatible LLM with LangGraph/LangChain and optional SQL-backed tools
• Flexible deployment: Run locally with Podman Compose, bare-metal development, or deploy to Kubernetes/OpenShift with Triton/KServe or OpenVINO Model Server
• Model customization: Train and deploy custom object detection models using the included Jupyter notebook workflow
• Annotation support: Optional Label Studio integration for creating and refining training datasets

Upload a video or connect an RTSP stream to see the AI detection in action. The dashboard displays:

• Live video feed with bounding boxes around detected objects
• Real-time safety and operational summaries
• Object tracking across frames
• Interactive chat for querying detection history

See it in action

Copy link

Interactive walkthrough

Architecture diagrams

Copy link

Static overview (SVG): docs/images/architecture.svg. Additional slides: docs/architecture-slides.html (see docs/architecture-slides-README.md). The workflow figures below are PNG exports from Mermaid sources (click a diagram to open the full-resolution PNG). See docs/mermaid.README.md for what the .mmd files are and how to regenerate the thumbnails and large images.

Video upload workflow

Copy link

Application workflow

Copy link

Components

Copy link

• Backend (Flask, OpenCV): Video decode, MJPEG output, and drawn overlays; inference over gRPC to OpenVINO Model Server (ovmsclient, local/CPU) or Triton via tritonclient (KServe / GPU path); multi-object tracking with BoxMOT (BoostTrack++); PostgreSQL for app configs, classes, tracks, and observations; MinIO for object storage; LLM chat with LangGraph / LangChain (OpenAI-compatible API) and optional read-only postgres-mcp for SQL tools; optional Arize Phoenix for tracing
• Frontend (React, React Router, Axios): Dashboard, source selection (RTSP / MP4 thumbnails), configuration page, and chat with Markdown rendering
• OpenVINO Model Server (OVMS): Model serving runtime; local stack also runs yolo-model-prep (Ultralytics-based export) to build the model repo from app/models/*.pt before OVMS starts
• MinIO: S3-compatible object storage for models, videos, uploads, and config-related objects
• PostgreSQL: Durable storage for multi-source configs and tracking data
• Data loader: Init container that seeds model and video objects into MinIO
• Label Studio (optional): Annotation UI using the same PostgreSQL and MinIO stack

Storage strategy

Copy link

All models and video files are stored in MinIO rather than baked into container images:

Requirements

Copy link

Minimum hardware expectations

Copy link

• KServe / Triton (RUNTIME_TYPE=kserve, e.g. make deploy / make deploy-gpu): plan for GPU-capable worker nodes appropriate to your model-serving footprint; sizing depends on model and cluster policy.
• OpenVINO Model Server (RUNTIME_TYPE=openvino, e.g. make deploy-openvino): targeted at CPU-oriented model serving—confirm node CPU/memory with your platform team.

Minimum software requirements

Copy link

• Podman + podman-compose for local container runs
• Docker (optional alternative)
• Helm (for Kubernetes/OpenShift deployment)
• OpenShift Client CLI (oc) when deploying to or operating against an OpenShift cluster

Required user permissions

Copy link

• Project / namespace permissions sufficient to install the Helm release (workloads, routes, PVCs, Services, etc.).
• Cluster administrator may be required when enabling OpenShift integrations that install SCCs or cluster-scoped bindings—see chart flags such as openshift.scc.enabled, openshift.scc.name, and openshift.roleBinding.*.

Configuration

Copy link

Copy .env.example to .env and fill in your values. The .env.example file contains the required OpenAI-compatible LLM variables: OPENAI_API_TOKEN, OPENAI_API_ENDPOINT, OPENAI_MODEL, and OPENAI_TEMPERATURE.

Important: When specifying OPENAI_API_ENDPOINT, include /v1 at the end (for example, https://your-api-endpoint.example.com/v1).

OpenShift/Kubernetes make deploy targets run check-openai-env and prompt for any missing OpenAI values, writing them to .env. Local workflows (make local-build-up, make dev-backend) do not run that prompt — create .env yourself before starting the backend so the chat stack can initialize.

Which make deploy variant to use (GPU vs CPU model serving, Label Studio) is covered under Deploy → Supported model-serving profiles.

Backend environment variables:

• PORT: backend port (default 8888)
• FLASK_DEBUG: set to true to enable debug mode
• CORS_ORIGINS: allowed origins, comma-separated or *

Frontend runtime config (app/frontend/public/env.js or mounted in containers):

• API_URL: backend base URL (example: http://localhost:8888)

Deploy

Copy link

The sections below cover cluster deployment (build, push, Helm install) and local workflows. Configure OpenAI-related variables in .env first — see Configuration.

Prerequisites

Copy link

• .env populated for OpenAI-compatible chat where used (see Configuration); cluster make deploy targets run check-openai-env interactively if values are missing.
• Container registry access for make push and make build-push-data as configured in your environment.
• Requirements satisfied for your target (Helm, oc for OpenShift, etc.).

Supported model-serving profiles

Copy link

All targets below use the same Helm chart and .env OpenAI settings; they differ only by model-serving runtime (RUNTIME_TYPE) and optional Label Studio.

Installation steps

Copy link

1. Clone this repository to your workstation (use your fork or upstream URL as appropriate).

2. Configure the application — copy .env.example to .env and set variables as described in Configuration.

3. Build and push images

# Build backend and frontend images
make build

# Push to registry
make push

# Build and push data loader image (contains model/video for MinIO upload)
make build-push-data

4. Install on the cluster — pick a profile from Supported model-serving profiles, then run one of:

make deploy NAMESPACE=<your-namespace>

make deploy-openvino NAMESPACE=<your-namespace>

make deploy-labelstudio NAMESPACE=<your-namespace>

make deploy-openvino-labelstudio NAMESPACE=<your-namespace>

5. Operate and tune — see Deployment workflow, Helm values, and OpenShift-specific chart options below.

Deployment workflow

Copy link

1. MinIO starts (from ai-architecture-charts dependency)
2. Backend Pod Init Container 1 (upload-data): Uploads model/video to MinIO
3. Backend Pod Init Container 2 (download-data): Downloads files from MinIO to PVC
4. Backend starts with MINIO_ENABLED=false, reads from PVC paths
5. Frontend connects to backend API

Helm values

Copy link

Override settings (from the repository root; chart path matches HELM_CHART in the root Makefile):

export HELM_CHART=$(grep '^HELM_CHART ?=' Makefile | sed 's/^HELM_CHART ?= //')
helm upgrade multimodal-monitoring "$HELM_CHART" \
  --set frontend.apiUrl=/api \
  --set backend.corsOrigins=http://your-frontend-host \
  --set storage.size=2Gi

OpenShift-specific options are included in the chart:

• Frontend Route: openshift.route.enabled and optional openshift.route.host
• Backend Route: openshift.backendRoute.enabled and optional openshift.backendRoute.host
• Label Studio Route: labelStudio.enabled, labelStudio.route.enabled, labelStudio.route.host
• Shared Route host (same host for frontend + backend): openshift.sharedHost
• NetworkPolicy: openshift.networkPolicy.enabled
• SCC/RoleBinding: openshift.scc.enabled, openshift.scc.name, openshift.roleBinding.*

Delete

Copy link

Remove the deployed application from your cluster:

make undeploy NAMESPACE=<your-namespace>

Local development (Podman Compose)

Copy link

Build and run

Copy link

make local-build-up

This starts:

1. MinIO - Object storage (ports 9000, 9001)
2. data-loader - Uploads model/video to MinIO (runs once)
3. backend - Flask API with MINIO_ENABLED=true (port 8888)
4. frontend - React app (port 3000)
5. Label Studio - Annotation UI backed by the same PostgreSQL + MinIO stack (port 8082)

Run without rebuild

Copy link

make local-up

Stop

Copy link

make local-down

Access

Copy link

• Frontend: http://localhost:3000
• Backend API: http://localhost:8888/api/
• MinIO Console: http://localhost:9001 (login: minioadmin / minioadmin)
• Label Studio: http://localhost:8082

Local development (no containers)

Copy link

Backend

Copy link

make dev-backend

Note: Requires model and video files in app/models/ and app/data/ directories.

Frontend

Copy link

make dev-frontend

Training a custom model

Copy link

To train a YOLO model for badge detection (or other object classes) using your own images:

1. Install JupyterLab:

   pip install jupyterlab
   

2. Run the training notebook:

   cd training
   jupyter lab
   

   Then open yolo_training.ipynb and run the cells in order.

The training/ folder includes an example dataset. See the detailed training README for the full training process, notebook steps, and dataset requirements.

API endpoints

Copy link

Example request

Copy link

curl -X POST http://localhost:8888/ask_question \
  -H 'Content-Type: application/json' \
  -d '{"question": "How many people are detected?"}'

Reference

Copy link

Related resources

Copy link

• Ultralytics YOLO - Object detection model framework
• OpenVINO Model Server - Model serving runtime documentation
• KServe - Kubernetes-based model serving platform
• Label Studio - Data annotation and labeling tool
• LangGraph - Framework for building stateful LLM applications

Project components

Copy link

• BoxMOT - Multi-object tracking library
• Arize Phoenix - LLM observability and tracing
• MinIO - S3-compatible object storage