Automate the repetitive technical work around the technician, not the technician out of the work.
Observe → Understand → Gather Evidence → Retrieve Knowledge →
Form Hypothesis → Test Hypothesis → Decide →
Execute Safe Action or Request Technician Action →
Verify → Document → Learn from Actual Resolution
HUMAN TECHNICIAN
│
┌─────────────┴─────────────┐
│ │
OPERATIONS UI AI ASSISTANT
(NOC Console & Timeline) (Chat & Slash Commands)
│ │
System state Reasoning
Evidence Investigation
Incidents Recommendations
Devices Tool use
Sites Explanation
│ │
└─────────────┬─────────────┘
│
POLICY ENGINE
(services/agent/policy.py)
│
┌─────────────┴─────────────┐
│ │
Safe automation Human action
(SAFE_REVERSIBLE) (APPROVAL / HUMAN_ONLY)
services/helpdesk/, Port 8001)open, investigating, pending_technician, pending_approval, resolved, closed.services/portal/, Port 8002)reconnect_stream, restart_service, retry_upload, clear_transient).services/agent/, Port 8003)POST /investigate/{asset_id}.data/knowledge/.frontend/, Port 5173)/): Metric cards, site health breakdown, active faults, and quick fault simulator modal./incidents): Filterable ticket table with severity and status badges./incidents/:id): Full ticket metadata, expandable diagnostic timeline, historical matches, and technician note submission./devices): Device status grid, ping/RTSP latency, and fault injection shortcuts./devices/:id): Comprehensive device specifications, real-time gauges, and live telemetry./sites): Multi-site status rollups and summary statistics./sites/:id): Interactive Topology Graph mapping network dependencies and correlating multi-camera outages./assistant): Conversational NOC chat with slash commands, tool execution step pills, and human-in-the-loop alerts./knowledge): Authoritative Markdown runbook browser./audit): Complete chronological log of autonomous and human actions.| Class | Examples | Automatic Action? | Policy Treatment |
|---|---|---|---|
| READ | ping_check, tcp_check, rtsp_check, get_logs, search_tickets |
✅ Always | Executed immediately for diagnostic evidence |
| SAFE_REVERSIBLE | reconnect_stream, restart_service, retry_upload, clear_transient |
✅ If Allow-Listed | Executed autonomously, followed by verification |
| APPROVAL_REQUIRED | reboot_host, update_config, firmware_upgrade, factory_reset |
❌ Never autonomous | Flagged with yellow badge, requires human approval |
| HUMAN_ONLY | hardware_repair, cable_replacement, lens_cleaning, credential_rotation |
❌ Never autonomous | Red badge alert, ticket escalated to technician |
Security Rule: Policy enforcement is strictly isolated in
services/agent/policy.py. Frontend buttons and chat prompts cannot bypass or override policy decisions.
Every investigation records and separates diagnostic facts:
| Field | Meaning | Rule |
|---|---|---|
observed_facts |
Raw check results (ping, RTSP, TCP, metrics) | Never invented; reflects real simulated telemetry |
historical_evidence |
Matching tickets from the helpdesk search | Extracted from actual past incidents |
documentation |
Runbook excerpts retrieved from SOPs | Sourced directly from data/knowledge/ |
hypothesis |
Diagnosed root cause identifier | Determined by deterministic diagnosis rules |
confidence |
Confidence level (0.0 to 1.0) | Based on conclusive vs. partial check results |
recommended_action |
Proposed remediation action | Derived from SOP and fault mapping |
safety_class |
READ / SAFE_REVERSIBLE / APPROVAL_REQUIRED / HUMAN_ONLY | Authoritatively assigned by policy.py |
actions_performed |
Actions actually triggered | Never represent an unperformed action as performed |
verification |
Post-action verification result | Confirmed by fresh telemetry check |
The conversational assistant supports quick-action slash commands:
/investigate <ASSET_ID>: Runs full diagnostic suite on asset, consults policy, and applies recovery or escalates./status <ASSET_ID>: Returns instant non-mutating telemetry snapshot./site <SITE_ID>: Evaluates site health, active devices, and checks for correlated multi-camera outages./history <QUERY>: Searches historical incidents for past resolutions./help: Lists supported commands, usage syntax, and safety guidelines./clear: Resets active chat conversation history without affecting backend tickets or device states.When executing commands, the UI renders real-time tool execution pills:
Running (Blue spinning indicator)Success (Green checkmark indicator)Failure (Red alert indicator)Blocked by Policy (Purple shield indicator)| Fault | Target | Auto-recoverable? | Safe Action | Policy Tier |
|---|---|---|---|---|
network_down |
Camera/NVR | No | — | HUMAN_ONLY |
rtsp_down |
Camera | Yes | reconnect_stream |
SAFE_REVERSIBLE |
rtsp_auth_failure |
Camera | No | — | HUMAN_ONLY |
wrong_rtsp_path |
Camera | No | — | HUMAN_ONLY |
poe_power_off |
Camera | No | — | HUMAN_ONLY |
high_cpu |
AI Box | No | — | APPROVAL_REQUIRED |
storage_full |
NVR | No | — | APPROVAL_REQUIRED |
intermittent_connectivity |
Camera | Yes | clear_transient |
SAFE_REVERSIBLE |
nvr_unavailable |
NVR | No | — | HUMAN_ONLY |
ai_box_service_failure |
AI Box | Yes | restart_service |
SAFE_REVERSIBLE |
cloud_sync_failure |
AI Box | Yes | retry_upload |
SAFE_REVERSIBLE |
multi_camera_site_outage |
Site | No | — | HUMAN_ONLY |
The full stack is covered by comprehensive automated tests:
pytest, 60 tests)pending_technician when faults are non-reversible.Vitest, 19 tests)/investigate, /status, /site, /history, and /clear state isolation.