KK
Screenshot
Control plane overview

Projects

Voice Agent

Orchestrator

Individual ProjectPrototype

Stack

  • Node.js
  • TypeScript
  • Angular
  • Voice AI
  • AI Runtime
  • Voice Streaming
  • Realtime & Pipeline
  • MCP
  • 3CX Call Control
  • RAG
  • SQLite-Vec
  • ONNX Runtime
  • Docker

Overview

Voice Agent Orchestrator is a standalone on-premises platform for building, evaluating, and operating AI-powered voice agents on telephony. Organizations can run the full conversation through a single speech-to-speech model, or use pipeline mode — a composable STT → LLM → TTS stack where each stage is configured independently. Knowledge retrieval, MCP tooling, and built-in evaluation are managed through a single control plane.

Problem

Voice AI integrations are often built around specific providers, runtimes, and deployment assumptions. As organizations introduce new models, local inference, knowledge systems, or hybrid architectures, business logic becomes increasingly coupled to implementation details.

The result is slower experimentation, duplicated integrations, and growing maintenance costs whenever the underlying AI stack changes. The challenge is not supporting a single provider or model, but enabling conversational systems to evolve without forcing the surrounding platform to evolve with them.

Approach

The platform is built around two interchangeable voice runtime architectures, deployed independently from the PBX — so the AI stack can scale and ship at its own pace without coupling to the communications core.

Realtime mode runs the full conversation through a single speech-to-speech provider. Pipeline mode composes independent STT, LLM, and TTS stages across cloud, on-premises, or hybrid deployments — with a unified runtime abstraction whether providers stream natively or connect over batch APIs.

Turn-taking, barge-in, and provider orchestration live inside the voice runtime rather than in telephony integration code.

The platform integrates with 3CX through the Call Control API for real-time call handling and uses the built-in 3CX MCP server as its primary tool execution layer. Core telephony capabilities — contact lookup, extension discovery, call transfers, and routing — are exposed to agents through MCP tools, with phonebook search leveraging 3CX's fuzzy linguistic matching to compensate for speech recognition errors.

Platform architecture

Control plane

Create multiple stacks of any architecture or deployment type, assign agents per stack — all running simultaneously in one runtime.

Agents

Agent A
Agent B
Agent C
Agent D

Configure voice, language, prompt, tooling, knowledge access, and more — per agent.

Voice processing architecture

The runtime voice path on every call — from incoming audio to spoken response.

IN

Audio in

3CX Call Control API · 8 kHz mono

getAudioStreamGET stream · 20 ms frames

PCM stream

VAD

Voice activity detection

Pre-STT · one VAD backend · barge-in

Silero VAD neural

Neural ONNX on raw PCM. Speech onset, silence windows, barge-in while the agent speaks.

or

Amplitude VAD

Peak amplitude threshold, zero dependencies. Zero-effort alternative when you don't need neural VAD.

Stop-speaking plan

Code-enforced · barge-in

When the caller can interrupt mid-reply — word-count threshold, post-interrupt backoff, and stricter VAD while the agent speaks.

speech-gated PCM

STT

Speech recognition

Cloud or on-prem · then stream or batch · optional upsample from 8 kHz

Cloud

Public vendor APIs.

  • StreamDeepgram · AssemblyAI · Speechmatics
  • BatchFlexible OpenAI-compatible endpoint

On-prem

Sidecars on your network — no cloud egress.

  • StreamLocal WS stream (sherpa-onnx sidecar)
  • BatchFlexible OpenAI-compatible endpoint

transcript

EOT

Turn commit

Patch focus · one authority before LLM

STT commits a transcript chunk to the LLM only after the caller's turn is complete. The STT model picks the path: vendor end-of-turn on the stream, or Smart Turn — a neural orchestrator module that scores turn completeness from caller speech after a VAD pause.

Smart Turn neural

Orchestrator module

Scores turn completeness after a pause in speech, then commits to the LLM. Used when the STT model has no built-in turn detection — e.g. Whisper, Nova, on-prem batch.

or

Provider EOT vendor

Shipped with the STT model

  • Deepgram FluxEnd-of-turn on the wire
  • AssemblyAI U3Built-in turn detection
  • SpeechmaticsBuilt-in turn detection

committed turn

LLM

Language model

Cloud or on-prem · OpenAI-compatible streaming

Cloud

Hosted inference and routing APIs.

  • GatewaysOpenAI · OpenRouter · Groq · xAI
  • ModelsThousands of OpenAI-compatible models

On-prem

Local or datacenter GPU — no cloud egress.

  • DevOllama · local models & edge
  • ServevLLM · SGLang · production GPU
Ollama

token stream

ACC

Sentence streaming

Orchestrator module · LLM deltas → TTS-sized chunks

Sentence accumulator

Orchestrator module

Buffers the LLM token stream into sentences and dispatches each to TTS as it completes — the caller hears the reply while generation continues. Parallel TTS, ordered playback to 3CX.

sentences

TTS

Text to speech

Cloud or on-prem · optional downsample to 8 kHz for 3CX

Cloud

Hosted voice APIs — WebSocket or HTTP.

  • StreamElevenLabs · Cartesia · Deepgram Aura
  • BatchFlexible OpenAI-compatible endpoint

On-prem

Local sidecar — no cloud egress.

  • StreamLocal WS stream (Kokoro WS sidecar)
  • BatchFlexible OpenAI-compatible endpoint

PCM out

OUT

Audio out

3CX Call Control API · barge-in loop

postAudioStreamPOST stream · 20 ms PCM

Capabilities

Operations Dashboard

  • Control and manage Realtime and Pipeline agent stacks from one place.
  • Mix local, cloud, and hybrid stages across STT, LLM, TTS, or a single speech-to-speech runtime.
  • Live Agent Activity — current and latest calls with transcripts, agent replies, and tool calls.
  • Outbound Missions & Automations — track active, scheduled, and in-call work.
  • Platform services health for Vex, Knowledge Base, and Ollama embeddings.
Operations dashboard
Operations dashboard

Agent Runtime

  • Compose agents from role, caller state, routing, prompt, tools, and playbooks — not a single system prompt.
  • Per-tool Behavior playbooks and Runtime hooks that enforce required state and call-routing rules before privileged tools execute.
  • Working memory — structured caller state shared across turns, tools, and consult legs.
  • Combine built-in call-control, MCP, and webhook tools with per-agent enablement.
  • One runtime for inbound calls and outbound missions.
  • Cross-agent handoffs and complex telephony operations such as attended transfer.
  • Role presets as a baseline — fully editable after create.
Agent configuration
Agent configuration

Knowledge Base

  • One-click local RAG deployment via Docker.
  • Built on SQLite, sqlite-vec, and Nomic embeddings.
  • Shared across all providers and runtime modes.
  • No migration required when switching models.
  • Compatible with any model that supports tool calling.
  • Upload and manage documents directly from the dashboard.
  • Supports PDF, Markdown, and plain text sources.
Knowledge Base
Knowledge base

MCP Integration

  • Built-in 3CX MCP server for telephony operations.
  • Contact lookup, extension discovery, routing, and call transfers.
  • Support for custom MCP servers through API-based integrations.
  • Shared across Realtime and Pipeline architectures.
  • Easily extend agent behavior without modifying core logic.
  • Easily configure per-agent tool filtering in agent settings after MCP server connection.
MCP Integration
MCP integration

Model Quality Evaluation

  • Available for Pipeline architectures.
  • Agent-vs-Agent evaluation framework.
  • Dynamic test scenarios based on agent configuration.
  • Knowledge Base, routing, and tool usage scenarios included automatically.
  • Detect regressions before deploying configuration changes.
  • Compare models under the same agent setup.
Model quality evaluation
Model quality evaluation

Stack Cost Analysis

  • Separate cost tracking for Realtime and Pipeline architectures.
  • Active sessions automatically archived after stack changes.
  • Compare historical configurations and provider combinations.
  • Automatic pricing synchronization where supported (e.g. OpenRouter).
  • Manual price overrides with update tracking for unsupported providers.
  • Optional infrastructure cost accounting for self-hosted components.
Stack Cost Analysis
Stack cost analysis

Performance Monitoring

  • Realtime response latency monitoring.
  • Pipeline latency breakdown across runtime components.
  • Turn-level performance tracking during conversations.
  • Quickly identify bottlenecks within the active stack.
  • Compare performance between different configurations.
  • Built-in diagnostics for optimization and troubleshooting.
Performance monitoring
Performance monitoring

Vex

  • Platform administrator copilot — text LLM separate from voice stacks on the phone.
  • Create and edit voice agents through a guided draft flow with preview and confirm.
  • Manage outbound missions, task board, and mission lifecycle from chat.
  • Schedule batch outbound automations — missed-call callbacks, daily dial runs, and similar flows.
  • PBX and custom MCP tools in chat when connectors are configured — call history, phonebook, external APIs.
  • Proactive platform alerts and on-demand documentation — separate from operator conversation tabs.
Platform administrator
Platform administrator
← All projects