Speech & Multimodal Pipelines
chakra-convai features a dual-pipeline architecture. Chakra Console assigns which pipelines a license is entitled to execute, with agent-level overrides governed remotely.
1. Managed chakra Pipeline
The chakra pipeline is Chakra Labs' fully managed, turn-based voice stack:
User Audio
│
▼
[Silero VAD] (Local Voice Activity Detection)
│
▼
[Chakra STT] (Speech-to-Text on Chakra GPU Fleet)
│
▼
[Managed LLM] (High-throughput OpenAI-compatible route on Fleet)
│
▼
[Chakra TTS] (Natural low-latency Text-to-Speech synthesis)
│
▼
Caller Audio Stream
Highlights
- Zero Provider Configuration: Clients configure no model names, no external LLM API keys, and no TTS audio hyper-parameters. Everything routes securely through the Chakra Gateway authenticated by
CHAKRA_LICENSE_KEY. - Fleet Optimization: Speech models and LLM engines are continuously tuned and upgraded on Chakra GPU clusters without client application redeployments.
- Vision Support: In web voice sessions, camera frames are packaged alongside speech transcripts to the fleet vision model.
2. Realtime gemini_live Pipeline
The gemini_live pipeline leverages Google's Gemini Live realtime multimodal WebSocket streaming:
User Audio & Video
│
▼
[LiveKit Audio/Video Tracks]
│
▼
[Google Gemini Live Session] (Bidirectional streaming)
│
▼
Synthesized Audio Return
Highlights
- Bidirectional Streaming: Ultra-low latency voice-to-voice streaming with natural barge-in interruptions.
- Native Video Track Ingestion: Streams camera and screen sharing directly to the Gemini Live multimodal model.
- Requirements: Requires a valid
GOOGLE_API_KEYset in your.env.
Language Enforcer & Policy
- Dynamic Language Detection: If an agent is not pinned to a specific language, the call language dynamically tracks the caller's spoken language based on turn transcript analysis (Sinhala, Tamil, or English).
- Pinned Language Enforcement: When pinned to a single language (e.g.
language = "si"),chakra-convaiinjects the language enforcement patch (language_policy.forced_enforcer_default) to ensure Gemini does not drift across languages during interruptions.
Pipeline Governance & Precedence
- Chakra Console Assignment: The signed response from
/v2/verifylists the authorized pipelines and per-agent pins. - Environment Variable:
CHAKRA_PIPELINEcan select between authorized pipelines. - Application Setting:
[pipeline] typeinchakra.tomlspecifies the application's default choice among licensed pipelines.
If an application requests a pipeline not permitted by the license,
chakra-convailogs a warning and falls back to the license's default pipeline.
Run chakra-convai check to view the resolved pipeline for each configured agent.