Skip to content

Speech & Multimodal Pipelines

chakra-convai features a dual-pipeline architecture. Chakra Console assigns which pipelines a license is entitled to execute, with agent-level overrides governed remotely.


1. Managed chakra Pipeline

The chakra pipeline is Chakra Labs' fully managed, turn-based voice stack:

User Audio
    │
    ▼
[Silero VAD] (Local Voice Activity Detection)
    │
    ▼
[Chakra STT] (Speech-to-Text on Chakra GPU Fleet)
    │
    ▼
[Managed LLM] (High-throughput OpenAI-compatible route on Fleet)
    │
    ▼
[Chakra TTS] (Natural low-latency Text-to-Speech synthesis)
    │
    ▼
Caller Audio Stream

Highlights

  • Zero Provider Configuration: Clients configure no model names, no external LLM API keys, and no TTS audio hyper-parameters. Everything routes securely through the Chakra Gateway authenticated by CHAKRA_LICENSE_KEY.
  • Fleet Optimization: Speech models and LLM engines are continuously tuned and upgraded on Chakra GPU clusters without client application redeployments.
  • Vision Support: In web voice sessions, camera frames are packaged alongside speech transcripts to the fleet vision model.

2. Realtime gemini_live Pipeline

The gemini_live pipeline leverages Google's Gemini Live realtime multimodal WebSocket streaming:

User Audio & Video
    │
    ▼
[LiveKit Audio/Video Tracks]
    │
    ▼
[Google Gemini Live Session] (Bidirectional streaming)
    │
    ▼
Synthesized Audio Return

Highlights

  • Bidirectional Streaming: Ultra-low latency voice-to-voice streaming with natural barge-in interruptions.
  • Native Video Track Ingestion: Streams camera and screen sharing directly to the Gemini Live multimodal model.
  • Requirements: Requires a valid GOOGLE_API_KEY set in your .env.

Language Enforcer & Policy

  • Dynamic Language Detection: If an agent is not pinned to a specific language, the call language dynamically tracks the caller's spoken language based on turn transcript analysis (Sinhala, Tamil, or English).
  • Pinned Language Enforcement: When pinned to a single language (e.g. language = "si"), chakra-convai injects the language enforcement patch (language_policy.forced_enforcer_default) to ensure Gemini does not drift across languages during interruptions.

Pipeline Governance & Precedence

  1. Chakra Console Assignment: The signed response from /v2/verify lists the authorized pipelines and per-agent pins.
  2. Environment Variable: CHAKRA_PIPELINE can select between authorized pipelines.
  3. Application Setting: [pipeline] type in chakra.toml specifies the application's default choice among licensed pipelines.

If an application requests a pipeline not permitted by the license, chakra-convai logs a warning and falls back to the license's default pipeline.

Run chakra-convai check to view the resolved pipeline for each configured agent.