Skip to content

Channels: Phone, Web Voice & Video, and Chat

chakra-convai supports three distinct channel modalities from a single shared codebase: 1. Phone (phone): Traditional telephony calls arriving through SIP trunks. 2. Web Voice & Video (web with mode: "voice"): Realtime voice streaming combined with user camera or screen sharing. 3. Web Chat (web with mode: "chat"): Low-latency typed text and photo messaging.

Each agent defined in chakra.toml declares its target channel ([[agents]] channel = "phone" | "web").


Channel Comparison

Feature Phone (phone) Web Voice (web_voice) Web Chat (web_chat)
Transport SIP / Telephony trunk via LiveKit SIP LiveKit WebRTC Room LiveKit WebRTC Room
User Input Spoken audio Audio + Camera / Screen Typed text + Photo uploads
Agent Output Synthesized speech Synthesized speech Text responses
Silence Handling Enabled (configurable check-ins) Enabled Disabled (uses idle timeout)
Vision Support N/A Full (Gemini video / Chakra vision frames) Photos attached as user messages
Talk Time Limits Enforced by daily caller limit Enforced by daily caller limit Excluded from talk limits
Identity Keying Caller caller ID (phone number) user.phone or user.id user.phone or user.id

1. Phone Channel

Configured with channel = "phone". Phone sessions are dialed through SIP into LiveKit rooms. The caller ID becomes the user identity, linking directly to PostgreSQL caller memory and daily call duration limits.

[[agents]]
name = "phone_agent"
channel = "phone"
language = "si"

2. Web Voice & Video

Web voice sessions connect an end-user from an iOS, Android, or Web client via WebRTC.

[[agents]]
name = "web_voice_agent"
channel = "web"

Multimodal Vision & Camera Frames

  • gemini_live Pipeline: The agent natively subscribes to the user's incoming video track.
  • chakra Pipeline: The engine captures user camera frames ([web] vision_frames, default 4, one frame per second) and attaches them to each user speech turn. The managed LLM inspects what the user showed while speaking. Once a turn is complete, earlier frames are evicted so subsequent turns only bill their own tokens.

3. Web Chat

When an end-user interacts over chat (or when room name starts with chat_room_prefix), the session runs in text/photo mode:

[web]
chat_idle_timeout_seconds = 600
chat_model = "gemini-2.5-flash"
  • Spoken greetings, VAD, and silence detectors are disabled.
  • Image uploads from the user are converted into multimodal prompts.
  • Sessions automatically close when inactive after chat_idle_timeout_seconds.

The Standard Session Data Format

Every web and mobile application sends session metadata inside the LiveKit room's metadata string:

{
  "chakra": {
    "version": 1,
    "mode": "voice",
    "language": "si",
    "user": {
      "id": "usr_99812",
      "phone": "0771234567",
      "name": "Nimal Perera"
    },
    "context": {
      "district": "Kandy",
      "crops": ["tomato", "chilli"],
      "account_tier": "premium"
    }
  }
}

Schema Details

  • mode: "voice" or "chat".
  • language: ISO code ("si", "ta", "en") or descriptive language name. Locked for the session when [web] keep_language = true.
  • user: User identity fields. user.phone takes precedence for caller memory, falling back to user.id.
  • context: Arbitrary key-value dictionary that is formatted and injected directly into the LLM system prompt on every turn.

Building Metadata in Python

Applications generating LiveKit room tokens in Python can use the built-in helper:

from chakra_convai import session_metadata

metadata_str = session_metadata(
    mode="voice",
    language="si",
    user_id="usr_99812",
    phone="0771234567",
    name="Nimal Perera",
    context={"district": "Kandy", "crops": ["tomato"]}
)