Caller journey / AI & speechAll pages
Caller journey

AI & speech

Choose where speech recognition, the synthetic voice and the language model run, then build AI agents that answer the phone for each job: a bank helpline, clinic appointments, university admissions.

Speech
Deepgram or your own server
Language model
OpenAI, Gemini, Claude, compatible
AI agents
8001 and 8010–8099
Call audio
8 kHz, 20 ms frames

What uses what

Two services do all the AI work, and each organization chooses its own. Admins set them in Integrations, under AI and speech: Speech, AI model and Call analysis. Anything an organization leaves empty uses the platform's accounts, if the operator configured them.

FeatureSpeechLanguage model
IVR promptsTyped prompts are voiced once, when you publish, and cached–
Speak from the answer (IVR)A sentence from your API is voiced during the call–
AI agentsHear the caller (recognition) and speak (voice)Decide what to say, streamed
Call transcriptsEach side of a recorded call is transcribed–
Call analysisUses the transcriptSummary, intent, topics, sentiment, urgency, flags
Write with AI (agent editor)–Drafts an agent's greeting, instructions and knowledge
Your serverSIPAudioSocketSTT · TTSper callstreamCallerphone or browserAsterisk8001 · 80xxAI agentPython serviceSpeechDeepgram / yoursDashboardkeys · wordingAI modelLLM provider
The AI agent runs on your server next to Asterisk; speech and the model are services you choose, public or inside your network.

Speech

ChoiceWhat you enterNotes
Platform defaultNothingThe platform's Deepgram account, when the operator set one.
DeepgramAn API key, a voice and a recognition modelThe key is checked with Deepgram before it is saved. Recognition defaults to nova-3, the voice to aura-2-thalia-en. Agents can pick other Aura voices: Thalia, Andromeda, Helena, Luna, Asteria, Apollo, Arcas, Aries, Orion, Zeus.
Your own speech serverIts base URL, an optional token and a voice nameFor banks and hospitals that keep audio inside their network. It must implement the two endpoints below.

Keys and tokens are stored encrypted and never shown again; leave the field empty to keep the stored one. Changes are audited and apply straight away: to the next transcript, the next AI call and the next IVR publish.

Your own speech server

Enter a base URL such as https://speech.yourbank.internal and, if your server wants one, a bearer token. The dashboard and the AI agent call two endpoints under it. The URL must use https and resolve to a public address, unless the installation sets WEBHOOK_ALLOW_PRIVATE=yes (on-premises installs), which allows private addresses and plain http.

POST{base}/stt

Speech to text. One caller utterance from the AI agent, or one whole side of a recorded call for a transcript.

Bodyaudio/wavrequired
A WAV file: PCM 16-bit, mono, 8000 Hz.
Authorizationheader
Bearer <token>, empty after Bearer when you set no token.
X-Languageheader
A language code such as en. Always sent for transcripts (the installation's TRANSCRIPT_LANGUAGE, default en); sent by the AI agent only when the operator fixed a language with AVR_LANGUAGE.
Responseapplication/jsonrequired
text, and optionally segments with start and end in seconds. Transcripts use the segments for timing; without them the whole text is one segment. The AI agent only reads text.
POST /sttHTTP
1POST /stt HTTP/1.12Host: speech.yourbank.internal3Authorization: Bearer <your token, empty when you set none>4Content-Type: audio/wav5X-Language: en6 7<WAV file: PCM 16-bit, mono, 8000 Hz>
POST{base}/tts

Text to speech. IVR prompts, sentences from an API answer, and everything an AI agent says.

Bodyapplication/jsonrequired
text, voice (the voice name set in Speech, or on the agent; may be empty) and sample_rate, always 8000.
Authorizationheader
Bearer <token>
Responseaudio/wavrequired
PCM 16-bit mono at 8000 Hz. Raw PCM 16-bit without a WAV header is accepted too.
POST /ttsHTTP
1POST /tts HTTP/1.12Host: speech.yourbank.internal3Authorization: Bearer <your token>4Content-Type: application/json5 6{ "text": "Welcome to Sunrise Bank.", "voice": "", "sample_rate": 8000 }

Time limits and retries

CallerEndpointWaits up toRetries
Dashboard: call transcripts/stt60 sThe background job retries a failed call later
Dashboard: IVR prompts, spoken answers/tts30 sNone: publishing shows the error
AI agent: live calls/stt and /tts30 s (AVR_HTTP_TIMEOUT)Once, after network errors and 5xx answers; never after a 4xx

On a live call every second counts: the caller hears silence while recognition and voice run. Host the server close to the phone system and aim for well under a second per request.

Check on save

When you save the connector, the dashboard sends /tts the text Test.. It must answer 2xx with audio within 10 seconds, or the settings are not saved.

Reference implementation

speech.pyPython
1# speech.py - a minimal speech server for the custom connector.2#   pip install fastapi uvicorn3#   SPEECH_TOKEN=change-me uvicorn speech:app --host 0.0.0.0 --port 8443 \4#       --ssl-keyfile key.pem --ssl-certfile cert.pem5import io, os, wave6from fastapi import FastAPI, Header, HTTPException, Request, Response7 8TOKEN = os.environ.get("SPEECH_TOKEN", "")9app = FastAPI()10 11def check(auth: str | None):12    if TOKEN and auth != f"Bearer {TOKEN}":13        raise HTTPException(401)14 15@app.post("/stt")16async def stt(request: Request, authorization: str | None = Header(None),17              x_language: str = Header("en")):18    check(authorization)19    with wave.open(io.BytesIO(await request.body())) as w:   # PCM16, mono, 8000 Hz20        pcm = w.readframes(w.getnframes())21    text = your_asr_model(pcm, sample_rate=8000, language=x_language)   # <- your model22    seconds = len(pcm) / 1600023    return {"text": text, "segments": [{"start": 0.0, "end": seconds, "text": text}]}24 25@app.post("/tts")26async def tts(body: dict, authorization: str | None = Header(None)):27    check(authorization)28    rate = int(body.get("sample_rate", 8000))29    pcm = your_tts_model(body["text"], voice=body.get("voice") or None,30                         sample_rate=rate)                             # <- PCM16 mono bytes31    out = io.BytesIO()32    with wave.open(out, "wb") as w:33        w.setnchannels(1); w.setsampwidth(2); w.setframerate(rate); w.writeframes(pcm)34    return Response(out.getvalue(), media_type="audio/wav")
Test your server before you connect itShell
1# the same check the dashboard makes when you save: a short text must come back as audio2curl -sS -X POST https://speech.yourbank.internal/tts \3  -H "Authorization: Bearer change-me" -H "Content-Type: application/json" \4  -d '{"text":"Test.","voice":"","sample_rate":8000}' -o test.wav && file test.wav5 6# recognition: send any 8 kHz mono WAV7curl -sS -X POST https://speech.yourbank.internal/stt \8  -H "Authorization: Bearer change-me" -H "Content-Type: audio/wav" -H "X-Language: en" \9  --data-binary @test.wav

Language model

ProviderAPI usedKey
OpenAIhttps://api.openai.com/v1Required
Google GeminiGoogle's OpenAI-compatible endpoint, …/v1beta/openaiRequired
Anthropic ClaudeThe Messages API, https://api.anthropic.comRequired
Custom (OpenAI-compatible)Your base URL: vLLM, Ollama, LM Studio, a gateway in front of your own model, Groq, OpenRouter …Optional
  • Model is the provider's model name, in letters, digits and . _ : / -.
  • Check on save: the dashboard calls GET {base}/models (for Anthropic, /v1/models); it must answer 2xx within 10 seconds.
  • Without an organization model, AI agents use the model configured for the platform's AI agent service, and call analysis and Write with AI use the platform's model, when the operator set them.

What a custom model server receives

The AI agent streams. It sends the agent's instructions as the system message and up to the last 12 turns of the conversation:

AI agent → your modelHTTP
1POST {base URL}/chat/completions2Authorization: Bearer {key}           # "none" when you left the key empty3Content-Type: application/json4 5{6  "model": "your-model",7  "stream": true,8  "max_tokens": 300,9  "temperature": 0.6,10  "messages": [11    { "role": "system", "content": "<the agent's instructions and knowledge>" },12    { "role": "user", "content": "I want to block my card" },13    { "role": "assistant", "content": "I can help with that. Is the card lost or stolen?" },14    { "role": "user", "content": "Stolen, this morning" }15  ]16}

Only choices[0].delta.content is read; the stream ends at data: [DONE]. The agent starts speaking at the first complete sentence, so a model that streams quickly sounds natural.

Call analysis and Write with AI send one request without streaming, waiting up to 60 seconds, and read the first choice's message:

Dashboard → your modelHTTP
1POST {base URL}/chat/completions2Authorization: Bearer {key}3Content-Type: application/json4 5{ "model": "your-model", "temperature": 0.2,6  "messages": [ { "role": "system", "content": "You review recorded customer-service phone calls ..." },7                { "role": "user",   "content": "Caller: ...\nAgent: ..." } ] }8 9-> { "choices": [ { "message": { "content": "{\"summary\": \"...\", ...}" } } ] }

AI agents

An AI agent is a voice the caller can talk to: it greets, answers from what you taught it, and ends the call politely. Each organization can have several, one per job. Open AI agents (under Configure) to create and edit them.

Dial codes and the default agent

  • Every agent has its own dial code from 8010 to 8099 (90 in all), given when it is created.
  • One agent is the default. It also answers 8001 and every destination set to Default AI agent. The default agent cannot be deleted; make another one the default first.
  • Callers reach an agent from a phone number's route (Numbers), a queue or ring group fallback, or an IVR AI agent block. The agent's page lists where it answers.

Use cases

A new agent starts from a use case, with wording you then edit. [Brackets] mark what to replace with your own facts.

Use caseWhat it does
ReceptionistGreets callers, answers common questions, takes a message
Bank helplineCards, loans and accounts, with strict security rules: never asks for a card number, PIN, OTP, password or CVV
Clinic appointmentsBooking requests, timings, doctors on duty; sends emergencies to the emergency number and gives no medical advice
University admissionsAdmissions, fees, deadlines, departments
Order status & deliveryWhere is my order, delivery times, returns
Payment remindersAnswers questions about dues and how to pay
After-hours messagesTakes a clear message when the office is closed
Customer surveyAsks a few short feedback questions
BlankStart from scratch

Settings

AI agent

Nameup to 80 charactersrequired
Only your team sees it.
Greetingup to 500 characters
The first thing the agent says. Empty uses the platform's greeting.Example: Welcome to Sunrise Bank customer support. This call may be recorded. How can I help you today?
Instructionsup to 8,000 characters
Its role, its tone and the rules it must never break. Short lines work best: on the phone it should answer in one or two sentences.
Knowledgeup to 30,000 characters
Facts it may tell callers: hours, branches, products, fees, policies. The agent is told to answer only from these and to say when it does not know.
Voicevoice name
A Deepgram Aura voice, or your speech server's voice name. Empty uses the organization's voice from Speech.
Languageoptional
Tells the agent to always speak this language unless the caller clearly prefers another. The voice must support it.
Goodbyeup to 300 characters
Said when the agent ends the call. Empty uses the platform's.

How the instructions reach the model

The agent's fields are joined into one system message, in this order:

System message for one callText
1<Instructions, as written on the agent>23Always speak Urdu, unless the caller clearly prefers another language.        <- when a Language is chosen45What you know about the organization (answer only from this; if something6is not here, say you don't know):7<Knowledge, as written on the agent>                                          <- when Knowledge is filled in89If the caller says goodbye or the conversation is finished, end your reply with [[HANGUP]].

How the AI agent handles a call

The AI agent is a Python service on the same server as Asterisk. Asterisk connects each AI call to it over AudioSocket, a plain TCP stream of 8 kHz, 16-bit audio in 20 ms frames. Before the greeting, the agent asks the dashboard (over the loopback interface) which organization and which agent the call is for, and gets that organization's keys and the agent's wording.

CallerCallerAsteriskAsteriskAI agentAI agentSpeechSpeechAI modelAI modelcalls 8010or an IVR AI block1AudioSocketcall id, then audio2fetches keys and wording from the dashboardgreeting audio4caller audio20 ms frames5700 ms of silence ends the turnutterance → text7transcript8instructions + historystream: true9tokens10first sentence → audio11audio framesnext sentences follow12caller talks over it for 240 ms: playback stops
One turn of conversation. The agent starts speaking at the first complete sentence, while the model is still writing the rest.

Listening

Voice activity detection measures the loudness of each 20 ms frame against a noise floor that adapts to the line. Speech starts after 100 ms above it; the turn ends after 700 ms of silence. Turns shorter than 300 ms (a cough, a click) are ignored, and a turn is cut at 15 seconds. 300 ms of audio before the start is kept so the first syllable is not lost.

Speaking

The model's answer is split into sentences as it streams in, and each sentence is voiced as soon as it is complete. With Deepgram, audio plays while it is still being generated; with other voices each sentence is synthesised in parallel and played in order. Answers are limited to 300 tokens.

Interruptions

  • Barge-in. When the caller talks over the agent for 240 ms, the agent stops speaking, drops the rest of its answer and listens. Pressing # also stops it speaking.
  • Without barge-in (an operator setting, for lines with strong echo) the agent ignores the caller while it speaks and for 500 ms after.

Ending the call

WhenWhat happens
The conversation is finishedThe model ends its reply with [[HANGUP]]; the agent speaks the reply, then hangs up.
The caller is silent for 40 sThe agent says its goodbye and hangs up.
The call reaches 15 minutesThe agent says its goodbye and hangs up.
Recognition fails“Sorry, I could not hear that. Could you repeat it?”
The model fails“Sorry, I'm having trouble right now. Please try again in a moment.”

The agent remembers the last 12 turns of the call. AI calls follow the organization's recording rules like any other call, so they can be transcribed and analysed too.

Operator settings

The listening and timing values above are the defaults. The operator can tune them for the whole installation:

avr.envConfig
1# /etc/voip-stack/avr.env (operator only; restart avr after a change)2AVR_VAD_END_SILENCE_MS=700     # silence that ends the caller's turn3AVR_BARGE_IN=yes               # the caller may interrupt the agent4AVR_BARGE_IN_MS=240            # how long they must talk over it5AVR_IDLE_SECONDS=40            # silence before the agent says goodbye6AVR_MAX_CALL_SECONDS=900       # longest AI call7AVR_MAX_CALLS=50               # calls at the same time

Call analysis

Call analysis turns every recorded call into a short report: what the caller wanted, how it ended and how they felt. It needs Speech and an AI model, and is switched on per organization.

  1. Switch it on

    In Integrations → Call analysis, turn it on and, if you like, list up to 50 keywords to watch for (each up to 40 characters), such as fraud, complaint, PTA, cancel.

  2. A recorded call ends

    Calls of at least 3 seconds with a recording are picked up about 30 seconds after they end. Calls are skipped when transcription is switched off for the organization or for the person who made or took the call (Call permissions).

  3. Transcribe and analyse

    The caller's side and your side are transcribed separately. The transcript (up to 12,000 characters) goes once to the language model, which returns a summary of at most two sentences, the caller's intent, one to three topics, sentiment (positive, neutral, negative), urgency (low, medium, high) and any of these flags: complaint, fraud, emergency, cancellation, escalation, technical issue, compliment. Keywords are found in the transcript as whole words, ignoring case. A failed step is retried.

Where the results show

PlaceWhat you see
The call's detailsThe summary, with the caller's mood, topics and flags.
AnalyticsWhat callers talk about: topics, caller mood, and a Needs attention list of unhappy or urgent callers, flagged issues and watched words.
REST APIA call's details include its transcript and summary.
Webhookstranscript.ready, call.analyzed and keyword.detected.
Data of a call.analyzed eventJSON
1{2  "recording": "<recording id>",3  "call_date": "2026-11-14 10:42:07",4  "from": "+923001234567",5  "to": "8201",6  "summary": "The caller reported a stolen debit card. The agent blocked it and ordered a replacement.",7  "intent": "block stolen card",8  "topics": ["card blocked", "card replacement"],9  "sentiment": "neutral",10  "urgency": "high",11  "flags": ["fraud"]12}

What is sent, and when

Nothing leaves your server except these requests, to the providers your organization chose. Recordings themselves stay on the server.

MomentSent to speechSent to the language model
A caller talks to an AI agentEach caller utterance (8 kHz audio) for recognition; each agent sentence (text) for the voiceThe agent's instructions and knowledge plus the conversation so far, streamed
An IVR is publishedThe text of each typed prompt, once; the audio is cached as a fileNothing
An IVR speaks an API answerThat sentence, once; the audio file is deleted after 10 minutesNothing
A recorded call is transcribedEach side of the call's audioNothing
Call analysisAs above, if not yet transcribedThe transcript text, once
Write with AINothingYour description and the agent's current wording

Next steps