AI & speech
Choose where speech recognition, the synthetic voice and the language model run, then build AI agents that answer the phone for each job: a bank helpline, clinic appointments, university admissions.
- Speech
- Deepgram or your own server
- Language model
- OpenAI, Gemini, Claude, compatible
- AI agents
- 8001 and 8010–8099
- Call audio
- 8 kHz, 20 ms frames
What uses what
Two services do all the AI work, and each organization chooses its own. Admins set them in Integrations, under AI and speech: Speech, AI model and Call analysis. Anything an organization leaves empty uses the platform's accounts, if the operator configured them.
| Feature | Speech | Language model |
|---|---|---|
| IVR prompts | Typed prompts are voiced once, when you publish, and cached | – |
| Speak from the answer (IVR) | A sentence from your API is voiced during the call | – |
| AI agents | Hear the caller (recognition) and speak (voice) | Decide what to say, streamed |
| Call transcripts | Each side of a recorded call is transcribed | – |
| Call analysis | Uses the transcript | Summary, intent, topics, sentiment, urgency, flags |
| Write with AI (agent editor) | – | Drafts an agent's greeting, instructions and knowledge |
Speech
| Choice | What you enter | Notes |
|---|---|---|
| Platform default | Nothing | The platform's Deepgram account, when the operator set one. |
| Deepgram | An API key, a voice and a recognition model | The key is checked with Deepgram before it is saved. Recognition defaults to nova-3, the voice to aura-2-thalia-en. Agents can pick other Aura voices: Thalia, Andromeda, Helena, Luna, Asteria, Apollo, Arcas, Aries, Orion, Zeus. |
| Your own speech server | Its base URL, an optional token and a voice name | For banks and hospitals that keep audio inside their network. It must implement the two endpoints below. |
Keys and tokens are stored encrypted and never shown again; leave the field empty to keep the stored one. Changes are audited and apply straight away: to the next transcript, the next AI call and the next IVR publish.
Your own speech server
Enter a base URL such as https://speech.yourbank.internal and, if your server wants one, a bearer token. The dashboard and the AI agent call two endpoints under it. The URL must use https and resolve to a public address, unless the installation sets WEBHOOK_ALLOW_PRIVATE=yes (on-premises installs), which allows private addresses and plain http.
Speech to text. One caller utterance from the AI agent, or one whole side of a recorded call for a transcript.
Bodyaudio/wavrequired- A WAV file: PCM 16-bit, mono, 8000 Hz.
AuthorizationheaderBearer <token>, empty afterBearerwhen you set no token.X-Languageheader- A language code such as
en. Always sent for transcripts (the installation'sTRANSCRIPT_LANGUAGE, defaulten); sent by the AI agent only when the operator fixed a language withAVR_LANGUAGE. Responseapplication/jsonrequiredtext, and optionallysegmentswithstartandendin seconds. Transcripts use the segments for timing; without them the whole text is one segment. The AI agent only readstext.
1POST /stt HTTP/1.12Host: speech.yourbank.internal3Authorization: Bearer <your token, empty when you set none>4Content-Type: audio/wav5X-Language: en6 7<WAV file: PCM 16-bit, mono, 8000 Hz>1{2 "text": "I want to block my card",3 "segments": [4 { "start": 0.42, "end": 1.96, "text": "I want to block my card" }5 ]6}Text to speech. IVR prompts, sentences from an API answer, and everything an AI agent says.
Bodyapplication/jsonrequiredtext,voice(the voice name set in Speech, or on the agent; may be empty) andsample_rate, always8000.AuthorizationheaderBearer <token>Responseaudio/wavrequired- PCM 16-bit mono at 8000 Hz. Raw PCM 16-bit without a WAV header is accepted too.
1POST /tts HTTP/1.12Host: speech.yourbank.internal3Authorization: Bearer <your token>4Content-Type: application/json5 6{ "text": "Welcome to Sunrise Bank.", "voice": "", "sample_rate": 8000 }1HTTP/1.1 200 OK2Content-Type: audio/wav3 4<WAV file: PCM 16-bit, mono, 8000 Hz> (raw PCM 16-bit without a header is accepted too)Time limits and retries
| Caller | Endpoint | Waits up to | Retries |
|---|---|---|---|
| Dashboard: call transcripts | /stt | 60 s | The background job retries a failed call later |
| Dashboard: IVR prompts, spoken answers | /tts | 30 s | None: publishing shows the error |
| AI agent: live calls | /stt and /tts | 30 s (AVR_HTTP_TIMEOUT) | Once, after network errors and 5xx answers; never after a 4xx |
On a live call every second counts: the caller hears silence while recognition and voice run. Host the server close to the phone system and aim for well under a second per request.
Check on save
When you save the connector, the dashboard sends /tts the text Test.. It must answer 2xx with audio within 10 seconds, or the settings are not saved.
Reference implementation
1# speech.py - a minimal speech server for the custom connector.2# pip install fastapi uvicorn3# SPEECH_TOKEN=change-me uvicorn speech:app --host 0.0.0.0 --port 8443 \4# --ssl-keyfile key.pem --ssl-certfile cert.pem5import io, os, wave6from fastapi import FastAPI, Header, HTTPException, Request, Response7 8TOKEN = os.environ.get("SPEECH_TOKEN", "")9app = FastAPI()10 11def check(auth: str | None):12 if TOKEN and auth != f"Bearer {TOKEN}":13 raise HTTPException(401)14 15@app.post("/stt")16async def stt(request: Request, authorization: str | None = Header(None),17 x_language: str = Header("en")):18 check(authorization)19 with wave.open(io.BytesIO(await request.body())) as w: # PCM16, mono, 8000 Hz20 pcm = w.readframes(w.getnframes())21 text = your_asr_model(pcm, sample_rate=8000, language=x_language) # <- your model22 seconds = len(pcm) / 1600023 return {"text": text, "segments": [{"start": 0.0, "end": seconds, "text": text}]}24 25@app.post("/tts")26async def tts(body: dict, authorization: str | None = Header(None)):27 check(authorization)28 rate = int(body.get("sample_rate", 8000))29 pcm = your_tts_model(body["text"], voice=body.get("voice") or None,30 sample_rate=rate) # <- PCM16 mono bytes31 out = io.BytesIO()32 with wave.open(out, "wb") as w:33 w.setnchannels(1); w.setsampwidth(2); w.setframerate(rate); w.writeframes(pcm)34 return Response(out.getvalue(), media_type="audio/wav")1# the same check the dashboard makes when you save: a short text must come back as audio2curl -sS -X POST https://speech.yourbank.internal/tts \3 -H "Authorization: Bearer change-me" -H "Content-Type: application/json" \4 -d '{"text":"Test.","voice":"","sample_rate":8000}' -o test.wav && file test.wav5 6# recognition: send any 8 kHz mono WAV7curl -sS -X POST https://speech.yourbank.internal/stt \8 -H "Authorization: Bearer change-me" -H "Content-Type: audio/wav" -H "X-Language: en" \9 --data-binary @test.wavLanguage model
| Provider | API used | Key |
|---|---|---|
| OpenAI | https://api.openai.com/v1 | Required |
| Google Gemini | Google's OpenAI-compatible endpoint, …/v1beta/openai | Required |
| Anthropic Claude | The Messages API, https://api.anthropic.com | Required |
| Custom (OpenAI-compatible) | Your base URL: vLLM, Ollama, LM Studio, a gateway in front of your own model, Groq, OpenRouter … | Optional |
- Model is the provider's model name, in letters, digits and
. _ : / -. - Check on save: the dashboard calls
GET {base}/models(for Anthropic,/v1/models); it must answer 2xx within 10 seconds. - Without an organization model, AI agents use the model configured for the platform's AI agent service, and call analysis and Write with AI use the platform's model, when the operator set them.
What a custom model server receives
The AI agent streams. It sends the agent's instructions as the system message and up to the last 12 turns of the conversation:
1POST {base URL}/chat/completions2Authorization: Bearer {key} # "none" when you left the key empty3Content-Type: application/json4 5{6 "model": "your-model",7 "stream": true,8 "max_tokens": 300,9 "temperature": 0.6,10 "messages": [11 { "role": "system", "content": "<the agent's instructions and knowledge>" },12 { "role": "user", "content": "I want to block my card" },13 { "role": "assistant", "content": "I can help with that. Is the card lost or stolen?" },14 { "role": "user", "content": "Stolen, this morning" }15 ]16}1data: {"choices":[{"delta":{"content":"I'm sorry to hear that."}}]}23data: {"choices":[{"delta":{"content":" Please block it now in the app,"}}]}45data: {"choices":[{"delta":{"content":" or press 1 in our main menu."}}]}67data: [DONE]Only choices[0].delta.content is read; the stream ends at data: [DONE]. The agent starts speaking at the first complete sentence, so a model that streams quickly sounds natural.
Call analysis and Write with AI send one request without streaming, waiting up to 60 seconds, and read the first choice's message:
1POST {base URL}/chat/completions2Authorization: Bearer {key}3Content-Type: application/json4 5{ "model": "your-model", "temperature": 0.2,6 "messages": [ { "role": "system", "content": "You review recorded customer-service phone calls ..." },7 { "role": "user", "content": "Caller: ...\nAgent: ..." } ] }8 9-> { "choices": [ { "message": { "content": "{\"summary\": \"...\", ...}" } } ] }AI agents
An AI agent is a voice the caller can talk to: it greets, answers from what you taught it, and ends the call politely. Each organization can have several, one per job. Open AI agents (under Configure) to create and edit them.
Dial codes and the default agent
- Every agent has its own dial code from
8010to8099(90 in all), given when it is created. - One agent is the default. It also answers
8001and every destination set to Default AI agent. The default agent cannot be deleted; make another one the default first. - Callers reach an agent from a phone number's route (Numbers), a queue or ring group fallback, or an IVR AI agent block. The agent's page lists where it answers.
Use cases
A new agent starts from a use case, with wording you then edit. [Brackets] mark what to replace with your own facts.
| Use case | What it does |
|---|---|
| Receptionist | Greets callers, answers common questions, takes a message |
| Bank helpline | Cards, loans and accounts, with strict security rules: never asks for a card number, PIN, OTP, password or CVV |
| Clinic appointments | Booking requests, timings, doctors on duty; sends emergencies to the emergency number and gives no medical advice |
| University admissions | Admissions, fees, deadlines, departments |
| Order status & delivery | Where is my order, delivery times, returns |
| Payment reminders | Answers questions about dues and how to pay |
| After-hours messages | Takes a clear message when the office is closed |
| Customer survey | Asks a few short feedback questions |
| Blank | Start from scratch |
Settings
AI agent
Nameup to 80 charactersrequired- Only your team sees it.
Greetingup to 500 characters- The first thing the agent says. Empty uses the platform's greeting.Example:
Welcome to Sunrise Bank customer support. This call may be recorded. How can I help you today? Instructionsup to 8,000 characters- Its role, its tone and the rules it must never break. Short lines work best: on the phone it should answer in one or two sentences.
Knowledgeup to 30,000 characters- Facts it may tell callers: hours, branches, products, fees, policies. The agent is told to answer only from these and to say when it does not know.
Voicevoice name- A Deepgram Aura voice, or your speech server's voice name. Empty uses the organization's voice from Speech.
Languageoptional- Tells the agent to always speak this language unless the caller clearly prefers another. The voice must support it.
Goodbyeup to 300 characters- Said when the agent ends the call. Empty uses the platform's.
How the instructions reach the model
The agent's fields are joined into one system message, in this order:
1<Instructions, as written on the agent>23Always speak Urdu, unless the caller clearly prefers another language. <- when a Language is chosen45What you know about the organization (answer only from this; if something6is not here, say you don't know):7<Knowledge, as written on the agent> <- when Knowledge is filled in89If the caller says goodbye or the conversation is finished, end your reply with [[HANGUP]].How the AI agent handles a call
The AI agent is a Python service on the same server as Asterisk. Asterisk connects each AI call to it over AudioSocket, a plain TCP stream of 8 kHz, 16-bit audio in 20 ms frames. Before the greeting, the agent asks the dashboard (over the loopback interface) which organization and which agent the call is for, and gets that organization's keys and the agent's wording.
Listening
Voice activity detection measures the loudness of each 20 ms frame against a noise floor that adapts to the line. Speech starts after 100 ms above it; the turn ends after 700 ms of silence. Turns shorter than 300 ms (a cough, a click) are ignored, and a turn is cut at 15 seconds. 300 ms of audio before the start is kept so the first syllable is not lost.
Speaking
The model's answer is split into sentences as it streams in, and each sentence is voiced as soon as it is complete. With Deepgram, audio plays while it is still being generated; with other voices each sentence is synthesised in parallel and played in order. Answers are limited to 300 tokens.
Interruptions
- Barge-in. When the caller talks over the agent for 240 ms, the agent stops speaking, drops the rest of its answer and listens. Pressing
#also stops it speaking. - Without barge-in (an operator setting, for lines with strong echo) the agent ignores the caller while it speaks and for 500 ms after.
Ending the call
| When | What happens |
|---|---|
| The conversation is finished | The model ends its reply with [[HANGUP]]; the agent speaks the reply, then hangs up. |
| The caller is silent for 40 s | The agent says its goodbye and hangs up. |
| The call reaches 15 minutes | The agent says its goodbye and hangs up. |
| Recognition fails | “Sorry, I could not hear that. Could you repeat it?” |
| The model fails | “Sorry, I'm having trouble right now. Please try again in a moment.” |
The agent remembers the last 12 turns of the call. AI calls follow the organization's recording rules like any other call, so they can be transcribed and analysed too.
Operator settings
The listening and timing values above are the defaults. The operator can tune them for the whole installation:
1# /etc/voip-stack/avr.env (operator only; restart avr after a change)2AVR_VAD_END_SILENCE_MS=700 # silence that ends the caller's turn3AVR_BARGE_IN=yes # the caller may interrupt the agent4AVR_BARGE_IN_MS=240 # how long they must talk over it5AVR_IDLE_SECONDS=40 # silence before the agent says goodbye6AVR_MAX_CALL_SECONDS=900 # longest AI call7AVR_MAX_CALLS=50 # calls at the same timeCall analysis
Call analysis turns every recorded call into a short report: what the caller wanted, how it ended and how they felt. It needs Speech and an AI model, and is switched on per organization.
Switch it on
In Integrations → Call analysis, turn it on and, if you like, list up to 50 keywords to watch for (each up to 40 characters), such as fraud, complaint, PTA, cancel.
A recorded call ends
Calls of at least 3 seconds with a recording are picked up about 30 seconds after they end. Calls are skipped when transcription is switched off for the organization or for the person who made or took the call (Call permissions).
Transcribe and analyse
The caller's side and your side are transcribed separately. The transcript (up to 12,000 characters) goes once to the language model, which returns a summary of at most two sentences, the caller's intent, one to three topics, sentiment (positive, neutral, negative), urgency (low, medium, high) and any of these flags: complaint, fraud, emergency, cancellation, escalation, technical issue, compliment. Keywords are found in the transcript as whole words, ignoring case. A failed step is retried.
Where the results show
| Place | What you see |
|---|---|
| The call's details | The summary, with the caller's mood, topics and flags. |
| Analytics | What callers talk about: topics, caller mood, and a Needs attention list of unhappy or urgent callers, flagged issues and watched words. |
| REST API | A call's details include its transcript and summary. |
| Webhooks | transcript.ready, call.analyzed and keyword.detected. |
1{2 "recording": "<recording id>",3 "call_date": "2026-11-14 10:42:07",4 "from": "+923001234567",5 "to": "8201",6 "summary": "The caller reported a stolen debit card. The agent blocked it and ordered a replacement.",7 "intent": "block stolen card",8 "topics": ["card blocked", "card replacement"],9 "sentiment": "neutral",10 "urgency": "high",11 "flags": ["fraud"]12}What is sent, and when
Nothing leaves your server except these requests, to the providers your organization chose. Recordings themselves stay on the server.
| Moment | Sent to speech | Sent to the language model |
|---|---|---|
| A caller talks to an AI agent | Each caller utterance (8 kHz audio) for recognition; each agent sentence (text) for the voice | The agent's instructions and knowledge plus the conversation so far, streamed |
| An IVR is published | The text of each typed prompt, once; the audio is cached as a file | Nothing |
| An IVR speaks an API answer | That sentence, once; the audio file is deleted after 10 minutes | Nothing |
| A recorded call is transcribed | Each side of the call's audio | Nothing |
| Call analysis | As above, if not yet transcribed | The transcript text, once |
| Write with AI | Nothing | Your description and the agent's current wording |