A Bangalore CA practice with 280 small-business clients was drowning in inbound calls about Tally support — license renewal, GST return queries, "data corrupt ho gaya," password reset. Two staff members handled 90+ calls a day; both were at burnout. We built them a voice IVR over a weekend that routes by language (Hindi, Tamil, English), classifies intent, creates a Tally Helpdesk ticket, and only escalates the genuinely complex cases. Stack: Twilio India, Sarvam Bulbul-v2, faster-whisper, Claude Sonnet 4.5. This post is the end-to-end build with real streaming + barge-in code.
## TL;DR — what this post delivers
A voice IVR for an SMB helpdesk that handles trilingual routing (Hindi/Tamil/English), STT-based intent classification (no DTMF press-1-for-X menus), ticket creation in Tally Helpdesk via REST, and human warm-transfer for complex cases. Streaming TTS + barge-in (caller can interrupt the bot). Built on Twilio Programmable Voice (₹0.35/min India inbound), Sarvam Bulbul-v2 for Hindi/Tamil TTS, faster-whisper medium for STT, Claude Sonnet 4.5 for the brain. Weekend build, 73% resolution rate, ₹2.80 per call.
## Why a voice IVR (and not just WhatsApp)
Two reasons. First, the CA practice's clients are mostly small-business owners aged 35-65 — many prefer phone calls over typing on WhatsApp, especially when stressed about a Tally crash. Second, voice IVR lets you start with intent classification before forcing the caller down a menu tree. The traditional "press 1 for renewal, press 2 for support" gets abandoned at 32% in our testing on Indian SMB clients; an open-ended "what brings you to us today?" with intent classification gets answered at 91%.
## The full call flow
## The streaming + barge-in code (Python)
This is the part most IVR tutorials skip. Without barge-in, the caller can't interrupt the bot — and on a slow Indian-English-speaking bot, that's frustrating fast. With streaming, the bot speaks as Claude generates, not after.
### Twilio TwiML for media streaming
### The Python WebSocket handler (FastAPI)
### The voice pipeline (with barge-in)
Three details that matter. The barge-in cancellation happens AT THE FIRST detected speech-frame, not after the user finishes — that's what makes the bot feel polite ("oh, you wanted to say something"). The TTS streaming flushes at sentence boundaries (using
## The Tally Helpdesk integration
Tally Solutions exposes a REST API for ticket creation through their TallyPrime Server. We POST a JSON payload with caller name (looked up from phone number against the practice's CRM), intent, transcript, and detected language.
## The build (hour by hour)
## Pre-launch checklist
280
Clients on the CA practice's books
73%
Calls fully resolved by IVR (no human)
₹2.80
Cost per 75-second IVR call
2 days
Weekend build time
Step 1 — Greeting + language detect
"Hello, [practice name] mein swagat hai. Aap kaisi madad chahte hain?" — bilingual opener. Caller's first 3 seconds tell us their preferred language.
Step 2 — Intent classification
Whisper transcribes the caller's free-form statement. Claude classifies into one of 8 intents: license_renewal, gst_query, data_corrupt, password_reset, install_help, e_invoice, training, other.
Step 3 — Resolution OR ticket
Bot answers if it's a known FAQ (license renewal steps, password reset link). Otherwise creates a Tally Helpdesk ticket with full context.
Step 4 — Human transfer (when needed)
For "data corrupt" or any explicit "agent" request, warm-transfer to the human staff with a brief context summary. Average transfer time: 8 seconds.
xml
python
from fastapi import FastAPI, WebSocket
import asyncio
import base64
from voice_pipeline import VoicePipeline
app = FastAPI()
@app.websocket('/twilio-stream')
async def twilio_stream(ws: WebSocket):
await ws.accept()
pipeline = VoicePipeline()
try:
async for msg in ws.iter_json():
if msg['event'] == 'start':
pipeline.call_sid = msg['start']['callSid']
pipeline.from_number = msg['start']['customParameters']['from']
asyncio.create_task(pipeline.run(ws))
elif msg['event'] == 'media':
# Inbound audio chunk from caller
audio_chunk = base64.b64decode(msg['media']['payload'])
await pipeline.on_audio_in(audio_chunk)
elif msg['event'] == 'stop':
await pipeline.shutdown()
break
except Exception as e:
print(f'Stream error: {e}')
await pipeline.shutdown()python
import asyncio
from typing import Optional
from anthropic import AsyncAnthropic
from sarvam_client import SarvamTTS
from whisper_client import StreamingWhisper
class VoicePipeline:
def __init__(self):
self.whisper = StreamingWhisper(model='medium', languages=['hi', 'ta', 'en'])
self.tts = SarvamTTS(voice='manan')
self.claude = AsyncAnthropic()
self.tts_task: Optional[asyncio.Task] = None
self.user_speaking = False
async def on_audio_in(self, audio_chunk: bytes):
# VAD: is the user speaking?
if self.whisper.is_speech(audio_chunk):
if not self.user_speaking:
self.user_speaking = True
# BARGE-IN: cancel any in-flight TTS
if self.tts_task and not self.tts_task.done():
self.tts_task.cancel()
print('Bot interrupted by caller')
await self.whisper.feed(audio_chunk)
async def run(self, ws):
# Greeting
await self.speak(ws, 'Namaste, Tally Helpdesk mein swagat hai. Aapki kya samasya hai?')
while True:
# Wait for caller to finish speaking
transcript = await self.whisper.get_completed_utterance()
if not transcript:
continue
self.user_speaking = False
detected_lang = self.whisper.last_detected_language
# Classify intent + generate reply
intent = await self.classify_intent(transcript, detected_lang)
if intent == 'transfer_to_human':
await self.transfer(ws, summary=transcript)
break
reply_stream = await self.generate_reply(transcript, intent, detected_lang)
# Stream TTS while Claude generates
self.tts_task = asyncio.create_task(self.stream_tts(ws, reply_stream, detected_lang))
try:
await self.tts_task
except asyncio.CancelledError:
print('TTS cancelled by barge-in')
async def stream_tts(self, ws, text_stream, language: str):
buffer = ''
async for chunk in text_stream:
buffer += chunk
# Flush at sentence boundary for natural cadence
if any(p in buffer for p in '.!?।'):
audio = await self.tts.synthesize(buffer, language=language)
await self.send_audio(ws, audio)
buffer = ''
if buffer:
audio = await self.tts.synthesize(buffer, language=language)
await self.send_audio(ws, audio)
async def speak(self, ws, text: str, language: str = 'hi'):
audio = await self.tts.synthesize(text, language=language)
await self.send_audio(ws, audio)
async def send_audio(self, ws, audio_bytes: bytes):
import base64
payload = base64.b64encode(audio_bytes).decode()
await ws.send_json({
'event': 'media',
'streamSid': self.stream_sid,
'media': {'payload': payload},
})। for Hindi punctuation) — buffering until the full Claude response would add 1-2 seconds of perceived delay. The language detection runs continuously through Whisper, so a caller switching from Hindi to English mid-call gets the right TTS voice.
## The intent classifier (Claude prompt)
python
INTENT_PROMPT = """You are an intent classifier for a Tally Helpdesk voice bot.
The caller said: "{transcript}"
Caller's language: {language}
Classify into ONE of these intents:
- license_renewal: questions about Tally license renewal, expiry, activation
- gst_query: GST return preparation, GSTR filing, GST configuration in Tally
- data_corrupt: Tally data file corruption, "data not opening", recovery requests
- password_reset: forgotten password, lock/unlock issues
- install_help: installation problems, version upgrade, setup
- e_invoice: e-invoice generation, IRN issues, GSTN integration
- training: requests for Tally training, "kaise sikhayenge"
- transfer_to_human: explicit request for human, "agent", "manager", complex multi-issue
- other: doesn't fit above categories
Respond with ONLY the intent name. No explanation.
"""
async def classify_intent(self, transcript: str, language: str) -> str:
response = await self.claude.messages.create(
model='claude-sonnet-4-5',
max_tokens=20,
messages=[{'role': 'user', 'content': INTENT_PROMPT.format(
transcript=transcript, language=language,
)}],
)
intent = response.content[0].text.strip()
return intent if intent in VALID_INTENTS else 'other'python
import httpx
async def create_tally_ticket(self, intent: str, transcript: str, language: str):
async with httpx.AsyncClient() as client:
# Lookup caller from CRM
customer = await self.lookup_customer_by_phone(self.from_number)
response = await client.post(
'https://tally-helpdesk.softechinfra.com/api/tickets',
headers={'Authorization': f'Bearer {os.environ["TALLY_TOKEN"]}'},
json={
'customer_id': customer['id'] if customer else None,
'phone': self.from_number,
'intent_category': intent,
'description': transcript,
'language': language,
'source': 'voice_ivr',
'priority': 'high' if intent == 'data_corrupt' else 'normal',
'call_sid': self.call_sid,
},
)
ticket = response.json()
return ticket['ticket_id']1
Saturday morning (4 hrs) — Twilio + WebSocket scaffolding
Buy Twilio India local number (₹110/month, KYC takes 2-5 days — start mid-week). Configure TwiML for media streaming. Stand up FastAPI WebSocket handler on Hetzner CCX13. Test with a hardcoded "hello" reply.
2
Saturday afternoon (4 hrs) — Whisper + Sarvam pipeline
Spin up faster-whisper medium on a Lambda Labs L4 (₹26/hour spot). Wire Sarvam Bulbul-v2 for TTS. Test full STT → TTS round trip with a static sentence. Initial p50: 1,400ms — too slow.
3
Saturday evening (3 hrs) — Streaming + barge-in
Add streaming TTS (flush at sentence boundary). Add VAD-triggered barge-in (cancel TTS task on first detected user speech). p50 drops to 1,100ms; conversation feels natural.
4
Sunday morning (3 hrs) — Intent classifier + Tally integration
Wire Claude Sonnet 4.5 intent classification with the 9-intent taxonomy. Build the customer-lookup-by-phone against the practice's existing CRM. POST tickets to Tally Helpdesk API. Smoke-test with real-sounding queries.
5
Sunday afternoon (3 hrs) — FAQ resolution paths
For 4 of the 9 intents (license_renewal, password_reset, install_help, training), the bot can resolve directly with a scripted answer + SMS link. Wire each FAQ flow with a Claude-generated dynamic response.
6
Sunday evening (2 hrs) — Human warm-transfer + go-live
Twilio Dial verb to warm-transfer the call to staff mobile, with a brief Claude-generated context summary played to the staff before bridging. Run 8 end-to-end test calls from different phones. Ship at 8pm.
- Twilio India number with Mumbai SIP termination requested (free, 24h)
- HTTPS + valid SSL on the WebSocket endpoint (mandatory for Twilio Streams)
- Whisper VAD threshold tuned to local accent (380-450ms silence works for India)
- TTS streaming flushes at sentence boundary, not full-response
- Barge-in cancels TTS task at first detected user speech
- Intent classifier prompt includes "transfer_to_human" as explicit option
- Customer lookup by phone before ticket creation
- Recording disclosure played at call start (TRAI compliance)
- Warm-transfer plays context summary to staff before bridging
- Smoke test from at least 5 carrier+device combinations before go-live
<Gather> primitive handles DTMF and we fall back to a 4-option menu.
### What's the resolution rate floor for this approach?
In our experience across 6 client builds, intent-based IVRs land at 60-78% resolution depending on the FAQ coverage and the callers' familiarity with the bot. The CA practice hit 73% with 4 FAQ-resolved intents. A practice with 8 FAQ-resolved intents would likely hit 80%+.
### Can I deploy this on-prem instead of Hetzner cloud?
Yes. Replace Hetzner with any Linux box that has 4GB RAM and a public-routable HTTPS endpoint. The L4 GPU for Whisper is the only hard requirement — and you can swap Whisper for OpenAI's hosted STT (slightly slower, 200ms more latency) if you can't run a GPU.
### How do you handle accents the bot consistently mishears?
Add the misheard transcripts as training data for prompt-level disambiguation. We track Whisper outputs that human staff later corrected, then add a "common variations" section to the intent prompt: "Note: 'jeem-essay-tee' usually means GST." Improves classification by 3-5pp without retraining the STT model.
### What does the recording disclosure say?
"Yeh call recording aur quality monitoring ke liye record ho rahi hai. Agar aap ise opt out karna chahte hain, kripya 9 dabayein." TRAI requires the disclosure; the opt-out path is best practice. Recordings stored in S3 ap-south-1 with 90-day retention.
Want a Voice IVR for Your CA Practice or Accounting Software Helpdesk?
We ship voice IVRs for Indian SMB helpdesks (Tally, Zoho Books, BUSY, custom billing apps) in 7-14 working days. Trilingual routing, intent classification, automatic ticket creation, warm transfer to your team. Typical project: ₹65,000-₹1,40,000 fixed scope. Per-call run cost from ₹2.10. First call is technical — with the engineer who would lead your build.
Book a 20-min CallTags:
Voice AIIVRTwilioSarvam AIWhisperTallyHelpdeskIndian SMB
