Arabic voice AI with an on-premise TTS pipeline

Context

Voice is the channel most customers in the region actually use, and Arabic is where most voice AI products are weakest. ZafarLabs wanted a customer-service voice assistant that felt native in Arabic and could be sold to clients who cannot use cloud speech services. That second requirement turns a product feature into a platform: the text-to-speech model has to be something we own, version, and deploy ourselves.

The constraint

Latency and language, at the same time. A voice conversation falls apart above roughly a second of round trip, and Arabic TTS quality varies wildly between models and voices. We needed a way to try models quickly, measure them on real Arabic prompts, and promote the winner into production, on cloud for some clients and fully on-premise for others.

How it works

The real-time path runs through a LiveKit agent: audio in over WebRTC, Whisper for speech to text, the LLM backend for the answer, and text to speech back to the caller. TTS serving is a swappable endpoint: a cloud voice for clients who allow it, an on-premise model for those who do not. Beside the real-time path sits the MLOps loop: experiments produce candidate TTS models, MLflow records them with their Arabic evaluation scores, and the registry is what production serving pulls from.

What I did

I built the end-to-end TTS pipeline: dataset handling, experiment tracking, evaluation on an Arabic prompt set with both automatic metrics and listening tests, model versioning, and inference deployment on Kubernetes with KServe. I built the real-time audio path and the monitoring for latency, drift and accuracy, and containerised every model so the same images run in the cloud and on a client’s rack.

I worked directly with the founder on this one. The conversations were about what we were selling, not how: the decision to make on-premise a first-class path rather than a special case came out of one of them, after a client meeting where the cloud version was a non-starter.

Outcome

Arabic-language customer interactions are handled end to end by the assistant, which reduced dependence on human agents for routine calls. New TTS candidates go from experiment to production through one pipeline instead of a manual hand-off, and the on-premise path is what unlocked clients who could not use cloud speech at all.