Design and Implementation of an AI-Based Multi-Character Audiobook Performance System

Abstract

Producing a high-quality, multi-character audiobook usually requires a professional studio and a team of voice actors, making it too expensive for individual authors. We developed EnsembleCast to bridge this gap by automating the entire pipeline from text analysis to final audio synthesis. Our system acts as an automated script reader, using large language models to identify character identities and emotions directly from the story. To keep the narration consistent, we implemented a custom matching algorithm that assigns voices based on character traits like age and gender. By keeping the heavy voice generation (IndexTTS) on the local machine, we ensure user privacy while achieving near real-time performance on standard consumer hardware.