Back to Projects
Done

Voice Cloning Demo

AITTSVoice CloningDockerPythonFastAPISelf-Hosted

This week I set a small personal goal: to clone my own voice and use it for text-to-speech. It turned out to be much harder than I expected 😅

My main requirements were:

  • should work on CPU
  • should be self-hosted
  • should need only 5-10 seconds of my voice for training

I tested several models and now I really understand why people buy powerful laptops 😂 Still, it was a great learning experience and in some cases the generated voice sounded surprisingly good.

Technical Implementation

The project uses a multi-worker architecture with isolated workers for different TTS models, all managed by supervisord in a single Docker container:

  • Gateway (FastAPI): Handles HTTP requests and job distribution
  • Coqui Worker: High-quality voice cloning
  • NeuTTS Worker: Advanced voice synthesis
  • Communication: File-based job queues

Model Comparison

Coqui XTTS2

The best results came from Coqui - it worked almost right away. The XTTS2 model is incredibly impressive and produces very natural-sounding voices that closely match the original. Even on CPU, the quality is outstanding.

NeuTTS Air

NeuTTS gives good results but somewhat unpredictable. Sometimes it produces excellent output, other times it needs multiple attempts. Still a solid option for experimentation.

Performance

GPU would definitely help with generation speed, but CPU-only setup is surprisingly viable. On MacBook Air M1 with 16GB RAM:

  • Initial setup: 3-10 minutes
  • Voice generation: 3-10 minutes per sample

Challenges

The most frustrating part was dealing with dependency conflicts. Different models require incompatible versions of NumPy and other libraries. Solved this by isolating workers in separate environments within the Docker container.

Try It Yourself

I shared my code on GitHub in case you want to experiment too: github.com/garfik/voice-cloning-demo

AI is an exciting field - always something new to try and learn.