# xtts

- Page: https://shared.btbytes.com/p/xtts
- Date: 2023-09-22
- Tags: xtts

Major Open Source Engine is Coqui. Others to consider -- Whisper, and Piper.


[XTTS model weights from Coqui](https://x.com/coqui_ai/status/1702369159550529863)  - [coqui-ai/TTS: 🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production](https://github.com/coqui-ai/tts)

[KoljaB/RealtimeTTS: Converts text to speech in realtime by identifying sentence fragments for immediate auditory feedback. Ideal for applications requiring instant audio responses.](https://github.com/KoljaB/RealtimeTTS)

[yl4579/StyleTTS2: StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://github.com/yl4579/StyleTTS2)

Related to above: [StyleTTS 2 | llm-tracker](https://llm-tracker.info/books/howto-guides/page/styletts-2) and a [shootoff](https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2); [via](https://news.ycombinator.com/item?id=38335255)


[vits](https://huggingface.co/docs/transformers/main/model_doc/vits) -- [via](https://x.com/reach_vb/status/1729620412965716071) "model is 40M parameter and 150MB in size, and works on-CPU runtime" <a id="vits" />

([Kim et al. 2021](https://arxiv.org/abs/2106.06103))

> It does its job for most on-device use cases like reading an article, practicing a language, etc.!! Here's how you can use it with Transformers !

Set up your environment: `pip install transformers accelerate phonemizer`

Initialize the model:

```python
import torch
from transformers import VitsModel, AutoTokenizer
model = VitsModel.from_pretrained( "kakao-enterprise/vits-vctk") tokenizer = AutoTokenizer.from_pretrained( "kakao-enterprise/vits-vctk")
# Pass the text you'd like to synthesise:
text = "Hey, it's Max the best doggo speaking!"
inputs = tokenizer(text, return_tensors="pt")

# Generate audio
with torch. no_grad():
    output = model(**inputs).waveform[0]
```

> Bonus: you'd soon be able to fine-tune them in your voice/ dataset too!


## Indian language TTS

... specially i'm interested in SOTA, Open Source, widely available Kannada models. If not, there is an opportunity to develop one.

[Indic TTS - Synthesis Docs](http://web.archive.org/web/20160611101746/http://www.iitm.ac.in/donlab/tts/synthDocs.php)
github - [AI4Bharat/Indic-TTS: Text-to-Speech for languages of India](https://github.com/AI4Bharat/Indic-TTS)

[Kannada (ಕನ್ನಡ) Text To Speech (TTS) Demo](https://hear2read.org/kannada-text-to-speech-demo)

[Bhashini Neural TTS](http://web.archive.org/web/20231207170429/https://tts.bhashini.ai/tts/)

<a href="#wlog-2023-12-06"><date class="wlogdate" id="wlog-2023-12-06">2023-12-06</date></a> I have asked [Mozilla Commonvoice](https://commonvoice.mozilla.org/en) to add Kannada Language Option.


## Books

[Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/) -- by  [Dan Jurafsky](http://web.stanford.edu/people/jurafsky/) and [James H. Martin](http://www.cs.colorado.edu/~martin/) -- <q>An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition</q>


## Teams

- Sesame -- [Crossing the uncanny valley of conversational voice](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice)

## References

- Kim, Kong, Son (2021). [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://arxiv.org/abs/2106.06103).