{"id":415482,"date":"2024-06-30T00:23:51","date_gmt":"2024-06-30T00:23:51","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=415482"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=415482","title":{"rendered":"<span>Multilingual Text-to-Speech Models for Indic Languages<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/d11\/ce4\/869\/d11ce4869a574fbbb023e43b1f0db9a6.jpg\" width=\"2285\" height=\"1285\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/d11\/ce4\/869\/d11ce4869a574fbbb023e43b1f0db9a6.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>In this article, we shall provide some background on how multilingual multi-speaker models work and test an Indic\u00a0<a href=\"https:\/\/github.com\/snakers4\/silero-models#indic-languages\" rel=\"noopener noreferrer nofollow\">TTS model<\/a>\u00a0that supports 9 languages and 17 speakers (Hindi, Malayalam, Manipuri, Bengali, Rajasthani, Tamil, Telugu, Gujarati, Kannada).<\/p>\n<p>It seems a bit counter-intuitive at first that one model can support so many languages and speakers provided that each Indic language has its own alphabet, but we shall see how it was implemented.<\/p>\n<p>Also, we shall list the specs of these models like supported sampling rates and try something cool \u2013 making speakers of different Indic languages speak Hindi. Please, if you are a native speaker of any of these languages, share your opinion on how these voices sound, both in their respective language and in Hindi.<\/p>\n<h3>Basic Background on Text-to-Speech<\/h3>\n<p>Text-to-speech\u00a0(<strong>TTS<\/strong>) is a broad subject, but we need to get a basic understanding of how it works in general or what are the main components.<\/p>\n<p>Unlike more traditional TTS models that relied on specific linguistic information as inputs, modern TTS models usually work with text or phoneme inputs. The main components of most modern TTS systems are (even newer fully end-to-end models still have similar components inside):<\/p>\n<ul>\n<li>\n<p>Text preprocessing module and cleaning plus some form of text to phoneme transcription, transcription, or transliteration;<\/p>\n<\/li>\n<li>\n<p>An acoustic model;<\/p>\n<\/li>\n<li>\n<p>A vocoder;<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/cf5\/b54\/e8e\/cf5b54e8ea4fc70170ba1812100df933.png\" width=\"692\" height=\"838\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/cf5\/b54\/e8e\/cf5b54e8ea4fc70170ba1812100df933.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Modern TTS models typically rely on curated graphemes or phonemes as input. This input is then passed into an\u00a0<a href=\"https:\/\/medium.com\/analytics-vidhya\/understanding-embedding-layer-in-keras-bbe3ff1327ce\" rel=\"noopener noreferrer nofollow\">embedding layer<\/a>\u00a0that maps this alphabet to a set of tensors.<\/p>\n<p>Usually,\u00a0<a href=\"https:\/\/medium.com\/analytics-vidhya\/understanding-the-mel-spectrogram-fca2afa2ce53\" rel=\"noopener noreferrer nofollow\">Mel Spectrograms<\/a>\u00a0are chosen as a modelling unit for the acoustic model and the vocoder. In simple terms, the acoustic model transfers input text embeddings into a Mel Spectrogram, which the vocoder turns into the actual audio.<\/p>\n<p>There is a whole plethora of different approaches and architectures for vocoders and acoustic models (please see the paper below for a comprehensive review), but this is out of scope for this article.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/af5\/c8e\/092\/af5c8e09220278843e5572fe9fa0baa0.png\" alt=\"Source:\u00a0http:\/\/arxiv.org\/abs\/2106.15561\" title=\"Source:\u00a0http:\/\/arxiv.org\/abs\/2106.15561\" width=\"1155\" height=\"1274\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/af5\/c8e\/092\/af5c8e09220278843e5572fe9fa0baa0.png\"\/><figcaption>Source:\u00a0http:\/\/arxiv.org\/abs\/2106.15561<\/figcaption><\/figure>\n<p>Each Indic language has its own alphabet and basically, there are only 2 ways we can combine them into one unified model (of course we also can train several distinct models):<\/p>\n<ul>\n<li>\n<p>Use some transliteration or transcription scheme;<\/p>\n<\/li>\n<li>\n<p>Combine the alphabets together and increase the number of embeddings;<\/p>\n<\/li>\n<\/ul>\n<p>The second approach is a bit problematic because the phonemes are very similar in closely related languages using such an approach may hurt the model\u2019s generalization and convergence.<\/p>\n<p>Also, each speaker should get his or her own embedding. As for the languages \u2013 it does not make sense to include language embeddings in case the number of speakers is not much larger than the number of languages.<\/p>\n<h3>How does it Work?<\/h3>\n<p>The secret sauce is that the model uses ISO romanization techniques supported by a widely used\u00a0<a href=\"https:\/\/aksharamukha.appspot.com\/converter\" rel=\"noopener noreferrer nofollow\">aksharamukha tool<\/a>\u00a0and its python\u00a0<a href=\"https:\/\/aksharamukha.appspot.com\/python\" rel=\"noopener noreferrer nofollow\">package<\/a>. This process is a transliteration (not a transcription) and some language\u2019s letters have a bit different sounds, but this works well enough as you will see.<\/p>\n<p><strong>To use this tool for all of these languages, for example using python, we need to:<\/strong><\/p>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<td>\n<p align=\"left\"><strong>Language<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>Romanization function<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">hindi<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Devanagari\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">malayalam<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Malayalam\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">manipuri<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Bengali\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">bengali<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Bengali\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">rajasthani<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Devanagari\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">tamil<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Tamil\u2019, \u2018ISO\u2019, orig_text, pre_options=[\u2018TamilTranscribe\u2019])<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">telugu<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Telugu\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">gujarati<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Gujarati\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">kannada<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Kannada\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p><strong>After converting the input text to ISO, now we can try the model:<\/strong><\/p>\n<pre><code class=\"python\">import torch from aksharamukha import transliterate  model, example_text = torch.hub.load(repo_or_dir='snakers4\/silero-models',   model='silero_tts',                                      language='indic',                                      speaker='v3_indic')  orig_text = \"\u092a\u094d\u0930\u0938\u093f\u0926\u094d\u0926 \u0915\u092c\u0940\u0930 \u0905\u0927\u094d\u092f\u0947\u0924\u093e, \u092a\u0941\u0930\u0941\u0937\u094b\u0924\u094d\u0924\u092e \u0905\u0917\u094d\u0930\u0935\u093e\u0932 \u0915\u093e \u092f\u0939 \u0936\u094b\u0927 \u0906\u0932\u0947\u0916, \u0909\u0938 \u0930\u093e\u092e\u093e\u0928\u0902\u0926 \u0915\u0940 \u0916\u094b\u091c \u0915\u0930\u0924\u093e \u0939\u0948\" roman_text = transliterate.process('Devanagari', 'ISO', orig_text) print(roman_text)  audio = model.apply_tts(roman_text,                          speaker='hindi_male')<\/code><\/pre>\n<p>The created audio is a PyTorch Tensor containing a 48 kHz (the default sampling rate) audio with the generated phrase. Of course, lower sampling rates are also available.<\/p>\n<p>The romanization is a lossless process, and for the majority of texts that I have tried, it\u00a0<strong>can be transliterated\u00a0<\/strong><a href=\"https:\/\/aksharamukha.appspot.com\/python\" rel=\"noopener noreferrer nofollow\"><strong>back and forth<\/strong><\/a><strong>\u00a0without any errors using the\u00a0aksharamukha tool<\/strong>.<\/p>\n<h3>Model Performance and Characteristics<\/h3>\n<p>After a bit of fiddling, prodding, and testing we can see that the model:<\/p>\n<ul>\n<li>\n<p>Can generate audio with a sampling rate of 8 000, 24 000, or 48 000 kHz;<\/p>\n<\/li>\n<li>\n<p>Works both on CPUs and GPUs. Also, it works even on old and outdated x86 CPUs without AVX2 instructions;<\/p>\n<\/li>\n<li>\n<p>Weighs about 50 MB for all speakers and languages (yes, really!);<\/p>\n<\/li>\n<\/ul>\n<p>If we properly measure the model speed on 1 and 4 CPU threads (turns out adding 6 or more threads does not really help), we can see that it quite quick (I measured the seconds of audio generated per second):<\/p>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<td>\n<p align=\"left\"><strong>Sampling Rate<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>1 CPU Thread<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>4 CPU Threads<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">8 kHz<\/p>\n<\/td>\n<td>\n<p align=\"left\">15 \u2014 25<\/p>\n<\/td>\n<td>\n<p align=\"left\">30 \u2014 60<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">24 kHz<\/p>\n<\/td>\n<td>\n<p align=\"left\">10<\/p>\n<\/td>\n<td>\n<p align=\"left\">15 \u2014 20<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">48 kHz<\/p>\n<\/td>\n<td>\n<p align=\"left\">5<\/p>\n<\/td>\n<td>\n<p align=\"left\">10<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p>The model also supports\u00a0<a href=\"https:\/\/github.com\/snakers4\/silero-models\/wiki\/SSML\" rel=\"noopener noreferrer nofollow\">SSML markdown<\/a>, but it is still a bit glitchy for some tags.<\/p>\n<h3>Audio Samples<\/h3>\n<p>Now let\u2019s generate some audio samples for each of the languages and listen to them:<\/p>\n<div class=\"tm-iframe_temp\" data-src=\"https:\/\/embedd.srv.habr.com\/iframe\/62bd96194122e46238f233b6\" data-style=\"\" id=\"62bd96194122e46238f233b6\" width=\"\"><\/div>\n<p>I am not a native speaker of these languages to evaluate the speech quality, but as far as TTS models go, most of these speakers sound passable. The most obvious problems can be traced to source material \u2013 mostly slow pace of speech and audio quality (some audios were noisy or upsampled from lower quality).<\/p>\n<p>I asked a native speaker from India (he speaks Hindi, Telugu,\u00a0Tamil and understands\u00a0Malayalam,\u00a0Bengali,\u00a0Kannada and\u00a0Gujarati\u00a0 to a certain extent) to evaluate the anecdotal quality, and this is what he replied:<\/p>\n<ul>\n<li>\n<p><strong>Hindi<\/strong>. The pronunciations were accurate. It sounded a bit mechanical (like reading out phrase by phrase), probably because that\u2019s how the speakers spoke it in the dataset;<\/p>\n<\/li>\n<li>\n<p><strong>Telugu\u00a0<\/strong>was perfect. Was natural and smooth with accurate pronunciations;<\/p>\n<\/li>\n<li>\n<p><strong>Tamil<\/strong>. The output could have been a bit smoother if it was generated for a single sentence. Nevertheless, it was still good;<\/p>\n<\/li>\n<li>\n<p><strong>Malayalam<\/strong>. The pauses and phrasing were somewhat uncomfortable to hear. Female output was better, probably because it has a bit higher speed;<\/p>\n<\/li>\n<li>\n<p><strong>Bengali\u00a0<\/strong>was good. The female sample was perfect;<\/p>\n<\/li>\n<li>\n<p><strong>Kannada. The male<\/strong>\u00a0sample was perfect. The female one had awkward pauses;<\/p>\n<\/li>\n<li>\n<p><strong>Gujarati\u00a0<\/strong>\u2013 seemed too mechanical;<\/p>\n<\/li>\n<\/ul>\n<h3>Non Hindi Speakers Speaking Hindi<\/h3>\n<p>Now for something extra. Let\u2019s see what happens if non-Hindi speakers are asked to speak Hindi!<\/p>\n<p>Take a listen:\u00a0<\/p>\n<div class=\"tm-iframe_temp\" data-src=\"https:\/\/embedd.srv.habr.com\/iframe\/62bd95ebb6b9a8627a8d4da9\" data-style=\"\" id=\"62bd95ebb6b9a8627a8d4da9\" width=\"\"><\/div>\n<p><strong>And here is what a native speaker thinks:<\/strong><\/p>\n<ul>\n<li>\n<p>In general, when we use a speaker of another language Y for an input text language X, it does not sound natural. And it also loses the accent of Y when in such cases. But neither does it have the accent of X;<\/p>\n<\/li>\n<li>\n<p>It was a plain read-out of the roman text using the voice of another speaker. Although the basic voice characteristics were maintained;<\/p>\n<\/li>\n<li>\n<p>But it was accurately able to pronounce some phonemes of X which they would have never encountered for Y. So definitely there\u2019s sharing of phonetics across languages, wherever non-ambiguous;<\/p>\n<\/li>\n<li>\n<p>The model is definitely useful when using appropriate languages for the respectively trained speakers;<\/p>\n<\/li>\n<li>\n<p>More precisely, it sounds like a person of native language Y trying hard to fluently read the script of language X, without knowing the language;<\/p>\n<\/li>\n<\/ul>\n<p>And what do you think? Do these conclusions hold? It is next to impossible for me to make these judgments apart from observing and fixing standard TTS problems shared by all non-tonal languages.<\/p>\n<h3>Conclusion<\/h3>\n<p>In this article, we have briefly touched upon the subject of how TTS models work and have shown that the ISO romanization scheme (provided by the aksharamukha tool)\u00a0 can serve as a decent foundation to build a universal Indic TTS model, that not only can \u201cspeak\u201d several Indic languages, but also can make speakers \u201cspeak\u201d a language that they originally did not even record.<\/p>\n<p>This did not work for the Assamese language, because some of its romanized phonemes sounded too different compared to other languages (according to our non-native ear).<\/p>\n<p>Also, a surprising finding is that the model generalizes that speakers can produce \u201cnon-native\u201d phonemes for other languages and speak \u201cunfamiliar\u201d languages not fluently, but at least diligently. And if you are a native speaker of any of these languages, what do you think? Hope you liked my article on Indic languages? Share in the comments below.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/674358\/\"> https:\/\/habr.com\/ru\/articles\/674358\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>In this article, we shall provide some background on how multilingual multi-speaker models work and test an Indic\u00a0<a href=\"https:\/\/github.com\/snakers4\/silero-models#indic-languages\" rel=\"noopener noreferrer nofollow\">TTS model<\/a>\u00a0that supports 9 languages and 17 speakers (Hindi, Malayalam, Manipuri, Bengali, Rajasthani, Tamil, Telugu, Gujarati, Kannada).<\/p>\n<p>It seems a bit counter-intuitive at first that one model can support so many languages and speakers provided that each Indic language has its own alphabet, but we shall see how it was implemented.<\/p>\n<p>Also, we shall list the specs of these models like supported sampling rates and try something cool \u2013 making speakers of different Indic languages speak Hindi. Please, if you are a native speaker of any of these languages, share your opinion on how these voices sound, both in their respective language and in Hindi.<\/p>\n<h3>Basic Background on Text-to-Speech<\/h3>\n<p>Text-to-speech\u00a0(<strong>TTS<\/strong>) is a broad subject, but we need to get a basic understanding of how it works in general or what are the main components.<\/p>\n<p>Unlike more traditional TTS models that relied on specific linguistic information as inputs, modern TTS models usually work with text or phoneme inputs. The main components of most modern TTS systems are (even newer fully end-to-end models still have similar components inside):<\/p>\n<ul>\n<li>\n<p>Text preprocessing module and cleaning plus some form of text to phoneme transcription, transcription, or transliteration;<\/p>\n<\/li>\n<li>\n<p>An acoustic model;<\/p>\n<\/li>\n<li>\n<p>A vocoder;<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>Modern TTS models typically rely on curated graphemes or phonemes as input. This input is then passed into an\u00a0<a href=\"https:\/\/medium.com\/analytics-vidhya\/understanding-embedding-layer-in-keras-bbe3ff1327ce\" rel=\"noopener noreferrer nofollow\">embedding layer<\/a>\u00a0that maps this alphabet to a set of tensors.<\/p>\n<p>Usually,\u00a0<a href=\"https:\/\/medium.com\/analytics-vidhya\/understanding-the-mel-spectrogram-fca2afa2ce53\" rel=\"noopener noreferrer nofollow\">Mel Spectrograms<\/a>\u00a0are chosen as a modelling unit for the acoustic model and the vocoder. In simple terms, the acoustic model transfers input text embeddings into a Mel Spectrogram, which the vocoder turns into the actual audio.<\/p>\n<p>There is a whole plethora of different approaches and architectures for vocoders and acoustic models (please see the paper below for a comprehensive review), but this is out of scope for this article.<\/p>\n<figure class=\"full-width\"><figcaption>Source:\u00a0http:\/\/arxiv.org\/abs\/2106.15561<\/figcaption><\/figure>\n<p>Each Indic language has its own alphabet and basically, there are only 2 ways we can combine them into one unified model (of course we also can train several distinct models):<\/p>\n<ul>\n<li>\n<p>Use some transliteration or transcription scheme;<\/p>\n<\/li>\n<li>\n<p>Combine the alphabets together and increase the number of embeddings;<\/p>\n<\/li>\n<\/ul>\n<p>The second approach is a bit problematic because the phonemes are very similar in closely related languages using such an approach may hurt the model\u2019s generalization and convergence.<\/p>\n<p>Also, each speaker should get his or her own embedding. As for the languages \u2013 it does not make sense to include language embeddings in case the number of speakers is not much larger than the number of languages.<\/p>\n<h3>How does it Work?<\/h3>\n<p>The secret sauce is that the model uses ISO romanization techniques supported by a widely used\u00a0<a href=\"https:\/\/aksharamukha.appspot.com\/converter\" rel=\"noopener noreferrer nofollow\">aksharamukha tool<\/a>\u00a0and its python\u00a0<a href=\"https:\/\/aksharamukha.appspot.com\/python\" rel=\"noopener noreferrer nofollow\">package<\/a>. This process is a transliteration (not a transcription) and some language\u2019s letters have a bit different sounds, but this works well enough as you will see.<\/p>\n<p><strong>To use this tool for all of these languages, for example using python, we need to:<\/strong><\/p>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<td>\n<p align=\"left\"><strong>Language<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>Romanization function<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">hindi<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Devanagari\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">malayalam<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Malayalam\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">manipuri<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Bengali\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">bengali<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Bengali\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">rajasthani<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Devanagari\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">tamil<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Tamil\u2019, \u2018ISO\u2019, orig_text, pre_options=[\u2018TamilTranscribe\u2019])<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">telugu<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Telugu\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">gujarati<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Gujarati\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">kannada<\/p>\n<\/td>\n<td>\n<p align=\"left\">transliterate.process(\u2018Kannada\u2019, \u2018ISO\u2019, orig_text)<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p><strong>After converting the input text to ISO, now we can try the model:<\/strong><\/p>\n<pre><code class=\"python\">import torch from aksharamukha import transliterate  model, example_text = torch.hub.load(repo_or_dir='snakers4\/silero-models',   model='silero_tts',                                      language='indic',                                      speaker='v3_indic')  orig_text = \"\u092a\u094d\u0930\u0938\u093f\u0926\u094d\u0926 \u0915\u092c\u0940\u0930 \u0905\u0927\u094d\u092f\u0947\u0924\u093e, \u092a\u0941\u0930\u0941\u0937\u094b\u0924\u094d\u0924\u092e \u0905\u0917\u094d\u0930\u0935\u093e\u0932 \u0915\u093e \u092f\u0939 \u0936\u094b\u0927 \u0906\u0932\u0947\u0916, \u0909\u0938 \u0930\u093e\u092e\u093e\u0928\u0902\u0926 \u0915\u0940 \u0916\u094b\u091c \u0915\u0930\u0924\u093e \u0939\u0948\" roman_text = transliterate.process('Devanagari', 'ISO', orig_text) print(roman_text)  audio = model.apply_tts(roman_text,                          speaker='hindi_male')<\/code><\/pre>\n<p>The created audio is a PyTorch Tensor containing a 48 kHz (the default sampling rate) audio with the generated phrase. Of course, lower sampling rates are also available.<\/p>\n<p>The romanization is a lossless process, and for the majority of texts that I have tried, it\u00a0<strong>can be transliterated\u00a0<\/strong><a href=\"https:\/\/aksharamukha.appspot.com\/python\" rel=\"noopener noreferrer nofollow\"><strong>back and forth<\/strong><\/a><strong>\u00a0without any errors using the\u00a0aksharamukha tool<\/strong>.<\/p>\n<h3>Model Performance and Characteristics<\/h3>\n<p>After a bit of fiddling, prodding, and testing we can see that the model:<\/p>\n<ul>\n<li>\n<p>Can generate audio with a sampling rate of 8 000, 24 000, or 48 000 kHz;<\/p>\n<\/li>\n<li>\n<p>Works both on CPUs and GPUs. Also, it works even on old and outdated x86 CPUs without AVX2 instructions;<\/p>\n<\/li>\n<li>\n<p>Weighs about 50 MB for all speakers and languages (yes, really!);<\/p>\n<\/li>\n<\/ul>\n<p>If we properly measure the model speed on 1 and 4 CPU threads (turns out adding 6 or more threads does not really help), we can see that it quite quick (I measured the seconds of audio generated per second):<\/p>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<td>\n<p align=\"left\"><strong>Sampling Rate<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>1 CPU Thread<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>4 CPU Threads<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">8 kHz<\/p>\n<\/td>\n<td>\n<p align=\"left\">15 \u2014 25<\/p>\n<\/td>\n<td>\n<p align=\"left\">30 \u2014 60<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">24 kHz<\/p>\n<\/td>\n<td>\n<p align=\"left\">10<\/p>\n<\/td>\n<td>\n<p align=\"left\">15 \u2014 20<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">48 kHz<\/p>\n<\/td>\n<td>\n<p align=\"left\">5<\/p>\n<\/td>\n<td>\n<p align=\"left\">10<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p>The model also supports\u00a0<a href=\"https:\/\/github.com\/snakers4\/silero-models\/wiki\/SSML\" rel=\"noopener noreferrer nofollow\">SSML markdown<\/a>, but it is still a bit glitchy for some tags.<\/p>\n<h3>Audio Samples<\/h3>\n<p>Now let\u2019s generate some audio samples for each of the languages and listen to them:<\/p>\n<div class=\"tm-iframe_temp\" data-src=\"https:\/\/embedd.srv.habr.com\/iframe\/62bd96194122e46238f233b6\" data-style=\"\" id=\"62bd96194122e46238f233b6\" width=\"\"><\/div>\n<p>I am not a native speaker of these languages to evaluate the speech quality, but as far as TTS models go, most of these speakers sound passable. The most obvious problems can be traced to source material \u2013 mostly slow pace of speech and audio quality (some audios were noisy or upsampled from lower quality).<\/p>\n<p>I asked a native speaker from India (he speaks Hindi, Telugu,\u00a0Tamil and understands\u00a0Malayalam,\u00a0Bengali,\u00a0Kannada and\u00a0Gujarati\u00a0 to a certain extent) to evaluate the anecdotal quality, and this is what he replied:<\/p>\n<ul>\n<li>\n<p><strong>Hindi<\/strong>. The pronunciations were accurate. It sounded a bit mechanical (like reading out phrase by phrase), probably because that\u2019s how the speakers spoke it in the dataset;<\/p>\n<\/li>\n<li>\n<p><strong>Telugu\u00a0<\/strong>was perfect. Was natural and smooth with accurate pronunciations;<\/p>\n<\/li>\n<li>\n<p><strong>Tamil<\/strong>. The output could have been a bit smoother if it was generated for a single sentence. Nevertheless, it was still good;<\/p>\n<\/li>\n<li>\n<p><strong>Malayalam<\/strong>. The pauses and phrasing were somewhat uncomfortable to hear. Female output was better, probably because it has a bit higher speed;<\/p>\n<\/li>\n<li>\n<p><strong>Bengali\u00a0<\/strong>was good. The female sample was perfect;<\/p>\n<\/li>\n<li>\n<p><strong>Kannada. The male<\/strong>\u00a0sample was perfect. The female one had awkward pauses;<\/p>\n<\/li>\n<li>\n<p><strong>Gujarati\u00a0<\/strong>\u2013 seemed too mechanical;<\/p>\n<\/li>\n<\/ul>\n<h3>Non Hindi Speakers Speaking Hindi<\/h3>\n<p>Now for something extra. Let\u2019s see what happens if non-Hindi speakers are asked to speak Hindi!<\/p>\n<p>Take a listen:\u00a0<\/p>\n<div class=\"tm-iframe_temp\" data-src=\"https:\/\/embedd.srv.habr.com\/iframe\/62bd95ebb6b9a8627a8d4da9\" data-style=\"\" id=\"62bd95ebb6b9a8627a8d4da9\" width=\"\"><\/div>\n<p><strong>And here is what a native speaker thinks:<\/strong><\/p>\n<ul>\n<li>\n<p>In general, when we use a speaker of another language Y for an input text language X, it does not sound natural. And it also loses the accent of Y when in such cases. But neither does it have the accent of X;<\/p>\n<\/li>\n<li>\n<p>It was a plain read-out of the roman text using the voice of another speaker. Although the basic voice characteristics were maintained;<\/p>\n<\/li>\n<li>\n<p>But it was accurately able to pronounce some phonemes of X which they would have never encountered for Y. So definitely there\u2019s sharing of phonetics across languages, wherever non-ambiguous;<\/p>\n<\/li>\n<li>\n<p>The model is definitely useful when using appropriate languages for the respectively trained speakers;<\/p>\n<\/li>\n<li>\n<p>More precisely, it sounds like a person of native language Y trying hard to fluently read the script of language X, without knowing the language;<\/p>\n<\/li>\n<\/ul>\n<p>And what do you think? Do these conclusions hold? It is next to impossible for me to make these judgments apart from observing and fixing standard TTS problems shared by all non-tonal languages.<\/p>\n<h3>Conclusion<\/h3>\n<p>In this article, we have briefly touched upon the subject of how TTS models work and have shown that the ISO romanization scheme (provided by the aksharamukha tool)\u00a0 can serve as a decent foundation to build a universal Indic TTS model, that not only can \u201cspeak\u201d several Indic languages, but also can make speakers \u201cspeak\u201d a language that they originally did not even record.<\/p>\n<p>This did not work for the Assamese language, because some of its romanized phonemes sounded too different compared to other languages (according to our non-native ear).<\/p>\n<p>Also, a surprising finding is that the model generalizes that speakers can produce \u201cnon-native\u201d phonemes for other languages and speak \u201cunfamiliar\u201d languages not fluently, but at least diligently. And if you are a native speaker of any of these languages, what do you think? Hope you liked my article on Indic languages? Share in the comments below.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/674358\/\"> https:\/\/habr.com\/ru\/articles\/674358\/<\/a><br \/><\/br><\/br><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-415482","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/415482","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=415482"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/415482\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=415482"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=415482"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=415482"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}