{"id":406389,"date":"2024-06-29T18:54:38","date_gmt":"2024-06-29T18:54:38","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=406389"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=406389","title":{"rendered":"<span>Modern Portable Voice Activity Detector Released<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-1\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/397\/ee1\/4d3\/397ee14d3e4d031ccb22d85eacc293e9.png\" alt=\"image\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/397\/ee1\/4d3\/397ee14d3e4d031ccb22d85eacc293e9.png\"\/><\/p>\n<p>  <\/p>\n<p>Currently, there are hardly any high quality \/ modern \/ free \/ public voice activity detectors except for WebRTC Voice Activity Detector (<a href=\"https:\/\/github.com\/wiseman\/py-webrtcvad\" rel=\"nofollow noopener noreferrer\">link<\/a>). WebRTC though starts to show its age and it suffers from many false positives.<\/p>\n<p>  <\/p>\n<p>Also in some cases it is crucial to be able to anonymize large-scale spoken corpora (i.e. remove personal data). Typically personal data is considered to be private \/ sensitive if it contains (i) a name (ii) some private ID. Name recognition is a highly subjective matter and it depends on locale and business case, but Voice Activity and Number Detection are quite general tasks.<\/p>\n<p>  <\/p>\n<p><strong>Key features:<\/strong><\/p>\n<p>  <\/p>\n<ul>\n<li>Modern, portable;<\/li>\n<li>Low memory footprint;<\/li>\n<li>Superior metrics to WebRTC;<\/li>\n<li>Trained on huge spoken corpora and noise \/ sound libraries;<\/li>\n<li>Slower than WebRTC, but fast enough for IOT \/ edge \/ mobile applications;<\/li>\n<li>Unlike WebRTC (which mostly tells silence from voice), our VAD can tell voice from noise \/ music \/ silence;<\/li>\n<li>PyTorch (JIT) and ONNX checkpoints;<\/li>\n<\/ul>\n<p>  <\/p>\n<p><strong>Typical use cases:<\/strong><\/p>\n<p>  <\/p>\n<ul>\n<li>Spoken corpora anonymization;<\/li>\n<li>Can be used together with WebRTC;<\/li>\n<li>Voice activity detection for IOT \/ edge \/ mobile use cases;<\/li>\n<li>Data cleaning and preparation, number and voice detection in general;<\/li>\n<li>PyTorch and ONNX can be used with a wide variety of deployment options and backends in mind; <\/li>\n<\/ul>\n<p><a name=\"habracut\"><\/a>  <\/p>\n<h2 id=\"getting-started\">Getting Started<\/h2>\n<p>  <\/p>\n<p>For each algorithm you can see the examples in the provided <a href=\"https:\/\/colab.research.google.com\/github\/snakers4\/silero-vad\/blob\/master\/silero-vad.ipynb\" rel=\"nofollow noopener noreferrer\">colab<\/a> or in the <a href=\"https:\/\/github.com\/snakers4\/silero-vad\" rel=\"nofollow noopener noreferrer\">repo<\/a> itself. For VAD we also provide streaming examples for a single stream and multiple streams.<\/p>\n<p>  <\/p>\n<pre><code class=\"python\">import torch torch.set_num_threads(1) from pprint import pprint  model, utils = torch.hub.load(repo_or_dir='snakers4\/silero-vad',                               model='silero_vad',                               force_reload=True)  (get_speech_ts,  _, read_audio,  _, _, _) = utils  files_dir = torch.hub.get_dir() + '\/snakers4_silero-vad_master\/files'  wav = read_audio(f'{files_dir}\/en.wav') # full audio # get speech timestamps from full audio file speech_timestamps = get_speech_ts(wav, model,                                   num_steps=4) pprint(speech_timestamps)<\/code><\/pre>\n<p>  <\/p>\n<h2 id=\"latency\">Latency<\/h2>\n<p>  <\/p>\n<p>All speed test were run on AMD Ryzen Threadripper 3960X using only 1 thread: <\/p>\n<p>  <\/p>\n<pre><code class=\"plaintext\">torch.set_num_threads(1) # pytorch ort_session.intra_op_num_threads = 1 # onnx ort_session.inter_op_num_threads = 1 # onnx<\/code><\/pre>\n<p>  <\/p>\n<p>Streaming latency depends on 2 factors:<\/p>\n<p>  <\/p>\n<ul>\n<li><strong>num_steps<\/strong> \u2014 number of windows to split each audio chunk into. Our post-processing class keeps previous chunk in memory (250 ms), so new chunk (also 250 ms) is appended to it. The resulting big chunk (500 ms) is split into <strong>num_steps<\/strong> overlapping windows, each 250 ms long;<\/li>\n<li><strong>number of audio streams<\/strong>;<\/li>\n<\/ul>\n<p>  <\/p>\n<p>So <strong>batch size<\/strong> for streaming is num_steps * number of audio streams. Time between receiving new audio chunks and getting results is shown in picture:<\/p>\n<p>  <\/p>\n<div class=\"scrollable-table\">\n<table>\n<thead>\n<tr>\n<th>Batch size<\/th>\n<th>Pytorch model time, ms<\/th>\n<th>Onnx model time, ms<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>2<\/strong><\/td>\n<td>9<\/td>\n<td>2<\/td>\n<\/tr>\n<tr>\n<td><strong>4<\/strong><\/td>\n<td>11<\/td>\n<td>4<\/td>\n<\/tr>\n<tr>\n<td><strong>8<\/strong><\/td>\n<td>14<\/td>\n<td>7<\/td>\n<\/tr>\n<tr>\n<td><strong>16<\/strong><\/td>\n<td>19<\/td>\n<td>12<\/td>\n<\/tr>\n<tr>\n<td><strong>40<\/strong><\/td>\n<td>36<\/td>\n<td>29<\/td>\n<\/tr>\n<tr>\n<td><strong>80<\/strong><\/td>\n<td>64<\/td>\n<td>55<\/td>\n<\/tr>\n<tr>\n<td><strong>120<\/strong><\/td>\n<td>96<\/td>\n<td>85<\/td>\n<\/tr>\n<tr>\n<td><strong>200<\/strong><\/td>\n<td>157<\/td>\n<td>137<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>  <\/p>\n<h2 id=\"throughput\">Throughput<\/h2>\n<p>  <\/p>\n<p><strong>RTS<\/strong> (seconds of audio processed per second, real time speed, or 1 \/ RTF) for full audio processing depends on <strong>num_steps<\/strong> (see previous paragraph) and <strong>batch size<\/strong> (bigger is better).<\/p>\n<p>  <\/p>\n<div class=\"scrollable-table\">\n<table>\n<thead>\n<tr>\n<th>Batch size<\/th>\n<th>num_steps<\/th>\n<th>Pytorch model RTS<\/th>\n<th>Onnx model RTS<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>40<\/strong><\/td>\n<td><strong>4<\/strong><\/td>\n<td>68<\/td>\n<td>86<\/td>\n<\/tr>\n<tr>\n<td><strong>40<\/strong><\/td>\n<td><strong>8<\/strong><\/td>\n<td>34<\/td>\n<td>43<\/td>\n<\/tr>\n<tr>\n<td><strong>80<\/strong><\/td>\n<td><strong>4<\/strong><\/td>\n<td>78<\/td>\n<td>91<\/td>\n<\/tr>\n<tr>\n<td><strong>80<\/strong><\/td>\n<td><strong>8<\/strong><\/td>\n<td>39<\/td>\n<td>45<\/td>\n<\/tr>\n<tr>\n<td><strong>120<\/strong><\/td>\n<td><strong>4<\/strong><\/td>\n<td>78<\/td>\n<td>88<\/td>\n<\/tr>\n<tr>\n<td><strong>120<\/strong><\/td>\n<td><strong>8<\/strong><\/td>\n<td>39<\/td>\n<td>44<\/td>\n<\/tr>\n<tr>\n<td><strong>200<\/strong><\/td>\n<td><strong>4<\/strong><\/td>\n<td>80<\/td>\n<td>91<\/td>\n<\/tr>\n<tr>\n<td><strong>200<\/strong><\/td>\n<td><strong>8<\/strong><\/td>\n<td>40<\/td>\n<td>46<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>  <\/p>\n<h2 id=\"vad-quality-benchmarks\">VAD Quality Benchmarks<\/h2>\n<p>  <\/p>\n<p>We use random 250 ms audio chunks for validation. Speech to non-speech ratio among chunks is about ~50\/50 (i.e. balanced). Speech chunks are sampled from real audios in four different languages (English, Russian, Spanish, German), then random background noise is added to some of them (~40%). <\/p>\n<p>  <\/p>\n<p>Since our VAD (only VAD, other networks are more flexible) was trained on chunks of the same length, model&#8217;s output is just one float from 0 to 1 \u2014 <strong>speech probability<\/strong>. We use speech probabilities as thresholds for precision-recall curve. This can be extended to 100 \u2014 150 ms. Less than 100 \u2014 150 ms cannot be distinguished as speech with confidence.<\/p>\n<p>  <\/p>\n<p><a href=\"https:\/\/github.com\/wiseman\/py-webrtcvad\" rel=\"nofollow noopener noreferrer\">Webrtc<\/a> splits audio into frames, each frame has corresponding number (0 <strong>or<\/strong> 1). We use 30ms frames for webrtc, so each 250 ms chunk is split into 8 frames, their <strong>mean<\/strong> value is used as a treshold for plot.<\/p>\n<p>  <\/p>\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/397\/ee1\/4d3\/397ee14d3e4d031ccb22d85eacc293e9.png\" alt=\"image\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/397\/ee1\/4d3\/397ee14d3e4d031ccb22d85eacc293e9.png\"\/><\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/537276\/\"> https:\/\/habr.com\/ru\/articles\/537276\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-1\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/397\/ee1\/4d3\/397ee14d3e4d031ccb22d85eacc293e9.png\" alt=\"image\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/397\/ee1\/4d3\/397ee14d3e4d031ccb22d85eacc293e9.png\"\/><\/p>\n<p>  <\/p>\n<p>Currently, there are hardly any high quality \/ modern \/ free \/ public voice activity detectors except for WebRTC Voice Activity Detector (<a href=\"https:\/\/github.com\/wiseman\/py-webrtcvad\" rel=\"nofollow noopener noreferrer\">link<\/a>). WebRTC though starts to show its age and it suffers from many false positives.<\/p>\n<p>  <\/p>\n<p>Also in some cases it is crucial to be able to anonymize large-scale spoken corpora (i.e. remove personal data). Typically personal data is considered to be private \/ sensitive if it contains (i) a name (ii) some private ID. Name recognition is a highly subjective matter and it depends on locale and business case, but Voice Activity and Number Detection are quite general tasks.<\/p>\n<p>  <\/p>\n<p><strong>Key features:<\/strong><\/p>\n<p>  <\/p>\n<ul>\n<li>Modern, portable;<\/li>\n<li>Low memory footprint;<\/li>\n<li>Superior metrics to WebRTC;<\/li>\n<li>Trained on huge spoken corpora and noise \/ sound libraries;<\/li>\n<li>Slower than WebRTC, but fast enough for IOT \/ edge \/ mobile applications;<\/li>\n<li>Unlike WebRTC (which mostly tells silence from voice), our VAD can tell voice from noise \/ music \/ silence;<\/li>\n<li>PyTorch (JIT) and ONNX checkpoints;<\/li>\n<\/ul>\n<p>  <\/p>\n<p><strong>Typical use cases:<\/strong><\/p>\n<p>  <\/p>\n<ul>\n<li>Spoken corpora anonymization;<\/li>\n<li>Can be used together with WebRTC;<\/li>\n<li>Voice activity detection for IOT \/ edge \/ mobile use cases;<\/li>\n<li>Data cleaning and preparation, number and voice detection in general;<\/li>\n<li>PyTorch and ONNX can be used with a wide variety of deployment options and backends in mind; <\/li>\n<\/ul>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-406389","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/406389","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=406389"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/406389\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=406389"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=406389"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=406389"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}