{"id":390402,"date":"2024-06-29T09:13:36","date_gmt":"2024-06-29T09:13:36","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=390402"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=390402","title":{"rendered":"<span>WebRTC in Docker. Struggling for resources<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/3e6\/614\/2a2\/3e66142a28ec0739417371fd95e9b9c2.jpg\" width=\"2000\" height=\"1335\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/3e6\/614\/2a2\/3e66142a28ec0739417371fd95e9b9c2.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>A few weeks ago we wrote an <a href=\"https:\/\/habr.com\/ru\/company\/flashphoner\/blog\/562244\/\">article about Docker and WebRTC servers<\/a> and talked about the intricacies of launching containers. Our readers (rightly) questioned whether Docker was a suitable tool for production, for the following reasons: <\/p>\n<ul>\n<li>\n<p>Docker does not optimally use and synchronize CPU resources between containers, which can cause RTP stream timings to shift. <\/p>\n<\/li>\n<li>\n<p>Using several containers will require mapping the ports of each of them to the ports of the host, thus creating a NAT behind a NAT, which is not always a convenient and stable construct. <\/p>\n<\/li>\n<\/ul>\n<p>When running tests for the article we didn&#8217;t run into such issues. However if the issues are outlined, it means someone has encountered them. We decided to look into this situation and find solutions. <\/p>\n<h3>Dividing the network<\/h3>\n<p>We will begin with the second task &#8212; configuring a network for Docker. Before containers can be created, we need to set up a Docker network. <\/p>\n<p>For the purposes of this article, we used public IP addresses. If the containers are operated via NAT, it will complicate the process, for you will also need to configure rules for port forwarding in the border gateway. <\/p>\n<p>With the ipvlan driver each container becomes a full network member, so we can create rules for port forwarding through NAT, just as if WCS were deployed on real hardware. So, keep in mind that, apart from the main NAT, no NATs are created for containers. <\/p>\n<p>We requested a network of 8 public elastic IPs from our ISP.<\/p>\n<pre><code class=\"bash\">1st address - network address:  147.75.76.160\/29 2nd address - address of the default gateway: 147.75.76.161 3rd address - host address: 147.75.76.162 4th address -  range address for Docker 147.75.76.163\/30 5th address -  address of the first container 147.75.76.164 6th address -  address of the second container 147.75.76.165 7th and 8th - backup addresses. <\/code><\/pre>\n<p>The first step is to create a ipvlan-based Docker network named &#171;new-testnet&#187;<\/p>\n<pre><code>docker network create -d ipvlan -o parent=enp0s3 \\ --subnet 147.75.76.160\/29 \\ --gateway 147.75.76.161 \\ --ip-range 147.75.76.163\/30 \\ new-testnet <\/code><\/pre>\n<p>where:<\/p>\n<ul>\n<li>\n<p><strong>ipvlan<\/strong> \u2014 network driver type;<\/p>\n<\/li>\n<li>\n<p><strong>parent=enp0s3<\/strong> \u2014 physical network interface (enp0s3), through which container traffic will go;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;subnet<\/strong> \u2014 subnet;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;gateway<\/strong> \u2014 default gateway for the subnet;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;ip-range<\/strong> \u2014 address range for the subnet, containing addresses that Docker will assign to containers.<\/p>\n<\/li>\n<\/ul>\n<h3>Preparing the load for the testing<\/h3>\n<p>Containers will be put under load using a WebRTC load test. Let&#8217;s prepare the settings: <\/p>\n<p>On the host, in the directory<\/p>\n<pre><code class=\"bash\">\/opt\/wcs\/conf\/<\/code><\/pre>\n<p>create the following files: <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\">flashphoner.properties<\/a> and <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a>, containing the lines: <\/p>\n<p>for <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\">flashphoner.properties<\/a>:<\/p>\n<pre><code class=\"bash\">#server ip ip                     = ip_local               =  #webrtc ports range media_port_from        =31001 media_port_to          =40000  #codecs codecs                   =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv codecs_exclude_sip       =mpeg4-generic,flv,mpv codecs_exclude_streaming =flv,telephone-event codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv  #websocket ports ws.port                 =8080 wss.port                =8443  wcs_activity_timer_timeout=86400000  wcs_agent_port_from=44001 wcs_agent_port_to=55000  global_bandwidth_check_enabled=true zgc_log_parser_enable=true zgc_log_time_format=yyyy-MM-dd'T'HH:mm:ss.SSSZ <\/code><\/pre>\n<p>This file will replace the original flashphoner.properties upon container launch.  The <strong>ip<\/strong> and <strong>ip_local<\/strong> variables will be filled with values specified during container creation by the <strong>EXTERNAL_IP<\/strong> and <strong>LOCAL_IP<\/strong> variables respectively. <\/p>\n<p>Next, we use special settings to broaden the address range:<\/p>\n<pre><code class=\"bash\">media_port_from=31001 media_port_to=40000 wcs_agent_port_from=44001 wcs_agent_port_to=55000<\/code><\/pre>\n<p>lengthen the test duration:<\/p>\n<pre><code class=\"bash\">wcs_activity_timer_timeout=86400000<\/code><\/pre>\n<p>and make it so the data on the speed of the network adapter and ZGC operation data are displayed on the statistics page:<\/p>\n<pre><code class=\"bash\">zgc_log_parser_enable=true zgc_log_time_format=yyyy-MM-dd'T'HH:mm:ss.SSSZ<\/code><\/pre>\n<p>In the <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties file<\/a>, configure ZGC usage and specify the heap size: <\/p>\n<pre><code class=\"bash\">### SERVER OPTIONS ### # Set this property to false to disable session debug -DsessionDebugEnabled=false # Disable SSLv3 -Djdk.tls.client.protocols=\"TLSv1,TLSv1.1,TLSv1.2\"   ### JVM OPTIONS ### -Xmx16g -Xms16g #-Xcheck:jni  # Can be a better GC setting to avoid long pauses  # Uncomment to fix multicast crosstalk problem when streams share multicast port -Djava.net.preferIPv4Stack=true  # Default monitoring port is 50999. Make sure the port is closed on firewall. Use ssh tunel for the monitoring. -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost  #-XX:ErrorFile=\/usr\/local\/FlashphonerWebCallServer\/logs\/error%p.log -Xlog:gc*:\/usr\/local\/FlashphonerWebCallServer\/logs\/gc-core-:time  # ZGC -XX:+UnlockExperimentalVMOptions -XX:+UseZGC  # Use System.gc() concurrently in CMS -XX:+ExplicitGCInvokesConcurrent # Disable System.gc() for RMI, for 10000 hours -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000<\/code><\/pre>\n<p>This file will also replace the <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a> file, which is in the container by default.<\/p>\n<p>We generally recommend to set the heap to take up 50% of the available RAM, but, in this case, the launch of two containers would take up the whole server RAM and it could result in unstable operation. This is why we will try a different approach. We shall allocate 25% of the available RAM to each container: <\/p>\n<pre><code class=\"bash\">### JVM OPTIONS ### -Xmx16g -Xms16g<\/code><\/pre>\n<p>Now, the set-up is finished, and we&#8217;re ready to launch.  (You can download the sample <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a> and <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\">flashphoner.properties<\/a> files in the &#171;Useful files&#187; section at the end of the article).<\/p>\n<h3>Allocating the resources and launching containers<\/h3>\n<p>Now we come back to the primary issue &#8212; that of the resource allocation to containers and of timing shifts. <\/p>\n<p>It is true that, when allocating CPU resources between containers via cgroups, the software within the containers may select the quota set by the scheduler, which results in jitters (unwanted deviations of the transmitted signal), which negatively affects RTP stream playback.<\/p>\n<p>Allocation of CPU resources and jitters are an issue not only for Docker, but for both &#171;classic&#187; hypervisor-based virtual machines and real hardware, which is why it is not really Docker-specific. When it comes to WebRTC, jitters are handled by an adaptive jitter buffer, which runs on the client and supports a fairly wide range (up to 1000 ms).<\/p>\n<p>In our case, timings are not really an issue, since the RTP stream is not linked to the server clock. RTP do not necessarily follow the encoder clock exactly, because the network is never flawless. All the discrepancies are handled with buffers in WebRTC. Once again, this is not a Docker-specific matter.<\/p>\n<p>In order to minimize the struggle for resources between containers we shall manually allocate processor cores to them. This way, there will be no conflicts or quotas of processor time. The way to do it is to use the following key when creating a container: <\/p>\n<pre><code class=\"bash\">--cpuset-cpus=<\/code><\/pre>\n<p>You can specify a list of cores (divided with commas) or a range of cores (separated by a hyphen). The first core is named &#171;0&#187;.<\/p>\n<p>Now let&#8217;s launch the first container:<\/p>\n<pre><code class=\"bash\">docker run --cpuset-cpus=0-15 \\ -v \/opt\/wcs\/conf:\/conf \\ -e PASSWORD=123Qwe \\ -e LICENSE=xxxx-xxxx-xxxx-xxxx-xxxx \\ -e LOCAL_IP=147.75.76.164 \\ -e EXTERNAL_IP=147.75.76.164 \\ --net new-testnet \\ --ip 147.75.76.164 \\ --name wcs-docker-test-1 \\ -d flashphoner\/webcallserver:latest<\/code><\/pre>\n<p> the keys are as follows: <\/p>\n<ul>\n<li>\n<p><strong>&#8212;cpuset-cpus=0-15<\/strong> &#8212; specifies that the container must use host cores 0 through 15 to run; <\/p>\n<\/li>\n<li>\n<p><strong>-v \/opt\/wcs\/conf:\/conf<\/strong> &#8212; attaches the directory with config files to the container; <\/p>\n<\/li>\n<li>\n<p><strong>PASSWORD<\/strong> \u2014 password to access the inner workings of the container via SSH. If this variable is not defined, it will not be possible to get into the container via SSH;<\/p>\n<\/li>\n<li>\n<p><strong>LICENSE<\/strong> \u2014 WCS license number. If this variable is not defined, the license can be activated through the web interface;<\/p>\n<\/li>\n<li>\n<p><strong>LOCAL_IP<\/strong> \u2014 IP address of the container in the Docker network, which will be logged into the ip_local parameter in the flashphoner.properties config file;<\/p>\n<\/li>\n<li>\n<p><strong>EXTERNAL_IP<\/strong> \u2014 IP address of the external network interface. It is entered into the IP parameter in the flashphoner.properties config file;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;net<\/strong> specifies the network within which the container will operate. Our container is launched in the testnet network;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;ip 147.75.76.164<\/strong> &#8212; address of the container in the Docker network; <\/p>\n<\/li>\n<li>\n<p><strong>&#8212;name wcs-docker-test-1<\/strong> &#8212; container name;<\/p>\n<\/li>\n<li>\n<p><strong>-d flashphoner\/webcallserver:latest<\/strong> &#8212; image for the container deployment<\/p>\n<\/li>\n<\/ul>\n<p>For the second container, we use a very similar command:<\/p>\n<pre><code class=\"bash\">docker run --cpuset-cpus=15-31 \\ -v \/opt\/wcs\/conf:\/conf \\ -e PASSWORD=123Qwe \\ -e LICENSE=xxxx-xxxx-xxxx-xxxx-xxxx \\ -e LOCAL_IP=147.75.76.165 \\ -e EXTERNAL_IP=147.75.76.165 \\ --net new-testnet \\ --ip 147.75.76.165 \\ --name wcs-docker-test-2 \\ -d flashphoner\/webcallserver:latest<\/code><\/pre>\n<p>Here, we specify a different core range and a different IP address for the container. You can also set a different password for the SSH, but it is not required. <\/p>\n<h3>Testing and evaluating the results<\/h3>\n<p>In the web interface of the first container we launch a console for WebRTC testing with stream capture <a href=\"http:\/\/147.75.76.164:9091\/client2\/examples\/demo\/streaming\/console\/console.html\">http:\/\/147.75.76.164:9091\/client2\/examples\/demo\/streaming\/console\/console.html<\/a>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/c75\/54e\/11b\/c7554e11b7c40fd6ca0aee06462818ca.jpg\" width=\"974\" height=\"583\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/c75\/54e\/11b\/c7554e11b7c40fd6ca0aee06462818ca.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>In the web interface of the second container we select &#171;Two-way Streaming&#187;: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/5e8\/667\/bf5\/5e8667bf55a7a2f883f62727eb301c7b.jpg\" width=\"784\" height=\"749\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/5e8\/667\/bf5\/5e8667bf55a7a2f883f62727eb301c7b.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>and then start the load test:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/310\/65a\/74d\/31065a74ddcc1d0053d8d7b3c3afb418.jpg\" width=\"974\" height=\"748\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/310\/65a\/74d\/31065a74ddcc1d0053d8d7b3c3afb418.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>Container performance is evaluated using graphs produced by the <a href=\"https:\/\/flashphoner.com\/10-important-webrtc-streaming-metrics-and-configuring-prometheus-grafana-monitoring\/\">Prometheus + Grafana<\/a> monitoring systems.  To receive data on the CPU load we have installed Prometheus Node Exporter on the host. Information on the container load and streams status is collected from the WCS server statistics page in the containers: <\/p>\n<pre><code class=\"bash\">http:\/\/147.75.76.164:8081\/?action=stat http:\/\/147.75.76.165:8081\/?action=stat<\/code><\/pre>\n<p>You can find the <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/Docker_WebRTC_test.json\">panel for Grafana<\/a> in the &#171;Useful files&#187; section.  <\/p>\n<p>The results of the test with container separation by core: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/7ea\/a01\/198\/7eaa01198a7ea020a7faeb2d9eb4f5f1.jpg\" width=\"1430\" height=\"873\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/7ea\/a01\/198\/7eaa01198a7ea020a7faeb2d9eb4f5f1.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>As we can see, the load on the test container that was receiving streams was slightly higher (singular peaks up to 20-25 units), compared to the container that was sending the streams.  At the same time, no stream degradation was detected and the control stream&#8217;s quality (see the screenshot below) was acceptable: there were no artifacts and sound stutters. All in all, it is fair to say that the containers handled the load well. <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/031\/06a\/ae2\/03106aae20c5ebfbeefe2bf5e53c00b9.jpg\" width=\"784\" height=\"749\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/031\/06a\/ae2\/03106aae20c5ebfbeefe2bf5e53c00b9.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>The output from the htop application, produced during the test, shows that the 32 cores used by the containers were active, while the rest of the cores were not:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/c42\/338\/4eb\/c423384ebc5d50c14b348258d3378609.jpg\" width=\"1021\" height=\"327\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/c42\/338\/4eb\/c423384ebc5d50c14b348258d3378609.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>Now, let&#8217;s relaunch the containers with no settings for container separation by core.<\/p>\n<p>The first container:<\/p>\n<pre><code class=\"bash\">docker run \\ -v \/opt\/wcs\/conf:\/conf \\ -e PASSWORD=123Qwe \\ -e LICENSE=xxxx-xxxx-xxxx-xxxx-xxxx \\ -e LOCAL_IP=147.75.76.164 \\ -e EXTERNAL_IP=147.75.76.164 \\ --net new-testnet \\ --ip 147.75.76.164 \\ --name wcs-docker-test-1 \\ -d flashphoner\/webcallserver:latest<\/code><\/pre>\n<p>The second container:<\/p>\n<pre><code class=\"bash\">docker run \\ -v \/opt\/wcs\/conf:\/conf \\ -e PASSWORD=123Qwe \\ -e LICENSE=xxxx-xxxx-xxxx-xxxx-xxxx \\ -e LOCAL_IP=147.75.76.165 \\ -e EXTERNAL_IP=147.75.76.165 \\ --net new-testnet \\ --ip 147.75.76.165 \\ --name wcs-docker-test-2 \\ -d flashphoner\/webcallserver:latest<\/code><\/pre>\n<p>Let&#8217;s start the load test again, using the same conditions, and look at the graphs: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/e9e\/4d0\/693\/e9e4d0693bc651857bf426a705abdede.jpg\" width=\"1428\" height=\"869\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e9e\/4d0\/693\/e9e4d0693bc651857bf426a705abdede.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>This time, the containers appeared to be &#171;more powerful&#187; \u2013 the cores were not manually limited, and each container had access to all the 40 cores of the host. That&#8217;s why they managed to capture more streams, but closer to the end of the test 1% of them became degraded. This means that with container separation by core containers run more smoothly. <\/p>\n<p>And thus, the testing has shown that containers can be configured to be full members of either the local or the global network. We also managed to configure resource allocation between containers in a way that makes it so they do not interfere with each other.  Keep in mind that the quantitative results of your testing may be different, because they depend on the specific task you aim to perform, the technical specifications of the system used, and its environment.<\/p>\n<p>Good streaming to you!<\/p>\n<h3>Useful files<\/h3>\n<p><a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\"> flashphoner.properties<\/a> <\/p>\n<p> <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a>  <\/p>\n<p><a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/Docker_WebRTC_test.json\">Docker_WebRTC_test.json<\/a><\/p>\n<h3>Links<\/h3>\n<p><a href=\"https:\/\/flashphoner.com\/support-web-call-server-in-docker\">WCS in Docker<\/a><\/p>\n<p><a href=\"https:\/\/docs.flashphoner.com\/display\/WCS52EN\/WCS+in+Docker\">Deployment WCS in Docker<\/a><\/p>\n<p><a href=\"https:\/\/hub.docker.com\/r\/flashphoner\/webcallserver\">WCS in DockerHub<\/a><\/p>\n<p><a href=\"https:\/\/flashphoner.com\/10-important-webrtc-streaming-metrics-and-configuring-prometheus-grafana-monitoring\/\">10 Important WebRTC Streaming Metrics and Configuring Prometheus + Grafana Monitoring<\/a><\/p>\n<p><a href=\"https:\/\/flashphoner.com\/what-kind-of-server-do-you-need-to-run-a-thousand-webrtc-streams\/\">What kind of server do you need to run a thousand WebRTC streams?<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/565610\/\"> https:\/\/habr.com\/ru\/articles\/565610\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>A few weeks ago we wrote an <a href=\"https:\/\/habr.com\/ru\/company\/flashphoner\/blog\/562244\/\">article about Docker and WebRTC servers<\/a> and talked about the intricacies of launching containers. Our readers (rightly) questioned whether Docker was a suitable tool for production, for the following reasons: <\/p>\n<ul>\n<li>\n<p>Docker does not optimally use and synchronize CPU resources between containers, which can cause RTP stream timings to shift. <\/p>\n<\/li>\n<li>\n<p>Using several containers will require mapping the ports of each of them to the ports of the host, thus creating a NAT behind a NAT, which is not always a convenient and stable construct. <\/p>\n<\/li>\n<\/ul>\n<p>When running tests for the article we didn&#8217;t run into such issues. However if the issues are outlined, it means someone has encountered them. We decided to look into this situation and find solutions. <\/p>\n<h3>Dividing the network<\/h3>\n<p>We will begin with the second task &#8212; configuring a network for Docker. Before containers can be created, we need to set up a Docker network. <\/p>\n<p>For the purposes of this article, we used public IP addresses. If the containers are operated via NAT, it will complicate the process, for you will also need to configure rules for port forwarding in the border gateway. <\/p>\n<p>With the ipvlan driver each container becomes a full network member, so we can create rules for port forwarding through NAT, just as if WCS were deployed on real hardware. So, keep in mind that, apart from the main NAT, no NATs are created for containers. <\/p>\n<p>We requested a network of 8 public elastic IPs from our ISP.<\/p>\n<pre><code class=\"bash\">1st address - network address:  147.75.76.160\/29 2nd address - address of the default gateway: 147.75.76.161 3rd address - host address: 147.75.76.162 4th address -  range address for Docker 147.75.76.163\/30 5th address -  address of the first container 147.75.76.164 6th address -  address of the second container 147.75.76.165 7th and 8th - backup addresses. <\/code><\/pre>\n<p>The first step is to create a ipvlan-based Docker network named &#171;new-testnet&#187;<\/p>\n<pre><code>docker network create -d ipvlan -o parent=enp0s3 \\ --subnet 147.75.76.160\/29 \\ --gateway 147.75.76.161 \\ --ip-range 147.75.76.163\/30 \\ new-testnet <\/code><\/pre>\n<p>where:<\/p>\n<ul>\n<li>\n<p><strong>ipvlan<\/strong> \u2014 network driver type;<\/p>\n<\/li>\n<li>\n<p><strong>parent=enp0s3<\/strong> \u2014 physical network interface (enp0s3), through which container traffic will go;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;subnet<\/strong> \u2014 subnet;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;gateway<\/strong> \u2014 default gateway for the subnet;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;ip-range<\/strong> \u2014 address range for the subnet, containing addresses that Docker will assign to containers.<\/p>\n<\/li>\n<\/ul>\n<h3>Preparing the load for the testing<\/h3>\n<p>Containers will be put under load using a WebRTC load test. Let&#8217;s prepare the settings: <\/p>\n<p>On the host, in the directory<\/p>\n<pre><code class=\"bash\">\/opt\/wcs\/conf\/<\/code><\/pre>\n<p>create the following files: <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\">flashphoner.properties<\/a> and <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a>, containing the lines: <\/p>\n<p>for <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\">flashphoner.properties<\/a>:<\/p>\n<pre><code class=\"bash\">#server ip ip                     = ip_local               =  #webrtc ports range media_port_from        =31001 media_port_to          =40000  #codecs codecs                   =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv codecs_exclude_sip       =mpeg4-generic,flv,mpv codecs_exclude_streaming =flv,telephone-event codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv  #websocket ports ws.port                 =8080 wss.port                =8443  wcs_activity_timer_timeout=86400000  wcs_agent_port_from=44001 wcs_agent_port_to=55000  global_bandwidth_check_enabled=true zgc_log_parser_enable=true zgc_log_time_format=yyyy-MM-dd'T'HH:mm:ss.SSSZ <\/code><\/pre>\n<p>This file will replace the original flashphoner.properties upon container launch.  The <strong>ip<\/strong> and <strong>ip_local<\/strong> variables will be filled with values specified during container creation by the <strong>EXTERNAL_IP<\/strong> and <strong>LOCAL_IP<\/strong> variables respectively. <\/p>\n<p>Next, we use special settings to broaden the address range:<\/p>\n<pre><code class=\"bash\">media_port_from=31001 media_port_to=40000 wcs_agent_port_from=44001 wcs_agent_port_to=55000<\/code><\/pre>\n<p>lengthen the test duration:<\/p>\n<pre><code class=\"bash\">wcs_activity_timer_timeout=86400000<\/code><\/pre>\n<p>and make it so the data on the speed of the network adapter and ZGC operation data are displayed on the statistics page:<\/p>\n<pre><code class=\"bash\">zgc_log_parser_enable=true zgc_log_time_format=yyyy-MM-dd'T'HH:mm:ss.SSSZ<\/code><\/pre>\n<p>In the <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties file<\/a>, configure ZGC usage and specify the heap size: <\/p>\n<pre><code class=\"bash\">### SERVER OPTIONS ### # Set this property to false to disable session debug -DsessionDebugEnabled=false # Disable SSLv3 -Djdk.tls.client.protocols=\"TLSv1,TLSv1.1,TLSv1.2\"   ### JVM OPTIONS ### -Xmx16g -Xms16g #-Xcheck:jni  # Can be a better GC setting to avoid long pauses  # Uncomment to fix multicast crosstalk problem when streams share multicast port -Djava.net.preferIPv4Stack=true  # Default monitoring port is 50999. Make sure the port is closed on firewall. Use ssh tunel for the monitoring. -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost  #-XX:ErrorFile=\/usr\/local\/FlashphonerWebCallServer\/logs\/error%p.log -Xlog:gc*:\/usr\/local\/FlashphonerWebCallServer\/logs\/gc-core-:time  # ZGC -XX:+UnlockExperimentalVMOptions -XX:+UseZGC  # Use System.gc() concurrently in CMS -XX:+ExplicitGCInvokesConcurrent # Disable System.gc() for RMI, for 10000 hours -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000<\/code><\/pre>\n<p>This file will also replace the <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a> file, which is in the container by default.<\/p>\n<p>We generally recommend to set the heap to take up 50% of the available RAM, but, in this case, the launch of two containers would take up the whole server RAM and it could result in unstable operation. This is why we will try a different approach. We shall allocate 25% of the available RAM to each container: <\/p>\n<pre><code class=\"bash\">### JVM OPTIONS ### -Xmx16g -Xms16g<\/code><\/pre>\n<p>Now, the set-up is finished, and we&#8217;re ready to launch.  (You can download the sample <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/wcs-core.properties\">wcs-core.properties<\/a> and <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/flashphoner.properties\">flashphoner.properties<\/a> files in the &#171;Useful files&#187; section at the end of the article).<\/p>\n<h3>Allocating the resources and launching containers<\/h3>\n<p>Now we come back to the primary issue &#8212; that of the resource allocation to containers and of timing shifts. <\/p>\n<p>It is true that, when allocating CPU resources between containers via cgroups, the software within the containers may select the quota set by the scheduler, which results in jitters (unwanted deviations of the transmitted signal), which negatively affects RTP stream playback.<\/p>\n<p>Allocation of CPU resources and jitters are an issue not only for Docker, but for both &#171;classic&#187; hypervisor-based virtual machines and real hardware, which is why it is not really Docker-specific. When it comes to WebRTC, jitters are handled by an adaptive jitter buffer, which runs on the client and supports a fairly wide range (up to 1000 ms).<\/p>\n<p>In our case, timings are not really an issue, since the RTP stream is not linked to the server clock. RTP do not necessarily follow the encoder clock exactly, because the network is never flawless. All the discrepancies are handled with buffers in WebRTC. Once again, this is not a Docker-specific matter.<\/p>\n<p>In order to minimize the struggle for resources between containers we shall manually allocate processor cores to them. This way, there will be no conflicts or quotas of processor time. The way to do it is to use the following key when creating a container: <\/p>\n<pre><code class=\"bash\">--cpuset-cpus=<\/code><\/pre>\n<p>You can specify a list of cores (divided with commas) or a range of cores (separated by a hyphen). The first core is named &#171;0&#187;.<\/p>\n<p>Now let&#8217;s launch the first container:<\/p>\n<pre><code class=\"bash\">docker run --cpuset-cpus=0-15 \\ -v \/opt\/wcs\/conf:\/conf \\ -e PASSWORD=123Qwe \\ -e LICENSE=xxxx-xxxx-xxxx-xxxx-xxxx \\ -e LOCAL_IP=147.75.76.164 \\ -e EXTERNAL_IP=147.75.76.164 \\ --net new-testnet \\ --ip 147.75.76.164 \\ --name wcs-docker-test-1 \\ -d flashphoner\/webcallserver:latest<\/code><\/pre>\n<p> the keys are as follows: <\/p>\n<ul>\n<li>\n<p><strong>&#8212;cpuset-cpus=0-15<\/strong> &#8212; specifies that the container must use host cores 0 through 15 to run; <\/p>\n<\/li>\n<li>\n<p><strong>-v \/opt\/wcs\/conf:\/conf<\/strong> &#8212; attaches the directory with config files to the container; <\/p>\n<\/li>\n<li>\n<p><strong>PASSWORD<\/strong> \u2014 password to access the inner workings of the container via SSH. If this variable is not defined, it will not be possible to get into the container via SSH;<\/p>\n<\/li>\n<li>\n<p><strong>LICENSE<\/strong> \u2014 WCS license number. If this variable is not defined, the license can be activated through the web interface;<\/p>\n<\/li>\n<li>\n<p><strong>LOCAL_IP<\/strong> \u2014 IP address of the container in the Docker network, which will be logged into the ip_local parameter in the flashphoner.properties config file;<\/p>\n<\/li>\n<li>\n<p><strong>EXTERNAL_IP<\/strong> \u2014 IP address of the external network interface. It is entered into the IP parameter in the flashphoner.properties config file;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;net<\/strong> specifies the network within which the container will operate. Our container is launched in the testnet network;<\/p>\n<\/li>\n<li>\n<p><strong>&#8212;ip 147.75.76.164<\/strong> &#8212; address of the container in the Docker network; <\/p>\n<\/li>\n<li>\n<p><strong>&#8212;name wcs-docker-test-1<\/strong> &#8212; container name;<\/p>\n<\/li>\n<li>\n<p><strong>-d flashphoner\/webcallserver:latest<\/strong> &#8212; image for the container deployment<\/p>\n<\/li>\n<\/ul>\n<p>For the second container, we use a very similar command:<\/p>\n<pre><code class=\"bash\">docker run --cpuset-cpus=15-31 \\ -v \/opt\/wcs\/conf:\/conf \\ -e PASSWORD=123Qwe \\ -e LICENSE=xxxx-xxxx-xxxx-xxxx-xxxx \\ -e LOCAL_IP=147.75.76.165 \\ -e EXTERNAL_IP=147.75.76.165 \\ --net new-testnet \\ --ip 147.75.76.165 \\ --name wcs-docker-test-2 \\ -d flashphoner\/webcallserver:latest<\/code><\/pre>\n<p>Here, we specify a different core range and a different IP address for the container. You can also set a different password for the SSH, but it is not required. <\/p>\n<h3>Testing and evaluating the results<\/h3>\n<p>In the web interface of the first container we launch a console for WebRTC testing with stream capture <a href=\"http:\/\/147.75.76.164:9091\/client2\/examples\/demo\/streaming\/console\/console.html\">http:\/\/147.75.76.164:9091\/client2\/examples\/demo\/streaming\/console\/console.html<\/a>:<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>In the web interface of the second container we select &#171;Two-way Streaming&#187;: <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>and then start the load test:<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>Container performance is evaluated using graphs produced by the <a href=\"https:\/\/flashphoner.com\/10-important-webrtc-streaming-metrics-and-configuring-prometheus-grafana-monitoring\/\">Prometheus + Grafana<\/a> monitoring systems.  To receive data on the CPU load we have installed Prometheus Node Exporter on the host. Information on the container load and streams status is collected from the WCS server statistics page in the containers: <\/p>\n<pre><code class=\"bash\">http:\/\/147.75.76.164:8081\/?action=stat http:\/\/147.75.76.165:8081\/?action=stat<\/code><\/pre>\n<p>You can find the <a href=\"https:\/\/flashphoner.com\/wp-content\/uploads\/2021\/07\/Docker_WebRTC_test.json\">panel for Grafana<\/a> in the<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-390402","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/390402","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=390402"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/390402\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=390402"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=390402"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=390402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}