{"id":389564,"date":"2024-06-29T08:40:52","date_gmt":"2024-06-29T08:40:52","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=389564"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-06-29T08:40:52","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=389564","title":{"rendered":"<span>Monitoring WebRTC streams with Prometheus and Grafana<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/6d6\/113\/c80\/6d6113c803b1aff53bf128fc24a09d6a.jpg\" width=\"1024\" height=\"576\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/6d6\/113\/c80\/6d6113c803b1aff53bf128fc24a09d6a.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>Monitoring systems are a vital tool for any system administrator, because they can be used to extract specific information from services, such that: <\/p>\n<ul>\n<li>\n<p>Provides a general idea regarding server loads;<\/p>\n<\/li>\n<li>\n<p>Shows whether the service is stable and whether it is operating properly.<\/p>\n<\/li>\n<\/ul>\n<p>This information can then be used to draw conclusions and make decisions on whether to scale the server power up or down, whether to broaden or narrow the communication channels, etc. For instance, if we see that the CPU load average is 0.1% while at 48 cores, then it becomes obvious that we are wasting resources that can be used for other purposes.  <\/p>\n<p>When it comes to WebRTC streaming, we are not just concerned with loads on CPU, memory and communication channels \u2013 the numbers of streams and viewers are also important, as are data on whether the stream quality is degrading or not.<\/p>\n<p>Some will say that monitoring systems are pointless, because all the data on WebRTC can be found in Chrome, the server data can be found on the server, and various costs calculators (for network usage, server hosting, etc.) are available in respective personal accounts. Imagine, however, what a nightmare it is to monitor all this in real time, while actively streaming. Without a doubt, it is not a user-friendly process, and, while streaming this way, it is easy to miss something important. <\/p>\n<p>Apart from parameter monitoring, a monitoring system can also be used to timely detect and even resolve typical issues that arise during streaming.<\/p>\n<p>All this is to say that a monitoring system is an absolute must-have. The only question is, which to opt for. <\/p>\n<h3>What parameters are we going to monitor?<\/h3>\n<p>The quality of the stream, as has already been discussed many times, is not constant and depends on many factors, such as the load on the server with or without transcoding, and the use of TCP or UDP transport protocols, and the presence of packet loss and\/or NACK feedbacks, etc. All these data for assessing stream quality can be obtained manually from various sources. <\/p>\n<p>Stream degradation is a condition of a video\/audio stream, in which the picture and sound quality is not satisfactory. There are artifacts, friezes, stuttering, or out of sync sound.  <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/5c5\/7e1\/95a\/5c57e195ad89066adf4e244a38cdfb2a.png\" width=\"811\" height=\"572\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/5c5\/7e1\/95a\/5c57e195ad89066adf4e244a38cdfb2a.png\"\/><figcaption><\/figcaption><\/figure>\n<h4>Parameter 1: CPU Load average<\/h4>\n<p>If the processor is loaded by more than 80%, and\/or the CPU load periodically rises to 100%, this means that the server is overloaded and does not have enough processing power to perform work operations, which will certainly lead to degradation of streams.<\/p>\n<p>You can estimate the current processor load using the htop software. Or top. Or mpstat. The screenshot below shows htop: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f00\/f71\/347\/f00f713474306a1ff47571e82bf350f0.PNG\" width=\"818\" height=\"269\"\/><figcaption><\/figcaption><\/figure>\n<p>Usually, the Load average parameter is estimated \u2013 the average value of the system load. This parameter is displayed as three values, which represent the average load data for the last 1, 5 and 15 minutes. The lower the value, the better.  <\/p>\n<pre><code class=\"bash\">load average:  4.55 4.22 4.18<\/code><\/pre>\n<p>For example, such values for a quad-core processor mean that it works at the limit of its capabilities for a long time (several hours), and such work will definitely lead to degradation of streams. <\/p>\n<h4>Parameter 2: Percentage of degraded streams<\/h4>\n<p>The next metric, which is directly related to the previous one: degraded_streams_percent &#8212; the percentage of degraded streams. A stream is degraded if the server does not have time to process the queue of this stream. The server is multithreading. The video data of the published stream is queued, and the data is sent in a separate thread. In case of a high load on the CPU and a lack of performance for encoding\/decoding or with internal locks, these queues can grow, and streams can degrade. Therefore, the percentage of degraded streams is an important metric that signals a loss of quality.<\/p>\n<p>You can find the value of degraded_streams_percent on the statistics page <a href=\"http:\/\/demo.flashphoner.com:8081\/?action=stat\">http:\/\/demo.flashphoner.com:8081\/?action=stat<\/a> (where demo.flashphoner.com is the address of the WCS server)<\/p>\n<p>The following few metrics are relevant to the Java Virtual Machine: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/aaf\/cb0\/6e7\/aafcb06e76b340fad046db8482fa86bf.png\" width=\"902\" height=\"344\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/aaf\/cb0\/6e7\/aafcb06e76b340fad046db8482fa86bf.png\"\/><figcaption><\/figcaption><\/figure>\n<h4>Parameter 3: Java GC pause<\/h4>\n<p>Garbage collection pause time. Like in the case of the Load average, the lower the number, the better. 100 ms is very bad. The situation when a Java machine starts the garbage collector is called \u201cStop the world,\u201d which means that your application will be completely stopped while the garbage collection is running. All server threads will be stopped for a moment, and this is enough to degrade the streams.<\/p>\n<h4>Parameter 4: Java Heap<\/h4>\n<p>The size of Java Heap memory used. If it approaches -Xmx and remains constantly at these values, this means the constant work of the garbage collector,  and as a result, high CPU consumption, possible degradation of streams, as well as an increase in the likelihood of the \u201cOutOfMemoryError\u201d error, which is a signal of an irreversible violation of the internal state server. If such an error occurs at least once, the server is already broken, so it is recommended to restart it. When an OutOfMemoryError appears, the Java machine can stop any thread, thereby disrupting server operation, although externally the server will continue to work and service connections.<\/p>\n<h4>Parameter 5: Physical memory<\/h4>\n<p>The -Xmx limit for heap does not guarantee a limit on RES physical memory usage. If the heap was given 16 GB, RES can take 32 GB or more, and this is not always a memory leak. Especially in heavy mixing and transcoding scenarios. Therefore, the \u201creal memory\u201d must be monitored and not be exhausted, since this will terminate the server process by the system itself.<\/p>\n<p>These metrics can be tracked using the Java Mission Control tool. Java Mission Control is a powerful tool for monitoring and debugging a Java machine that comes with the JDK and runs on the user&#8217;s (administrator&#8217;s) PC.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/95a\/f82\/f4f\/95af82f4f057d6e3c1be7864ec96bbfe.jpg\" width=\"1363\" height=\"706\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/95a\/f82\/f4f\/95af82f4f057d6e3c1be7864ec96bbfe.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<h4>Parameter 6: Transcoder queues and transcoding time<\/h4>\n<p>Transcoding is a separate scenario with a high load on server resources. In paragraph 2, we already wrote about Degraded streams. Now, this is a later metric. In the case of transcoders, first the metrics of the queues of transcoders will show the excess and only then the Degraded streams will react. Transcoder queue metrics show the number of frames accumulated in the decode\/encoding queue. Frames are heavy, and 30 accumulated frames means 1 second of latency with a frame rate of 30 FPS. The decoding\/encoding time tells us how quickly the corresponding operations go. For example, at 30 FPS, the encoding operation for one frame should take no more than 1\/30 second = 33 milliseconds. Otherwise, the transcoder will not be able to deliver frames on time, which will ruin the stream.<\/p>\n<p>Decoding\/encoding queue metrics:<\/p>\n<p>transcoding_video_decoding_max_queue_size <\/p>\n<p>transcoding_video_encoding_max_queue_size<\/p>\n<p>Maximum decoding\/encoding time metrics<\/p>\n<p>transcoding_video_decoding_max_time <\/p>\n<p>transcoding_video_encoding_max_time<\/p>\n<p>These metrics are available on the statistics page:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/7ea\/ff2\/9a7\/7eaff29a73057126ea3d86a09ddf6721.png\" width=\"902\" height=\"426\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/7ea\/ff2\/9a7\/7eaff29a73057126ea3d86a09ddf6721.png\"\/><figcaption><\/figcaption><\/figure>\n<h4>Number of free ports for streaming<\/h4>\n<p>Metric: ports_media_free<\/p>\n<p>An important metric for highly loaded servers. Ports are allocated for each stream. If the ports in the range run out, adding new streams on the server will cause an error. <\/p>\n<p>Information on the number of ports can be found on the statistics page:  <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/2ba\/77b\/4c6\/2ba77b4c655f047395c0db093488bc2b.png\" width=\"902\" height=\"426\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/2ba\/77b\/4c6\/2ba77b4c655f047395c0db093488bc2b.png\"\/><figcaption><\/figcaption><\/figure>\n<h4>Stream audio and video synchronization<\/h4>\n<p>Metric: streams_synchronization<\/p>\n<p>An important metric, which is also called lipsync (synchronization of lips and voice in video broadcasts). Out of sync can be caused by a variety of reasons, ranging from poor quality stream source to server performance issues. With an out of sync of 100 milliseconds or more, the human ear and eye will already notice that something is wrong. The metric allows you to see this moment in numbers, in milliseconds for each stream; a positive value of the metric shows that audio is playing ahead of the video, and a negative value indicates that audio is lagging behind the video.  <\/p>\n<p>The screenshot below shows normal synchronization of audio and video in streams. The metric shows values from +2 to +79 milliseconds.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/013\/450\/2c3\/0134502c3c505cb6d6d2f73e9e566506.png\" width=\"902\" height=\"415\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/013\/450\/2c3\/0134502c3c505cb6d6d2f73e9e566506.png\"\/><figcaption><\/figcaption><\/figure>\n<h4>Parameter 9: Number of open transcoders<\/h4>\n<p>Transcoders, namely encoders, have the maximum impact on server load. Therefore, it is important to control their number. If your script does not involve the use of transcoding (for example, streaming without changing the resolution), then the number of decoders and encoders should be zero.<\/p>\n<p>Metrics:<\/p>\n<p>native_resources.video_decoders <\/p>\n<p>native_resources.video_encoders<\/p>\n<p>In the screenshot below, transcoding is not used.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/7c6\/dcd\/dc1\/7c6dcddc11ef4c68e196abcbc50535e8.png\" width=\"902\" height=\"311\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/7c6\/dcd\/dc1\/7c6dcddc11ef4c68e196abcbc50535e8.png\"\/><figcaption><\/figcaption><\/figure>\n<h4>Parameter 10: Total load on transcoders<\/h4>\n<p>The number of transcoders is a good metric, but the two encoders can differ by an order of magnitude in terms of CPU utilization. For example, stream encoding 240p 15 FPS and 1080p 60 FPS. Therefore, to get an idea of the loading, it is important to consider not only the number of encoders, but their weight. The encoder weight is: w x h x FPS, where w and h is the image resolution. As a result, the metric allows to display the real load of the server by encoders, and one heavyweight encoder can show a higher load than 10 lightweight ones.<\/p>\n<p>Metrics:<\/p>\n<p>transcoding_video_decoding_load <\/p>\n<p>transcoding_video_encoding_load<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/340\/75e\/866\/34075e8668f911b153220329bba48a85.png\" width=\"902\" height=\"426\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/340\/75e\/866\/34075e8668f911b153220329bba48a85.png\"\/><figcaption><\/figcaption><\/figure>\n<h3>Preparation for monitoring system deployment<\/h3>\n<p>If your system uses a firewall, you need to open ports <\/p>\n<ul>\n<li>\n<p>TCP 9090 \u2014 http for Prometheus server;<\/p>\n<\/li>\n<li>\n<p>TCP 9093 \u2014 http for alert manager;<\/p>\n<\/li>\n<li>\n<p>TCP and UDP 9094 \u2014 for alert manager;<\/p>\n<\/li>\n<li>\n<p>TCP 9100 \u2014 for node_exporter;<\/p>\n<\/li>\n<li>\n<p>TCP 3000 \u2014 http for Grafana server.<\/p>\n<\/li>\n<\/ul>\n<p>For example, using iptables:<\/p>\n<pre><code class=\"bash\">iptables -I INPUT 1 -p tcp --match multiport --dports 9090,9093,9094,9100 -j ACCEPT iptables -A INPUT -p udp --dport 9094 -j ACCEPT<\/code><\/pre>\n<p>If you are using a Red Hat-based operating system, SELinux will probably be enabled by default. You can disable it using the commands <\/p>\n<pre><code class=\"bash\">sudo setenforce 0 sudo sed -i 's\/^SELINUX=.*\/SELINUX=disabled\/g' \/etc\/selinux\/config<\/code><\/pre>\n<p>We are now ready to install Prometheus and Grafana. <\/p>\n<h3>Prometheus: installation and setup<\/h3>\n<p>Prometheus does not have its own repositories for popular distributions, and the official operating system repositories tend to lag behind the upstream. Therefore, we will look at the manual installation. This method is suitable for any Linux distributions running on systemd (Ubuntu, Debian, Centos, Arch, etc.). You will need to download the source, create a user, manually copy the necessary files, assign rights, and create a unit for autorun. <\/p>\n<p>You can download the source from the [official download page] <a href=\"https:\/\/prometheus.io\/download\/\">https:\/\/prometheus.io\/download\/<\/a>).  Copy the link to the Linux package and download it using wget: <\/p>\n<pre><code class=\"bash\">wget https:\/\/github.com\/prometheus\/prometheus\/releases\/download\/v2.21.0\/prometheus-2.21.0.linux-amd64.tar.gz<\/code><\/pre>\n<p>Create directories to copy the files for Prometheus:<\/p>\n<pre><code class=\"bash\">mkdir \/etc\/prometheus mkdir \/var\/lib\/prometheus<\/code><\/pre>\n<p>Unpack the downloaded archive: <\/p>\n<pre><code class=\"bash\">tar zxvf prometheus-*.linux-amd64.tar.gz<\/code><\/pre>\n<p>Go to the directory with the unpacked files:<\/p>\n<pre><code class=\"bash\">cd prometheus-*.linux-amd64<\/code><\/pre>\n<p>Distribute files into directories:<\/p>\n<pre><code class=\"bash\">cp prometheus promtool \/usr\/local\/bin\/ cp -r console_libraries consoles prometheus.yml \/etc\/prometheus<\/code><\/pre>\n<p>Create a user that will start the monitoring system:<\/p>\n<pre><code class=\"bash\">sudo useradd --no-create-home --shell \/bin\/false prometheus<\/code><\/pre>\n<p>Set the owner for the directories that we created in the previous step:<\/p>\n<pre><code class=\"bash\">sudo chown -R prometheus:prometheus \/etc\/prometheus \/var\/lib\/prometheus<\/code><\/pre>\n<p>Set the owner for the copied files:<\/p>\n<pre><code class=\"bash\">sudo chown prometheus:prometheus \/usr\/local\/bin\/{prometheus,promtool}<\/code><\/pre>\n<p>Configure Prometheus autorun:<\/p>\n<p>Create a prometheus.service file: <\/p>\n<pre><code class=\"bash\">sudo nano \/etc\/systemd\/system\/prometheus.service<\/code><\/pre>\n<p>Place the following text in it: <\/p>\n<pre><code class=\"bash\">[Unit] Description=Prometheus Service After=network.target  [Service] User=prometheus Group=prometheus Type=simple ExecStart=\/usr\/local\/bin\/prometheus \\ --config.file \/etc\/prometheus\/prometheus.yml \\ --storage.tsdb.path \/var\/lib\/prometheus\/ \\ --web.console.templates=\/etc\/prometheus\/consoles \\ --web.console.libraries=\/etc\/prometheus\/console_libraries ExecReload=\/bin\/kill -HUP $MAINPID Restart=on-failure  [Install] WantedBy=multi-user.target<\/code><\/pre>\n<p>Re-read the systemd configuration:<\/p>\n<pre><code class=\"bash\">sudo systemctl daemon-reload<\/code><\/pre>\n<p>Allow autorun:<\/p>\n<pre><code class=\"bash\">sudo systemctl enable prometheus<\/code><\/pre>\n<p>After creating autorun, start Prometheus as a service: <\/p>\n<pre><code class=\"bash\">sudo systemctl start prometheus<\/code><\/pre>\n<p>In the browser, open http:\/\/&lt;IP-\u0430\u0434\u0440\u0435\u0441 \u0441\u0435\u0440\u0432\u0435\u0440\u0430>:9090, and if everything is done correctly, we get to the Prometheus monitoring system console. So far, it is useless, but you can already press and see the menus.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/374\/9fc\/aec\/3749fcaec7829e90271edf12049e3fa8.png\" width=\"902\" height=\"552\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/374\/9fc\/aec\/3749fcaec7829e90271edf12049e3fa8.png\"\/><figcaption><\/figcaption><\/figure>\n<p>So, let&#8217;s connect WCS to monitoring. Open the prometheus.yml settings file for editing<\/p>\n<pre><code class=\"bash\">sudo nano \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n<p>Add a description of the monitoring configuration to the file:<\/p>\n<pre><code class=\"bash\">scrape_configs:   - job_name: 'flashphoner'     metrics_path: '\/'     params:       action: [stat]       format: [prometheus]     static_configs:       - targets: ['WCS_address:8081']<\/code><\/pre>\n<p>where:<\/p>\n<ul>\n<li>\n<p>WCS_address &#8212; WCS server address;<\/p>\n<\/li>\n<li>\n<p>8081 &#8212; WCS server port for displaying statistics.<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/01e\/ffe\/e47\/01effee479abd3f7572ae72ba8227d65.png\" width=\"902\" height=\"802\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/01e\/ffe\/e47\/01effee479abd3f7572ae72ba8227d65.png\"\/><figcaption><\/figcaption><\/figure>\n<p>After changing the settings of prometheus.yml, restart the service: <\/p>\n<pre><code class=\"bash\">sudo systemctl restart prometheus<\/code><\/pre>\n<p>Return to the Prometheus console, select the &#171;Status => Targets&#187; menu item:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/8f7\/d86\/ac8\/8f7d86ac8806c03187458b7f99eeb35d.png\" width=\"904\" height=\"543\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/8f7\/d86\/ac8\/8f7d86ac8806c03187458b7f99eeb35d.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Make sure that Prometheus receives data from WCS (the target is in the &#171;UP&#187; state):<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/43a\/12a\/5d5\/43a12a5d503b22f80991b3b807bb49fb.png\" width=\"904\" height=\"495\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/43a\/12a\/5d5\/43a12a5d503b22f80991b3b807bb49fb.png\"\/><figcaption><\/figcaption><\/figure>\n<p>This concludes the monitoring setup. Let&#8217;s move on to setting up charts. <\/p>\n<h3>Grafana:  installation and setup<\/h3>\n<p>Grafana is a web interface for various time series DBMSs such as Graphite, InfluxDB and Prometheus. In general, Grafana can draw beautiful graphs using information from Prometheus. Earlier we saw that Prometheus also had its own web interface, but it was extremely minimalistic and rather inconvenient. Therefore, even the Prometheus developers themselves recommend using Grafana. Well, who are we to argue with the developers? <\/p>\n<p>Let&#8217;s get started. <\/p>\n<p>Grafana as well as Prometheus will be installed manually using the deb package. Find a link to a fresh package and download it using wget at <a href=\"https:\/\/grafana.com\/grafana\/download\">https:\/\/grafana.com\/grafana\/download<\/a>:<\/p>\n<pre><code class=\"bash\">wget https:\/\/dl.grafana.com\/oss\/release\/grafana_7.2.1_amd64.deb<\/code><\/pre>\n<p>Unpack it: <\/p>\n<pre><code class=\"bash\">sudo dpkg -i grafana_7.2.1_amd64.deb<\/code><\/pre>\n<p>Allow autorun:<\/p>\n<pre><code class=\"bash\">sudo systemctl enable grafana-server<\/code><\/pre>\n<p>Run it:<\/p>\n<pre><code class=\"bash\">sudo systemctl start grafana-server<\/code><\/pre>\n<p>Open the web interface at http:\/\/&lt;IP-\u0430\u0434\u0440\u0435\u0441 \u0441\u0435\u0440\u0432\u0435\u0440\u0430>:3000. Default login and password: admin\/admin. When you first log in, Grafana will immediately prompt you to change your password. <\/p>\n<p>Change the password and add Prometheus as a data source:<\/p>\n<p>Click the \u201cAdd data source\u201d button:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/793\/6d3\/85f\/7936d385f8e1fae11ffb727bb5a08496.png\" width=\"904\" height=\"506\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/793\/6d3\/85f\/7936d385f8e1fae11ffb727bb5a08496.png\"\/><figcaption><\/figcaption><\/figure>\n<p>and select Prometheus:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/c56\/61f\/746\/c5661f7462a81520d5e5ab1ce6e22ace.png\" width=\"904\" height=\"526\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/c56\/61f\/746\/c5661f7462a81520d5e5ab1ce6e22ace.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Specify the parameters for the connection. It is enough to specify the address and port of the Prometheus Web interface. Press the Save &amp; Test button to save:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/922\/f55\/6e9\/922f556e9cd6eb393f784e38b697d749.png\" width=\"904\" height=\"1011\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/922\/f55\/6e9\/922f556e9cd6eb393f784e38b697d749.png\"\/><figcaption><\/figcaption><\/figure>\n<p>After successful connection, create a new panel for charts. Choose from the menu on the left side of the Create \u2013 Dashboard window: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/ece\/897\/62b\/ece89762ba8a8a4883e0381b42bc42d7.png\" width=\"904\" height=\"493\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/ece\/897\/62b\/ece89762ba8a8a4883e0381b42bc42d7.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Click the \u201cAdd new panel\u201d button:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/16c\/98c\/321\/16c98c32169787d6f515c537dd396171.png\" width=\"904\" height=\"505\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/16c\/98c\/321\/16c98c32169787d6f515c537dd396171.png\"\/><figcaption><\/figcaption><\/figure>\n<p>On the \u201cQuery\u201d tab, select the previously created Prometheus (1) as a data source and select the metrics we are interested in from the list (2): <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/22a\/402\/bb9\/22a402bb9a2b476b3577aa9d04ff1b78.png\" width=\"904\" height=\"840\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/22a\/402\/bb9\/22a402bb9a2b476b3577aa9d04ff1b78.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Then you can go through the rest of the points by selecting specific parameters and the type of chart. After that, save the panel setting (3).<\/p>\n<p>Set the dashboard name and save: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/fb1\/ec5\/28d\/fb1ec528df56aaeb21d30a4ce1d49c6e.png\" width=\"904\" height=\"540\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/fb1\/ec5\/28d\/fb1ec528df56aaeb21d30a4ce1d49c6e.png\"\/><figcaption><\/figcaption><\/figure>\n<p>In the screenshot above, the graph displays all available metrics for stream_stats. <\/p>\n<p>You can make a graph for a specific metric that interests you. To do this, you need to specify its name in the \u201cMetrics\u201d field. For example, in the screenshot below, we selected the \u201cNumber of incoming RTSP streams\u201d metric:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/7f5\/ecc\/fab\/7f5eccfab9e7945f028afc33265c29a0.png\" width=\"904\" height=\"840\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/7f5\/ecc\/fab\/7f5eccfab9e7945f028afc33265c29a0.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Now let&#8217;s create graphs for the key metrics that we discussed at the beginning of this article.  <\/p>\n<p>Metric selection lines: <\/p>\n<p>Degraded streams percentage:<\/p>\n<pre><code class=\"bash\">degraded_streams_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"degraded_streams\"} degraded_streams_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"degraded_streams_percent\"}<\/code><\/pre>\n<p>Java heap:<\/p>\n<pre><code class=\"bash\">core_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"core_heap_memory_used\"}<\/code><\/pre>\n<p>Physical memory usage for Java:<\/p>\n<pre><code class=\"bash\">core_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"core_java_freePhysicalMemorySize\"} core_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"core_java_totalPhysicalMemorySize\"}<\/code><\/pre>\n<p>Transcoder queues and transcoding time:<\/p>\n<pre><code class=\"bash\">transcoding_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"transcoding_video_decoding_max_queue_size\"} transcoding_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"transcoding_video_encoding_max_queue_size\"} transcoding_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"transcoding_video_decoding_max_time\"} transcoding_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"transcoding_video_encoding_max_time\"}<\/code><\/pre>\n<p>Number of free ports for streaming:<\/p>\n<pre><code class=\"bash\">port_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"ports_media_free\"}<\/code><\/pre>\n<p>Audio and video synchronization in streams:<\/p>\n<pre><code class=\"bash\">streams_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", name=\"s1\", param=\"streams_synchronization\"}<\/code><\/pre>\n<p>Allow autorun:<\/p>\n<pre><code class=\"bash\">sudo systemctl enable node_exporter<\/code><\/pre>\n<p>Run the service:<\/p>\n<pre><code class=\"bash\">sudo systemctl start node_exporter<\/code><\/pre>\n<p>Open a web browser and go to http:\/\/:9100\/metrics \u2014 we will see the metrics collected by node_exporter:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/355\/d4b\/3ba\/355d4b3ba7095b15e3ca62977262df25.png\" width=\"902\" height=\"455\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/355\/d4b\/3ba\/355d4b3ba7095b15e3ca62977262df25.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Now let&#8217;s connect node_exporter on WCS to Prometheus server. <\/p>\n<p>On the Prometheus server, open the prometheus.yml configuration file for editing: <\/p>\n<pre><code class=\"bash\">sudo nano \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n<p>and add a new task to collect data from node_exporter to WCS. Be careful with the indentation in the yml file: <\/p>\n<pre><code class=\"bash\"> - job_name: 'WCS_node_exporter'     scrape_interval: 5s     static_configs:     - targets: ['demo.flashphoner.com:9100']<\/code><\/pre>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/deb\/623\/fd9\/deb623fd91888061b84e97ec68ebda33.png\" width=\"901\" height=\"658\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/deb\/623\/fd9\/deb623fd91888061b84e97ec68ebda33.png\"\/><figcaption><\/figcaption><\/figure>\n<p>After changing the settings file prometheus.yml, restart the service: <\/p>\n<pre><code class=\"bash\">sudo systemctl restart prometheus<\/code><\/pre>\n<p>Go to Grafana and add a graph for CPU load to the previously created dashboard:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/665\/bf0\/b3c\/665bf0b3cb359721351e847952e88fe2.png\" width=\"906\" height=\"837\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/665\/bf0\/b3c\/665bf0b3cb359721351e847952e88fe2.png\"\/><figcaption><\/figcaption><\/figure>\n<p>CPU metrics:<\/p>\n<pre><code class=\"bash\">node_load1 node_load5 node_load15<\/code><\/pre>\n<h3>If the required parameters are unavailable<\/h3>\n<p>WCS provides the ability to collect statistics using an external script.<\/p>\n<p>To enable this function, you need to specify the path to the script file for collecting statistics in the &#171;flashphoner.properties&#187; file, for example, for the custom_stats.sh file located in the \/usr\/local\/FlashphonerWebCallServer\/bin directory, it can be written as follows:<\/p>\n<pre><code class=\"bash\">custom_stats_script=custom_stats.sh <\/code><\/pre>\n<p>If you have a different path to the script file, then you need to specify it in full:<\/p>\n<pre><code class=\"bash\">custom_stats_script=\/path\/to\/custom_stats.sh<\/code><\/pre>\n<p>Let&#8217;s write a script that collects pause data for the Java garbage collector<\/p>\n<pre><code class=\"bash\">#!\/bin\/bash  WCS_HOME=\"\/usr\/local\/FlashphonerWebCallServer\" LAST_LOG=$(ls -t ${WCS_HOME}\/logs\/ | grep gc-core | head -1) LOG=\"${WCS_HOME}\/logs\/${LAST_LOG}\" JAVA_VER=$(java -version 2>&amp;1 | head -n 1 | awk -F '\"' '{print $2}')   TYPE_GC=\"$(grep -Pv '^(#|$)' ${WCS_HOME}\/conf\/wcs-core.properties | grep -oE 'ConcMarkSweepGC|ZGC')\"  #GC if [[ $JAVA_VER != \"1\"[0-9]* ]]; then         gc_pause=$(grep 'Allocation Failure' $LOG | tail -1 | awk -F'->' '{print $3}' | sed -rn 's\/([0-9]+)K\\(([0-9]+)K\\), ([0-9]+.[0-9]+).*\/\\3\/p' | tr , . | awk '{printf \"%f\\n\", $1 * 1000 }')         echo \"gc_pause=$gc_pause\" #ZGC elif [[ $JAVA_VER == \"1\"[0-9]* ]]; then     if [[ $TYPE_GC == \"ConcMarkSweepGC\" ]]; then         gc_pause=$(grep 'Allocation Failure' $LOG | tail -1 | awk '{print $8}' | sed 's\/ms$\/\/')         echo \"gc_pause=$gc_pause\"     fi     if [[ $TYPE_GC == \"ZGC\" ]]; then         gc_pause=$(grep '.*GC.*Pause' $LOG | awk -F 'Pause Mark Start|End|Relocate Start' '{print $2}' | tail -3 | sed 's\/ms$\/\/' | awk '{a=$1; getline;b=$1;getline;c=$1;getline;t=a+b+c;print t}')         echo \"gc_pause=$gc_pause\"     fi fi<\/code><\/pre>\n<p>This script requests for Java version installed in the system and, according to this information, finds lines with data on pauses for garbage collection for GC or ZGC in the log and outputs this value (in milliseconds) to the \u201cgc_pause\u201d parameter. <\/p>\n<p>The statistics that we have collected using an external script are displayed on the statistics page <a href=\"http:\/\/demo.flashphoner.com:8081\/?action=stat\">http:\/\/demo.flashphoner.com:8081\/?action=stat<\/a> in the \u201cCustom info\u201d section:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/4c3\/573\/a16\/4c3573a1618f7fbd337e36648fe26a06.png\" width=\"902\" height=\"279\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/4c3\/573\/a16\/4c3573a1618f7fbd337e36648fe26a06.png\"\/><figcaption><\/figcaption><\/figure>\n<p>The previously configured monitoring system Prometheus takes it from there. <\/p>\n<p>Now let&#8217;s add a graph for this metric to the dashboard in Grafana: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/07d\/21a\/f25\/07d21af256731ea5718d04e3c284cc17.png\" width=\"902\" height=\"837\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/07d\/21a\/f25\/07d21af256731ea5718d04e3c284cc17.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Metric for Java GC pause:<\/p>\n<pre><code class=\"bash\">custom_stats{instance=\"demo.flashphoner.com:8081\", job=\"flashphoner\", param=\"gc_pause\"}<\/code><\/pre>\n<p>General view of a dashboard with graphs for monitoring WCS performance metrics <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/87e\/143\/004\/87e143004f25b78471f8cd43debefe86.PNG\" width=\"1463\" height=\"1016\"\/><figcaption><\/figcaption><\/figure>\n<p>We have collected and built graphs for server performance metrics. Now let&#8217;s look at how you can build graphs for the parameters of a specific stream that will be published on the server. <\/p>\n<h3>Collecting parameters for WebRTC<\/h3>\n<p>If you execute a REST request<\/p>\n<pre><code class=\"bash\">\/stream\/metrics<\/code><\/pre>\n<p>the result will be a fairly extensive list of metrics. And, if everything is clear from the metrics like \u201cVIDEO_HEIGHT,\u201d \u201cVIDEO_WIDTH,\u201d \u201cVIDEO_CODEC\u201d and \u201cAUDIO_CODEC,\u201d then I propose to get acquainted with other metrics in more detail <\/p>\n<p>The metrics \u201cAUDIO_SYNC\u201d and \u201cVIDEO_SYNC\u201d are familiar metrics for audio and video synchronization in published streams.<\/p>\n<p>The metric  \u201cVIDEO_K_FRAMES\u201d \u2014 \u041athe number of the video\u2019s key frames (I-frames). A key frame contains all information about the image in the frame and does not depend on other frames. Keyframes have the lowest compression ratio. If the browser does not send keyframes evenly, this can lead to freezes when playing the stream. The following is closely related to this metric:<\/p>\n<p>The metric \u201cVIDEO_PLI\u201d \u2014 the number of PLI requests for keyframes from the browser. PLI can be sent when the stream receiver has lost a full frame or multiple frames. A large number of PLI requests indicates a bad channel. <\/p>\n<p>The metrics \u201cVIDEO_P_FRAMES\u201d and \u201cVIDEO_B_FRAMES\u201d \u2014 the number of P-frames (intermediate or tween) and B-frames (bidirectional), respectively. A tween can refer to blocks of an image in a previous key or previous tweens. This allows you to make intermediate frames smaller than the key ones: they contain less information about the image. A bi-directional frame does not contain the entire image either. But unlike a tween, it can also refer to subsequent blocks, up to the next tween \u2013 hence its name. Bidirectional frames take up even less space than intermediate frames. The method of decoding a bidirectional frame is even more complicated than decoding an intermediate frame: first, the next frame after the bidirectional intermediate frame is decoded according to the scheme described above; the bidirectional frame is then decoded using the information about neighboring frames. <\/p>\n<p>It turns out that the more often B frames appear in a stream, the more resources are required to decode this stream, which can negatively affect the video quality. Moreover, browsers cannot play B-frames correctly, and if there are bidirectional frames, the image will twitch.<\/p>\n<p>The metrics \u201cAUDIO_RATE\u201d and \u201cVIDEO_RATE\u201d show the bitrate of audio or video. <\/p>\n<p>The metric \u201cVIDEO_FPS\u201d refers to video frame rate.<\/p>\n<p>Three more metrics related to packet loss on communication channels:<\/p>\n<p>\u201cVIDEO_NACK\u201d \u2014 the number of NACK requests. <a href=\"https:\/\/flashphoner.com\/what-is-webrtc-nack\/\">NACK<\/a> \u2014 it is a way for the receiving party to indicate that it has not received a particular packet or list of packets. <\/p>\n<p>\u201cVIDEO_LOST\u201d \u2014 the number of video packets lost.<\/p>\n<p>\u201cAUDIO_LOST\u201d \u2014 the number of audio packets lost.<\/p>\n<p>A large number of NACK requests and lost packets also mean problems with communication channels. This is manifested by freezing in the stream, loss of sound and other unpleasant stuff.<\/p>\n<p>The aforementioned parameters for published streams are available in the web interface and on the statistics page: <\/p>\n<pre><code class=\"bash\">http:\/\/demo.flashphoner.com:8081\/?action=stat&amp;format=prometheus<\/code><\/pre>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e25\/716\/772\/e25716772b386cf0996d284af9964c2a.png\" width=\"817\" height=\"365\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e25\/716\/772\/e25716772b386cf0996d284af9964c2a.png\"\/><figcaption><\/figcaption><\/figure>\n<p>The parameters are displayed on the statistics page if the server has at least one published stream. Now, let us add them to the data that Prometheus monitors. To do this, in the settings file prometheus.yml<\/p>\n<pre><code class=\"bash\">sudo nano \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n<p>add a new task to collect data on metrics for a stream. Be careful with the indentation in the yml file:<\/p>\n<pre><code> - job_name: 'stream_metrics'     metrics_path: '\/'     params:       action: [stat]       format: [prometheus]       groups: [publish_streams]     static_configs:     - targets: ['demo.flashphoner.com:8081']<\/code><\/pre>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/aa1\/960\/23e\/aa196023e913ee42aa8821ad591d59be.png\" width=\"901\" height=\"658\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/aa1\/960\/23e\/aa196023e913ee42aa8821ad591d59be.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Save the configuration and restart Prometheus:<\/p>\n<pre><code class=\"bash\">sudo systemctl restart prometheus<\/code><\/pre>\n<p>Moving on to Grafana. I propose to create a new panel for stream metric charts. Select Create \u2013 Dashboard from the menu on the left side of the window <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/313\/9ed\/392\/3139ed392b37c99d4c4062f7f7c18a58.png\" width=\"904\" height=\"493\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/313\/9ed\/392\/3139ed392b37c99d4c4062f7f7c18a58.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Click the \u201cAdd new panel\u201d button<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/949\/04f\/4f0\/94904f4f0ddd6dc7fc8d53e3834d1a7a.png\" width=\"904\" height=\"505\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/949\/04f\/4f0\/94904f4f0ddd6dc7fc8d53e3834d1a7a.png\"\/><figcaption><\/figcaption><\/figure>\n<p>On the \u201cQuery\u201d tab, select Prometheus as a data source and select publish => publish_streams from the list of metrics: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/87c\/ab8\/44e\/87cab844ea465818c906361ce38a9ce3.png\" width=\"902\" height=\"780\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/87c\/ab8\/44e\/87cab844ea465818c906361ce38a9ce3.png\"\/><figcaption><\/figcaption><\/figure>\n<p>As a result, we get graphs for all available metrics for each published stream. For clarity, you can display the metrics of interest on a separate graph. For example, a list of metrics for a stream published on demo.flashphoner.com named \u201cstream1\u201d<\/p>\n<pre><code class=\"bash\">publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"AUDIO_CODEC\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"AUDIO_LOST\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"AUDIO_RATE\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"AUDIO_SYNC\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_B_FRAMES\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_CODEC\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_FPS\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_HEIGHT\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_K_FRAMES\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_LOST\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_NACK\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_PLI\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_P_FRAMES\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_RATE\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_SYNC\"} publish_streams{instance=\"demo.flashphoner.com:8081\", job=\"stream_metrics\", name=\"stream1\", param=\"VIDEO_WIDTH\"}<\/code><\/pre>\n<p>An example of such a dashboard is below:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/c55\/7cd\/770\/c557cd770fc70102efeb82e4b04c61b9.PNG\" width=\"1457\" height=\"983\"\/><figcaption><\/figcaption><\/figure>\n<p>As you can see, Prometheus is an interesting and promising tool. Our article cannot claim to be a complete description of this monitoring system, because we have reviewed a special case here, applicable to monitoring WCS and streams. For example, with Prometheus you can also set up alerts, which we did not consider. After reading this article, you now know how Prometheus and Grafana are used, and you can successfully apply them in practice. <\/p>\n<p>Have a nice streaming!<\/p>\n<h3>Links<\/h3>\n<p><a href=\"https:\/\/demo.flashphoner.com\/admin\/login.html\">Demo<\/a><\/p>\n<p><a href=\"https:\/\/flashphoner.com\/amazon-ec2-support-in-web-call-server\">WCS on Amazon EC2<\/a><\/p>\n<p><a href=\"https:\/\/flashphoner.com\/web-call-server-on-digital-ocean-marketplace\">WCS on DigitalOcean<\/a><\/p>\n<p><a href=\"https:\/\/flashphoner.com\/support-web-call-server-in-docker\">WCS in Docker<\/a><\/p>\n<p><a href=\"https:\/\/docs.flashphoner.com\/display\/WCS52EN\/Quick+deployment+and+testing+of+the+server\">Quick deployment and testing of the server<\/a><\/p>\n<p><a href=\"https:\/\/docs.flashphoner.com\/display\/WCS52EN\/Load+and+resource+usage+information\">Load and resource usage information<\/a><\/p>\n<p><a href=\"https:\/\/docs.flashphoner.com\/display\/WCS52EN\/Connecting+from+Prometheus\">Monitoring with Prometheus<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/568790\/\"> https:\/\/habr.com\/ru\/articles\/568790\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>Monitoring systems are a vital tool for any system administrator, because they can be used to extract specific information from services, such that: <\/p>\n<ul>\n<li>\n<p>Provides a general idea regarding server loads;<\/p>\n<\/li>\n<li>\n<p>Shows whether the service is stable and whether it is operating properly.<\/p>\n<\/li>\n<\/ul>\n<p>This information can then be used to draw conclusions and make decisions on whether to scale the server power up or down, whether to broaden or narrow the communication channels, etc. For instance, if we see that the CPU load average is 0.1% while at 48 cores, then it becomes obvious that we are wasting resources that can be used for other purposes.  <\/p>\n<p>When it comes to WebRTC streaming, we are not just concerned with loads on CPU, memory and communication channels \u2013 the numbers of streams and viewers are also important, as are data on whether the stream quality is degrading or not.<\/p>\n<p>Some will say that monitoring systems are pointless, because all the data on WebRTC can be found in Chrome, the server data can be found on the server, and various costs calculators (for network usage, server hosting, etc.) are available in respective personal accounts. Imagine, however, what a nightmare it is to monitor all this in real time, while actively streaming. Without a doubt, it is not a user-friendly process, and, while streaming this way, it is easy to miss something important. <\/p>\n<p>Apart from parameter monitoring, a monitoring system can also be used to timely detect and even resolve typical issues that arise during streaming.<\/p>\n<p>All this is to say that a monitoring system is an absolute must-have. The only question is, which to opt for. <\/p>\n<h3>What parameters are we going to monitor?<\/h3>\n<p>The quality of the stream, as has already been discussed many times, is not constant and depends on many factors, such as the load on the server with or without transcoding, and the use of TCP or UDP transport protocols, and the presence of packet loss and\/or NACK feedbacks, etc. All these data for assessing stream quality can be obtained manually from various sources. <\/p>\n<p>Stream degradation is a condition of a video\/audio stream, in which the picture and sound quality is not satisfactory. There are artifacts, friezes, stuttering, or out of sync sound.  <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Parameter 1: CPU Load average<\/h4>\n<p>If the processor is loaded by more than 80%, and\/or the CPU load periodically rises to 100%, this means that the server is overloaded and does not have enough processing power to perform work operations, which will certainly lead to degradation of streams.<\/p>\n<p>You can estimate the current processor load using the htop software. Or top. Or mpstat. The screenshot below shows htop: <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>Usually, the Load average parameter is estimated \u2013 the average value of the system load. This parameter is displayed as three values, which represent the average load data for the last 1, 5 and 15 minutes. The lower the value, the better.  <\/p>\n<pre><code class=\"bash\">load average:  4.55 4.22 4.18<\/code><\/pre>\n<p>For example, such values for a quad-core processor mean that it works at the limit of its capabilities for a long time (several hours), and such work will definitely lead to degradation of streams. <\/p>\n<h4>Parameter 2: Percentage of degraded streams<\/h4>\n<p>The next metric, which is directly related to the previous one: degraded_streams_percent &#8212; the percentage of degraded streams. A stream is degraded if the server does not have time to process the queue of this stream. The server is multithreading. The video data of the published stream is queued, and the data is sent in a separate thread. In case of a high load on the CPU and a lack of performance for encoding\/decoding or with internal locks, these queues can grow, and streams can degrade. Therefore, the percentage of degraded streams is an important metric that signals a loss of quality.<\/p>\n<p>You can find the value of degraded_streams_percent on the statistics page <a href=\"http:\/\/demo.flashphoner.com:8081\/?action=stat\">http:\/\/demo.flashphoner.com:8081\/?action=stat<\/a> (where demo.flashphoner.com is the address of the WCS server)<\/p>\n<p>The following few metrics are relevant to the Java Virtual Machine: <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Parameter 3: Java GC pause<\/h4>\n<p>Garbage collection pause time. Like in the case of the Load average, the lower the number, the better. 100 ms is very bad. The situation when a Java machine starts the garbage collector is called \u201cStop the world,\u201d which means that your application will be completely stopped while the garbage collection is running. All server threads will be stopped for a moment, and this is enough to degrade the streams.<\/p>\n<h4>Parameter 4: Java Heap<\/h4>\n<p>The size of Java Heap memory used. If it approaches -Xmx and remains constantly at these values, this means the constant work of the garbage collector,  and as a result, high CPU consumption, possible degradation of streams, as well as an increase in the likelihood of the \u201cOutOfMemoryError\u201d error, which is a signal of an irreversible violation of the internal state server. If such an error occurs at least once, the server is already broken, so it is recommended to restart it. When an OutOfMemoryError appears, the Java machine can stop any thread, thereby disrupting server operation, although externally the server will continue to work and service connections.<\/p>\n<h4>Parameter 5: Physical memory<\/h4>\n<p>The -Xmx limit for heap does not guarantee a limit on RES physical memory usage. If the heap was given 16 GB, RES can take 32 GB or more, and this is not always a memory leak. Especially in heavy mixing and transcoding scenarios. Therefore, the \u201creal memory\u201d must be monitored and not be exhausted, since this will terminate the server process by the system itself.<\/p>\n<p>These metrics can be tracked using the Java Mission Control tool. Java Mission Control is a powerful tool for monitoring and debugging a Java machine that comes with the JDK and runs on the user&#8217;s (administrator&#8217;s) PC.<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Parameter 6: Transcoder queues and transcoding time<\/h4>\n<p>Transcoding is a separate scenario with a high load on server resources. In paragraph 2, we already wrote about Degraded streams. Now, this is a later metric. In the case of transcoders, first the metrics of the queues of transcoders will show the excess and only then the Degraded streams will react. Transcoder queue metrics show the number of frames accumulated in the decode\/encoding queue. Frames are heavy, and 30 accumulated frames means 1 second of latency with a frame rate of 30 FPS. The decoding\/encoding time tells us how quickly the corresponding operations go. For example, at 30 FPS, the encoding operation for one frame should take no more than 1\/30 second = 33 milliseconds. Otherwise, the transcoder will not be able to deliver frames on time, which will ruin the stream.<\/p>\n<p>Decoding\/encoding queue metrics:<\/p>\n<p>transcoding_video_decoding_max_queue_size <\/p>\n<p>transcoding_video_encoding_max_queue_size<\/p>\n<p>Maximum decoding\/encoding time metrics<\/p>\n<p>transcoding_video_decoding_max_time <\/p>\n<p>transcoding_video_encoding_max_time<\/p>\n<p>These metrics are available on the statistics page:<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Number of free ports for streaming<\/h4>\n<p>Metric: ports_media_free<\/p>\n<p>An important metric for highly loaded servers. Ports are allocated for each stream. If the ports in the range run out, adding new streams on the server will cause an error. <\/p>\n<p>Information on the number of ports can be found on the statistics page:  <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Stream audio and video synchronization<\/h4>\n<p>Metric: streams_synchronization<\/p>\n<p>An important metric, which is also called lipsync (synchronization of lips and voice in video broadcasts). Out of sync can be caused by a variety of reasons, ranging from poor quality stream source to server performance issues. With an out of sync of 100 milliseconds or more, the human ear and eye will already notice that something is wrong. The metric allows you to see this moment in numbers, in milliseconds for each stream; a positive value of the metric shows that audio is playing ahead of the video, and a negative value indicates that audio is lagging behind the video.  <\/p>\n<p>The screenshot below shows normal synchronization of audio and video in streams. The metric shows values from +2 to +79 milliseconds.<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Parameter 9: Number of open transcoders<\/h4>\n<p>Transcoders, namely encoders, have the maximum impact on server load. Therefore, it is important to control their number. If your script does not involve the use of transcoding (for example, streaming without changing the resolution), then the number of decoders and encoders should be zero.<\/p>\n<p>Metrics:<\/p>\n<p>native_resources.video_decoders <\/p>\n<p>native_resources.video_encoders<\/p>\n<p>In the screenshot below, transcoding is not used.<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h4>Parameter 10: Total load on transcoders<\/h4>\n<p>The number of transcoders is a good metric, but the two encoders can differ by an order of magnitude in terms of CPU utilization. For example, stream encoding 240p 15 FPS and 1080p 60 FPS. Therefore, to get an idea of the loading, it is important to consider not only the number of encoders, but their weight. The encoder weight is: w x h x FPS, where w and h is the image resolution. As a result, the metric allows to display the real load of the server by encoders, and one heavyweight encoder can show a higher load than 10 lightweight ones.<\/p>\n<p>Metrics:<\/p>\n<p>transcoding_video_decoding_load <\/p>\n<p>transcoding_video_encoding_load<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h3>Preparation for monitoring system deployment<\/h3>\n<p>If your system uses a firewall, you need to open ports <\/p>\n<ul>\n<li>\n<p>TCP 9090 \u2014 http for Prometheus server;<\/p>\n<\/li>\n<li>\n<p>TCP 9093 \u2014 http for alert manager;<\/p>\n<\/li>\n<li>\n<p>TCP and UDP 9094 \u2014 for alert manager;<\/p>\n<\/li>\n<li>\n<p>TCP 9100 \u2014 for node_exporter;<\/p>\n<\/li>\n<li>\n<p>TCP 3000 \u2014 http for Grafana server.<\/p>\n<\/li>\n<\/ul>\n<p>For example, using iptables:<\/p>\n<pre><code class=\"bash\">iptables -I INPUT 1 -p tcp --match multiport --dports 9090,9093,9094,9100 -j ACCEPT iptables -A INPUT -p udp --dport 9094 -j ACCEPT<\/code><\/pre>\n<p>If you are using a Red Hat-based operating system, SELinux will probably be enabled by default. You can disable it using the commands <\/p>\n<pre><code class=\"bash\">sudo setenforce 0 sudo sed -i 's\/^SELINUX=.*\/SELINUX=disabled\/g' \/etc\/selinux\/config<\/code><\/pre>\n<p>We are now ready to install Prometheus and Grafana. <\/p>\n<h3>Prometheus: installation and setup<\/h3>\n<p>Prometheus does not have its own repositories for popular distributions, and the official operating system repositories tend to lag behind the upstream. Therefore, we will look at the manual installation. This method is suitable for any Linux<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-389564","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/389564","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=389564"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/389564\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=389564"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=389564"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=389564"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}