{"id":385738,"date":"2024-06-29T06:17:21","date_gmt":"2024-06-29T06:17:21","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=385738"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=385738","title":{"rendered":"<span>Prometheus in Action: from default counters to SLO-related queries<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h2>A Gentle Intro<\/h2>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/73d\/ffb\/cb3\/73dffbcb3f6469ec1a746bbbf6e61de1.png\" width=\"150\" height=\"149\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/73d\/ffb\/cb3\/73dffbcb3f6469ec1a746bbbf6e61de1.png\"\/><figcaption><\/figcaption><\/figure>\n<p>All Prometheus metrics are based on <strong>time series<\/strong> &#8212; streams of timestamped values belonging to the same metric. Each time series is uniquely identified by its metric name and optional key-value pairs called labels. The metric name specifies some characteristics of the measured system, such as <code>http_requests_total<\/code> &#8212; the total number of received HTTP requests. In practice, you often will be interested in some subset of the values of a metric, for example, in the number of requests received by a particular endpoint; and here is where the labels come in handy. We can partition a metric by adding <code>endpoint<\/code> label and see the statics for a particular endpoint: <code>http_requests_total{endpoint=\"api\/status\"}.<\/code> Every metric has two automatically created labels: <code>job_name<\/code> and <code>instance<\/code>. We see their roles in the next section.<\/p>\n<p>Prometheus provides a functional query language called PromQL. The result of the query might be evaluated to one of four types:<\/p>\n<ul>\n<li>\n<p><strong>Scalar<\/strong> (aka float)<\/p>\n<\/li>\n<li>\n<p><strong><em>String<\/em><\/strong><em> (currently unused)<\/em><\/p>\n<\/li>\n<li>\n<p><strong>Instant Vector<\/strong> &#8212; a set of time series that have exactly one value per timestamp.<\/p>\n<\/li>\n<li>\n<p><strong>Range Vector<\/strong> &#8212; a set of time series that have a range of values between two timestamps.<\/p>\n<\/li>\n<\/ul>\n<p>At first glance, <strong>Instant Vector<\/strong> might look like an array, and <strong>Range Vector<\/strong> as a matrix.<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/78a\/6eb\/f23\/78a6ebf2306a7ce5d82e83490334654f.jpg\" width=\"1576\" height=\"634\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/78a\/6eb\/f23\/78a6ebf2306a7ce5d82e83490334654f.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>If that would be the case, then a Range Vector for a single time series &#171;downgrades&#187; to an Instant Vector. However, that&#8217;s not the case: the difference between a Range Vector and an Instant Vector is not in the number of tracked time series but in the relation between a value of the metric and the corresponding timestamp. In Instant Vector a time series has a single value at the timestamp, in Range Vector a time series has an arbitrary number of values between two timestamps. Therefore, we can more accurately visualize a Range Vector in the following form:<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/407\/683\/349\/407683349c1c2f0ec09669cd941d65ff.jpg\" width=\"1677\" height=\"595\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/407\/683\/349\/407683349c1c2f0ec09669cd941d65ff.jpg\" data-blurred=\"true\"\/><figcaption><\/figcaption><\/figure>\n<p>The difference between an instant vector and a range vector will be more clearly seen in the action: let&#8217;s instrument a Go application and see how can Prometheus help us with insights.<\/p>\n<p>A <strong>Counter<\/strong> is a metric that only goes up, for example, a number of the incoming HTTP requests.<\/p>\n<p>A <strong>Gauge<\/strong> is a metric that can have an arbitrary float value, for example, current CPU usage.<\/p>\n<p>A <strong>Histogram<\/strong> summarizes observations into statistical buckets, for example, we can track an API&#8217;s response times. Why would one need a histogram and can&#8217;t just measure the average response time? This is because we are not always interested in the average response times only. Let&#8217;s say we have an SLO and 95% of the requests should take no longer than 300 ms. We need to single out all requests that took 300 ms or less, count their amount, and divide it by the total number of the request. To do so in Prometheus, we configure a histogram to have a bucket with an upper limit of 0.3 seconds for a response. Later we see how can we configure an alert if we fail to comply with the SLO.<\/p>\n<p>There is also a <strong>Summary,<\/strong> however it is left mostly for historical reasons and serves the same purpose as a histogram.<\/p>\n<h2>Instrumenting a Go application<\/h2>\n<p>Let&#8217;s start <strong>without<\/strong> a Prometheus server.<\/p>\n<pre><code class=\"go\">package main  import (     \"net\/http\"      \"github.com\/prometheus\/client_golang\/prometheus\/promhttp\" )  func main() {     http.Handle(\"\/metrics\", promhttp.Handler())     http.ListenAndServe(\":2112\", nil) } <\/code><\/pre>\n<p><code>promhttp.Handler<\/code> returns an http.Handler which is already instrumented with the default metrics. Let&#8217;s see what we get right out of the box by issuing a GET request to the <code>\/metrics<\/code> endpoint.<\/p>\n<blockquote>\n<p><code>http localhost:2112\/metrics<\/code><\/p>\n<\/blockquote>\n<pre><code class=\"bash\"># HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile=\"0\"} 0 go_gc_duration_seconds{quantile=\"0.25\"} 0 go_gc_duration_seconds{quantile=\"0.5\"} 0 go_gc_duration_seconds{quantile=\"0.75\"} 0 go_gc_duration_seconds{quantile=\"1\"} 0 go_gc_duration_seconds_sum 0 go_gc_duration_seconds_count 0 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 8 # HELP go_info Information about the Go environment. # TYPE go_info gauge go_info{version=\"go1.15.5\"} 1 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge  [truncated] <\/code><\/pre>\n<p>As we can see we have a rather impressive list of metrics of all types: counters, gauges, and summaries. <\/p>\n<p>One of particular interest is a gauge for currently existing goroutines:<\/p>\n<pre><code class=\"bash\"># HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 7 <\/code><\/pre>\n<p>Note, that it is the number of <strong>existing<\/strong> goroutines, but not the <strong>running<\/strong> ones: some of the goroutines might be in the suspended state. One of the goroutines is our <code>main<\/code> function; the rest are running helper functions from <code>runtime<\/code> package, and are responsible for tasks such as garbage collection. Later we see how we can use this metric to reveal potential memory leaks. <\/p>\n<p>At the end very end, we see a familiar metric for the received HTTP request with label <code>code<\/code>.<\/p>\n<pre><code>promhttp_metric_handler_requests_total{code=\"200\"} 1 promhttp_metric_handler_requests_total{code=\"500\"} 0 promhttp_metric_handler_requests_total{code=\"503\"} 0 <\/code><\/pre>\n<p>At the next curl request, the metric will be increased by one.<\/p>\n<p>Now, this is already looking promising, but how can we instrument <strong>our own<\/strong> code? Let&#8217;s add a simple metric for the processed orders; as this number can only increase, the counter type is a natural choice.<\/p>\n<pre><code class=\"go\">package main  import (     \"net\/http\"     \"time\"      \"github.com\/prometheus\/client_golang\/prometheus\"     \"github.com\/prometheus\/client_golang\/prometheus\/promauto\"     \"github.com\/prometheus\/client_golang\/prometheus\/promhttp\" )  var counter = promauto.NewCounter(prometheus.CounterOpts{     Name: \"orders_processed\", })  func main() {     go func() {         for {             counter.Inc()             \/\/ simulate some processing function             time.Sleep(time.Second)         }     }()      http.Handle(\"\/metrics\", promhttp.Handler())     http.ListenAndServe(\":2112\", nil) } <\/code><\/pre>\n<p><code>prometheus<\/code> package provides metrics data types, and <code>promauto<\/code> package automates some routine tasks, such as metric registration.<\/p>\n<p>Now, we can see how well our processing goes:<\/p>\n<blockquote>\n<p>http localhost:2112\/metrics | grep orders_processed<\/p>\n<\/blockquote>\n<pre><code class=\"bash\"># HELP orders_processed # TYPE orders_processed counter orders_processed 7 <\/code><\/pre>\n<p>If we issue the request again we can verify that the processing is up and running:<\/p>\n<pre><code class=\"bash\"># HELP orders_processed # TYPE orders_processed counter orders_processed 10 <\/code><\/pre>\n<p>If you have a feeling that these values are not that illuminating, then you are on the right track. <strong>Prometheus is a statistical instrument<\/strong> at its core: it is intended to work with the trends, and not with the individual results. Later we see how to get some insights with <code>rate<\/code> and <code>increase<\/code> functions. <\/p>\n<p>Now, we have already instrumented our application <strong>and have not yet run Prometheus itself.<\/strong> Why do we need one? A Prometheus server allows us to aggregate data from several instances of our application. It scrapes our application and collects the values of metrics. Here is a simple Prometheus configuration:<\/p>\n<pre><code class=\"json\">scrape_configs:   - job_name: myapp     scrape_interval: 10s     static_configs:       - targets:           - 127.0.0.1:2112 <\/code><\/pre>\n<p>Prometheus server will scrape our application on port 2112 every 10 seconds and collects the metrics. Let&#8217;s bind this configuration to a docker container and run a Prometheus server:<\/p>\n<pre><code class=\"bash\">docker run \\     --network \"host\" \\     -p 9090:9090 \\     -v $HOME\/GolandProjects\/go-learn\/prometheus\/prometheus.yml:\/etc\/prometheus\/prometheus.yml \\     prom\/prometheus <\/code><\/pre>\n<p>Note, that we have to enable networking in <code>host<\/code> mode: our docker container needs to access <code>metric<\/code> endpoint in our application which is outside of the docker network. <\/p>\n<p>Now we can access Prometheus UI on <a href=\"http:\/\/localhost:9090\" rel=\"noopener noreferrer nofollow\">localhost:9090<\/a>. Here we can see that we are successfully scraping metrics from our application. <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/d0a\/3bf\/e58\/d0a3bfe582da4f798e6a0ea8601221a9.png\" width=\"1453\" height=\"459\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/d0a\/3bf\/e58\/d0a3bfe582da4f798e6a0ea8601221a9.png\"\/><figcaption><\/figcaption><\/figure>\n<p>How many goroutines do we have? In a table view we see the last recorded value:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/b45\/8fd\/d34\/b458fdd34576521cb00bdfede6a07c72.png\" width=\"933\" height=\"349\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/b45\/8fd\/d34\/b458fdd34576521cb00bdfede6a07c72.png\"\/><figcaption><\/figcaption><\/figure>\n<p>And in a table view we can see the trend:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f47\/967\/c98\/f47967c98ebb48710a3091cb8690b6c3.png\" width=\"942\" height=\"831\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f47\/967\/c98\/f47967c98ebb48710a3091cb8690b6c3.png\"\/><figcaption><\/figcaption><\/figure>\n<p>The result of this query (<code>go_goroutines<\/code>) is an instant vector that contains a single time series. What will be the result for the same query when we run a second instance of our application? Then <code>instance<\/code> label will have two possible values for two instances, therefore we get two time series. And as noted above we still have an instant vector, not a range vector. <\/p>\n<p>Let&#8217;s add one more target to our configuration and start the second instance of our application at port <code>21112<\/code>.<\/p>\n<pre><code class=\"json\">scrape_configs:   - job_name: myapp     scrape_interval: 10s     static_configs:       - targets:           - 127.0.0.1:2112           - 127.0.0.1:21112 <\/code><\/pre>\n<p>Here we can see the number of goroutines for both our instances. <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e3f\/819\/d8c\/e3f819d8c4a475d88bfcadf878b90789.png\" width=\"925\" height=\"817\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e3f\/819\/d8c\/e3f819d8c4a475d88bfcadf878b90789.png\"\/><figcaption><\/figcaption><\/figure>\n<p>If we provide a time range we get a range vector: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/6e6\/a34\/dba\/6e6a34dba183d4aedd110730d73497bc.png\" width=\"1723\" height=\"507\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/6e6\/a34\/dba\/6e6a34dba183d4aedd110730d73497bc.png\"\/><figcaption><\/figcaption><\/figure>\n<p>In this case, we requested a <strong>range<\/strong> of values between two timestamps: now and five minutes in the past with a step equal to 1 minute. In the output, symbol <code>@<\/code>separates a value of a metric from its timestamp.<\/p>\n<p>Neither <code>Prometheus<\/code> no <code>Grafana<\/code> can picture the graph for a range vector. However, all function that takes a range vector as input returns an instant vector which in turn can be pictured. <\/p>\n<h2>Prometheus Functions and Modifiers<\/h2>\n<p>The two most useful range functions are <code>increase<\/code> and <code>rate;<\/code> both of them should be used only with counters. Let&#8217;s have a look at our <code>orders_processed<\/code> metric. <\/p>\n<p>Prometheus is happy to tell us that one of the instances has processed ~7000 orders so far. <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/85c\/67e\/3f3\/85c67e3f392f27ae56917ed44f38a5d0.png\" width=\"936\" height=\"320\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/85c\/67e\/3f3\/85c67e3f392f27ae56917ed44f38a5d0.png\"\/><figcaption><\/figcaption><\/figure>\n<p>What insights can we get from this number? Actually, not that much. For example, it would be more interesting to know how many orders did we processed recently. Here we calculate the increase in the number of processed orders during the last 5 minutes. <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/6d7\/c95\/733\/6d7c95733b6ef90273ff2bdce5de941e.png\" width=\"939\" height=\"328\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/6d7\/c95\/733\/6d7c95733b6ef90273ff2bdce5de941e.png\"\/><figcaption><\/figcaption><\/figure>\n<p>This number could shed some light on the health of our application if we knew how many orders we expected to process under normal conditions. Let&#8217;s compare it with some reference point in the past: let&#8217;s imagine that we just deployed a nightly build of our application, and two hours ago we were running a thoroughly tested stable version.<\/p>\n<p>Modifier <code>offset<\/code>changes the time offset for the vector in the query:  <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/fb5\/ce0\/867\/fb5ce08670a34f50d067f6ed5abf4ffd.png\" width=\"934\" height=\"359\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/fb5\/ce0\/867\/fb5ce08670a34f50d067f6ed5abf4ffd.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Currently, we have roughly the same increase in the number of orders as 2 hours ago: therefore, we can <strong>assume<\/strong> that our new build didn&#8217;t break anything   (yet). <\/p>\n<p>What is our current rate of order processing? <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/0b9\/c13\/f61\/0b9c13f61a4a32515ad54d6d158f32ef.png\" width=\"935\" height=\"255\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/0b9\/c13\/f61\/0b9c13f61a4a32515ad54d6d158f32ef.png\"\/><figcaption><\/figcaption><\/figure>\n<p>We processed 197 orders during the last 5 minutes, therefore the per-second average rate of increase is calculated as <code>197 orders \/ 5 * 60 seconds<\/code> and is equal to ~0.7 orders per second. Prometheus has in-build function <code>rate<\/code> for this purpose:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/5f9\/4f3\/e1c\/5f94f3e1cbbccae5dc437047d044cd05.png\" width=\"934\" height=\"246\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/5f9\/4f3\/e1c\/5f94f3e1cbbccae5dc437047d044cd05.png\"\/><figcaption><\/figcaption><\/figure>\n<p>As you can see from the graphs <code>rate<\/code> and <code>increase<\/code> are basically reveals the same pattern.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/07f\/d95\/4ae\/07fd954aef83b38080fed4cb09d3ba2e.png\" width=\"923\" height=\"731\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/07f\/d95\/4ae\/07fd954aef83b38080fed4cb09d3ba2e.png\"\/><figcaption><\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/caa\/d0e\/cde\/caad0ecde82ac792d65be352935d6cbc.png\" width=\"918\" height=\"725\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/caa\/d0e\/cde\/caad0ecde82ac792d65be352935d6cbc.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Now, how can we calculate the total rate of order processing for all instances of the application? We need to sum the rates of all instances: <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/67a\/554\/a64\/67a554a64f40b41cba9c6f1a9019922f.png\" width=\"918\" height=\"306\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/67a\/554\/a64\/67a554a64f40b41cba9c6f1a9019922f.png\"\/><figcaption><\/figcaption><\/figure>\n<p>However, if we try to sum rates of different instances we get <code>Empty query result<\/code>. This happens because before the application of a binary operation Prometheus selects time series that have exactly the same set of labels from the left and right operands. In our case, Prometheus didn&#8217;t find matching time series due to the difference in <code>instance<\/code> label. <\/p>\n<p>We can use <code>on<\/code> keyword to specify that we only want to match by <code>job<\/code> label (and effectively ignore the difference in <code>instance<\/code> label)<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/6bd\/c67\/c0f\/6bdc67c0fcacdbec9d4a9b1d0b8f5231.png\" width=\"934\" height=\"252\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/6bd\/c67\/c0f\/6bdc67c0fcacdbec9d4a9b1d0b8f5231.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Or we can use <code>ignoring<\/code> keyword to explicitly ignore <code>instance<\/code> label:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/48a\/5f0\/f7f\/48a5f0f7fd810664deaaac3aca9ed4cc.png\" width=\"908\" height=\"284\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/48a\/5f0\/f7f\/48a5f0f7fd810664deaaac3aca9ed4cc.png\"\/><figcaption><\/figcaption><\/figure>\n<p>What happens if we have ten instances of our application instead of two? Especially, taking into consideration that Kubernetes is free to kill any instance at any time, and spin up a new one. Prometheus offers aggregate functions that relieve us from the manual bookkeeping:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e98\/052\/112\/e98052112e5aa8b339f952d1e144cee3.png\" width=\"930\" height=\"688\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e98\/052\/112\/e98052112e5aa8b339f952d1e144cee3.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Note, that we applied aggregate operaion <code>sum<\/code> to <code>rate<\/code> function and not vise versa. We always take <code>rate<\/code> first and then apply the aggregation; otherwise, <code>rate<\/code> cannot detect counters restarts when the application restarts. As <a href=\"https:\/\/www.robustperception.io\/rate-then-sum-never-sum-then-rate\" rel=\"noopener noreferrer nofollow\">Robust Perception Blog<\/a> puts it: <\/p>\n<blockquote>\n<p>The only mathematical operations you can safely directly apply to a counter&#8217;s values are rate, irate, increase, and resets. Anything else will cause you problems.<\/p>\n<\/blockquote>\n<p><code>orders_processed<\/code> is a rather generic metric; it could possibly have another label <code>application<\/code> that partitions it by different applications. In this case query <code>sum(rate(orders_processed[5m]))<\/code> returns the total rate for all applications which would be rather useless. We calculate the rate per application by using <code>by<\/code> keyword: <code>sum by (application) (rate(orders_processed[5m]))<\/code>.<\/p>\n<p>We can check a potential memory leak by comparing the current total number of goroutines for the application with the total number of goroutines recorded an hour ago:<\/p>\n<pre><code>sum(rate(go_goroutines[5m] offset 1h)) \/  sum(rate(go_goroutines[5m])) <\/code><\/pre>\n<h2>SLO-related queries<\/h2>\n<h3>API Errors Rate<\/h3>\n<p>Let&#8217;s say we have counter <code>http_requests_total<\/code> that tracks the number of the received requests and has label <code>status_code<\/code>.<\/p>\n<p>We can determine the rate for the failed requests due to server errors:<\/p>\n<pre><code>rate(http_requests_total{status_code=~\"5.*\"}[5m]) <\/code><\/pre>\n<p>We calculate the error ratio by dividing the rate of failed request on the total rate:<\/p>\n<pre><code>sum(rate(http_requests_total{status_code=~\"5.*\"}[5m])  \/ sum(rate(http_requests_total[5m]) <\/code><\/pre>\n<p>Finally, we can check whether the proportion of failed API requests is larger than 10%<\/p>\n<pre><code>sum(rate(http_requests_total{status_code=~\"5.*\"}[5m])) \/     sum(rate(http_requests_total[5m]))     > 0.1 <\/code><\/pre>\n<h3>Request Latency<\/h3>\n<p>Another metric <code>response_latency_ms<\/code> tracks the latency of the API responses in ms. We can check whether 95% of the responses take less or equal than 5 seconds.<\/p>\n<pre><code>histogram_quantile(     0.95,      sum(rate(request_latency_ms_bucket[5m])) by (le) ) \/ 1e3 > 5 <\/code><\/pre>\n<p><code>le<\/code> (less or equal) &#8212; is a required label that denotes an inclusive upper limit for the bucket, in our case 0.95-quantile. We divide the query result by <code>1e3<\/code> (1000) to convert from milliseconds to seconds.<\/p>\n<h2>Conclusion<\/h2>\n<p>As we can Prometheus is a powerful and flexible tool. This article covers its usage from basic instrumentation of a Go application to SLO-related PromQl queries.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/538692\/\"> https:\/\/habr.com\/ru\/articles\/538692\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h2>A Gentle Intro<\/h2>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>All Prometheus metrics are based on <strong>time series<\/strong> &#8212; streams of timestamped values belonging to the same metric. Each time series is uniquely identified by its metric name and optional key-value pairs called labels. The metric name specifies some characteristics of the measured system, such as <code>http_requests_total<\/code> &#8212; the total number of received HTTP requests. In practice, you often will be interested in some subset of the values of a metric, for example, in the number of requests received by a particular endpoint; and here is where the labels come in handy. We can partition a metric by adding <code>endpoint<\/code> label and see the statics for a particular endpoint: <code>http_requests_total{endpoint=\"api\/status\"}.<\/code> Every metric has two automatically created labels: <code>job_name<\/code> and <code>instance<\/code>. We see their roles in the next section.<\/p>\n<p>Prometheus provides a functional query language called PromQL. The result of the query might be evaluated to one of four types:<\/p>\n<ul>\n<li>\n<p><strong>Scalar<\/strong> (aka float)<\/p>\n<\/li>\n<li>\n<p><strong><em>String<\/em><\/strong><em> (currently unused)<\/em><\/p>\n<\/li>\n<li>\n<p><strong>Instant Vector<\/strong> &#8212; a set of time series that have exactly one value per timestamp.<\/p>\n<\/li>\n<li>\n<p><strong>Range Vector<\/strong> &#8212; a set of time series that have a range of values between two timestamps.<\/p>\n<\/li>\n<\/ul>\n<p>At first glance, <strong>Instant Vector<\/strong> might look like an array, and <strong>Range Vector<\/strong> as a matrix.<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>If that would be the case, then a Range Vector for a single time series &#171;downgrades&#187; to an Instant Vector. However, that&#8217;s not the case: the difference between a Range Vector and an Instant Vector is not in the number of tracked time series but in the relation between a value of the metric and the corresponding timestamp. In Instant Vector a time series has a single value at the timestamp, in Range Vector a time series has an arbitrary number of values between two timestamps. Therefore, we can more accurately visualize a Range Vector in the following form:<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>The difference between an instant vector and a range vector will be more clearly seen in the action: let&#8217;s instrument a Go application and see how can Prometheus help us with insights.<\/p>\n<p>A <strong>Counter<\/strong> is a metric that only goes up, for example, a number of the incoming HTTP requests.<\/p>\n<p>A <strong>Gauge<\/strong> is a metric that can have an arbitrary float value, for example, current CPU usage.<\/p>\n<p>A <strong>Histogram<\/strong> summarizes observations into statistical buckets, for example, we can track an API&#8217;s response times. Why would one need a histogram and can&#8217;t just measure the average response time? This is because we are not always interested in the average response times only. Let&#8217;s say we have an SLO and 95% of the requests should take no longer than 300 ms. We need to single out all requests that took 300 ms or less, count their amount, and divide it by the total number of the request. To do so in Prometheus, we configure a histogram to have a bucket with an upper limit of 0.3 seconds for a response. Later we see how can we configure an alert if we fail to comply with the SLO.<\/p>\n<p>There is also a <strong>Summary,<\/strong> however it is left mostly for historical reasons and serves the same purpose as a histogram.<\/p>\n<h2>Instrumenting a Go application<\/h2>\n<p>Let&#8217;s start <strong>without<\/strong> a Prometheus server.<\/p>\n<pre><code class=\"go\">package main  import (     \"net\/http\"      \"github.com\/prometheus\/client_golang\/prometheus\/promhttp\" )  func main() {     http.Handle(\"\/metrics\", promhttp.Handler())     http.ListenAndServe(\":2112\", nil) } <\/code><\/pre>\n<p><code>promhttp.Handler<\/code> returns an http.Handler which is already instrumented with the default metrics. Let&#8217;s see what we get right out of the box by issuing a GET request to the <code>\/metrics<\/code> endpoint.<\/p>\n<blockquote>\n<p><code>http localhost:2112\/metrics<\/code><\/p>\n<\/blockquote>\n<pre><code class=\"bash\"># HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile=\"0\"} 0 go_gc_duration_seconds{quantile=\"0.25\"} 0 go_gc_duration_seconds{quantile=\"0.5\"} 0 go_gc_duration_seconds{quantile=\"0.75\"} 0 go_gc_duration_seconds{quantile=\"1\"} 0 go_gc_duration_seconds_sum 0 go_gc_duration_seconds_count 0 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 8 # HELP go_info Information about the Go environment. # TYPE go_info gauge go_info{version=\"go1.15.5\"} 1 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge  [truncated] <\/code><\/pre>\n<p>As we can see we have a rather impressive list of metrics of all types: counters, gauges, and summaries. <\/p>\n<p>One of particular interest is a gauge for currently existing goroutines:<\/p>\n<pre><code class=\"bash\"># HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 7 <\/code><\/pre>\n<p>Note, that it is the number of <strong>existing<\/strong> goroutines, but not the <strong>running<\/strong> ones: some of the goroutines might be in the suspended state. One of the goroutines is our <code>main<\/code> function; the rest are running helper functions from <code>runtime<\/code> package, and are responsible for tasks such as garbage collection. Later we see how we can use this metric to reveal potential memory leaks. <\/p>\n<p>At the end very end, we see a familiar metric for the received HTTP request with label <code>code<\/code>.<\/p>\n<pre><code>promhttp_metric_handler_requests_total{code=\"200\"} 1 promhttp_metric_handler_requests_total{code=\"500\"} 0 promhttp_metric_handler_requests_total{code=\"503\"} 0 <\/code><\/pre>\n<p>At the next curl request, the metric will be increased by one.<\/p>\n<p>Now, this is already looking promising, but how can we instrument <strong>our own<\/strong> code? Let&#8217;s add a simple metric for the processed orders; as this number can only increase, the counter type is a natural choice.<\/p>\n<pre><code class=\"go\">package main  import (     \"net\/http\"     \"time\"      \"github.com\/prometheus\/client_golang\/prometheus\"     \"github.com\/prometheus\/client_golang\/prometheus\/promauto\"     \"github.com\/prometheus\/client_golang\/prometheus\/promhttp\" )  var counter = promauto.NewCounter(prometheus.CounterOpts{     Name: \"orders_processed\", })  func main() {     go func() {         for {             counter.Inc()             \/\/ simulate some processing function             time.Sleep(time.Second)         }     }()      http.Handle(\"\/metrics\", promhttp.Handler())     http.ListenAndServe(\":2112\", nil) } <\/code><\/pre>\n<p><code>prometheus<\/code> package provides metrics data types, and <code>promauto<\/code> package automates some routine tasks, such as metric registration.<\/p>\n<p>Now, we can see how well our processing goes:<\/p>\n<blockquote>\n<p>http localhost:2112\/metrics | grep orders_processed<\/p>\n<\/blockquote>\n<pre><code class=\"bash\"># HELP orders_processed # TYPE orders_processed counter orders_processed 7 <\/code><\/pre>\n<p>If we issue the request again we can verify that the processing is up and running:<\/p>\n<pre><code class=\"bash\"># HELP orders_processed # TYPE orders_processed counter orders_processed 10 <\/code><\/pre>\n<p>If you have a feeling that these values are not that illuminating, then you are on the right track. <strong>Prometheus is a statistical instrument<\/strong> at its core: it is intended to work with the trends, and not with the individual results. Later we see how to get some insights with <code>rate<\/code> and <code>increase<\/code> functions. <\/p>\n<p>Now, we have already instrumented our application <strong>and have not yet run Prometheus itself.<\/strong> Why do we need one? A Prometheus server allows us to aggregate data from several instances of our application. It scrapes our application and collects the values of metrics. Here is a simple Prometheus configuration:<\/p>\n<pre><code class=\"json\">scrape_configs:   - job_name: myapp     scrape_interval: 10s     static_configs:       - targets:           - 127.0.0.1:2112 <\/code><\/pre>\n<p>Prometheus server will scrape our application on port 2112 every 10 seconds and collects the metrics. Let&#8217;s bind this configuration to a docker container and run a Prometheus server:<\/p>\n<pre><code class=\"bash\">docker run \\     --network \"host\" \\     -p 9090:9090 \\     -v $HOME\/GolandProjects\/go-learn\/prometheus\/prometheus.yml:\/etc\/prometheus\/prometheus.yml \\     prom\/prometheus <\/code><\/pre>\n<p>Note, that we have to enable networking in <code>host<\/code> mode: our docker container needs to access <code>metric<\/code> endpoint in our application which is outside of the docker network. <\/p>\n<p>Now we can access Prometheus UI on <a href=\"http:\/\/localhost:9090\" rel=\"noopener noreferrer nofollow\">localhost:9090<\/a>. Here we can see that we are successfully scraping metrics from our application. <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>How many goroutines do we have? In a table view we see the last recorded value:<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>And in a table view we can see the trend:<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>The result of this query (<code>go_goroutines<\/code>) is an instant vector that contains a single time series. What will be the result for the same query when we run a second instance of our application? Then <code>instance<\/code> label will have two possible values for two instances, therefore we get two time series. And as noted above we still have an instant vector, not a range vector. <\/p>\n<p>Let&#8217;s add one more target to our configuration and start the second instance of our application at port <code>21112<\/code>.<\/p>\n<pre><code class=\"json\">scrape_configs:   - job_name: myapp     scrape_interval: 10s     static_configs:       - targets:           - 127.0.0.1:2112           - 127.0.0.1:21112 <\/code><\/pre>\n<p>Here we can see the number of goroutines for both our instances. <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>If we provide a time range we get a range vector: <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>In this case, we requested a <strong>range<\/strong> of values between two timestamps: now and five minutes in the past with a step equal to 1 minute. In the output, symbol <code>@<\/code>separates a value of a metric from its timestamp.<\/p>\n<p>Neither <code>Prometheus<\/code> no <code>Grafana<\/code> can picture the graph for a range vector. However, all function that takes a range vector as input returns an instant vector which in turn can be pictured. <\/p>\n<h2>Prometheus Functions and Modifiers<\/h2>\n<p>The two most useful range functions are <code>increase<\/code> and <code>rate;<\/code> both of them should be used only with counters. Let&#8217;s have a look at our <code>orders_processed<\/code> metric. <\/p>\n<p>Prometheus is happy to tell us that one of the instances has processed ~7000 orders so far. <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>What insights can we get from this number? Actually, not that much. For example, it would be more interesting to know how many orders did we processed recently. Here we calculate the increase in the number of processed orders during the last 5 minutes. <\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>This number could shed some light on the health of our application if we knew how many orders we expected to process under normal conditions. Let&#8217;s compare it with some reference point in the past: let&#8217;s imagine that we just deployed a nightly build of our application, and two hours ago we were running a thoroughly tested stable version.<\/p>\n<p>Modifier <code>offset<\/code>changes the time offset for the vector<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-385738","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/385738","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=385738"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/385738\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=385738"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=385738"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=385738"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}