{"id":405501,"date":"2024-06-29T18:21:37","date_gmt":"2024-06-29T18:21:37","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=405501"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-06-29T18:21:37","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=405501","title":{"rendered":"<span>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0430\u0435\u043c VictoriaMetrics \u0432 Deckhouse \u0438 \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0435\u043c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f \u0432 Telegram<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>\u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c, \u043a\u0430\u043a \u0432 Kubernetes-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435 \u043f\u043e\u0434 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435\u043c Deckhouse \u0443\u0434\u043e\u0431\u043d\u043e \u0438 \u0431\u044b\u0441\u0442\u0440\u043e \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0441 \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f\u043c\u0438 \u0432 Telegram. \u0412\u043e\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c\u0441\u044f VictoriaMetrics \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043c\u0435\u0442\u0440\u0438\u043a, \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0434\u0430\u0448\u0431\u043e\u0440\u0434 \u0432 Grafana, \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0430\u043b\u0435\u0440\u0442 \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u043e\u043f\u043e\u0432\u0435\u0449\u0435\u043d\u0438\u0435.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e06\/223\/840\/e06223840031cfd44c29472850aee956.png\" width=\"1560\" height=\"897\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e06\/223\/840\/e06223840031cfd44c29472850aee956.png\"\/><\/figure>\n<h2>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u043e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u044f<\/h2>\n<p>\u041d\u0430\u043c \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u044e\u0442\u0441\u044f:<\/p>\n<ul>\n<li>\n<p>\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Kubernetes \u0441 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u043c <a href=\"https:\/\/deckhouse.ru\/gs\/index.html\"><u>Deckhouse<\/u><\/a> \u0438 \u0432\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/\"><u>\u043c\u043e\u0434\u0443\u043b\u0435\u043c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430<\/u><\/a>.<\/p>\n<\/li>\n<li>\n<p>kubectl. \u041a\u043e\u043c\u0430\u043d\u0434\u044b \u043c\u043e\u0436\u043d\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u043a\u0430\u043a \u043d\u0430 master-\u0443\u0437\u043b\u0435, \u0442\u0430\u043a \u0438 \u0441 \u043b\u044e\u0431\u043e\u0433\u043e \u0434\u0440\u0443\u0433\u043e\u0433\u043e \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430, <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/150-user-authn\/faq.html#%D0%BA%D0%B0%D0%BA-%D1%8F-%D0%BC%D0%BE%D0%B3%D1%83-%D1%81%D0%B3%D0%B5%D0%BD%D0%B5%D1%80%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D1%82%D1%8C-kubeconfig-%D0%B4%D0%BB%D1%8F-%D0%B4%D0%BE%D1%81%D1%82%D1%83%D0%BF%D0%B0-%D0%BA-kubernetes-api\"><u>\u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0432 \u0434\u043e\u0441\u0442\u0443\u043f<\/u><\/a> \u043a API \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/helm.sh\/docs\/intro\/install\/\"><u>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 Helm<\/u><\/a>.<\/p>\n<\/li>\n<li>\n<p>\u0423\u0447\u0435\u0442\u043d\u0430\u044f \u0437\u0430\u043f\u0438\u0441\u044c Telegram.<\/p>\n<\/li>\n<\/ul>\n<h2>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 VictoriaMetrics<\/h2>\n<p>\u0420\u0430\u0437\u0432\u0435\u0440\u043d\u0451\u043c <a href=\"https:\/\/victoriametrics.com\/\"><u>VictoriaMetrics<\/u><\/a> \u0432 \u043f\u0440\u043e\u0441\u0442\u0440\u0430\u043d\u0441\u0442\u0432\u0435 \u0438\u043c\u0435\u043d <code>victoria-metrics-test<\/code>, \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0432 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0435 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0434\u043b\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0438 \u0438\u0437 Helm-\u0447\u0430\u0440\u0442\u0430 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0435\u0439 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e:<\/p>\n<pre><code>helm repo add vm https:\/\/victoriametrics.github.io\/helm-charts\/ &amp;&amp; \\ helm repo update &amp;&amp; \\ helm upgrade --install victoria-metrics vm\/victoria-metrics-single --namespace victoria-metrics-test --create-namespace<\/code><\/pre>\n<h2>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 VictoriaMetrics \u043a Prometheus<\/h2>\n<p>\u0412\u043d\u0435\u0448\u043d\u0435\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u043c\u0435\u0442\u0440\u0438\u043a \u0432 Deckhouse \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0430\u0435\u0442\u0441\u044f \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#prometheusremotewrite\"><u>PrometheusRemoteWrite<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: PrometheusRemoteWrite metadata:   name: victoria-metrics spec:   url: http:\/\/victoria-metrics-victoria-metrics-single-server.victoria-metrics-test.svc.cluster.local:8428\/api\/v1\/write<\/code><\/pre>\n<p><em>\u0412 \u043c\u0438\u043d\u0438\u043c\u0430\u043b\u044c\u043d\u043e\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u0432 \u0440\u0435\u0441\u0443\u0440\u0441\u0435 PrometheusRemoteWrite \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u0430\u0434\u0440\u0435\u0441 VictoriaMetrics. \u0422\u0430\u043a\u0436\u0435 \u043f\u0440\u0438 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e\u0441\u0442\u0438 \u043c\u043e\u0436\u043d\u043e \u0432\u0432\u0435\u0441\u0442\u0438 \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u043b\u044f \u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u0438 \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043f\u0440\u0435\u043e\u0431\u0440\u0430\u0437\u043e\u0432\u0430\u043d\u0438\u0435 label\u2019\u043e\u0432 \u043f\u0435\u0440\u0435\u0434 \u043e\u0442\u043f\u0440\u0430\u0432\u043a\u043e\u0439 \u0434\u0430\u043d\u043d\u044b\u0445<\/em>.<\/p>\n<p><em>\u0412\u0441\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u043d\u044b\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b <\/em><a href=\"https:\/\/github.com\/flant\/examples\/tree\/master\/2023\/03-monitoring\"><em><u>\u0432 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438<\/u><\/em><\/a><em>.<\/em><\/p>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/prometheus-remote-write.yaml<\/code><\/pre>\n<p><em>\u0417\u0434\u0435\u0441\u044c \u043c\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043b\u0438 \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 \u0437\u0430\u0440\u0430\u043d\u0435\u0435 template \u0438\u0437 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u044f \u043d\u0430 GitHub. \u042d\u0442\u043e \u043d\u0435 \u043e\u0431\u044f\u0437\u0430\u0442\u0435\u043b\u044c\u043d\u043e \u2014 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u0442\u044c \u0444\u0430\u0439\u043b \u0441 \u043d\u0443\u0436\u043d\u044b\u043c \u0441\u043e\u0434\u0435\u0440\u0436\u0438\u043c\u044b\u043c \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435 \u043c\u043e\u0436\u043d\u043e \u043b\u044e\u0431\u044b\u043c \u0443\u0434\u043e\u0431\u043d\u044b\u043c \u0441\u043f\u043e\u0441\u043e\u0431\u043e\u043c.<\/em><\/p>\n<p>\u041f\u043e\u0434\u043e\u0436\u0434\u0435\u043c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u043c\u0438\u043d\u0443\u0442 \u0438 \u0443\u0431\u0435\u0434\u0438\u043c\u0441\u044f, \u0447\u0442\u043e \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u043f\u043e\u0441\u0442\u0443\u043f\u0430\u044e\u0442 \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440. \u0414\u043b\u044f \u044d\u0442\u043e\u0433\u043e \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e \u043f\u043e\u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c \u0441\u0442\u0430\u0442\u0443\u0441 \u0431\u0430\u0437\u044b \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u0435\u0442\u0440\u0438\u043a <strong>\u043d\u0430 master-\u0443\u0437\u043b\u0435<\/strong> \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u044b:<\/p>\n<pre><code>curl $(kubectl --namespace victoria-metrics-test get ep -l \"app=server\" -o jsonpath=\"{.items[0].subsets[0].addresses[0].ip}\"):8428\/api\/v1\/status\/tsdb<\/code><\/pre>\n<p>\u0412 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0435 \u0434\u043e\u043b\u0436\u043d\u043e \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u044c\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043d\u043e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0435:<\/p>\n<pre><code class=\"json\">{\"status\":\"success\",\"data\":{\"totalSeries\":100477,\"totalLabelValuePairs\":1099074,\"seriesCountByMetricName\":[{\"name\":\"apiserver_request_slo_duration_seconds_bucket\",\"value\":15576},{\"name\":\"apiserver_request_duration_seconds_bucket\",\"value\":8496},{\"name\":\"etcd_request_duration_seconds_bucket\",\"value\":4284},{\"name\":\"apiserver_response_sizes_bucket\",\"value\":3080},{\"name\":\"apiserver_watch_events_sizes_bucket\",\"value\":1872},{\"name\":\"trivy_vulnerability_id\",\"value\":1541},{\"name\":\"workqueue_queue_duration_seconds_bucket\",\"value\":1089},{\"name\":\"workqueue_work_duration_seconds_bucket\",\"value\":1089},{\"name\":\"container_memory_failures_total\",\"value\":904},{\"name\":\"scheduler_plugin_execution_duration_seconds_bucket\",\"value\":903}],\"seriesCountByLabelName\":[{\"name\":\"__name__\",\"value\":100477},{\"name\":\"prometheus\",\"value\":100477},{\"name\":\"job\",\"value\":100171},{\"name\":\"instance\",\"value\":99736},{\"name\":\"tier\",\"value\":81434},{\"name\":\"service\",\"value\":57725},{\"name\":\"le\",\"value\":51720},{\"name\":\"resource\",\"value\":36606},{\"name\":\"version\",\"value\":34823},{\"name\":\"namespace\",\"value\":33844}],\"seriesCountByFocusLabelValue\":[],\"seriesCountByLabelValuePair\":[{\"name\":\"prometheus=deckhouse\",\"value\":100475},{\"name\":\"tier=cluster\",\"value\":81434},{\"name\":\"service=kubernetes\",\"value\":45705},{\"name\":\"instance=192.168.199.9:6443\",\"value\":45196},{\"name\":\"job=kube-apiserver\",\"value\":45196},{\"name\":\"component=apiserver\",\"value\":31879},{\"name\":\"version=v1\",\"value\":19859},{\"name\":\"scope=cluster\",\"value\":19344},{\"name\":\"container=kube-rbac-proxy\",\"value\":17796},{\"name\":\"__name__=apiserver_request_slo_duration_seconds_bucket\",\"value\":15576}],\"labelValueCountByLabelName\":[{\"name\":\"__name__\",\"value\":1870},{\"name\":\"name\",\"value\":693},{\"name\":\"resource\",\"value\":538},{\"name\":\"le\",\"value\":379},{\"name\":\"type\",\"value\":323},{\"name\":\"secret\",\"value\":282},{\"name\":\"hook\",\"value\":201},{\"name\":\"kind\",\"value\":198},{\"name\":\"controller_name\",\"value\":173},{\"name\":\"installed_version\",\"value\":167}]}}<\/code><\/pre>\n<p><code>\"status\":\"success\"<\/code> \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442 \u043d\u0430 \u0442\u043e, \u0447\u0442\u043e \u0432\u0441\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043a\u0430\u043a \u043d\u0443\u0436\u043d\u043e; \u0435\u0441\u043b\u0438 <code>totalSeries<\/code> \u0431\u043e\u043b\u044c\u0448\u0435 \u043d\u0443\u043b\u044f, \u0442\u043e \u0434\u0430\u043d\u043d\u044b\u0435 \u0443\u0436\u0435 \u043d\u0430\u0447\u0430\u043b\u0438 \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u0442\u044c\u0441\u044f.<\/p>\n<p>\u041f\u0440\u043e\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0441\u0431\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a \u043c\u043e\u0436\u043d\u043e \u0447\u0435\u0440\u0435\u0437 \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 VictoriaMetrics. \u0414\u043b\u044f \u044d\u0442\u043e\u0433\u043e \u043f\u0440\u043e\u0431\u0440\u043e\u0441\u044c\u0442\u0435 \u043f\u043e\u0440\u0442 VictoriaMetrics \u043d\u0430 \u0441\u0432\u043e\u0439 \u043a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440:<\/p>\n<pre><code>export POD_NAME=$(kubectl get pods --namespace victoria-metrics-test -l \"app=server\" -o jsonpath=\"{.items[0].metadata.name}\") &amp;&amp;  kubectl --namespace victoria-metrics-test port-forward $POD_NAME 8428<\/code><\/pre>\n<p><em>\u0414\u043b\u044f \u0443\u0441\u043f\u0435\u0448\u043d\u043e\u0433\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u044d\u0442\u043e\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u043d\u0430 \u043c\u0430\u0448\u0438\u043d\u0435, \u0441 \u043a\u043e\u0442\u043e\u0440\u043e\u0439 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u0437\u0430\u043f\u0440\u043e\u0441, \u0434\u043e\u043b\u0436\u0435\u043d \u0431\u044b\u0442\u044c \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d \u0443\u0434\u0430\u043b\u0435\u043d\u043d\u044b\u0439 \u0434\u043e\u0441\u0442\u0443\u043f \u0434\u043b\u044f kubectl.<\/em><\/p>\n<p>\u0412\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u0441\u0442\u0430\u043d\u0435\u0442 \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <a href=\"http:\/\/localhost:8428\"><u>http:\/\/localhost:8428<\/u><\/a>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/d51\/26a\/d06\/d5126ad061653603f279fbd136eae144.png\" alt=\"\" title=\"\" width=\"651\" height=\"384\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/d51\/26a\/d06\/d5126ad061653603f279fbd136eae144.png\"\/><\/figure>\n<p>\u041f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u043d\u0430 \u0430\u0434\u0440\u0435\u0441 <a href=\"http:\/\/localhost:8428\/api\/v1\/status\/tsdb\"><u>http:\/\/localhost:8428\/api\/v1\/status\/tsdb<\/u><\/a> \u2014\u00a0\u0442\u0430\u043c \u0442\u0430\u043a\u0436\u0435 \u0434\u043e\u043b\u0436\u0435\u043d \u0431\u044b\u0442\u044c \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0435\u043d \u0441\u0442\u0430\u0442\u0443\u0441 <code>\"status\":\"success\"<\/code>; \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 <code>totalSeries<\/code> \u0431\u043e\u043b\u044c\u0448\u0435 \u043d\u0443\u043b\u044f \u0431\u0443\u0434\u0435\u0442 \u0441\u0438\u0433\u043d\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043e\u0431 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\u043c \u0441\u0431\u043e\u0440\u0435 \u043c\u0435\u0442\u0440\u0438\u043a:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/79e\/8e5\/2c4\/79e8e52c49bc020544ab70b9a38249ed.png\" alt=\"\" title=\"\" width=\"808\" height=\"427\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/79e\/8e5\/2c4\/79e8e52c49bc020544ab70b9a38249ed.png\"\/><\/figure>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u0445\u0440\u0430\u043d\u044f\u0442\u0441\u044f \u0432 VictoriaMectrics. \u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u043c \u044d\u0442\u043e \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u043a Grafana.<\/p>\n<h2>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 VictoriaMetrics \u043a Grafana<\/h2>\n<h3>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u043d\u043e\u0432\u043e\u0433\u043e datasource<\/h3>\n<p>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 datasource \u043a Grafana \u0432 Deckhouse \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#grafanaadditionaldatasource\"><u>GrafanaAdditionalDatasource<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: GrafanaAdditionalDatasource metadata:   name: victoria-metrics spec:   access: Proxy   basicAuth: false   jsonData:     timeInterval: 30s   type: prometheus   url: http:\/\/victoria-metrics-victoria-metrics-single-server.victoria-metrics-test.svc.cluster.local:8428\/<\/code><\/pre>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u0435\u0433\u043e \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/grafana-additional-datasource.yaml<\/code><\/pre>\n<p>Deckhouse \u043f\u0435\u0440\u0435\u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442 Grafana \u0441 \u043d\u043e\u0432\u043e\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0435\u0439, \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u043c \u0435\u0435 \u0433\u043e\u0442\u043e\u0432\u043d\u043e\u0441\u0442\u044c:<\/p>\n<pre><code>kubectl -n d8-monitoring get po -l app=grafana<\/code><\/pre>\n<p>\u0421\u0442\u0430\u0442\u0443\u0441 \u0434\u043e\u043b\u0436\u0435\u043d \u0431\u044b\u0442\u044c <code>Running<\/code>:<\/p>\n<pre><code>kubectl -n d8-monitoring get po -l app=grafana NAME                       READY   STATUS    RESTARTS   AGE grafana-56df555c67-glzqr   3\/3     Running   0          67s<\/code><\/pre>\n<h3>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f<\/h3>\n<p>\u0417\u0430\u0439\u0434\u0435\u043c \u0432 \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 Grafana \u0438 \u043f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u043d\u0430 \u0432\u043a\u043b\u0430\u0434\u043a\u0443 <em>Configuration<\/em> -> <em>Data sources<\/em>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/94f\/995\/fa4\/94f995fa44a260e78b04e0cc5d4b7e0c.png\" alt=\"\" title=\"\" width=\"640\" height=\"581\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/94f\/995\/fa4\/94f995fa44a260e78b04e0cc5d4b7e0c.png\"\/><\/figure>\n<p>\u0412 \u0441\u043f\u0438\u0441\u043a\u0435 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b\u0445 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u043e\u0432 \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0430\u0442\u044c\u0441\u044f <em>victoria-metrics<\/em>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/79f\/1cb\/a32\/79f1cba32b3270024d9eadf0dd8a3f40.png\" alt=\"\" title=\"\" width=\"987\" height=\"515\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/79f\/1cb\/a32\/79f1cba32b3270024d9eadf0dd8a3f40.png\"\/><\/figure>\n<p>\u041e\u0442\u043a\u0440\u043e\u0435\u043c \u043c\u0435\u043d\u044e <em>Explore<\/em>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/23a\/a7a\/6e9\/23aa7a6e95723837cd10a85088ec5303.png\" alt=\"\" title=\"\" width=\"640\" height=\"581\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/23a\/a7a\/6e9\/23aa7a6e95723837cd10a85088ec5303.png\"\/><\/figure>\n<p>\u0412\u044b\u0431\u0435\u0440\u0435\u043c <em>victoria-metrics<\/em> \u0432 \u043a\u0430\u0447\u0435\u0441\u0442\u0432\u0435 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u0430 \u0434\u0430\u043d\u043d\u044b\u0445:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/c9b\/9e1\/08b\/c9b9e108b278b1361604182b3425f3ef.png\" alt=\"\" title=\"\" width=\"640\" height=\"581\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/c9b\/9e1\/08b\/c9b9e108b278b1361604182b3425f3ef.png\"\/><\/figure>\n<p>\u0420\u0430\u0441\u043a\u0440\u043e\u0435\u043c <em>Metrics browser<\/em>: \u0432 \u043d\u0435\u043c \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u044c\u0441\u044f \u0441\u043f\u0438\u0441\u043e\u043a \u043c\u0435\u0442\u0440\u0438\u043a, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0445 \u0438\u0437 VictoriaMetrics:<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/48e\/eef\/89f\/48eeef89fea39b19e399fbbb94df456d.png\" alt=\"\" title=\"\" width=\"434\" height=\"407\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/48e\/eef\/89f\/48eeef89fea39b19e399fbbb94df456d.png\"\/><\/figure>\n<p>\u0412\u0441\u0435 \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u043e \u0438 \u0433\u043e\u0442\u043e\u0432\u043e \u043a \u0440\u0430\u0431\u043e\u0442\u0435.<\/p>\n<h2>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0430\u043b\u0435\u0440\u0442\u0430<\/h2>\n<p>\u041d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0435 \u043e \u0441\u043e\u0431\u044b\u0442\u0438\u044f\u0445, \u0441\u043e\u0437\u0434\u0430\u0432 \u043d\u043e\u0432\u044b\u0439 \u0430\u043b\u0435\u0440\u0442.\u00a0<\/p>\n<p>\u0412 Deckhouse \u0430\u043b\u0435\u0440\u0442\u044b \u043e\u043f\u0438\u0441\u044b\u0432\u0430\u044e\u0442\u0441\u044f \u0432 CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#customprometheusrules\"><u>CustomPrometheusRules<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: CustomPrometheusRules metadata:   name: always-firing-alert spec:   groups:   - name: cluster-state-alert.rules     rules:       - alert: PrometheusCanScrapeTragets         annotations:           description: This is a fake alert only for a demo.           summary: The alert shows that Prometheus can scrape targets.         expr: |           up{job=\"deckhouse\"}<\/code><\/pre>\n<p><em>\u0421\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0439 \u0430\u043b\u0435\u0440\u0442 \u00ab\u0431\u0435\u0441\u043f\u043e\u043b\u0435\u0437\u043d\u044b\u0439\u00bb \u2014 \u043e\u043d \u0432\u0441\u0435\u0433\u0434\u0430 \u0430\u043a\u0442\u0438\u0432\u043d\u044b\u0439, \u043d\u043e \u0445\u043e\u0440\u043e\u0448\u043e \u043f\u043e\u0434\u0445\u043e\u0434\u0438\u0442 \u0434\u043b\u044f \u0446\u0435\u043b\u0435\u0439 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f.<\/em><\/p>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u0435\u0433\u043e \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/custom-prometheus-rule.yaml<\/code><\/pre>\n<p>\u0414\u043b\u044f \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u043f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u0432 Prometheus \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <code>&lt;\u0430\u0434\u0440\u0435\u0441_Grafana>\/prometheus\/<\/code> \u0438 \u043e\u0442\u043a\u0440\u043e\u0435\u043c \u0432\u043a\u043b\u0430\u0434\u043a\u0443 <em>Alerts<\/em>. \u0412 \u0441\u0442\u0440\u043e\u043a\u0435 \u043f\u043e\u0438\u0441\u043a\u0430 \u0432\u0432\u0435\u0434\u0435\u043c \u0438\u043c\u044f \u0430\u043b\u0435\u0440\u0442\u0430 \u2014 <em>PrometheusCanScrapeTragets<\/em>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/5a3\/f93\/719\/5a3f937194415b8d73cf42d86e4483bb.png\" alt=\"\" title=\"\" width=\"799\" height=\"531\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/5a3\/f93\/719\/5a3f937194415b8d73cf42d86e4483bb.png\"\/><\/figure>\n<p>\u0410\u043b\u0435\u0440\u0442 \u0441\u043e\u0437\u0434\u0430\u043d.<\/p>\n<h2>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u0448\u0431\u043e\u0440\u0434\u0430 \u0432 Grafana<\/h2>\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0434\u0430\u0448\u0431\u043e\u0440\u0434 <em>Services Up<\/em> \u0432 \u043a\u0430\u0442\u0430\u043b\u043e\u0433\u0435 <em>Services<\/em> (\u0447\u0435\u0440\u0435\u0437 CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#grafanadashboarddefinition\"><u>GrafanaDashboardDefinition<\/u><\/a>):<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: GrafanaDashboardDefinition metadata:   name: up-services spec:   folder: Services   definition: |     {       \"annotations\": {         \"list\": [           {             \"builtIn\": 1,             \"datasource\": {               \"type\": \"grafana\",               \"uid\": \"-- Grafana --\"             },             \"enable\": true,             \"hide\": true,             \"iconColor\": \"rgba(0, 211, 255, 1)\",             \"name\": \"Annotations &amp; Alerts\",             \"target\": {               \"limit\": 100,               \"matchAny\": false,               \"tags\": [],               \"type\": \"dashboard\"             },             \"type\": \"dashboard\"           }         ]       },       \"editable\": false,       \"fiscalYearStartMonth\": 0,       \"graphTooltip\": 0,       \"id\": 31,       \"links\": [],       \"liveNow\": false,       \"panels\": [         {           \"datasource\": {             \"type\": \"prometheus\",             \"uid\": \"P0D6E4079E36703EB\"           },           \"fieldConfig\": {             \"defaults\": {               \"color\": {                 \"mode\": \"palette-classic\"               },               \"custom\": {                 \"axisLabel\": \"\",                 \"axisPlacement\": \"auto\",                 \"barAlignment\": 0,                 \"drawStyle\": \"bars\",                 \"fillOpacity\": 15,                 \"gradientMode\": \"none\",                 \"hideFrom\": {                   \"legend\": false,                   \"tooltip\": false,                   \"viz\": false                 },                 \"lineInterpolation\": \"linear\",                 \"lineWidth\": 10,                 \"pointSize\": 5,                 \"scaleDistribution\": {                   \"type\": \"linear\"                 },                 \"showPoints\": \"never\",                 \"spanNulls\": false,                 \"stacking\": {                   \"group\": \"A\",                   \"mode\": \"normal\"                 },                 \"thresholdsStyle\": {                   \"mode\": \"off\"                 }               },               \"mappings\": [],               \"thresholds\": {                 \"mode\": \"absolute\",                 \"steps\": [                   {                     \"color\": \"green\",                     \"value\": null                   },                   {                     \"color\": \"red\",                     \"value\": 80                   }                 ]               }             },             \"overrides\": []           },           \"gridPos\": {             \"h\": 26,             \"w\": 24,             \"x\": 0,             \"y\": 0           },           \"id\": 2,           \"options\": {             \"legend\": {               \"calcs\": [                   \"lastNotNull\"               ],               \"displayMode\": \"table\",               \"placement\": \"right\"             },             \"tooltip\": {               \"mode\": \"single\",               \"sort\": \"none\"             }           },           \"pluginVersion\": \"8.5.2\",           \"targets\": [             {               \"datasource\": {                 \"type\": \"prometheus\",                 \"uid\": \"P0D6E4079E36703EB\"               },               \"editorMode\": \"code\",               \"exemplar\": false,               \"expr\": \"sum by (job, scrape_endpoint, scrape_source) (up)\",               \"format\": \"time_series\",               \"instant\": false,               \"legendFormat\": \"{{ job }} {{ scrape_source }}\",               \"range\": true,               \"refId\": \"A\"             }           ],           \"title\": \"Up\",           \"transformations\": [],           \"type\": \"timeseries\"         }       ],       \"refresh\": \"30s\",       \"schemaVersion\": 36,       \"style\": \"dark\",       \"tags\": [],       \"templating\": {         \"list\": []       },       \"time\": {         \"from\": \"now-3h\",         \"to\": \"now\"       },       \"timepicker\": {         \"refresh_intervals\": [             \"30s\"         ]       },       \"timezone\": \"\",       \"title\": \"Services Up\",       \"uid\": \"f_8jGXenz\",       \"version\": 1,       \"weekStart\": \"\"     }<\/code><\/pre>\n<p>\u0418 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u0435\u0433\u043e \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/grafana-dashboard-definition.yaml<\/code><\/pre>\n<p>\u041f\u0440\u043e\u0432\u0435\u0440\u0438\u043c, \u0447\u0442\u043e \u0432\u0441\u0435 \u043e\u0442\u0440\u0430\u0431\u043e\u0442\u0430\u043b\u043e, \u043f\u0435\u0440\u0435\u0439\u0434\u044f \u043f\u043e \u043f\u0443\u0442\u0438 <em>Services <\/em>-><em> Services Up<\/em> \u043d\u0430 \u0432\u043a\u043b\u0430\u0434\u043a\u0435 <em>Dashboards<\/em>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f03\/e13\/f5b\/f03e13f5b2aa6db21e724267dbf55e9d.png\" alt=\"\" title=\"\" width=\"662\" height=\"663\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f03\/e13\/f5b\/f03e13f5b2aa6db21e724267dbf55e9d.png\"\/><\/figure>\n<p>\u0412 \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u043e\u043c \u0434\u0430\u0448\u0431\u043e\u0440\u0434\u0435 \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u0441\u044f \u0432\u043e\u0442 \u0442\u0430\u043a\u043e\u0439 \u00ab\u0447\u0430\u0441\u0442\u043e\u043a\u043e\u043b\u00bb \u0430\u043b\u0435\u0440\u0442\u043e\u0432, \u043f\u0440\u0438\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u043a\u043e \u0432\u0441\u0435\u043c \u0441\u0435\u0440\u0432\u0438\u0441\u0430\u043c Deckhouse \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f75\/670\/f7c\/f75670f7c6b399f1dcc4d1a4ffbada7a.png\" alt=\"\" title=\"\" width=\"1088\" height=\"411\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f75\/670\/f7c\/f75670f7c6b399f1dcc4d1a4ffbada7a.png\"\/><\/figure>\n<h2>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0439 \u0432 Telegram<\/h2>\n<h3>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 \u043a Telegram<\/h3>\n<p><em>\u0415\u0441\u043b\u0438 \u0443 \u0432\u0430\u0441 \u0443\u0436\u0435 \u0435\u0441\u0442\u044c ID \u0447\u0430\u0442\u0430, \u0432 \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0432\u044b \u0445\u043e\u0442\u0438\u0442\u0435 \u043f\u043e\u043b\u0443\u0447\u0430\u0442\u044c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f, \u0438 \u0442\u043e\u043a\u0435\u043d \u0431\u043e\u0442\u0430, \u0447\u0435\u0440\u0435\u0437 \u043a\u043e\u0442\u043e\u0440\u043e\u0433\u043e \u0432\u044b \u0445\u043e\u0442\u0438\u0442\u0435 \u043e\u0442\u043f\u0440\u0430\u0432\u043b\u044f\u0442\u044c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f, \u0442\u043e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0439 \u044d\u0442\u0430\u043f \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0438 \u043f\u0435\u0440\u0435\u0439\u0442\u0438 \u0441\u0440\u0430\u0437\u0443 \u043a \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0435.<\/em><\/p>\n<h4>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0431\u043e\u0442\u0430 \u0438 \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u0435 ID \u0447\u0430\u0442\u0430<\/h4>\n<p><em>\u041f\u043e\u0434\u0440\u043e\u0431\u043d\u0443\u044e \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u044e \u043f\u043e \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044e \u0431\u043e\u0442\u0430 \u043c\u043e\u0436\u043d\u043e \u043d\u0430\u0439\u0442\u0438 <\/em><a href=\"https:\/\/core.telegram.org\/bots\"><em><u>\u0432 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438 Telegram<\/u><\/em><\/a><em>.<\/em><\/p>\n<p>\u0412 \u0441\u0442\u0440\u043e\u043a\u0435 \u043f\u043e\u0438\u0441\u043a\u0430 \u043a\u043b\u0438\u0435\u043d\u0442\u0430 Telegram \u0432\u0432\u0435\u0434\u0435\u043c \u0430\u0434\u0440\u0435\u0441 <a href=\"https:\/\/t.me\/botfather\"><u>https:\/\/t.me\/botfather<\/u><\/a> \u0438 \u043d\u0430\u0439\u0434\u0435\u043c \u0431\u043e\u0442\u0430 <code>@BotFather<\/code>. \u041d\u0435 \u043f\u0435\u0440\u0435\u043f\u0443\u0442\u0430\u0439\u0442\u0435 \u0435\u0433\u043e \u0441 \u0434\u0440\u0443\u0433\u0438\u043c\u0438 \u2014 \u0443 \u043d\u0430\u0441\u0442\u043e\u044f\u0449\u0435\u0433\u043e \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0430\u0435\u0442\u0441\u044f \u0437\u043d\u0430\u0447\u043e\u043a \u0432\u0435\u0440\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u0438:<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f3d\/694\/1c0\/f3d6941c0b626dac21589bb232d30f54.png\" alt=\"\" title=\"\" width=\"379\" height=\"243\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f3d\/694\/1c0\/f3d6941c0b626dac21589bb232d30f54.png\"\/><\/figure>\n<p>\u0412\u044b\u0431\u0435\u0440\u0435\u043c \u0435\u0433\u043e \u0438 \u043d\u0430\u0436\u043c\u0435\u043c Start. \u0417\u0430\u0442\u0435\u043c \u043e\u0442\u043f\u0440\u0430\u0432\u0438\u043c \u0431\u043e\u0442\u0443 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0435 <code>\/newbot<\/code> \u0438 \u043e\u0442\u0432\u0435\u0442\u0438\u043c \u043d\u0430 \u043f\u043e\u0441\u0442\u0443\u043f\u0438\u0432\u0448\u0438\u0435 \u0432\u043e\u043f\u0440\u043e\u0441\u044b, \u0432\u0432\u0435\u0434\u044f \u0438\u043c\u044f \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0435\u043c\u043e\u0433\u043e \u0431\u043e\u0442\u0430 \u0438 \u0435\u0433\u043e \u043d\u0438\u043a.<\/p>\n<p>\u041f\u0440\u0438\u043c\u0435\u0440 \u0434\u0438\u0430\u043b\u043e\u0433\u0430 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u0431\u043e\u0442\u0430 <code>@mytestalert2023bot<\/code>:<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/b50\/23b\/6c6\/b5023b6c6275057680af31898a86ba2c.png\" alt=\"\" title=\"\" width=\"511\" height=\"519\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/b50\/23b\/6c6\/b5023b6c6275057680af31898a86ba2c.png\"\/><\/figure>\n<p>\u041c\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u043c \u0442\u043e\u043a\u0435\u043d \u0431\u043e\u0442\u0430 \u0438 \u0441\u0441\u044b\u043b\u043a\u0443. \u041f\u0440\u043e\u0439\u0434\u0435\u043c \u043f\u043e \u043d\u0435\u0439 \u043a \u0434\u0438\u0430\u043b\u043e\u0433\u0443 \u0441 \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u043c \u0431\u043e\u0442\u043e\u043c (\u0432 \u043f\u0440\u0438\u043c\u0435\u0440\u0435 \u044d\u0442\u043e <code>http:\/\/t.me\/mytestalert2023bot<\/code>) \u0438 \u043d\u0430\u0436\u043c\u0435\u043c Start.<\/p>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043d\u0443\u0436\u043d\u043e \u0443\u0437\u043d\u0430\u0442\u044c ID \u0447\u0430\u0442\u0430, \u043a\u0443\u0434\u0430 \u0431\u0443\u0434\u0443\u0442 \u043e\u0442\u043f\u0440\u0430\u0432\u043b\u044f\u0442\u044c\u0441\u044f \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u044f.\u00a0<\/p>\n<p><em>\u042d\u0442\u043e \u043c\u043e\u0433\u0443\u0442 \u0431\u044b\u0442\u044c \u043a\u0430\u043a \u043b\u0438\u0447\u043d\u044b\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u044f, \u0442\u0430\u043a \u0438 \u0433\u0440\u0443\u043f\u043f\u0430 \u0438\u043b\u0438 \u043a\u0430\u043d\u0430\u043b \u0432 Telegram. \u0412\u043e \u0432\u0442\u043e\u0440\u043e\u043c \u0441\u043b\u0443\u0447\u0430\u0435 \u043d\u0443\u0436\u043d\u043e \u043f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0432 \u043d\u0438\u0445 \u0431\u043e\u0442\u0430, \u0430 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0435 \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u0442\u044c ID \u043a\u0430\u043d\u0430\u043b\u0430.<\/em><\/p>\n<p>\u0423\u0437\u043d\u0430\u0435\u043c ID \u0443\u0447\u0435\u0442\u043d\u043e\u0439 \u0437\u0430\u043f\u0438\u0441\u0438. \u0421\u0434\u0435\u043b\u0430\u0442\u044c \u044d\u0442\u043e \u043c\u043e\u0436\u043d\u043e, \u0432\u044b\u0431\u0440\u0430\u0432 \u0431\u043e\u0442\u0430 <code>@getmyid_bot<\/code> \u0438 \u043d\u0430\u0436\u0430\u0432 <em>Start<\/em>. \u0412 \u043e\u0442\u0432\u0435\u0442 \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u0441\u044f ID \u0443\u0447\u0435\u0442\u043d\u043e\u0439 \u0437\u0430\u043f\u0438\u0441\u0438 Telegram:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/0fb\/fe5\/50c\/0fbfe550c7f23f0c312bc8ec8b69be48.png\" alt=\"\" title=\"\" width=\"684\" height=\"486\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/0fb\/fe5\/50c\/0fbfe550c7f23f0c312bc8ec8b69be48.png\"\/><\/figure>\n<p>\u041f\u0440\u043e\u0432\u0435\u0440\u0438\u043c \u0440\u0430\u0431\u043e\u0442\u0443 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0439 \u0447\u0435\u0440\u0435\u0437 API Telegram.<\/p>\n<p>\u0412\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0443\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u0443, \u0443\u043a\u0430\u0437\u0430\u0432 \u0432\u043c\u0435\u0441\u0442\u043e XXX \u0442\u043e\u043a\u0435\u043d \u0431\u043e\u0442\u0430 (\u043e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435 \u043d\u0430 \u043f\u0440\u0435\u0444\u0438\u043a\u0441 <code>bot<\/code> \u2014 \u0442\u043e\u043a\u0435\u043d \u0438\u0434\u0435\u0442 \u043f\u043e\u0441\u043b\u0435 \u043d\u0435\u0433\u043e), \u0430 \u0432\u043c\u0435\u0441\u0442\u043e YYY \u2014\u00a0 ID \u0443\u0447\u0435\u0442\u043d\u043e\u0439 \u0437\u0430\u043f\u0438\u0441\u0438 Telegram:<\/p>\n<pre><code>curl -X POST \"https:\/\/api.telegram.org\/botXXX\/sendMessage\" -d \"chat_id=YYY&amp;text=text for test\"<\/code><\/pre>\n<p>\u0412 \u043e\u0442\u0432\u0435\u0442 \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u0441\u044f \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0435 \u043e\u0431 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\u0439 \u043e\u0442\u043f\u0440\u0430\u0432\u043a\u0435:<\/p>\n<pre><code>{\"ok\":true,\"result\":{\"message_id\":2,\"from\":{\"id\":2222222222,\"is_bot\":true,\"first_name\":\"mytestbot\",\"username\":\"mytestalert2023bot\"},\"chat\":{\"id\":2222222222,\"first_name\":\"Joe\",\"type\":\"private\"},\"date\":1674327896,\"text\":\"text for test\"}}<\/code><\/pre>\n<p>\u2026 \u0430 \u043d\u0430\u043c \u043f\u0440\u0438\u0434\u0435\u0442 \u043b\u0438\u0447\u043d\u043e\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0435 \u043e\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u043e\u0433\u043e \u0431\u043e\u0442\u0430.<\/p>\n<h3>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u043e\u0442\u043f\u0440\u0430\u0432\u043a\u0438 \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0439<\/h3>\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c Secret <code>telegram-bot-secret<\/code>, \u0443\u043a\u0430\u0437\u0430\u0432 \u0442\u043e\u043a\u0435\u043d \u0431\u043e\u0442\u0430:<\/p>\n<pre><code>kubectl create secret generic -n d8-monitoring telegram-bot-secret --from-literal=token=XXX<\/code><\/pre>\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0440\u0435\u0441\u0443\u0440\u0441 <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#customalertmanager\"><u>CustomAlertmanager<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1alpha1 kind: CustomAlertmanager metadata:   name: telegram spec:   type: Internal   internal:     route:       groupBy: [ 'job' ]       groupWait: 30s       groupInterval: 5m       repeatInterval: 12h       receiver: 'telegram'     receivers:       - name: telegram         telegramConfigs:           - botToken:               key: token               name: telegram-bot-secret             chatID: 111<\/code><\/pre>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u0435\u0433\u043e \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/alertmanager.yaml<\/code><\/pre>\n<p>\u0412\u00a0 \u043f\u0440\u043e\u0441\u0442\u0440\u0430\u043d\u0441\u0442\u0432\u0435 \u0438\u043c\u0435\u043d <code>db-monitoring<\/code> \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u0441\u044f Pod \u0441 AlertManager. \u041f\u0440\u043e\u0432\u0435\u0440\u0438\u043c, \u0447\u0442\u043e \u043e\u043d \u0438\u043c\u0435\u0435\u0442 \u0441\u0442\u0430\u0442\u0443\u0441 <code>Running<\/code>:<\/p>\n<pre><code>kubectl -n d8-monitoring get po -l alertmanager<\/code><\/pre>\n<p>\u041f\u0440\u0438\u043c\u0435\u0440 \u0432\u044b\u0432\u043e\u0434\u0430:<\/p>\n<pre><code>NAME                      READY   STATUS    RESTARTS       AGE alertmanager-telegram-0   3\/3     Running   1 (4m1s ago)   3m57s<\/code><\/pre>\n<p>\u0421\u0435\u0439\u0447\u0430\u0441 \u0432 \u0436\u0443\u0440\u043d\u0430\u043b\u0435 Alertmanager\u2019\u0430 \u0431\u0443\u0434\u0443\u0442 \u043e\u0448\u0438\u0431\u043a\u0438 \u043e\u0442\u043f\u0440\u0430\u0432\u043a\u0438, \u0442. \u043a. \u043d\u0435 \u0443\u043a\u0430\u0437\u0430\u043d ID \u0447\u0430\u0442\u0430, \u0432 \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043d\u0443\u0436\u043d\u043e \u043e\u0442\u043f\u0440\u0430\u0432\u043b\u044f\u0442\u044c \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u044f. \u041f\u043e\u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c \u043b\u043e\u0433 \u043c\u043e\u0436\u043d\u043e \u043a\u043e\u043c\u0430\u043d\u0434\u043e\u0439:<\/p>\n<pre><code>kubectl -n d8-monitoring logs alertmanager-telegram-0 -c alertmanager<\/code><\/pre>\n<p>ID \u0447\u0430\u0442\u0430 \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442\u0441\u044f \u0432 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u0435 <code>spec.internal.receivers.telegramConfigs.chatID<\/code> CustomAlertmanager. <\/p>\n<p>\u0420\u0435\u0441\u0443\u0440\u0441 \u043c\u043e\u0436\u043d\u043e \u043b\u0438\u0431\u043e \u043e\u0442\u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0432\u0440\u0443\u0447\u043d\u0443\u044e (<code>kubectl edit customalertmanager telegram<\/code>), \u043b\u0438\u0431\u043e \u0432\u043e\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u043e\u0439, \u0443\u043a\u0430\u0437\u0430\u0432 \u0432 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0439 <code>CHAT_ID<\/code> ID \u0443\u0447\u0435\u0442\u043d\u043e\u0439 \u0437\u0430\u043f\u0438\u0441\u0438 Telegram:<\/p>\n<pre><code>CHAT_ID=2222222222 &amp;&amp; \\ kubectl patch customalertmanager telegram --type json -p \"[{\\\"op\\\": \\\"replace\\\", \\\"path\\\": \\\"\/spec\/internal\/receivers\/0\/telegramConfigs\/0\/chatID\\\", \\\"value\\\": ${CHAT_ID}}]\"<\/code><\/pre>\n<p>Alertmanager \u043e\u0431\u043d\u043e\u0432\u0438\u0442 \u0441\u0432\u043e\u044e \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e \u0438 \u043f\u0440\u0438\u0448\u043b\u0435\u0442 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0435 \u043e\u0431 \u0430\u043b\u0435\u0440\u0442\u0435 <em>PrometheusCanScrapeTragets, <\/em>\u0441\u043e\u0437\u0434\u0430\u043d\u043d\u043e\u043c \u0440\u0430\u043d\u0435\u0435, \u0430 \u0442\u0430\u043a\u0436\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u044f \u043e \u0434\u0440\u0443\u0433\u0438\u0445 \u0430\u043b\u0435\u0440\u0442\u0430\u0445, \u0430\u043a\u0442\u0438\u0432\u043d\u044b\u0445 \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435.<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/bac\/82a\/546\/bac82a546d57001be0d7ebbbbc927d18.png\" alt=\"\" title=\"\" width=\"496\" height=\"655\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/bac\/82a\/546\/bac82a546d57001be0d7ebbbbc927d18.png\"\/><\/figure>\n<h2>\u0423\u0431\u0438\u0440\u0430\u0435\u043c \u0437\u0430 \u0441\u043e\u0431\u043e\u0439<\/h2>\n<p>\u0414\u043b\u044f \u0443\u0434\u0430\u043b\u0435\u043d\u0438\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0445 \u0432\u044b\u0448\u0435 \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0442\u0435:<\/p>\n<pre><code>kubectl delete prometheusremotewrite victoria-metrics kubectl delete grafanaadditionaldatasource victoria-metrics kubectl delete customprometheusrules always-firing-alert kubectl delete grafanadashboarddefinitions up-services kubectl delete customalertmanager telegram helm uninstall victoria-metrics -n victoria-metrics-test kubectl delete ns victoria-metrics-test<\/code><\/pre>\n<h2>\u0417\u0430\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435<\/h2>\n<p>\u041c\u044b \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0435\u043b\u0438, \u043a\u0430\u043a \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u0442\u044c \u043d\u043e\u0432\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u043c\u0435\u0442\u0440\u0438\u043a \u043a \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0443 \u043f\u043e\u0434 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435\u043c Deckhouse, \u0441\u043e\u0437\u0434\u0430\u0442\u044c \u0430\u043b\u0435\u0440\u0442 \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f \u0432 Telegram. \u0417\u0434\u0435\u0441\u044c \u0441\u0442\u043e\u0438\u0442 \u043e\u0431\u0440\u0430\u0442\u0438\u0442\u044c \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0432\u0441\u0435 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u044e\u0442\u0441\u044f \u0447\u0435\u0440\u0435\u0437 custom resources. \u0422\u0430\u043a\u043e\u0439 \u0434\u0435\u043a\u043b\u0430\u0440\u0430\u0442\u0438\u0432\u043d\u044b\u0439 \u043f\u043e\u0434\u0445\u043e\u0434 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u0435\u0442 \u043e\u0431\u0449\u0435\u043c\u0443 \u0442\u0440\u0435\u043d\u0434\u0443, \u043f\u0440\u0438 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u043e\u043f\u0438\u0441\u0430\u043d\u0438\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0445\u0440\u0430\u043d\u0438\u0442\u0441\u044f \u0432 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438 \u0438 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0439 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f \u043a\u0430\u043a \u0435\u0434\u0438\u043d\u0441\u0442\u0432\u0435\u043d\u043d\u044b\u0439 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a (<a href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/526102\/\"><u>GitOps<\/u><\/a>).<\/p>\n<p>\u0421 \u043f\u0440\u0435\u0434\u043b\u043e\u0436\u0435\u043d\u0438\u044f\u043c\u0438 \u0438 \u0432\u043e\u043f\u0440\u043e\u0441\u0430\u043c\u0438 \u0436\u0434\u0435\u043c \u0432\u0430\u0441 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u0430 \u0442\u0430\u043a\u0436\u0435 \u0432 Telegram-\u0447\u0430\u0442\u0435 <a href=\"https:\/\/t.me\/deckhouse_ru\"><u>deckhouse_ru<\/u><\/a>, \u0433\u0434\u0435 \u0432\u0430\u043c \u0432\u0441\u0435\u0433\u0434\u0430 \u043f\u043e\u043c\u043e\u0433\u0443\u0442. \u0422\u0430\u043a\u0436\u0435 \u0431\u0443\u0434\u0435\u043c \u0440\u0430\u0434\u044b Issues (\u0438, \u043a\u043e\u043d\u0435\u0447\u043d\u043e, \u0437\u0432\u0451\u0437\u0434\u0430\u043c) <a href=\"https:\/\/github.com\/deckhouse\/deckhouse\"><u>\u0432 GitHub-\u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438 Deckhouse<\/u><\/a>.<\/p>\n<h2>P.S.<\/h2>\n<p>\u0427\u0438\u0442\u0430\u0439\u0442\u0435 \u0432 \u043d\u0430\u0448\u0435\u043c \u0431\u043b\u043e\u0433\u0435:<\/p>\n<ul>\n<li>\n<p><a href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/704586\/\"><u>\u00ab\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043c\u0435\u0436\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0433\u043e \u0432\u0437\u0430\u0438\u043c\u043e\u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f Kubernetes \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043f\u0440\u043e\u0442\u043e\u043a\u043e\u043b\u0430 NetFlow\u00bb<\/u><\/a>;<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/340728\/\"><u>\u00ab\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0441 Prometheus \u0432 Kubernetes \u0437\u0430 15 \u043c\u0438\u043d\u0443\u0442\u00bb<\/u><\/a>;<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/412901\/\"><u>\u00ab\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0438 Kubernetes (\u043e\u0431\u0437\u043e\u0440 \u0438 \u0432\u0438\u0434\u0435\u043e \u0434\u043e\u043a\u043b\u0430\u0434\u0430)\u00bb<\/u><\/a>.<\/p>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/726528\/\"> https:\/\/habr.com\/ru\/articles\/726528\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>\u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c, \u043a\u0430\u043a \u0432 Kubernetes-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435 \u043f\u043e\u0434 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435\u043c Deckhouse \u0443\u0434\u043e\u0431\u043d\u043e \u0438 \u0431\u044b\u0441\u0442\u0440\u043e \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0441 \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f\u043c\u0438 \u0432 Telegram. \u0412\u043e\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c\u0441\u044f VictoriaMetrics \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043c\u0435\u0442\u0440\u0438\u043a, \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0434\u0430\u0448\u0431\u043e\u0440\u0434 \u0432 Grafana, \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0430\u043b\u0435\u0440\u0442 \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u043e\u043f\u043e\u0432\u0435\u0449\u0435\u043d\u0438\u0435.<\/p>\n<figure class=\"full-width\"><\/figure>\n<h2>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u043e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u044f<\/h2>\n<p>\u041d\u0430\u043c \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u044e\u0442\u0441\u044f:<\/p>\n<ul>\n<li>\n<p>\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Kubernetes \u0441 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u043c <a href=\"https:\/\/deckhouse.ru\/gs\/index.html\"><u>Deckhouse<\/u><\/a> \u0438 \u0432\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/\"><u>\u043c\u043e\u0434\u0443\u043b\u0435\u043c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430<\/u><\/a>.<\/p>\n<\/li>\n<li>\n<p>kubectl. \u041a\u043e\u043c\u0430\u043d\u0434\u044b \u043c\u043e\u0436\u043d\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u043a\u0430\u043a \u043d\u0430 master-\u0443\u0437\u043b\u0435, \u0442\u0430\u043a \u0438 \u0441 \u043b\u044e\u0431\u043e\u0433\u043e \u0434\u0440\u0443\u0433\u043e\u0433\u043e \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430, <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/150-user-authn\/faq.html#%D0%BA%D0%B0%D0%BA-%D1%8F-%D0%BC%D0%BE%D0%B3%D1%83-%D1%81%D0%B3%D0%B5%D0%BD%D0%B5%D1%80%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D1%82%D1%8C-kubeconfig-%D0%B4%D0%BB%D1%8F-%D0%B4%D0%BE%D1%81%D1%82%D1%83%D0%BF%D0%B0-%D0%BA-kubernetes-api\"><u>\u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0432 \u0434\u043e\u0441\u0442\u0443\u043f<\/u><\/a> \u043a API \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/helm.sh\/docs\/intro\/install\/\"><u>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 Helm<\/u><\/a>.<\/p>\n<\/li>\n<li>\n<p>\u0423\u0447\u0435\u0442\u043d\u0430\u044f \u0437\u0430\u043f\u0438\u0441\u044c Telegram.<\/p>\n<\/li>\n<\/ul>\n<h2>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 VictoriaMetrics<\/h2>\n<p>\u0420\u0430\u0437\u0432\u0435\u0440\u043d\u0451\u043c <a href=\"https:\/\/victoriametrics.com\/\"><u>VictoriaMetrics<\/u><\/a> \u0432 \u043f\u0440\u043e\u0441\u0442\u0440\u0430\u043d\u0441\u0442\u0432\u0435 \u0438\u043c\u0435\u043d <code>victoria-metrics-test<\/code>, \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0432 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0435 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0434\u043b\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0438 \u0438\u0437 Helm-\u0447\u0430\u0440\u0442\u0430 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0435\u0439 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e:<\/p>\n<pre><code>helm repo add vm https:\/\/victoriametrics.github.io\/helm-charts\/ &amp;&amp; \\ helm repo update &amp;&amp; \\ helm upgrade --install victoria-metrics vm\/victoria-metrics-single --namespace victoria-metrics-test --create-namespace<\/code><\/pre>\n<h2>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 VictoriaMetrics \u043a Prometheus<\/h2>\n<p>\u0412\u043d\u0435\u0448\u043d\u0435\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u043c\u0435\u0442\u0440\u0438\u043a \u0432 Deckhouse \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0430\u0435\u0442\u0441\u044f \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#prometheusremotewrite\"><u>PrometheusRemoteWrite<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: PrometheusRemoteWrite metadata:   name: victoria-metrics spec:   url: http:\/\/victoria-metrics-victoria-metrics-single-server.victoria-metrics-test.svc.cluster.local:8428\/api\/v1\/write<\/code><\/pre>\n<p><em>\u0412 \u043c\u0438\u043d\u0438\u043c\u0430\u043b\u044c\u043d\u043e\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u0432 \u0440\u0435\u0441\u0443\u0440\u0441\u0435 PrometheusRemoteWrite \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u0430\u0434\u0440\u0435\u0441 VictoriaMetrics. \u0422\u0430\u043a\u0436\u0435 \u043f\u0440\u0438 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e\u0441\u0442\u0438 \u043c\u043e\u0436\u043d\u043e \u0432\u0432\u0435\u0441\u0442\u0438 \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u043b\u044f \u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u0438 \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043f\u0440\u0435\u043e\u0431\u0440\u0430\u0437\u043e\u0432\u0430\u043d\u0438\u0435 label\u2019\u043e\u0432 \u043f\u0435\u0440\u0435\u0434 \u043e\u0442\u043f\u0440\u0430\u0432\u043a\u043e\u0439 \u0434\u0430\u043d\u043d\u044b\u0445<\/em>.<\/p>\n<p><em>\u0412\u0441\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u043d\u044b\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b <\/em><a href=\"https:\/\/github.com\/flant\/examples\/tree\/master\/2023\/03-monitoring\"><em><u>\u0432 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438<\/u><\/em><\/a><em>.<\/em><\/p>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/prometheus-remote-write.yaml<\/code><\/pre>\n<p><em>\u0417\u0434\u0435\u0441\u044c \u043c\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043b\u0438 \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 \u0437\u0430\u0440\u0430\u043d\u0435\u0435 template \u0438\u0437 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u044f \u043d\u0430 GitHub. \u042d\u0442\u043e \u043d\u0435 \u043e\u0431\u044f\u0437\u0430\u0442\u0435\u043b\u044c\u043d\u043e \u2014 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u0442\u044c \u0444\u0430\u0439\u043b \u0441 \u043d\u0443\u0436\u043d\u044b\u043c \u0441\u043e\u0434\u0435\u0440\u0436\u0438\u043c\u044b\u043c \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435 \u043c\u043e\u0436\u043d\u043e \u043b\u044e\u0431\u044b\u043c \u0443\u0434\u043e\u0431\u043d\u044b\u043c \u0441\u043f\u043e\u0441\u043e\u0431\u043e\u043c.<\/em><\/p>\n<p>\u041f\u043e\u0434\u043e\u0436\u0434\u0435\u043c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u043c\u0438\u043d\u0443\u0442 \u0438 \u0443\u0431\u0435\u0434\u0438\u043c\u0441\u044f, \u0447\u0442\u043e \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u043f\u043e\u0441\u0442\u0443\u043f\u0430\u044e\u0442 \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440. \u0414\u043b\u044f \u044d\u0442\u043e\u0433\u043e \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e \u043f\u043e\u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c \u0441\u0442\u0430\u0442\u0443\u0441 \u0431\u0430\u0437\u044b \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u0435\u0442\u0440\u0438\u043a <strong>\u043d\u0430 master-\u0443\u0437\u043b\u0435<\/strong> \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u044b:<\/p>\n<pre><code>curl $(kubectl --namespace victoria-metrics-test get ep -l \"app=server\" -o jsonpath=\"{.items[0].subsets[0].addresses[0].ip}\"):8428\/api\/v1\/status\/tsdb<\/code><\/pre>\n<p>\u0412 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0435 \u0434\u043e\u043b\u0436\u043d\u043e \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u044c\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043d\u043e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0435:<\/p>\n<pre><code class=\"json\">{\"status\":\"success\",\"data\":{\"totalSeries\":100477,\"totalLabelValuePairs\":1099074,\"seriesCountByMetricName\":[{\"name\":\"apiserver_request_slo_duration_seconds_bucket\",\"value\":15576},{\"name\":\"apiserver_request_duration_seconds_bucket\",\"value\":8496},{\"name\":\"etcd_request_duration_seconds_bucket\",\"value\":4284},{\"name\":\"apiserver_response_sizes_bucket\",\"value\":3080},{\"name\":\"apiserver_watch_events_sizes_bucket\",\"value\":1872},{\"name\":\"trivy_vulnerability_id\",\"value\":1541},{\"name\":\"workqueue_queue_duration_seconds_bucket\",\"value\":1089},{\"name\":\"workqueue_work_duration_seconds_bucket\",\"value\":1089},{\"name\":\"container_memory_failures_total\",\"value\":904},{\"name\":\"scheduler_plugin_execution_duration_seconds_bucket\",\"value\":903}],\"seriesCountByLabelName\":[{\"name\":\"__name__\",\"value\":100477},{\"name\":\"prometheus\",\"value\":100477},{\"name\":\"job\",\"value\":100171},{\"name\":\"instance\",\"value\":99736},{\"name\":\"tier\",\"value\":81434},{\"name\":\"service\",\"value\":57725},{\"name\":\"le\",\"value\":51720},{\"name\":\"resource\",\"value\":36606},{\"name\":\"version\",\"value\":34823},{\"name\":\"namespace\",\"value\":33844}],\"seriesCountByFocusLabelValue\":[],\"seriesCountByLabelValuePair\":[{\"name\":\"prometheus=deckhouse\",\"value\":100475},{\"name\":\"tier=cluster\",\"value\":81434},{\"name\":\"service=kubernetes\",\"value\":45705},{\"name\":\"instance=192.168.199.9:6443\",\"value\":45196},{\"name\":\"job=kube-apiserver\",\"value\":45196},{\"name\":\"component=apiserver\",\"value\":31879},{\"name\":\"version=v1\",\"value\":19859},{\"name\":\"scope=cluster\",\"value\":19344},{\"name\":\"container=kube-rbac-proxy\",\"value\":17796},{\"name\":\"__name__=apiserver_request_slo_duration_seconds_bucket\",\"value\":15576}],\"labelValueCountByLabelName\":[{\"name\":\"__name__\",\"value\":1870},{\"name\":\"name\",\"value\":693},{\"name\":\"resource\",\"value\":538},{\"name\":\"le\",\"value\":379},{\"name\":\"type\",\"value\":323},{\"name\":\"secret\",\"value\":282},{\"name\":\"hook\",\"value\":201},{\"name\":\"kind\",\"value\":198},{\"name\":\"controller_name\",\"value\":173},{\"name\":\"installed_version\",\"value\":167}]}}<\/code><\/pre>\n<p><code>\"status\":\"success\"<\/code> \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442 \u043d\u0430 \u0442\u043e, \u0447\u0442\u043e \u0432\u0441\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043a\u0430\u043a \u043d\u0443\u0436\u043d\u043e; \u0435\u0441\u043b\u0438 <code>totalSeries<\/code> \u0431\u043e\u043b\u044c\u0448\u0435 \u043d\u0443\u043b\u044f, \u0442\u043e \u0434\u0430\u043d\u043d\u044b\u0435 \u0443\u0436\u0435 \u043d\u0430\u0447\u0430\u043b\u0438 \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u0442\u044c\u0441\u044f.<\/p>\n<p>\u041f\u0440\u043e\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0441\u0431\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a \u043c\u043e\u0436\u043d\u043e \u0447\u0435\u0440\u0435\u0437 \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 VictoriaMetrics. \u0414\u043b\u044f \u044d\u0442\u043e\u0433\u043e \u043f\u0440\u043e\u0431\u0440\u043e\u0441\u044c\u0442\u0435 \u043f\u043e\u0440\u0442 VictoriaMetrics \u043d\u0430 \u0441\u0432\u043e\u0439 \u043a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440:<\/p>\n<pre><code>export POD_NAME=$(kubectl get pods --namespace victoria-metrics-test -l \"app=server\" -o jsonpath=\"{.items[0].metadata.name}\") &amp;&amp;  kubectl --namespace victoria-metrics-test port-forward $POD_NAME 8428<\/code><\/pre>\n<p><em>\u0414\u043b\u044f \u0443\u0441\u043f\u0435\u0448\u043d\u043e\u0433\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u044d\u0442\u043e\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u043d\u0430 \u043c\u0430\u0448\u0438\u043d\u0435, \u0441 \u043a\u043e\u0442\u043e\u0440\u043e\u0439 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u0437\u0430\u043f\u0440\u043e\u0441, \u0434\u043e\u043b\u0436\u0435\u043d \u0431\u044b\u0442\u044c \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d \u0443\u0434\u0430\u043b\u0435\u043d\u043d\u044b\u0439 \u0434\u043e\u0441\u0442\u0443\u043f \u0434\u043b\u044f kubectl.<\/em><\/p>\n<p>\u0412\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u0441\u0442\u0430\u043d\u0435\u0442 \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <a href=\"http:\/\/localhost:8428\"><u>http:\/\/localhost:8428<\/u><\/a>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u041f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u043d\u0430 \u0430\u0434\u0440\u0435\u0441 <a href=\"http:\/\/localhost:8428\/api\/v1\/status\/tsdb\"><u>http:\/\/localhost:8428\/api\/v1\/status\/tsdb<\/u><\/a> \u2014\u00a0\u0442\u0430\u043c \u0442\u0430\u043a\u0436\u0435 \u0434\u043e\u043b\u0436\u0435\u043d \u0431\u044b\u0442\u044c \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0435\u043d \u0441\u0442\u0430\u0442\u0443\u0441 <code>\"status\":\"success\"<\/code>; \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 <code>totalSeries<\/code> \u0431\u043e\u043b\u044c\u0448\u0435 \u043d\u0443\u043b\u044f \u0431\u0443\u0434\u0435\u0442 \u0441\u0438\u0433\u043d\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043e\u0431 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\u043c \u0441\u0431\u043e\u0440\u0435 \u043c\u0435\u0442\u0440\u0438\u043a:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u0445\u0440\u0430\u043d\u044f\u0442\u0441\u044f \u0432 VictoriaMectrics. \u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u043c \u044d\u0442\u043e \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u043a Grafana.<\/p>\n<h2>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 VictoriaMetrics \u043a Grafana<\/h2>\n<h3>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u043d\u043e\u0432\u043e\u0433\u043e datasource<\/h3>\n<p>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 datasource \u043a Grafana \u0432 Deckhouse \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#grafanaadditionaldatasource\"><u>GrafanaAdditionalDatasource<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: GrafanaAdditionalDatasource metadata:   name: victoria-metrics spec:   access: Proxy   basicAuth: false   jsonData:     timeInterval: 30s   type: prometheus   url: http:\/\/victoria-metrics-victoria-metrics-single-server.victoria-metrics-test.svc.cluster.local:8428\/<\/code><\/pre>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u0435\u0433\u043e \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/grafana-additional-datasource.yaml<\/code><\/pre>\n<p>Deckhouse \u043f\u0435\u0440\u0435\u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442 Grafana \u0441 \u043d\u043e\u0432\u043e\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0435\u0439, \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u043c \u0435\u0435 \u0433\u043e\u0442\u043e\u0432\u043d\u043e\u0441\u0442\u044c:<\/p>\n<pre><code>kubectl -n d8-monitoring get po -l app=grafana<\/code><\/pre>\n<p>\u0421\u0442\u0430\u0442\u0443\u0441 \u0434\u043e\u043b\u0436\u0435\u043d \u0431\u044b\u0442\u044c <code>Running<\/code>:<\/p>\n<pre><code>kubectl -n d8-monitoring get po -l app=grafana NAME                       READY   STATUS    RESTARTS   AGE grafana-56df555c67-glzqr   3\/3     Running   0          67s<\/code><\/pre>\n<h3>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f<\/h3>\n<p>\u0417\u0430\u0439\u0434\u0435\u043c \u0432 \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 Grafana \u0438 \u043f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u043d\u0430 \u0432\u043a\u043b\u0430\u0434\u043a\u0443 <em>Configuration<\/em> -> <em>Data sources<\/em>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0412 \u0441\u043f\u0438\u0441\u043a\u0435 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b\u0445 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u043e\u0432 \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0430\u0442\u044c\u0441\u044f <em>victoria-metrics<\/em>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u041e\u0442\u043a\u0440\u043e\u0435\u043c \u043c\u0435\u043d\u044e <em>Explore<\/em>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0412\u044b\u0431\u0435\u0440\u0435\u043c <em>victoria-metrics<\/em> \u0432 \u043a\u0430\u0447\u0435\u0441\u0442\u0432\u0435 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u0430 \u0434\u0430\u043d\u043d\u044b\u0445:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0420\u0430\u0441\u043a\u0440\u043e\u0435\u043c <em>Metrics browser<\/em>: \u0432 \u043d\u0435\u043c \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u044c\u0441\u044f \u0441\u043f\u0438\u0441\u043e\u043a \u043c\u0435\u0442\u0440\u0438\u043a, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0445 \u0438\u0437 VictoriaMetrics:<\/p>\n<figure class=\"\"><\/figure>\n<p>\u0412\u0441\u0435 \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u043e \u0438 \u0433\u043e\u0442\u043e\u0432\u043e \u043a \u0440\u0430\u0431\u043e\u0442\u0435.<\/p>\n<h2>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0430\u043b\u0435\u0440\u0442\u0430<\/h2>\n<p>\u041d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0435 \u043e \u0441\u043e\u0431\u044b\u0442\u0438\u044f\u0445, \u0441\u043e\u0437\u0434\u0430\u0432 \u043d\u043e\u0432\u044b\u0439 \u0430\u043b\u0435\u0440\u0442.\u00a0<\/p>\n<p>\u0412 Deckhouse \u0430\u043b\u0435\u0440\u0442\u044b \u043e\u043f\u0438\u0441\u044b\u0432\u0430\u044e\u0442\u0441\u044f \u0432 CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#customprometheusrules\"><u>CustomPrometheusRules<\/u><\/a>:<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: CustomPrometheusRules metadata:   name: always-firing-alert spec:   groups:   - name: cluster-state-alert.rules     rules:       - alert: PrometheusCanScrapeTragets         annotations:           description: This is a fake alert only for a demo.           summary: The alert shows that Prometheus can scrape targets.         expr: |           up{job=\"deckhouse\"}<\/code><\/pre>\n<p><em>\u0421\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0439 \u0430\u043b\u0435\u0440\u0442 \u00ab\u0431\u0435\u0441\u043f\u043e\u043b\u0435\u0437\u043d\u044b\u0439\u00bb \u2014 \u043e\u043d \u0432\u0441\u0435\u0433\u0434\u0430 \u0430\u043a\u0442\u0438\u0432\u043d\u044b\u0439, \u043d\u043e \u0445\u043e\u0440\u043e\u0448\u043e \u043f\u043e\u0434\u0445\u043e\u0434\u0438\u0442 \u0434\u043b\u044f \u0446\u0435\u043b\u0435\u0439 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f.<\/em><\/p>\n<p>\u041f\u0440\u0438\u043c\u0435\u043d\u0438\u043c \u0435\u0433\u043e \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435:<\/p>\n<pre><code>kubectl create -f https:\/\/raw.githubusercontent.com\/flant\/examples\/master\/2023\/03-monitoring\/custom-prometheus-rule.yaml<\/code><\/pre>\n<p>\u0414\u043b\u044f \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u043f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u0432 Prometheus \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <code>&lt;\u0430\u0434\u0440\u0435\u0441_Grafana>\/prometheus\/<\/code> \u0438 \u043e\u0442\u043a\u0440\u043e\u0435\u043c \u0432\u043a\u043b\u0430\u0434\u043a\u0443 <em>Alerts<\/em>. \u0412 \u0441\u0442\u0440\u043e\u043a\u0435 \u043f\u043e\u0438\u0441\u043a\u0430 \u0432\u0432\u0435\u0434\u0435\u043c \u0438\u043c\u044f \u0430\u043b\u0435\u0440\u0442\u0430 \u2014 <em>PrometheusCanScrapeTragets<\/em>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0410\u043b\u0435\u0440\u0442 \u0441\u043e\u0437\u0434\u0430\u043d.<\/p>\n<h2>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u0448\u0431\u043e\u0440\u0434\u0430 \u0432 Grafana<\/h2>\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0434\u0430\u0448\u0431\u043e\u0440\u0434 <em>Services Up<\/em> \u0432 \u043a\u0430\u0442\u0430\u043b\u043e\u0433\u0435 <em>Services<\/em> (\u0447\u0435\u0440\u0435\u0437 CustomResource <a href=\"https:\/\/deckhouse.ru\/documentation\/v1\/modules\/300-prometheus\/cr.html#grafanadashboarddefinition\"><u>GrafanaDashboardDefinition<\/u><\/a>):<\/p>\n<pre><code class=\"yaml\">--- apiVersion: deckhouse.io\/v1 kind: GrafanaDashboardDefinition metadata:   name: up-services spec:   folder: Services   definition: |     {       \"annotations\": {         \"list\": [           {             \"builtIn\": 1,             \"datasource\": {               \"type\": \"grafana\",               \"uid\": \"-- Grafana --\"             },             \"enable\": true,             \"hide\": true,             \"iconColor\": \"rgba(0, 211, 255, 1)\",             \"name\": \"Annotations &amp; Alerts\",             \"target\": {               \"limit\": 100,               \"matchAny\": false,               \"tags\": [],               \"type\": \"dashboard\"             },             \"type\": \"dashboard\"           }         ]       },       \"editable\": false,       \"fiscalYearStartMonth\": 0,       \"graphTooltip\": 0,       \"id\": 31,       \"links\": [],       \"liveNow\": false,       \"panels\": [         {           \"datasource\": {             \"type\": \"prometheus\",             \"uid\": \"P0D6E4079E36703EB\"           },           \"fieldConfig\": {             \"defaults\": {               \"color\": {                 \"mode\": \"palette-classic\"               },               \"custom\": {                 \"axisLabel\": \"\",                 \"axisPlacement\": \"auto\",                 \"barAlignment\": 0,                 \"drawStyle\": \"bars\",                 \"fillOpacity\": 15,                 \"gradientMode\": \"none\",                 \"hideFrom\": {                   \"legend\": false,                   \"tooltip\": false,                   \"viz\": false                 },                 \"lineInterpolation\": \"linear\",                 \"lineWidth\": 10,                 \"pointSize\": 5,                 \"scaleDistribution\": {                   \"type\": \"linear\"                 },                 \"showPoints\": \"never\",                 \"spanNulls\": false,                 \"stacking\": {                   \"group\": \"A\",                   \"mode\": \"normal\"                 },                 \"thresholdsStyle\": {                   \"mode\": \"off\"                 }               },               \"mappings\": [],               \"thresholds\": {                 \"mode\": \"absolute\",                 \"steps\": [                   {                     \"color\": \"green\",                     \"value\": null                   },                   {                     \"color\": \"red\",                     \"value\": 80                   }                 ]               }          <\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-405501","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/405501","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=405501"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/405501\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=405501"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=405501"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=405501"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}