{"id":303128,"date":"2020-05-06T15:00:48","date_gmt":"2020-05-06T15:00:48","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=303128"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=303128","title":{"rendered":"Prometheus: \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 HTTP \u0447\u0435\u0440\u0435\u0437 Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440"},"content":{"rendered":"\n<div class=\"post__text post__text-html post__text_v1\" id=\"post-content-body\" data-io-article-url=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/500448\/\"><b><i>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u0412 \u043c\u0430\u0435 OTUS \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u0435\u0442 <a href=\"https:\/\/otus.pw\/UXML\/\">\u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0443\u043c \u043f\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0443 \u0438 \u043b\u043e\u0433\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e<\/a>, \u043a\u0430\u043a \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0442\u0430\u043a \u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e Zabbix, Prometheus, Grafana \u0438 ELK. \u0412 \u0441\u0432\u044f\u0437\u0438 \u0441 \u044d\u0442\u0438\u043c \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u0434\u0435\u043b\u0438\u043c\u0441\u044f \u043f\u043e\u043b\u0435\u0437\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c \u043f\u043e \u0442\u0435\u043c\u0435.<\/i><\/b><\/p>\n<hr>\n<p>  <a href=\"https:\/\/github.com\/prometheus\/blackbox_exporter\">Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440<\/a> \u0434\u043b\u044f Prometheus \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0440\u0435\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0432\u043d\u0435\u0448\u043d\u0438\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0447\u0435\u0440\u0435\u0437 HTTP, HTTPS, DNS, TCP, ICMP. \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u043f\u043e\u043a\u0430\u0436\u0443 \u0432\u0430\u043c, \u043a\u0430\u043a \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 HTTP\/HTTPS \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430. \u0417\u0430\u043f\u0443\u0441\u043a\u0430\u0442\u044c Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440 \u0431\u0443\u0434\u0435\u043c \u0432 Kubernetes.<a name=\"habracut\"><\/a><\/p>\n<h3>\u041e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u0435<\/h3>\n<p>  \u041d\u0430\u043c \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0435:<\/p>\n<ul>\n<li>Kubernetes<\/li>\n<li>Prometheus Operator<\/li>\n<\/ul>\n<p>  <\/p>\n<h4>\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430<\/h4>\n<p>  \u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u0443\u0435\u043c Blackbox \u0447\u0435\u0440\u0435\u0437 <code>ConfigMap<\/code> \u0434\u043b\u044f \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 <code>http<\/code> \u043c\u043e\u0434\u0443\u043b\u044f \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430 \u0432\u0435\u0431-\u0441\u043b\u0443\u0436\u0431.<\/p>\n<pre><code class=\"go\">apiVersion: v1 kind: ConfigMap metadata:   name: prometheus-blackbox-exporter   labels:     app: prometheus-blackbox-exporter data:   blackbox.yaml: |     modules:       http_2xx:         http:           no_follow_redirects: false           preferred_ip_protocol: ip4           valid_http_versions:           - HTTP\/1.1           - HTTP\/2           valid_status_codes: []         prober: http         timeout: 5s<\/code><\/pre>\n<p>  \u041c\u043e\u0434\u0443\u043b\u044c <code>http_2xx<\/code> \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f \u0434\u043b\u044f \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e \u0432\u0435\u0431-\u0441\u0435\u0440\u0432\u0438\u0441 \u0432\u043e\u0437\u0432\u0440\u0430\u0449\u0430\u0435\u0442 \u043a\u043e\u0434 \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u044f HTTP 2xx. \u041f\u043e\u0434\u0440\u043e\u0431\u043d\u0435\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430 \u043e\u043f\u0438\u0441\u0430\u043d\u0430 \u0432 <a href=\"https:\/\/github.com\/prometheus\/blackbox_exporter\/blob\/master\/CONFIGURATION.md\">\u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438<\/a>.<\/p>\n<h4>\u0414\u0435\u043f\u043b\u043e\u0439 blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430 \u0432 Kubernetes \u043a\u043b\u0430\u0441\u0442\u0435\u0440<\/h4>\n<p>  \u041e\u043f\u0438\u0448\u0438\u0442\u0435 <code>Deployment<\/code> \u0438 <code>Service<\/code> \u0434\u043b\u044f \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f \u0432 Kubernetes.<\/p>\n<pre><code class=\"go\">--- kind: Service apiVersion: v1 metadata:   name: prometheus-blackbox-exporter   labels:     app: prometheus-blackbox-exporter spec:   type: ClusterIP   ports:     - name: http       port: 9115       protocol: TCP   selector:     app: prometheus-blackbox-exporter  --- apiVersion: apps\/v1 kind: Deployment metadata:   name: prometheus-blackbox-exporter   labels:     app: prometheus-blackbox-exporter spec:   replicas: 1   selector:     matchLabels:       app: prometheus-blackbox-exporter   template:     metadata:       labels:         app: prometheus-blackbox-exporter     spec:       restartPolicy: Always       containers:         - name: blackbox-exporter           image: &quot;prom\/blackbox-exporter:v0.15.1&quot;           imagePullPolicy: IfNotPresent           securityContext:             readOnlyRootFilesystem: true             runAsNonRoot: true             runAsUser: 1000           args:             - &quot;--config.file=\/config\/blackbox.yaml&quot;           resources:             {}           ports:             - containerPort: 9115               name: http           livenessProbe:             httpGet:               path: \/health               port: http           readinessProbe:             httpGet:               path: \/health               port: http           volumeMounts:             - mountPath: \/config               name: config         - name: configmap-reload           image: &quot;jimmidyson\/configmap-reload:v0.2.2&quot;           imagePullPolicy: &quot;IfNotPresent&quot;           securityContext:             runAsNonRoot: true             runAsUser: 65534           args:             - --volume-dir=\/etc\/config             - --webhook-url=http:\/\/localhost:9115\/-\/reload           resources:             {}           volumeMounts:             - mountPath: \/etc\/config               name: config               readOnly: true       volumes:         - name: config           configMap:             name: prometheus-blackbox-exporter<\/code><\/pre>\n<p>  Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440 \u043c\u043e\u0436\u043d\u043e \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442\u044c \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u044b. \u041f\u0440\u043e\u0441\u0442\u0440\u0430\u043d\u0441\u0442\u0432\u043e \u0438\u043c\u0435\u043d <code>monitoring<\/code> \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0441\u044f \u043a Prometheus Operator.<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring apply -f blackbox-exporter.yaml<\/code><\/pre>\n<p>  \u0423\u0431\u0435\u0434\u0438\u0442\u0435\u0441\u044c, \u0447\u0442\u043e \u0432\u0441\u0435 \u0441\u043b\u0443\u0436\u0431\u044b \u0437\u0430\u043f\u0443\u0449\u0435\u043d\u044b, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044f \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0443\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u0443:<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter<\/code><\/pre>\n<p>  <\/p>\n<h4>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 Blackbox<\/h4>\n<p>  \u0412\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u043f\u043e\u043b\u0443\u0447\u0438\u0442\u044c \u0434\u043e\u0441\u0442\u0443\u043f \u043a \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441\u0443 Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e <code>port-forward<\/code>:<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring port-forward svc\/prometheus-blackbox-exporter 9115:9115<\/code><\/pre>\n<p>  \u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u0442\u0435\u0441\u044c \u043a \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441\u0443 Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430 \u0447\u0435\u0440\u0435\u0437 \u0432\u0435\u0431-\u0431\u0440\u0430\u0443\u0437\u0435\u0440 \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <a href=\"http:\/\/localhost\">localhost<\/a>:9115.<\/p>\n<p>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/webt\/0z\/cd\/9s\/0zcd9smy_0fxhrht76dd6piazj0.png\"><\/p>\n<p>  \u0415\u0441\u043b\u0438 \u0432\u044b \u043f\u0435\u0440\u0435\u0439\u0434\u0435\u0442\u0435 \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <a href=\"http:\/\/localhost:9115\/probe?module=http_2xx&amp;target=https:\/\/www.google.com\">http:\/\/localhost:9115\/probe?module=http_2xx&amp;target=https:\/\/www.google.com<\/a>, \u0442\u043e \u0443\u0432\u0438\u0434\u0438\u0442\u0435 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0443\u043a\u0430\u0437\u0430\u043d\u043d\u043e\u0433\u043e URL (<a href=\"https:\/\/www.google.com\">https:\/\/www.google.com<\/a>).<\/p>\n<p>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/webt\/yo\/wj\/st\/yowjstp0dw_xdegczrum1lq2vti.png\"><\/p>\n<p>  \u0417\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u043c\u0435\u0442\u0440\u0438\u043a\u0438 <code>probe_success<\/code> \u0440\u0430\u0432\u043d\u043e\u0435 1 \u043e\u0437\u043d\u0430\u0447\u0430\u0435\u0442 \u0443\u0441\u043f\u0435\u0448\u043d\u0443\u044e \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0443. \u0417\u043d\u0430\u0447\u0435\u043d\u0438\u0435 0 \u0433\u043e\u0432\u043e\u0440\u0438\u0442 \u043e\u0431 \u043e\u0448\u0438\u0431\u043a\u0435.<\/p>\n<h4>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 Prometheus<\/h4>\n<p>  \u041f\u043e\u0441\u043b\u0435 \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f BlackBox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430, \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0435\u043c Prometheus \u0432 <code>prometheus-additional.yaml<\/code>.<\/p>\n<pre><code class=\"go\">- job_name: 'kube-api-blackbox'   scrape_interval: 1w   metrics_path: \/probe   params:     module: [http_2xx]   static_configs:    - targets:       - https:\/\/www.google.com       - http:\/\/www.example.com       - https:\/\/prometheus.io   relabel_configs:    - source_labels: [__address__]      target_label: __param_target    - source_labels: [__param_target]      target_label: instance    - target_label: __address__      replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.<\/code><\/pre>\n<p>  \u0413\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c <code>Secret<\/code>, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044f \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0443\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u0443.<\/p>\n<pre><code class=\"go\">PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64) cat &lt;&lt; EOF | kubectl --namespace=monitoring apply -f - apiVersion: v1 kind: Secret metadata:   name: additional-scrape-configs type: Opaque data:   prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG EOF<\/code><\/pre>\n<p>  \u0423\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u043c <code>additional-scrape-configs<\/code> \u0434\u043b\u044f Prometheus Operator, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044f <code>additionalScrapeConfigs<\/code>.<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring edit prometheuses k8s ... spec:   additionalScrapeConfigs:     key: prometheus-additional.yaml     name: additional-scrape-configs<\/code><\/pre>\n<p>  \u0417\u0430\u0445\u043e\u0434\u0438\u043c \u0432 \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 Prometheus, \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0435\u043c \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u0438 \u0446\u0435\u043b\u0438.<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring port-forward svc\/prometheus-k8s 9090:9090<\/code><\/pre>\n<p>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/webt\/pi\/9e\/kz\/pi9ekzhhjnts1m1goe6lz7eewaw.png\"><\/p>\n<p>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/webt\/h5\/8p\/sc\/h58psc4y8guijvdjqhbujdlexpa.png\"><\/p>\n<p>  \u0412\u0438\u0434\u0438\u043c \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u0438 \u0446\u0435\u043b\u0438 Blackbox.<\/p>\n<h4>\u0414\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u043f\u0440\u0430\u0432\u0438\u043b \u0434\u043b\u044f \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0439 (alert)<\/h4>\n<p>  \u0414\u043b\u044f \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u044f \u043e\u043f\u043e\u0432\u0435\u0449\u0435\u043d\u0438\u0439 \u043e\u0442 Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u043f\u0440\u0430\u0432\u0438\u043b\u0430 \u0432 Prometheus Operator.<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules ...   - name: blackbox-exporter     rules:     - alert: ProbeFailed       expr: probe_success == 0       for: 5m       labels:         severity: error       annotations:         summary: &quot;Probe failed (instance {{ $labels.instance }})&quot;         description: &quot;Probe failed\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;     - alert: SlowProbe       expr: avg_over_time(probe_duration_seconds[1m]) &gt; 1       for: 5m       labels:         severity: warning       annotations:         summary: &quot;Slow probe (instance {{ $labels.instance }})&quot;         description: &quot;Blackbox probe took more than 1s to complete\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;     - alert: HttpStatusCode       expr: probe_http_status_code &lt;= 199 OR probe_http_status_code &gt;= 400       for: 5m       labels:         severity: error       annotations:         summary: &quot;HTTP Status Code (instance {{ $labels.instance }})&quot;         description: &quot;HTTP status code is not 200-399\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;     - alert: SslCertificateWillExpireSoon       expr: probe_ssl_earliest_cert_expiry - time() &lt; 86400 * 30       for: 5m       labels:         severity: warning       annotations:         summary: &quot;SSL certificate will expire soon (instance {{ $labels.instance }})&quot;         description: &quot;SSL certificate expires in 30 days\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;     - alert: SslCertificateHasExpired       expr: probe_ssl_earliest_cert_expiry - time()  &lt;= 0       for: 5m       labels:         severity: error       annotations:         summary: &quot;SSL certificate has expired (instance {{ $labels.instance }})&quot;         description: &quot;SSL certificate has expired already\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;     - alert: HttpSlowRequests       expr: avg_over_time(probe_http_duration_seconds[1m]) &gt; 1       for: 5m       labels:         severity: warning       annotations:         summary: &quot;HTTP slow requests (instance {{ $labels.instance }})&quot;         description: &quot;HTTP request took more than 1s\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;     - alert: SlowPing       expr: avg_over_time(probe_icmp_duration_seconds[1m]) &gt; 1       for: 5m       labels:         severity: warning       annotations:         summary: &quot;Slow ping (instance {{ $labels.instance }})&quot;         description: &quot;Blackbox ping took more than 1s\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;<\/code><\/pre>\n<p>  \u0412 \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441\u0435 Prometheus \u043f\u0435\u0440\u0435\u0439\u0434\u0438\u0442\u0435 \u0432 \u0440\u0430\u0437\u0434\u0435\u043b Status =&gt; Rules \u0438 \u043d\u0430\u0439\u0434\u0438\u0442\u0435 \u043f\u0440\u0430\u0432\u0438\u043b\u0430 \u043e\u043f\u043e\u0432\u0435\u0449\u0435\u043d\u0438\u0439 \u0434\u043b\u044f blackbox-exporter.<\/p>\n<p>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/webt\/ku\/q1\/lt\/kuq1ltgw-9ggcch0r6nrjnsh_ug.png\"><\/p>\n<h4>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0439 \u043e\u0431 \u0438\u0441\u0442\u0435\u0447\u0435\u043d\u0438\u0438 \u0441\u0440\u043e\u043a\u0430 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f SSL-\u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432 Kubernetes API Server <\/h4>\n<p>  \u0414\u0430\u0432\u0430\u0439\u0442\u0435 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0438\u0441\u0442\u0435\u0447\u0435\u043d\u0438\u044f \u0441\u0440\u043e\u043a\u0430 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f SSL-\u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432 Kubernetes API Server. \u041e\u043d \u0431\u0443\u0434\u0435\u0442 \u043e\u0442\u043f\u0440\u0430\u0432\u043b\u044f\u0442\u044c \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u044f \u043e\u0434\u0438\u043d \u0440\u0430\u0437 \u0432 \u043d\u0435\u0434\u0435\u043b\u044e.<\/p>\n<p>  \u0414\u043e\u0431\u0430\u0432\u043b\u044f\u0435\u043c \u043c\u043e\u0434\u0443\u043b\u044c Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430 \u0434\u043b\u044f Kubernetes API Server Authentication.<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter ...       kube-api:         http:           method: GET           no_follow_redirects: false           preferred_ip_protocol: ip4           tls_config:             insecure_skip_verify: false             ca_file: \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt           bearer_token_file: \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token           valid_http_versions:           - HTTP\/1.1           - HTTP\/2           valid_status_codes: []         prober: http         timeout: 5s<\/code><\/pre>\n<p>  \u0414\u043e\u0431\u0430\u0432\u043b\u044f\u0435\u043c scrape-\u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e Prometheus<\/p>\n<pre><code class=\"go\">- job_name: 'kube-api-blackbox'   metrics_path: \/probe   params:     module: [kube-api]   static_configs:    - targets:       - https:\/\/kubernetes.default.svc\/api   relabel_configs:    - source_labels: [__address__]      target_label: __param_target    - source_labels: [__param_target]      target_label: instance    - target_label: __address__      replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.<\/code><\/pre>\n<p>  \u041f\u0440\u0438\u043c\u0435\u043d\u044f\u0435\u043c Prometheus Secret<\/p>\n<pre><code class=\"go\">PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64) cat &lt;&lt; EOF | kubectl --namespace=monitoring apply -f - apiVersion: v1 kind: Secret metadata:   name: additional-scrape-configs type: Opaque data:   prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG EOF<\/code><\/pre>\n<p>  \u0414\u043e\u0431\u0430\u0432\u043b\u044f\u0435\u043c \u043f\u0440\u0430\u0432\u0438\u043b\u0430 \u043e\u043f\u043e\u0432\u0435\u0449\u0435\u043d\u0438\u0439<\/p>\n<pre><code class=\"go\">kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules ...   - name: k8s-api-server-cert-expiry     rules:     - alert: K8sAPIServerSSLCertExpiringAfterThreeMonths       expr: probe_ssl_earliest_cert_expiry{job=&quot;kube-api-blackbox&quot;} - time() &lt; 86400 * 90        for: 1w       labels:         severity: warning       annotations:         summary: &quot;Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})&quot;         description: &quot;Kubernetes API Server SSL certificate expires in 90 days\\n  VALUE = {{ $value }}\\n  LABELS: {{ $labels }}&quot;<\/code><\/pre>\n<p>  <\/p>\n<h3>\u041f\u043e\u043b\u0435\u0437\u043d\u044b\u0435 \u0441\u0441\u044b\u043b\u043a\u0438<\/h3>\n<p>  <\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/prometheus\/blackbox_exporter\">github.com\/prometheus\/blackbox_exporter<\/a><\/li>\n<li><a href=\"https:\/\/awesome-prometheus-alerts.grep.to\/rules#blackbox\">awesome-prometheus-alerts.grep.to\/rules#blackbox<\/a><\/li>\n<\/ul>\n<p>  <\/p>\n<hr>\n<p>  <a href=\"https:\/\/otus.pw\/UXML\/\">\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0438 \u043b\u043e\u0433\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0432 Docker<\/a><\/p>\n<hr>\n<\/div>\n<p> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/500448\/\"> https:\/\/habr.com\/ru\/company\/otus\/blog\/500448\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n<div class=\"post__text post__text-html post__text_v1\" id=\"post-content-body\" data-io-article-url=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/500448\/\"><b><i>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u0412 \u043c\u0430\u0435 OTUS \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u0435\u0442 <a href=\"https:\/\/otus.pw\/UXML\/\">\u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0443\u043c \u043f\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0443 \u0438 \u043b\u043e\u0433\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e<\/a>, \u043a\u0430\u043a \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0442\u0430\u043a \u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e Zabbix, Prometheus, Grafana \u0438 ELK. \u0412 \u0441\u0432\u044f\u0437\u0438 \u0441 \u044d\u0442\u0438\u043c \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u0434\u0435\u043b\u0438\u043c\u0441\u044f \u043f\u043e\u043b\u0435\u0437\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c \u043f\u043e \u0442\u0435\u043c\u0435.<\/i><\/b><\/p>\n<hr>\n<p>  <a href=\"https:\/\/github.com\/prometheus\/blackbox_exporter\">Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440<\/a> \u0434\u043b\u044f Prometheus \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0440\u0435\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0432\u043d\u0435\u0448\u043d\u0438\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0447\u0435\u0440\u0435\u0437 HTTP, HTTPS, DNS, TCP, ICMP. \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u043f\u043e\u043a\u0430\u0436\u0443 \u0432\u0430\u043c, \u043a\u0430\u043a \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 HTTP\/HTTPS \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440\u0430. \u0417\u0430\u043f\u0443\u0441\u043a\u0430\u0442\u044c Blackbox \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440 \u0431\u0443\u0434\u0435\u043c \u0432 Kubernetes.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-303128","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/303128","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=303128"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/303128\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=303128"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=303128"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=303128"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}