{"id":403124,"date":"2024-06-29T16:55:44","date_gmt":"2024-06-29T16:55:44","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=403124"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=403124","title":{"rendered":"Monitoring and Logging external services in Kubernetes<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\"image\"<\/p>\n

Greetings to all!<\/p>\n

As a matter of fact I haven’t found so far any generalized guide on logging and monitoring of metrics from external systems to Kubernetes<\/a> on the web. Here I want to share with you my own version. First of all it is supposed that you have already got working Prometheus<\/a> and other services.<\/p>\n

As an example I use third-party data for stateful service RDBMS PostgreSQL<\/a> in a Docker<\/a> container. In our company we also use the Helm<\/a> package manager, and you’ll find how it works below. You can see same examples in this article below. You can see same examples in this article below. For a complete solution we create a nested chart.
<\/a> <\/p>\n

Logging<\/h4>\n

Many companies use ELK stack for centralized logging (Elasticsearch<\/a> + Logstash<\/a> + kibana<\/a>). In our case we don’t need to index the content and I apply lightweight Loki<\/a>. It is available and can be installed as a Helm package. We added it as subchart having changed values for ingress & pv.<\/p>\n

values.yaml<\/b> <\/p>\n
\n
ingress:   enabled: true   annotations:      kubernetes.io\/ingress.class: nginx   hosts:     - host: kube-loki.example.domain       paths:          - \/   tls: []  ....  persistence:   type: pvc   enabled: true   accessModes:     - ReadWriteOnce   size: 100Gi   finalizers:     - kubernetes.io\/pvc-protection   existingClaim: \"pv-loki\" <\/code><\/pre>\n
  <\/div>\n<\/p><\/div>\n
  We use Loki Docker Logging Driver<\/a> for log sending to instance Loki<\/a>.
  You have to set it up as a Docker<\/a> plugin on all hosts from which you want to get logs. There are several ways to use plugin. My choice is to put it in the yaml file of Docker Compose<\/a> which is a part of Ansible<\/a> playbook.<\/p>\n
                         postgres.yaml<\/b>                         <\/p>\n
\n
    - name: Run containers       docker_compose:         project_name: main-postgres         definition:           version: '3.7'           services:             p:               image: \"{{ postgres_version }}\"               container_name: postgresql               restart: always               volumes:                 - \"{{ postgres_dir }}\/data:\/var\/lib\/postgresql\/data\"                 - \"{{ postgres_dir }}\/postgres_init_scripts:\/docker-entrypoint-initdb.d\"               environment:                 POSTGRES_PASSWORD: {{ postgres_pass }}                 POSTGRES_USER: {{ postgres_user }}               ports:                 - \"{{ postgres_ip }}:{{ postgres_port }}:5432\"               logging:                 driver: \"loki\"                 options:                   loki-url: \"{{ loki_url }}\"                   loki-batch-size: \"{{ loki_batch_size }}\"                   loki-retries: \"{{ loki_retries }}\" ... <\/code><\/pre>\n
  where is loki_url: kube-loki.example.domain\/loki\/api\/v1\/push<\/a>   <\/div>\n<\/p><\/div>\n
  <\/p>\n
Metrics<\/h4>\n
  PostgreSQL metrics are collected to Prometheus<\/a> with a postgres_exporter<\/a>. See extension of Ansible<\/a> playbook file below.<\/p>\n
                         postgres.yaml<\/b>                         <\/p>\n
\n
...             pexp:               image: \"wrouesnel\/postgres_exporter\"               container_name: pexporter               restart: unless-stopped               environment:                 DATA_SOURCE_NAME: \"postgresql:\/\/{{ postgres_user }}:{{ postgres_pass }}@p:5432\/postgres?sslmode=disable\"               ports:                 - \"{{ postgres_ip }}:{{ postgres_exporter_port }}:9187\"               logging:                 driver: \"json-file\"                 options:                   max-size: \"5m\" ... <\/code><\/pre>\n
  <\/div>\n<\/p><\/div>\n
  For better clarity the names of external stateful services are presented with Endpoints.<\/p>\n
                         postgres-service.yaml<\/b>                         <\/p>\n
\n
apiVersion: v1 kind: Endpoints metadata:   name: postgres-exporter subsets:   - addresses:       - ip: {{ .Values.service.postgres.ip }}     ports:       - port: {{ .Values.service.postgres.port }}         protocol: TCP --- apiVersion: v1 kind: Service metadata:   name: postgres-exporter   labels:     chart:  \"{{ .Chart.Name }}-{{ .Chart.Version | replace \"+\" \"_\" }}\" spec:   ports:     - protocol: TCP       port: {{ .Values.service.postgres.port }}       targetPort: {{ .Values.service.postgres.port }} <\/code><\/pre>\n
  <\/div>\n<\/p><\/div>\n
  Prometheus settings for getting data from postgres_exporter are tuned by means of value editing in the subchart.<\/p>\n
                         values.yaml<\/b>                         <\/p>\n
\n
scrape_configs: ...   - job_name: postgres-exporter     static_configs:       - targets:           - postgres-exporter.applicationnamespace.svc.cluster.local:9187         labels:           alias: postgres ... <\/code><\/pre>\n
  <\/div>\n<\/p><\/div>\n
  For visualisation of data install a proper dashboard for Grafana<\/a> and add datasource. That can be done with values of Grafana subchart. See how it looks below.<\/p>\n
                         How it looks<\/b>                         <\/p>\n
  <\/div>\n<\/p><\/div>\n
  I hope my first brief article in English will help you to understand basic ideas of this solution and save your time when adjusting monitoring and logging procedures of external services in Loki\/Prometheus in Kubernetes.<\/div>\n<\/div>\n<\/div>\n
<\/div>\n

 \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438  https:\/\/habr.com\/ru\/articles\/506278\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
<\/div>\n
\n
\n
\n
\"image\"<\/p>\n
  Greetings to all!<\/p>\n
  As a matter of fact I haven’t found so far any generalized guide on logging and monitoring of metrics from external systems to Kubernetes<\/a> on the web. Here I want to share with you my own version. First of all it is supposed that you have already got working Prometheus<\/a> and other services.<\/p>\n
  As an example I use third-party data for stateful service RDBMS PostgreSQL<\/a> in a Docker<\/a> container. In our company we also use the Helm<\/a> package manager, and you’ll find how it works below. You can see same examples in this article below. You can see same examples in this article below. For a complete solution we create a nested chart.   <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-403124","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/403124","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=403124"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/403124\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=403124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=403124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=403124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}