{"id":341394,"date":"2022-11-18T03:00:06","date_gmt":"2022-11-18T03:00:06","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=341394"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=341394","title":{"rendered":"<span>Kubernetes the hard way<\/span>"},"content":{"rendered":"<div><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0414\u043e\u0431\u0440\u044b\u0439 \u041a\u043e\u0442\u00a0<a href=\"https:\/\/t.me\/Dobry_kot\" rel=\"noopener noreferrer nofollow\">Telegram<\/a>.<\/p>\n<p>\u041e\u0442 \u043a\u043e\u043b\u043b\u0435\u043a\u0442\u0438\u0432\u0430 FR-Solutions \u0438 \u043f\u0440\u0438 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u043a\u0435\u00a0<a href=\"https:\/\/habr.com\/users\/irbgeo\" rel=\"noopener noreferrer nofollow\">@irbgeo<\/a>\u00a0<a href=\"https:\/\/t.me\/irbgeo\" rel=\"noopener noreferrer nofollow\">Telegram<\/a>\u00a0: \u041f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0441\u0435\u0440\u0438\u044e \u0441\u0442\u0430\u0442\u0435\u0439 \u043e  K8S.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/58b\/103\/4eb\/58b1034eb64ffed83f0199bdcbc4e866.png\" width=\"1407\" height=\"783\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/58b\/103\/4eb\/58b1034eb64ffed83f0199bdcbc4e866.png\"\/><figcaption><\/figcaption><\/figure>\n<p>\u0426\u0435\u043b\u0438 \u0434\u0430\u043d\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438:<\/p>\n<ol>\n<li>\n<p>\u0410\u043a\u0442\u0443\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043f\u043e\u0440\u044f\u0434\u043e\u043a \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f kubernetes, \u043e\u043f\u0438\u0441\u0430\u043d\u043d\u044b\u0439 \u0432\u0441\u0435\u043c\u0438 \u043b\u044e\u0431\u0438\u043c\u044b\u043c <a href=\"https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way\" rel=\"noopener noreferrer nofollow\">Kelsey Hightower<\/a>.<\/p>\n<\/li>\n<li>\n<p>\u0414\u043e\u043a\u0430\u0437\u0430\u0442\u044c \u0447\u0442\u043e &#171;kubernetes \u044d\u0442\u043e \u0432\u0441\u0435\u0433\u043e 5-\u0431\u0438\u043d\u0430\u0440\u0435\u0439&#187; \u0438 &#171;kubernetes \u044d\u0442\u043e \u043f\u0440\u043e\u0441\u0442\u043e&#187; &#8212; \u044d\u0442\u043e \u043d\u0435\u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0435 \u0441\u0443\u0436\u0434\u0435\u043d\u0438\u0435.<\/p>\n<\/li>\n<li>\n<p>\u0414\u043e\u0431\u0430\u0432\u0438\u0442\u044c <a href=\"https:\/\/github.com\/fraima\/key-keeper\" rel=\"noopener noreferrer nofollow\">Key-keeper<\/a> \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e kubernetes \u0434\u043b\u044f \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430\u043c\u0438.<\/p>\n<\/li>\n<\/ol>\n<h2>\u0418\u0437 \u0447\u0435\u0433\u043e \u0441\u043e\u0441\u0442\u043e\u0438\u0442 Kubernetes?<\/h2>\n<p>\u0412\u0441\u0435 \u043c\u044b \u043f\u043e\u043c\u043d\u0438\u043c \u0448\u0443\u0442\u043a\u0443 &#171;kubernetes \u044d\u0442\u043e \u0432\u0441\u0435\u0433\u043e 5-\u0431\u0438\u043d\u0430\u0440\u0435\u0439&#187;:<\/p>\n<ol>\n<li>\n<p>etcd<\/p>\n<\/li>\n<li>\n<p>kube-apiserver<\/p>\n<\/li>\n<li>\n<p>kube-controller-manager<\/p>\n<\/li>\n<li>\n<p>kube-scheduler<\/p>\n<\/li>\n<li>\n<p>kubelet<\/p>\n<\/li>\n<\/ol>\n<p>\u041d\u043e, \u0435\u0441\u043b\u0438 \u043c\u044b \u0431\u0443\u0434\u0435\u043c \u043e\u043f\u0435\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0442\u043e\u043b\u044c\u043a\u043e \u0438\u043c\u0438, \u0442\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440 \u0432\u044b \u043d\u0435 \u0441\u043e\u0431\u0435\u0440\u0435\u0442\u0435. <strong>\u041f\u043e\u0447\u0435\u043c\u0443 \u0436\u0435?<\/strong><\/p>\n<p><strong><u>kubelet -\u0443 \u0442\u0440\u0435\u0431\u0443\u044e\u0442\u0441\u044f \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0435 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u044b \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>Container Runtime Interface &#8212; CRI (containerd, cri-o, docker, etc.). <\/p>\n<\/li>\n<\/ol>\n<p><strong><u>CRI &#8212; \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>RUNC \u0431\u0438\u0431\u043b\u0438\u043e\u0442\u0435\u043a\u0430, \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430\u043c\u0438. <\/p>\n<\/li>\n<\/ol>\n<p><strong><u>Certificates:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>(cfssl, kubeadm, key-keeper) \u0442\u0440\u0435\u0431\u0443\u044e\u0442\u0441\u044f \u0434\u043b\u044f \u0432\u044b\u043f\u0443\u0441\u043a\u0430 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432.<\/p>\n<\/li>\n<\/ol>\n<p><strong><u>\u041f\u0440\u043e\u0447\u0435\u0435:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>kubectl (\u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 kubernetes)             &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<li>\n<p>crictl (\u0434\u043b\u044f \u0443\u0434\u043e\u0431\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b \u0441 CRI)              &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<li>\n<p>etcdctl (\u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 etcd \u043d\u0430 \u043c\u0430\u0441\u0442\u0435\u0440\u0430\u0445)   &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<li>\n<p>kubeadm (\u0434\u043b\u044f \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430)          &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<\/ol>\n<p><strong><u>\u0422\u0430\u043a\u0438\u043c \u043e\u0431\u0440\u0430\u0437\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442\u044c kubernetes, \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u043c\u0438\u043d\u0438\u043c\u0443\u043c 8 \u0431\u0438\u043d\u0430\u0440\u0435\u0439.<\/u><\/strong><\/p>\n<h2>\u042d\u0442\u0430\u043f\u044b \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 K8S<\/h2>\n<ol>\n<li>\n<p>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 linux \u043c\u0430\u0448\u0438\u043d, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u0431\u0443\u0434\u0435\u0442 \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442 control-plane \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<\/li>\n<li>\n<p>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043d\u0430 \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0445 linux \u043c\u0430\u0448\u0438\u043d\u0430\u0445:<\/p>\n<ol>\n<li>\n<p>\u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0431\u0430\u0437\u043e\u0432\u044b\u0445 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 (\u0434\u043b\u044f \u043e\u0431\u0441\u043b\u0443\u0436\u0438\u0432\u0430\u043d\u0438\u044f linux).<\/p>\n<\/li>\n<li>\n<p>\u0440\u0430\u0431\u043e\u0442\u0430 \u0441 modprobe.<\/p>\n<\/li>\n<li>\n<p>\u0440\u0430\u0431\u043e\u0442\u0430 \u0441 sysctls.<\/p>\n<\/li>\n<li>\n<p>\u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u044b\u0445 \u0434\u043b\u044f \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0431\u0438\u043d\u0430\u0440\u0435\u0439.<\/p>\n<\/li>\n<li>\n<p>\u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 \u0434\u043b\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0445 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432.<\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 Vault \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0430.<\/p>\n<\/li>\n<li>\n<p>\u0413\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u044f static-pod \u043c\u0430\u043d\u0438\u0444\u0435\u0441\u0442\u043e\u0432.<\/p>\n<\/li>\n<li>\n<p>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<\/li>\n<\/ol>\n<p>\u041a\u0430\u043a \u0432\u0438\u0434\u0438\u043c, \u0432\u0441\u0435\u0433\u043e 5-\u0442\u044c \u044d\u0442\u0430\u043f\u043e\u0432 &#8212; \u043d\u0438\u0447\u0435\u0433\u043e \u0441\u043b\u043e\u0436\u043d\u043e\u0433\u043e)<\/p>\n<h2>\u041d\u0443 \u0447\u0442\u043e, \u0434\u0430\u0432\u0430\u0439\u0442\u0435 \u043d\u0430\u0447\u043d\u0435\u043c!<\/h2>\n<p>1) <strong>\u0421\u043e\u0437\u0434\u0430\u0435\u043c 3 \u0423\u0437\u043b\u0430 \u043f\u043e\u0434 \u043c\u0430\u0441\u0442\u0435\u0440\u0430 \u0438 \u043f\u0440\u0438\u0432\u044f\u0437\u044b\u0432\u0430\u0435\u043c \u043a \u043d\u0438\u043c DNS \u0438\u043c\u0435\u043d\u0430 \u043f\u043e \u043c\u0430\u0441\u043a\u0435:<\/strong><\/p>\n<p><code>master-${INDEX}.${CLUSTER_NAME}.${BASE_DOMAIN}<\/code><\/p>\n<p>** <strong>\u0412\u0410\u0416\u041d\u041e<\/strong>: <code>${INDEX}<\/code> \u0434\u043e\u043b\u0436\u0435\u043d \u043d\u0430\u0447\u0438\u043d\u0430\u0442\u044c\u0441\u044f \u0441 <strong>0<\/strong> \u0438\u0437-\u0437\u0430 \u0440\u0435\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438 \u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0438\u043d\u0434\u0435\u043a\u0441\u043e\u0432 \u0432 \u043c\u043e\u0434\u0443\u043b\u0435 \u0442\u0435\u0440\u0440\u0430\u0444\u043e\u0440\u043c \u0434\u043b\u044f VAULT, \u043d\u043e \u043e \u043d\u0435\u043c \u043f\u043e\u0437\u0436\u0435.<\/p>\n<details class=\"spoiler\">\n<summary>environments<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## REQUIRED VARS:  export BASE_DOMAIN=dobry-kot.ru export CLUSTER_NAME=example export BASE_CLUSTER_DOMAIN=${CLUSTER_NAME}.${BASE_DOMAIN}  # \u041f\u043e\u0440\u0442\u044b \u0434\u043b\u044f ETCD export ETCD_SERVER_PORT=\"2379\" export ETCD_PEER_PORT=\"2380\" export ETCD_METRICS_PORT=\"2381\"  # \u041f\u043e\u0440\u0442\u044b \u0434\u043b\u044f KUBERNETES export KUBE_APISERVER_PORT=\"6443\" export KUBE_CONTROLLER_MANAGER_PORT=\"10257\" export KUBE_SCHEDULER_PORT=\"10259\"  # \u0423\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0435 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 1, 3, 5 export MASTER_COUNT=1  # \u0414\u043b\u044f Kube-apiserver  export ETCD_SERVERS=$(echo \\ $(for INDEX in `seq 0 $(($MASTER_COUNT-1))`; \\ do \\ echo https:\/\/master-${INDEX}.${BASE_CLUSTER_DOMAIN}:${ETCD_SERVER_PORT} ; \\ done) |  sed \"s\/,\/\/\" |  sed \"s\/ \/,\/g\")  # \u0414\u043b\u044f \u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f ETCD \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 export ETCD_INITIAL_CLUSTER=$(echo \\ $(for INDEX in `seq 0 $(($MASTER_COUNT-1))`; \\ do \\ echo master-${INDEX}.${BASE_CLUSTER_DOMAIN}=https:\/\/master-${INDEX}.${BASE_CLUSTER_DOMAIN}:${ETCD_PEER_PORT} ; \\ done) |  sed \"s\/,\/\/\" |  sed \"s\/ \/,\/g\")   export KUBERNETES_VERSION=\"v1.23.12\" export ETCD_VERSION=\"3.5.3-0\" export ETCD_TOOL_VERSION=\"v3.5.5\" export RUNC_VERSION=\"v1.1.3\" export CONTAINERD_VERSION=\"1.6.8\" export CRICTL_VERSION=$(echo $KUBERNETES_VERSION |  sed -r 's\/^v([0-9]*).([0-9]*).([0-9]*)\/v\\1.\\2.0\/')  export BASE_K8S_PATH=\"\/etc\/kubernetes\"  export SERVICE_CIDR=\"29.64.0.0\/16\" # \u041d\u0435 \u043e\u0431\u0438\u0436\u0430\u0439\u0442\u0435\u0441\u044c - regexp \u0441\u0430\u043c\u0438 \u043d\u0430\u043f\u0438\u0448\u0438\u0442\u0435) export SERVICE_DNS=\"29.64.0.10\"  export VAULT_MASTER_TOKEN=\"hvs.vy0dqWuHkJpiwtYhw4yPT6cC\" export VAULT_SERVER=\"http:\/\/193.32.219.99:9200\/\"  export VAULT_MASTER_TOKEN=\"root\" export VAULT_SERVER=\"http:\/\/master-0.${CLUSTER_NAME}.${BASE_DOMAIN}:9200\/\"<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0415\u0441\u043b\u0438 \u0432\u044b \u0438\u0437\u0443\u0430\u0447\u0430\u043b\u0438 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044e \u043e\u0442 <a href=\"https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way\" rel=\"noopener noreferrer nofollow\">Kelsey Hightower<\/a>, \u0442\u043e \u0437\u0430\u043c\u0435\u0447\u0430\u043b\u0438, \u0447\u0442\u043e \u0432 \u043e\u0441\u043d\u043e\u0432\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 \u043b\u0435\u0436\u0430\u0442 ip \u0430\u0434\u0440\u0435\u0441\u0430 \u0443\u0437\u043b\u043e\u0432. \u0414\u0430\u043d\u043d\u044b\u0439 \u043f\u043e\u0434\u0445\u043e\u0434 \u0440\u0430\u0431\u043e\u0447\u0438\u0439, \u043d\u043e \u043c\u0435\u043d\u0435\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u044b\u0439, \u0434\u043b\u044f \u043f\u0440\u043e\u0441\u0442\u043e\u0442\u044b \u043e\u0431\u0441\u043b\u0443\u0436\u0438\u0432\u0430\u043d\u0438\u044f \u0438 \u0434\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0435\u0439 \u0448\u0430\u0431\u043b\u043e\u043d\u0438\u0437\u0430\u0446\u0438\u0438 \u043b\u0443\u0447\u0448\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0437\u0430\u0440\u0430\u043d\u0435\u0435 \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u044b\u0435 \u043d\u0430\u043c FQDN \u043c\u0430\u0441\u043a\u0438, \u043a\u0430\u043a \u044f \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u043b \u0434\u043b\u044f \u043c\u0430\u0441\u0442\u0435\u0440\u043e\u0432 \u0432\u044b\u0448\u0435.<\/p>\n<p>2) <strong>\u0421\u043a\u0430\u0447\u0438\u0432\u0430\u0435\u043c \u0432\u0441\u0435 \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u044b\u0435 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c K8S \u0431\u0438\u043d\u0430\u0440\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b.<\/strong><\/p>\n<ul>\n<li>\n<p>\u0412 \u0434\u0430\u043d\u043d\u043e\u043c \u0441\u0435\u0442\u0430\u043f\u0435 \u044f \u043d\u0435 \u0431\u0443\u0434\u0443 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c RPM \u0438\u043b\u0438 DEB \u043f\u0430\u043a\u0435\u0442\u044b, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0441\u0442\u0430\u0440\u0430\u0442\u044c\u0441\u044f \u0434\u0435\u0442\u0430\u043b\u044c\u043d\u043e \u043f\u043e\u043a\u0430\u0437\u0430\u0442\u044c, \u0438\u0437 \u0447\u0435\u0433\u043e \u0441\u043e\u0441\u0442\u043e\u0438\u0442 \u0432\u0441\u044f \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u044f.<\/p>\n<\/li>\n<\/ul>\n<details class=\"spoiler\">\n<summary>download components<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. wget -O \/usr\/bin\/key-keeper   \"https:\/\/storage.yandexcloud.net\/m.images\/key-keeper-T2?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=YCAJEhOlYpv1GRY7hghCojNX5%2F20221020%2Fru-central1%2Fs3%2Faws4_request&amp;X-Amz-Date=20221020T123413Z&amp;X-Amz-Expires=2592000&amp;X-Amz-Signature=138701723B70343E38D82791A28AD1DB87040677F7C94D83610FF26ED9AF1954&amp;X-Amz-SignedHeaders=host\" wget -O \/usr\/bin\/kubectl       https:\/\/storage.googleapis.com\/kubernetes-release\/release\/${KUBERNETES_VERSION}\/bin\/linux\/amd64\/kubectl wget -O \/usr\/bin\/kubelet       https:\/\/storage.googleapis.com\/kubernetes-release\/release\/${KUBERNETES_VERSION}\/bin\/linux\/amd64\/kubelet wget -O \/usr\/bin\/kubeadm       https:\/\/storage.googleapis.com\/kubernetes-release\/release\/${KUBERNETES_VERSION}\/bin\/linux\/amd64\/kubeadm wget -O \/usr\/bin\/runc          https:\/\/github.com\/opencontainers\/runc\/releases\/download\/${RUNC_VERSION}\/runc.amd64 wget -O \/tmp\/etcd.tar.gz       https:\/\/github.com\/etcd-io\/etcd\/releases\/download\/${ETCD_TOOL_VERSION}\/etcd-${ETCD_TOOL_VERSION}-linux-amd64.tar.gz wget -O \/tmp\/containerd.tar.gz https:\/\/github.com\/containerd\/containerd\/releases\/download\/v${CONTAINERD_VERSION}\/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz wget -O \/tmp\/crictl.tar.gz     https:\/\/github.com\/kubernetes-sigs\/cri-tools\/releases\/download\/${CRICTL_VERSION}\/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz  chmod +x \/usr\/bin\/key-keeper  chmod +x \/usr\/bin\/kubelet  chmod +x \/usr\/bin\/kubectl  chmod +x \/usr\/bin\/kubeadm chmod +x \/usr\/bin\/runc  mkdir -p \/tmp\/containerd mkdir -p \/tmp\/etcd  tar -C \"\/tmp\/etcd\"        -xvf \/tmp\/etcd.tar.gz tar -C \"\/tmp\/containerd\"  -xvf \/tmp\/containerd.tar.gz tar -C \"\/usr\/bin\"         -xvf \/tmp\/crictl.tar.gz  cp \/tmp\/etcd\/etcd*\/etcdctl \/usr\/bin\/ cp \/tmp\/containerd\/bin\/*   \/usr\/bin\/<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>3) <strong>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432:<\/strong><\/p>\n<p>\u0421\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0432 \u043d\u0430\u0448\u0435\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u0432\u0441\u0435\u0433\u043e 3 (key-keeper, kubelet, containerd)<\/p>\n<details class=\"spoiler\">\n<summary>containerd.service<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE FOR CONTAINERD  cat &lt;&lt;EOF > \/etc\/systemd\/system\/containerd.service [Unit] Description=containerd container runtime Documentation=https:\/\/containerd.io After=network.target  [Service] ExecStartPre=\/sbin\/modprobe overlay ExecStart=\/usr\/bin\/containerd Restart=always RestartSec=5 Delegate=yes KillMode=process OOMScoreAdjust=-999 LimitNOFILE=1048576 LimitNPROC=infinity LimitCORE=infinity  [Install] WantedBy=multi-user.target EOF <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>key-keeper.service<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE FOR KEY-KEEPER cat &lt;&lt;EOF > \/etc\/systemd\/system\/key-keeper.service [Unit] Description=key-keeper-agent  Wants=network-online.target After=network-online.target  [Service] ExecStart=\/usr\/bin\/key-keeper -config-dir ${BASE_K8S_PATH}\/pki -config-regexp .*vault-config   Restart=always StartLimitInterval=0 RestartSec=10  [Install] WantedBy=multi-user.target EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet.service<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE FOR KUBELET cat &lt;&lt;EOF > \/etc\/systemd\/system\/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=https:\/\/kubernetes.io\/docs\/home\/ Wants=network-online.target After=network-online.target   [Service] ExecStart=\/usr\/bin\/kubelet  Restart=always StartLimitInterval=0 RestartSec=10  [Install] WantedBy=multi-user.target EOF <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet.d\/conf<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE-CONFIG FOR KUBELET  mkdir -p \/etc\/systemd\/system\/kubelet.service.d  cat &lt;&lt;EOF > \/etc\/systemd\/system\/kubelet.service.d\/10-fraima.conf [Service] EnvironmentFile=-${BASE_K8S_PATH}\/kubelet\/service\/kubelet-args.env  ExecStart= ExecStart=\/usr\/bin\/kubelet \\ \\$KUBELET_HOSTNAME \\ \\$KUBELET_CNI_ARGS \\ \\$KUBELET_RUNTIME_ARGS \\ \\$KUBELET_AUTH_ARGS \\ \\$KUBELET_CONFIGS_ARGS \\ \\$KUBELET_BASIC_ARGS \\ \\$KUBELET_KUBECONFIG_ARGS EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet-args.env<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE-CONFIG FOR KUBELET  mkdir -p  ${BASE_K8S_PATH}\/kubelet\/service\/  cat &lt;&lt;EOF > ${BASE_K8S_PATH}\/kubelet\/service\/kubelet-args.env KUBELET_HOSTNAME=\"\" KUBELET_BASIC_ARGS=\"     --register-node=true     --cloud-provider=external     --image-pull-progress-deadline=2m     --feature-gates=RotateKubeletServerCertificate=true     --cert-dir=\/etc\/kubernetes\/pki\/certs\/kubelet     --authorization-mode=Webhook     --v=2 \" KUBELET_AUTH_ARGS=\"     --anonymous-auth=\"false\" \" KUBELET_CNI_ARGS=\"     --cni-bin-dir=\/opt\/cni\/bin     --cni-conf-dir=\/etc\/cni\/net.d     --network-plugin=cni \" KUBELET_CONFIGS_ARGS=\"     --config=${BASE_K8S_PATH}\/kubelet\/config.yaml     --root-dir=\/var\/lib\/kubelet     --register-node=true     --image-pull-progress-deadline=2m     --v=2 \" KUBELET_KUBECONFIG_ARGS=\"     --kubeconfig=${BASE_K8S_PATH}\/kubelet\/kubeconfig \" KUBELET_RUNTIME_ARGS=\"     --container-runtime=remote     --container-runtime-endpoint=\/run\/containerd\/containerd.sock     --pod-infra-container-image=k8s.gcr.io\/pause:3.6 \" EOF<\/code><\/pre>\n<p>** \u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0435\u0441\u043b\u0438 \u0432\u044b \u0432 \u043f\u0435\u0440\u0441\u043f\u0435\u043a\u0442\u0438\u0432\u0435 \u0431\u0443\u0434\u0435\u0442\u0435 \u0440\u0430\u0437\u0432\u043e\u0440\u0430\u0447\u0438\u0432\u0430\u0442\u044c K8S \u0432 \u043e\u0431\u043b\u0430\u043a\u0435 \u0438 \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0435\u0433\u043e \u0441 \u043d\u0438\u043c, \u0442\u043e \u0441\u0442\u0430\u0432\u044c\u0442\u0435  &#8212;cloud-provider=external<\/p>\n<p>*** \u041f\u043e\u043b\u0435\u0437\u043d\u043e\u0439 \u0444\u0438\u0447\u0435\u0439 \u043c\u043e\u0436\u0435\u0442 \u043e\u043a\u0430\u0437\u0430\u0442\u044c\u0441\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043b\u0435\u0439\u0431\u043b\u0438\u043d\u0433 \u043d\u043e\u0434\u044b \u043f\u0440\u0438 \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0430\u0446\u0438\u0438 \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435<br \/><code>--node-labels=node.kubernetes.io\/master,foo=bar<br \/><\/code><br \/><strong><u>\u041d\u0438\u0436\u0435 \u043f\u0440\u0438\u0432\u0435\u0434\u0435\u043d \u0441\u043f\u0438\u0441\u043e\u043a \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u0445 \u043c\u0435\u0442\u043e\u043a, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u043e\u0436\u043d\u043e \u043c\u0435\u043d\u044f\u0442\u044c:<\/u><\/strong><br \/><a href=\"http:\/\/kubelet.kubernetes.io\" rel=\"noopener noreferrer nofollow\">kubelet.kubernetes.io<\/a><br \/><a href=\"http:\/\/node.kubernetes.io\" rel=\"noopener noreferrer nofollow\">node.kubernetes.io<\/a><br \/><a href=\"http:\/\/beta.kubernetes.io\/arch\" rel=\"noopener noreferrer nofollow\">beta.kubernetes.io\/arch<\/a>,<br \/><a href=\"http:\/\/beta.kubernetes.io\/instance-type\" rel=\"noopener noreferrer nofollow\">beta.kubernetes.io\/instance-type<\/a>,<br \/><a href=\"http:\/\/beta.kubernetes.io\/os\" rel=\"noopener noreferrer nofollow\">beta.kubernetes.io\/os<\/a>,<br \/><a href=\"http:\/\/failure-domain.beta.kubernetes.io\/region\" rel=\"noopener noreferrer nofollow\">failure-domain.beta.kubernetes.io\/region<\/a>,<br \/><a href=\"http:\/\/failure-domain.beta.kubernetes.io\/zone\" rel=\"noopener noreferrer nofollow\">failure-domain.beta.kubernetes.io\/zone<\/a>,<br \/><a href=\"http:\/\/kubernetes.io\/arch\" rel=\"noopener noreferrer nofollow\">kubernetes.io\/arch<\/a>,<br \/><a href=\"http:\/\/kubernetes.io\/hostname\" rel=\"noopener noreferrer nofollow\">kubernetes.io\/hostname<\/a>,<br \/><a href=\"http:\/\/kubernetes.io\/os\" rel=\"noopener noreferrer nofollow\">kubernetes.io\/os<\/a>,<br \/><a href=\"http:\/\/node.kubernetes.io\/instance-type\" rel=\"noopener noreferrer nofollow\">node.kubernetes.io\/instance-type<\/a>,<br \/><a href=\"http:\/\/topology.kubernetes.io\/region\" rel=\"noopener noreferrer nofollow\">topology.kubernetes.io\/region<\/a>,<br \/><a href=\"http:\/\/topology.kubernetes.io\/zone\" rel=\"noopener noreferrer nofollow\">topology.kubernetes.io\/zone<\/a><\/p>\n<p><strong><u>\u0414\u043b\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u0430, \u043d\u0435\u043b\u044c\u0437\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u0435 \u043b\u0435\u0439\u0431\u043b \u043d\u0435 \u0438\u0437 \u0441\u043f\u0438\u0441\u043a\u0430:<br \/><\/u><\/strong><code>--node-labels=node-role.kubernetes.io\/master<\/code><\/p>\n<\/div>\n<\/details>\n<p>4) <strong>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 Vault.<\/strong><\/p>\n<p>\u041a\u0430\u043a \u043c\u044b \u0440\u0430\u043d\u0435\u0435 \u043f\u0438\u0441\u0430\u043b\u0438, \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b \u0431\u0443\u0434\u0435\u043c \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0447\u0435\u0440\u0435\u0437 \u0446\u0435\u043d\u0442\u0440\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d\u043d\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u0449\u0435 Vault.<\/p>\n<p>\u0414\u043b\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u0430 \u043c\u044b \u0440\u0430\u0437\u043c\u0435\u0441\u0442\u0438\u043c \u043e\u043f\u043e\u0440\u043d\u044b\u0439 <strong>Vault server<\/strong> \u043d\u0430 <code>master-0<\/code> \u0432 \u0440\u0435\u0436\u0438\u043c\u0435 <code>dev<\/code> \u0441 \u0443\u0436\u0435 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u043c \u0441\u0442\u043e\u0440\u0435\u0434\u0436\u043e\u043c \u0438 \u0434\u0435\u0444\u043e\u043b\u0442\u043d\u044b\u043c \u0442\u043e\u043a\u0435\u043d\u043e\u043c \u0434\u043b\u044f \u0443\u0434\u043e\u0431\u0441\u0442\u0432\u0430.<\/p>\n<details class=\"spoiler\">\n<summary>Vault<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON MASTER-0. export VAULT_VERSION=\"1.12.1\" export VAULT_ADDR=${VAULT_SERVER} export VAULT_TOKEN=${VAULT_MASTER_TOKEN}  wget -O \/tmp\/vault_${VAULT_VERSION}_linux_amd64.zip https:\/\/releases.hashicorp.com\/vault\/${VAULT_VERSION}\/vault_${VAULT_VERSION}_linux_amd64.zip unzip \/tmp\/vault_${VAULT_VERSION}_linux_amd64.zip -d \/usr\/bin <\/code><\/pre>\n<pre><code class=\"bash\">## RUN ON MASTER-0. cat &lt;&lt;EOF > \/etc\/systemd\/system\/vault.service [Unit] Description=Vault secret management tool After=consul.service   [Service] PermissionsStartOnly=true ExecStart=\/usr\/bin\/vault server -log-level=debug -dev -dev-root-token-id=\"${VAULT_MASTER_TOKEN}\" -dev-listen-address=0.0.0.0:9200 Restart=on-failure LimitMEMLOCK=infinity  [Install] WantedBy=multi-user.target EOF<\/code><\/pre>\n<pre><code class=\"bash\">## RUN ON MASTER-0. #enable Vault PKI secret engine  vault secrets enable -path=pki-root pki  #set default ttl vault secrets tune -max-lease-ttl=87600h pki-root  #generate root CA vault write -format=json pki-root\/root\/generate\/internal \\ common_name=\"ROOT PKI\" ttl=8760h<\/code><\/pre>\n<p>*\u041f\u0440\u043e\u0448\u0443 \u043e\u0431\u0440\u0430\u0442\u0438\u0442\u044c \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0435\u0441\u043b\u0438 \u0432\u044b \u043d\u0430\u0445\u043e\u0434\u0438\u0442\u0435\u0441\u044c \u043d\u0430 \u0442\u0435\u0440\u0440\u0438\u0442\u043e\u0440\u0438\u0438 \u0420\u043e\u0441\u0441\u0438\u0438, \u0443 \u0432\u0430\u0441 \u0431\u0443\u0434\u0443\u0442 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b \u0441 \u0434\u043e\u0441\u0442\u0443\u043f\u043e\u043c \u0434\u043b\u044f \u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044e Vault \u0438 Terrraform.<\/p>\n<p>** pki-root\/root\/generate\/<strong>internal<\/strong> &#8212; \u0423\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442, \u0447\u0442\u043e \u0441\u0444\u043e\u0440\u043c\u0438\u0440\u0443\u0435\u0442\u0441\u044f CA, \u0438 \u0432 response \u043f\u0440\u0438\u043b\u0435\u0442\u0438\u0442 \u0442\u043e\u043b\u044c\u043a\u043e \u043f\u0443\u0431\u043b\u0438\u0447\u043d\u044b\u0439 \u043a\u043b\u044e\u0447, \u043f\u0440\u0438\u0432\u0430\u0442\u043d\u044b\u0439 \u0431\u0443\u0434\u0435\u0442 \u0437\u0430\u043a\u0440\u044b\u0442.<\/p>\n<p>*** pki-root &#8212; \u0431\u0430\u0437\u043e\u0432\u043e\u0435 \u043d\u0430\u0438\u043c\u0435\u043d\u043e\u0432\u0430\u043d\u0438\u0435 \u0441\u0435\u0439\u0444\u0430 \u0434\u043b\u044f Root-CA, \u0441\u043c\u0435\u043d\u0430 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0441\u044f \u0447\u0435\u0440\u0435\u0437 \u043a\u0430\u0441\u0442\u043e\u043c\u0438\u0437\u0430\u0446\u0438\u044e terraform \u043c\u043e\u0434\u0443\u043b\u044f, \u043e \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u0431\u0443\u0434\u0435\u043c \u0433\u043e\u0432\u043e\u0440\u0438\u0442\u044c \u043d\u0438\u0436\u0435.<\/p>\n<p>**** \u0414\u0430\u043d\u043d\u0430\u044f \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u044f vault \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442\u0430 \u043a\u0430\u043a \u043e\u0431\u0437\u043e\u0440\u043d\u0430\u044f \u0438 \u043d\u0435 \u043c\u043e\u0436\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0434\u043b\u044f \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u0438\u0432\u043d\u043e\u0439 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0438.<\/p>\n<\/p>\n<\/div>\n<\/details>\n<p>\u041e\u0442\u043b\u0438\u0447\u043d\u043e, Vault \u043c\u044b \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u043b\u0438, \u0442\u0435\u043f\u0435\u0440\u044c \u043d\u0443\u0436\u043d\u043e \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u0438\u0442\u044c \u0440\u043e\u043b\u0438, \u043f\u043e\u043b\u0438\u0442\u0438\u043a\u0438 \u0438 \u0434\u043e\u0441\u0442\u0443\u043f\u044b \u0432 \u043d\u0435\u043c \u0434\u043b\u044f <strong>key-keeper.<\/strong><\/p>\n<p>\u0414\u043b\u044f \u044d\u0442\u043e\u0433\u043e \u0432\u043e\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c\u0441\u044f \u043d\u0430\u0448\u0438\u043c \u043c\u043e\u0434\u0443\u043b\u0435\u043c \u0434\u043b\u044f <strong>Terraform<\/strong>.<\/p>\n<details class=\"spoiler\">\n<summary>Terraform<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON MASTER-0. export TERRAFORM_VERSION=\"1.3.4\"  wget -O \/tmp\/terraform_${TERRAFORM_VERSION}_linux_amd64.zip https:\/\/releases.hashicorp.com\/terraform\/${TERRAFORM_VERSION}\/terraform_${TERRAFORM_VERSION}_linux_amd64.zip  unzip \/tmp\/terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d \/usr\/bin<\/code><\/pre>\n<pre><code class=\"bash\">## RUN ON MASTER-0. mkdir terraform  cat &lt;&lt;EOF > terraform\/main.tf terraform {   required_version = \">= 0.13\"  }  provider \"vault\" {          address = \"http:\/\/127.0.0.1:9200\/\"     token = \"${VAULT_MASTER_TOKEN}\" }   variable \"master-instance-count\" {   type = number   default = 1 }  variable \"base_domain\" {   type = string   default = \"${BASE_DOMAIN}\" }  variable \"cluster_name\" {   type = string   default = \"${CLUSTER_NAME}\" }  variable \"vault_server\" {   type = string   default = \"http:\/\/master-0.${BASE_CLUSTER_DOMAIN}:9200\/\" }  # \u0414\u0430\u043d\u043d\u044b\u0439 \u043c\u043e\u0434\u0443\u043b\u044c \u0433\u0435\u043d\u0435\u0440\u0438\u0442 \u0432\u0435\u0441\u044c \u043d\u0430\u0431\u043e\u0440 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445,  # \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u0432 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0445 \u0441\u0442\u0430\u0442\u044c\u044f\u0445 \u0438 \u043c\u043e\u0434\u0443\u043b\u044f\u0445. module \"k8s-global-vars\" {     source = \"git::https:\/\/github.com\/fraima\/kubernetes.git\/\/modules\/k8s-config-vars\"     cluster_name          = var.cluster_name     base_domain           = var.base_domain     master_instance_count = var.master-instance-count     vault_server          = var.vault_server }  # \u0422\u0443\u0442 \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442 \u0432\u0441\u044f \u043c\u0430\u0433\u0438\u044f \u0441 Vault. module \"k8s-vault\" {     source = \"git::https:\/\/github.com\/fraima\/kubernetes.git\/\/modules\/k8s-vault\"     k8s_global_vars   = module.k8s-global-vars } EOF<\/code><\/pre>\n<pre><code class=\"bash\">cd terraform  terraform init --upgrade terraform plan terraform apply<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0412 \u0431\u0430\u0437\u043e\u0432\u044b\u0439 \u043d\u0430\u0431\u043e\u0440 Vault \u043a\u043e\u043d\u0442\u0435\u043d\u0442\u0430 \u0431\u043e\u0435\u0432\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0432\u0445\u043e\u0434\u0438\u0442:<\/p>\n<ol>\n<li>\n<p>\u0421\u0435\u0439\u0444\u044b \u043f\u043e\u0434 etcd, kubernetes, frotend-proxy. (* \u0421\u0435\u0439\u0444\u044b \u0434\u043b\u044f PKI \u0441\u043e\u0437\u0434\u0430\u044e\u0442\u0441\u044f \u043f\u043e \u043c\u0430\u0441\u043a\u0430\u043c):<\/p>\n<ol>\n<li>\n<p><code>clusters\/${CLUSTER_NAME}\/pki\/etcd<\/code><\/p>\n<\/li>\n<li>\n<p><code>clusters\/${CLUSTER_NAME}\/pki\/kubernetes-ca<\/code><\/p>\n<\/li>\n<li>\n<p><code>clusters\/${CLUSTER_NAME}\/pki\/front-proxy<\/code><\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>\u0421\u0435\u0439\u0444 Key Value \u0434\u043b\u044f \u0441\u0435\u043a\u0440\u0435\u0442\u043e\u0432<\/p>\n<ol>\n<li>\n<p><code>clusters\/${CLUSTER_NAME}\/kv\/<\/code><\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>\u0420\u043e\u043b\u0438 \u0434\u043b\u044f \u0437\u0430\u043a\u0430\u0437\u0430 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432 (\u043b\u0438\u043d\u043a\u0438 \u0432\u0435\u0434\u0443\u0442 \u043d\u0430 \u043e\u043f\u0438\u0441\u0430\u043d\u0438\u0435 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430)<\/p>\n<ol>\n<li>\n<p>ETCD:<\/p>\n<ol>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L743\" rel=\"noopener noreferrer nofollow\"><strong>etcd-client<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L636\" rel=\"noopener noreferrer nofollow\"><s>etcd-server<\/s><\/a><s>  (\u0432 \u0434\u0430\u043d\u043d\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u043d\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f)<\/s><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L652\" rel=\"noopener noreferrer nofollow\"><strong>etcd-peer<\/strong><\/a><\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>Kubernetes-ca:<\/p>\n<ol>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L111\" rel=\"noopener noreferrer nofollow\"><s>bootstrappers-client<\/s><\/a><s> (\u0432 \u0434\u0430\u043d\u043d\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u043d\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f)<\/s><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L148\" rel=\"noopener noreferrer nofollow\"><strong>kube-controller-manager-client<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L179\" rel=\"noopener noreferrer nofollow\"><strong>kube-controller-manager-server<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L234\" rel=\"noopener noreferrer nofollow\"><strong>kube-apiserver-kubelet-client<\/strong><\/a><strong> **<\/strong><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L268\" rel=\"noopener noreferrer nofollow\"><strong>kubeadm-client<\/strong><\/a> (\u0432 \u0434\u0430\u043d\u043d\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f \u043a\u0430\u043a cluster-admin)<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L306\" rel=\"noopener noreferrer nofollow\"><strong><s>kube-apiserver-cluster-admin-client<\/s><\/strong><\/a><strong><s> *** <\/s><\/strong><s> (\u0432 \u0434\u0430\u043d\u043d\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u043d\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f)<\/s><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L344\" rel=\"noopener noreferrer nofollow\"><strong>kube-apiserver<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L410\" rel=\"noopener noreferrer nofollow\"><strong>kube-scheduler-server<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L464\" rel=\"noopener noreferrer nofollow\"><strong>kube-scheduler-client<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L497\" rel=\"noopener noreferrer nofollow\"><s>kubelet-peer-k8s-certmanager <\/s><\/a><s>(\u0412 \u0434\u0430\u043d\u043d\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u043d\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0435\u0442\u0441\u044f)<\/s><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L519\" rel=\"noopener noreferrer nofollow\"><strong>kubelet-server<\/strong><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L575\" rel=\"noopener noreferrer nofollow\"><strong>kubelet-client<\/strong><\/a><\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>Front-proxy:<\/p>\n<ol>\n<li>\n<p><a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L803\" rel=\"noopener noreferrer nofollow\"><strong>front-proxy-client<\/strong><\/a><\/p>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>\u041f\u043e\u043b\u0438\u0442\u0438\u043a\u0438 \u0434\u043e\u0441\u0442\u0443\u043f\u0430 \u043a \u0440\u043e\u043b\u044f\u043c \u0438\u0437 \u041f.2<\/p>\n<\/li>\n<li>\n<p>\u0410\u043f\u043f\u0440\u043e\u043b\u0438 \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0443\u043f\u0430 \u043a\u043b\u0438\u0435\u043d\u0442\u043e\u0432.<\/p>\n<ol>\n<li>\n<p>\u041f\u0443\u0442\u044c \u0434\u043e Approle \u0444\u043e\u0440\u043c\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u043f\u043e \u043c\u0430\u0441\u043a\u0435 &#8212; <code>clusters\/${CLUSTER_NAME}\/approle<\/code><\/p>\n<\/li>\n<li>\n<p>\u0418\u043c\u044f Approle \u0444\u043e\u0440\u043c\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u043f\u043e \u043c\u0430\u0441\u043a\u0435 &#8212; <code>${CERT_ROLE}-${MASTER_NAME}<\/code><\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>\u0412\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0435 \u0442\u043e\u043a\u0435\u043d\u044b.<\/p>\n<\/li>\n<li>\n<p>\u041a\u043b\u044e\u0447\u0438 \u0448\u0438\u0444\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0434\u043b\u044f \u043f\u043e\u0434\u043f\u0438\u0441\u0438 jwt \u0442\u043e\u043a\u0435\u043d\u043e\u0432 \u043e\u0442 \u0441\u0435\u0440\u0432\u0438\u0441\u043d\u044b\u0445 \u0430\u043a\u043a\u0430\u0443\u043d\u0442\u043e\u0432.<\/p>\n<\/li>\n<\/ol>\n<p>** \u0421\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442 <a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L234\" rel=\"noopener noreferrer nofollow\"><strong>kube-apiserver-kubelet-client<\/strong><\/a><strong> <\/strong>\u0432\u043e \u0432\u0441\u0435\u0445 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u044f\u0445 \u043e\u0431\u044b\u0447\u043d\u043e \u0438\u043c\u0435\u0435\u0442 \u043f\u0440\u0438\u0432\u0438\u043b\u0435\u0433\u0438\u0438 <strong>cluster-admin<\/strong>, \u0432 \u0434\u0430\u043d\u043d\u043e\u0439 \u0436\u0435 \u0441\u0438\u0442\u0443\u0430\u0446\u0438\u0438, \u043f\u043e \u0434\u0435\u0444\u043e\u043b\u0442\u0443 \u043e\u043d \u043d\u0435 \u0438\u043c\u0435\u0435\u0442 \u043f\u0440\u0430\u0432 \u0438 \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f ClusterRolebinding \u0434\u043b\u044f \u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b \u0441 kubelet-\u0430\u043c\u0438 \u043d\u043e\u0434, \u043d\u043e \u043e\u0431 \u044d\u0442\u043e\u043c \u043f\u043e\u0437\u0436\u0435 (\u0441\u043c\u043e\u0442\u0440\u0438\u0442\u0435 \u0432 \u043a\u043e\u043d\u0446\u0435 \u0441\u0442\u0430\u0442\u044c\u0438 \u0432 \u0431\u043b\u043e\u043a\u0435 <strong>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430<\/strong>).<\/p>\n<p>*** <a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L268\" rel=\"noopener noreferrer nofollow\"><strong>kubeadm-client<\/strong><\/a> <strong> <\/strong>\u043f\u043e \u0434\u0435\u0444\u043e\u043b\u0442\u0443 \u0438\u043c\u0435\u0435\u0442 \u043f\u0440\u0430\u0432\u0430 cluster-admin. \u0412 \u044d\u0442\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u043e\u043d \u0431\u0443\u0434\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a\u0430\u043a \u043a\u043b\u0438\u0435\u043d\u0442 \u0434\u043e\u0441\u0442\u0443\u043f\u0430 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440\u0430 \u0434\u043b\u044f \u043f\u0435\u0440\u0432\u0438\u0447\u043d\u043e\u0439 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<p><strong>5) \u041f\u0440\u0438\u0441\u0442\u0443\u043f\u0430\u0435\u043c \u043a \u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 \u0434\u043b\u044f \u043d\u0430\u0448\u0438\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432.<\/strong><\/p>\n<p><strong>** <\/strong>\u041d\u0430\u043f\u043e\u043c\u0438\u043d\u0430\u044e, \u0447\u0442\u043e \u0438\u0445 \u0432\u0441\u0435\u0433\u043e 3  (<strong>key-keeper<\/strong>, <strong>kubelet<\/strong>, <strong>containerd<\/strong>).<br \/>*** <strong>containerd<\/strong> (\u0440\u0430\u0441\u0441\u043c\u0430\u0442\u0440\u0438\u0432\u0430\u0442\u044c \u043d\u0435 \u0431\u0443\u0434\u0435\u043c, \u0442.\u043a. \u043e\u043d \u0441\u0430\u043c \u0433\u0435\u043d\u0435\u0440\u0438\u0442 \u0431\u0430\u0437\u043e\u0432\u044b\u0439 \u043a\u043e\u043d\u0444\u0438\u0433 \u0438 \u0432 \u0431\u043e\u043b\u044c\u0448\u0438\u043d\u0441\u0442\u0432\u0435 \u0441\u043b\u0443\u0447\u0430\u0435\u0432 \u0435\u0433\u043e \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e)<\/p>\n<h3>\u041d\u0430\u0447\u043d\u0435\u043c \u0441 Key-keeper<\/h3>\n<p>\u0421\u043e \u0441\u043f\u0435\u0446\u0438\u0444\u0438\u043a\u043e\u0439 \u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u043a\u043e\u043d\u0444\u0438\u0433\u0430 \u043c\u043e\u0436\u043d\u043e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0432\u043e\u0442 \u0432 \u044d\u0442\u043e\u043c <a href=\"https:\/\/github.com\/fraima\/key-keeper\" rel=\"noopener noreferrer nofollow\"><strong>README<\/strong><\/a><strong>.<\/strong><\/p>\n<p>\u041a\u043e\u043d\u0444\u0438\u0433 \u043e\u0447\u0435\u043d\u044c \u0434\u043b\u0438\u043d\u043d\u044b\u0439 \u0442\u0430\u043a, \u0447\u0442\u043e \u043d\u0435 \u0443\u0434\u0438\u0432\u043b\u044f\u0439\u0442\u0435\u0441\u044c&#8230; .<\/p>\n<details class=\"spoiler\">\n<summary>key-keeper.issuers<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. # \u0414\u043b\u044f \u043a\u0430\u0436\u0434\u043e\u0439 \u043d\u043e\u0434\u044b \u0441\u0432\u043e\u0435 \u0438\u043c\u044f!!!! export MASTER_NAME=\"master-0\"<\/code><\/pre>\n<p>\u0412 \u043f\u0435\u0440\u0432\u043e\u0439 \u0447\u0430\u0441\u0442\u0438 \u043a\u043e\u043d\u0444\u0438\u0433\u0430 \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u043c \u0438\u043c\u044f \u043d\u043e\u0434\u044b, \u0432\u0441\u0435 \u043e\u0441\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0435 \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u043b\u0438 \u0432\u044b\u0448\u0435.<\/p>\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p ${BASE_K8S_PATH}\/pki\/  cat &lt;&lt;EOF > ${BASE_K8S_PATH}\/pki\/vault-config --- issuers:    - name: kube-apiserver-sa     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-apiserver-sa-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver-sa\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver-sa\/role-id       resource:         kv:           path: clusters\/${CLUSTER_NAME}\/kv       timeout: 15s    - name: etcd-ca     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: etcd-ca-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/etcd-ca\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/etcd-ca\/role-id       resource:         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/etcd\"         rootCAPath: \"clusters\/${CLUSTER_NAME}\/pki\/root\"       timeout: 15s    - name: etcd-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: etcd-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/etcd-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/etcd-client\/role-id       resource:         role: etcd-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/etcd\"       timeout: 15s    - name: etcd-peer     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: etcd-peer-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/etcd-peer\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/etcd-peer\/role-id       resource:         role: etcd-peer         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/etcd\"       timeout: 15s    - name: front-proxy-ca     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: front-proxy-ca-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/front-proxy-ca\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/front-proxy-ca\/role-id       resource:         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/front-proxy\"         rootCAPath: \"clusters\/${CLUSTER_NAME}\/pki\/root\"       timeout: 15s    - name: front-proxy-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: front-proxy-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/front-proxy-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/front-proxy-client\/role-id       resource:         role: front-proxy-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/front-proxy\"       timeout: 15s    - name: kubernetes-ca     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kubernetes-ca-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubernetes-ca\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubernetes-ca\/role-id       resource:         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"         rootCAPath: \"clusters\/${CLUSTER_NAME}\/pki\/root\"       timeout: 15s    - name: kube-apiserver     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-apiserver-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver\/role-id       resource:         role: kube-apiserver         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kube-apiserver-cluster-admin-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-apiserver-cluster-admin-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver-cluster-admin-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver-cluster-admin-client\/role-id       resource:         role: kube-apiserver-cluster-admin-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kube-apiserver-kubelet-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-apiserver-kubelet-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver-kubelet-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-apiserver-kubelet-client\/role-id       resource:         role: kube-apiserver-kubelet-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kube-controller-manager-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-controller-manager-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-controller-manager-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-controller-manager-client\/role-id       resource:         role: kube-controller-manager-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kube-controller-manager-server     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-controller-manager-server-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-controller-manager-server\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-controller-manager-server\/role-id       resource:         role: kube-controller-manager-server         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kube-scheduler-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-scheduler-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-scheduler-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-scheduler-client\/role-id       resource:         role: kube-scheduler-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kube-scheduler-server     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kube-scheduler-server-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-scheduler-server\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kube-scheduler-server\/role-id       resource:         role: kube-scheduler-server         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kubeadm-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kubeadm-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubeadm-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubeadm-client\/role-id       resource:         role: kubeadm-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kubelet-client     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kubelet-client-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubelet-client\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubelet-client\/role-id       resource:         role: kubelet-client         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s    - name: kubelet-server     vault:       server: ${VAULT_SERVER}       auth:         caBundle:          tlsInsecure: true         bootstrap:           file: \/var\/lib\/key-keeper\/bootstrap.token         appRole:           name: kubelet-server-${MASTER_NAME}           path: \"clusters\/${CLUSTER_NAME}\/approle\"           secretIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubelet-server\/secret-id           roleIDLocalPath: \/var\/lib\/key-keeper\/vault\/kubelet-server\/role-id       resource:         role: kubelet-server         CAPath: \"clusters\/${CLUSTER_NAME}\/pki\/kubernetes\"       timeout: 15s EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>key-keeper.certs<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. cat &lt;&lt;EOF >> ${BASE_K8S_PATH}\/pki\/vault-config certificates:    - name: etcd-ca     issuerRef:       name: etcd-ca     isCa: true     ca:       exportedKey: false       generate: false     hostPath: \"${BASE_K8S_PATH}\/pki\/ca\"    - name: kube-apiserver-etcd-client     issuerRef:       name: etcd-client     spec:       subject:         commonName: \"system:kube-apiserver-etcd-client\"       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\"     withUpdate: true    - name: etcd-peer     issuerRef:       name: etcd-peer     spec:       subject:         commonName: \"system:etcd-peer\"       usage:         - server auth         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ipAddresses:         interfaces:           - lo           - eth*       ttl: 10m       hostnames:         - localhost         - $HOSTNAME         - \"${MASTER_NAME}.${BASE_CLUSTER_DOMAIN}\"     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/etcd\"     withUpdate: true    - name: etcd-server     issuerRef:       name: etcd-peer     spec:       subject:         commonName: \"system:etcd-server\"       usage:         - server auth         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ipAddresses:         static:           - 127.0.1.1         interfaces:           - lo           - eth*       ttl: 10m       hostnames:         - localhost         - $HOSTNAME         - \"${MASTER_NAME}.${BASE_CLUSTER_DOMAIN}\"     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/etcd\"     withUpdate: true    - name: front-proxy-ca     issuerRef:       name: front-proxy-ca     isCa: true     ca:       exportedKey: false       generate: false     hostPath: \"${BASE_K8S_PATH}\/pki\/ca\"    - name: front-proxy-client     issuerRef:       name: front-proxy-client     spec:       subject:         commonName: \"custom:kube-apiserver-front-proxy-client\"       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\"     withUpdate: true    - name: kubernetes-ca     issuerRef:       name: kubernetes-ca     isCa: true     ca:       exportedKey: false       generate: false     hostPath: \"${BASE_K8S_PATH}\/pki\/ca\"    - name: kube-apiserver     issuerRef:       name: kube-apiserver     spec:       subject:         commonName: \"custom:kube-apiserver\"       usage:         - server auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ipAddresses:         static:           - 29.64.0.1         interfaces:           - lo           - eth*         dnsLookup:           - api.${BASE_CLUSTER_DOMAIN}       ttl: 10m       hostnames:         - localhost         - kubernetes         - kubernetes.default         - kubernetes.default.svc         - kubernetes.default.svc.cluster         - kubernetes.default.svc.cluster.local     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\"     withUpdate: true    - name: kube-apiserver-kubelet-client     issuerRef:       name: kube-apiserver-kubelet-client     spec:       subject:         commonName: \"custom:kube-apiserver-kubelet-client\"       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\"     withUpdate: true    - name: kube-controller-manager-client     issuerRef:       name: kube-controller-manager-client     spec:       subject:         commonName: \"system:kube-controller-manager\"       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-controller-manager\"     withUpdate: true    - name: kube-controller-manager-server     issuerRef:       name: kube-controller-manager-server     spec:       subject:         commonName: \"custom:kube-controller-manager\"       usage:         - server auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ipAddresses:         interfaces:           - lo           - eth*       ttl: 10m       hostnames:         - localhost         - kube-controller-manager.default         - kube-controller-manager.default.svc         - kube-controller-manager.default.svc.cluster         - kube-controller-manager.default.svc.cluster.local     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-controller-manager\"     withUpdate: true    - name: kube-scheduler-client     issuerRef:       name: kube-scheduler-client     spec:       subject:         commonName: \"system:kube-scheduler\"       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-scheduler\"     withUpdate: true    - name: kube-scheduler-server     issuerRef:       name: kube-scheduler-server     spec:       subject:         commonName: \"custom:kube-scheduler\"       usage:         - server auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ipAddresses:         interfaces:           - lo           - eth*       ttl: 10m       hostnames:         - localhost         - kube-scheduler.default         - kube-scheduler.default.svc         - kube-scheduler.default.svc.cluster         - kube-scheduler.default.svc.cluster.local     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-scheduler\"     withUpdate: true    - name: kubeadm-client     issuerRef:       name: kubeadm-client     spec:       subject:         commonName: \"custom:kubeadm-client\"         organizationalUnit:           - system:masters       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\"     withUpdate: true    - name: kubelet-client     issuerRef:       name: kubelet-client     spec:       subject:         commonName: \"system:node:${MASTER_NAME}-${CLUSTER_NAME}\"         organization:           - system:nodes       usage:         - client auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ttl: 10m     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kubelet\"     withUpdate: true    - name: kubelet-server     issuerRef:       name: kubelet-server     spec:       subject:         commonName: \"system:node:${MASTER_NAME}-${CLUSTER_NAME}\"       usage:         - server auth       privateKey:         algorithm: \"RSA\"         encoding: \"PKCS1\"         size: 4096       ipAddresses:         interfaces:           - lo           - eth*       ttl: 10m       hostnames:         - localhost         - $HOSTNAME         - \"${MASTER_NAME}.${BASE_CLUSTER_DOMAIN}\"     renewBefore: 7m     hostPath: \"${BASE_K8S_PATH}\/pki\/certs\/kubelet\"     withUpdate: true  secrets:   - name: kube-apiserver-sa     issuerRef:       name: kube-apiserver-sa     key: private       hostPath: ${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\/kube-apiserver-sa.pem    - name: kube-apiserver-sa     issuerRef:       name: kube-apiserver-sa     key: public       hostPath: ${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\/kube-apiserver-sa.pub EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>** \u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b \u0432\u044b\u043f\u0443\u0441\u043a\u0430\u044e\u0442\u0441\u044f \u0441 <strong>ttl<\/strong>=10\u043c\u0438\u043d\u0443\u0442 \u0438 <strong>renewBefore<\/strong>=7\u043c\u0438\u043d\u0443\u0442, \u044d\u0442\u043e \u043e\u0437\u043d\u0430\u0447\u0430\u0435\u0442, \u0447\u0442\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442 \u0431\u0443\u0434\u0435\u0442 \u043f\u0435\u0440\u0435\u0432\u044b\u043f\u0443\u0441\u043a\u0430\u0442\u044c\u0441\u044f \u043a\u0430\u0436\u0434\u044b\u0435 3 \u043c\u0438\u043d\u0443\u0442\u044b. \u0422\u0430\u043a\u0438\u0435 \u043c\u0430\u043b\u044b\u0435 \u0438\u043d\u0442\u0435\u0440\u0432\u0430\u043b\u044b \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u044b, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u043a\u0430\u0437\u0430\u0442\u044c \u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0441\u0442\u044c \u0440\u0430\u0431\u043e\u0442\u044b \u0444\u0443\u043d\u043a\u0446\u0438\u0438 \u043f\u0435\u0440\u0435\u0432\u044b\u043f\u0443\u0441\u043a\u0430 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430. <strong>(\u0418\u0437\u043c\u0435\u043d\u0438\u0442\u0435 \u0438\u0445 \u043d\u0430 \u0430\u043a\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0435 \u0434\u043b\u044f \u0432\u0430\u0441 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044f.)<\/strong><\/p>\n<p>*** \u0421 \u0432\u0435\u0440\u0441\u0438\u0438 <strong>1.22 Kubernetes<\/strong> (\u043d\u0438\u0436\u0435 \u043d\u0435 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u043b) \u0432\u0441\u0435 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u044b \u0443\u043c\u0435\u044e\u0442 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b \u043d\u0430 \u0444\u0430\u0439\u043b\u043e\u0432\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u0435 \u0438\u0437\u043c\u0435\u043d\u0438\u043b\u0438\u0441\u044c \u0438 \u043f\u0435\u0440\u0435\u0447\u0438\u0442\u044b\u0432\u0430\u0442\u044c \u0438\u0445 \u0431\u0435\u0437 \u043f\u0435\u0440\u0435\u0437\u0430\u043f\u0443\u0441\u043a\u0430.<\/p>\n<details class=\"spoiler\">\n<summary>key-keeper.token<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p \/var\/lib\/key-keeper\/  cat &lt;&lt;EOF > \/var\/lib\/key-keeper\/bootstrap.token ${VAULT_MASTER_TOKEN} EOF<\/code><\/pre>\n<p>** \u041d\u0435 \u0443\u0434\u0438\u0432\u043b\u044f\u0439\u0442\u0435\u0441\u044c, \u0447\u0442\u043e \u0432 \u044d\u0442\u043e\u043c \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u043e\u043c \u0444\u0430\u0439\u043b\u0435 \u043c\u0430\u0441\u0442\u0435\u0440 \u043a\u043b\u044e\u0447 \u043e\u0442 Vault Server, \u043a\u0430\u043a \u044f \u0433\u043e\u0432\u043e\u0440\u0438\u043b \u0440\u0430\u043d\u0435\u0435 &#8212; \u044d\u0442\u043e \u0443\u043f\u0440\u043e\u0449\u0451\u043d\u043d\u0430\u044f \u0432\u0435\u0440\u0441\u0438\u044f\u00a0\u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438.<\/p>\n<p>*** \u0415\u0441\u043b\u0438 \u0447\u0443\u0442\u044c \u0433\u043b\u0443\u0431\u0436\u0435 \u0438\u0437\u0443\u0447\u0438\u0442\u0435 \u043d\u0430\u0448 \u043c\u043e\u0434\u0443\u043b\u044c Vault \u0434\u043b\u044f Terraform, \u0442\u043e \u043f\u043e\u0439\u043c\u0435\u0442\u0435, \u0447\u0442\u043e \u0442\u0430\u043c \u0441\u043e\u0437\u0434\u0430\u044e\u0442\u0441\u044f \u0432\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0435 \u0442\u043e\u043a\u0435\u043d\u044b, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0443\u0436\u043d\u043e \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u0442\u044c \u0432 bootstrap \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0435 key-keeper. \u0414\u043b\u044f \u043a\u0430\u0436\u0434\u043e\u0433\u043e issuer \u0441\u0432\u043e\u0439 \u0442\u043e\u043a\u0435\u043d.  \u041f\u0440\u0438\u043c\u0435\u0440 -> <a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-templates\/cloud-init\/templates\/cloud-init-kubeadm-master.tftpl#L115\" rel=\"noopener noreferrer nofollow\">https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-templates\/cloud-init\/templates\/cloud-init-kubeadm-master.tftpl#L115<\/a><\/p>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0411\u043e\u043b\u044c\u0448\u0430\u044f \u0447\u0430\u0441\u0442\u0438 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u0438, \u043e\u043f\u0438\u0441\u044b\u0432\u0430\u044e\u0449\u0430\u044f \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a, \u0430 \u043d\u0435 \u0438\u043d\u0430\u0447\u0435, \u043f\u0440\u0438\u0432\u0435\u0434\u0435\u043d\u0430 \u0432 \u0441\u0442\u0430\u0442\u044c\u044f\u0445:<\/p>\n<p><a href=\"https:\/\/habr.com\/ru\/post\/673730\/\" rel=\"noopener noreferrer nofollow\">\u0421\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b K8S \u0438\u043b\u0438 \u043a\u0430\u043a \u0440\u0430\u0441\u043f\u0443\u0442\u0430\u0442\u044c \u0432\u0435\u0440\u043c\u0438\u0448\u0435\u043b\u044c \u0427\u0430\u0441\u0442\u044c 1<\/a><\/p>\n<p><a href=\"https:\/\/habr.com\/ru\/post\/695344\/\" rel=\"noopener noreferrer nofollow\">\u0421\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b K8S \u0438\u043b\u0438 \u043a\u0430\u043a \u0440\u0430\u0441\u043f\u0443\u0442\u0430\u0442\u044c \u0432\u0435\u0440\u043c\u0438\u0448\u0435\u043b\u044c \u0427\u0430\u0441\u0442\u044c 2<\/a><\/p>\n<p>\u0412\u0430\u0436\u043d\u043e\u0439 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u044c\u044e \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0442\u043e, \u0447\u0442\u043e \u043c\u044b \u0431\u043e\u043b\u044c\u0448\u0435 \u043d\u0435 \u0437\u0430\u0434\u0443\u043c\u044b\u0432\u0430\u0435\u043c\u0441\u044f \u043e \u043f\u0440\u043e\u0442\u0443\u0445\u0430\u044e\u0449\u0438\u0445 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430\u0445, Key-keeper \u0431\u0435\u0440\u0435\u0442 \u043d\u0430 \u0441\u0435\u0431\u044f \u044d\u0442\u0443 \u0437\u0430\u0434\u0430\u0447\u0443, \u043e\u0442 \u043d\u0430\u0441 \u0442\u043e\u043b\u044c\u043a\u043e \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0438 \u0430\u043b\u0435\u0440\u0442\u044b, \u0434\u043b\u044f \u043e\u0442\u0441\u043b\u0435\u0436\u0438\u0432\u0430\u043d\u0438\u044f \u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b.<\/p>\n<h2>Kubelet config<\/h2>\n<details class=\"spoiler\">\n<summary>config.yaml<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p ${BASE_K8S_PATH}\/kubelet  cat &lt;&lt;EOF >> ${BASE_K8S_PATH}\/kubelet\/config.yaml apiVersion: kubelet.config.k8s.io\/v1beta1 authentication:   anonymous:     enabled: false   webhook:     cacheTTL: 0s     enabled: true   x509:     clientCAFile: \"${BASE_K8S_PATH}\/pki\/ca\/kubernetes-ca.pem\"  tlsCertFile: ${BASE_K8S_PATH}\/pki\/certs\/kubelet\/kubelet-server.pem tlsPrivateKeyFile: ${BASE_K8S_PATH}\/pki\/certs\/kubelet\/kubelet-server-key.pem  authorization:   mode: Webhook   webhook:     cacheAuthorizedTTL: 0s     cacheUnauthorizedTTL: 0s cgroupDriver: systemd clusterDNS:   - \"${SERVICE_DNS}\" clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration logging:   flushFrequency: 0   options:     json:       infoBufferSize: \"0\"   verbosity: 0 memorySwap: {} nodeStatusReportFrequency: 1s nodeStatusUpdateFrequency: 1s resolvConf: \/run\/systemd\/resolve\/resolv.conf rotateCertificates: false runtimeRequestTimeout: 0s serverTLSBootstrap: true shutdownGracePeriod: 15s shutdownGracePeriodCriticalPods: 5s staticPodPath: \"${BASE_K8S_PATH}\/manifests\" streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s containerLogMaxSize: 50Mi maxPods: 250 kubeAPIQPS: 50 kubeAPIBurst: 100 podPidsLimit: 4096 serializeImagePulls: false systemReserved:   ephemeral-storage: 1Gi featureGates:   APIPriorityAndFairness: true   DownwardAPIHugePages: true   PodSecurity: true   CSIMigrationAWS: false   CSIMigrationAzureFile: false   CSIMigrationGCE: false   CSIMigrationvSphere: false rotateCertificates: false serverTLSBootstrap: true tlsMinVersion: VersionTLS12 tlsCipherSuites:   - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256   - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256   - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384   - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256   - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 allowedUnsafeSysctls:   - \"net.core.somaxconn\" evictionSoft:    memory.available: 3Gi    nodefs.available: 25%   nodefs.inodesFree: 15%   imagefs.available: 30%   imagefs.inodesFree: 25% evictionSoftGracePeriod:     memory.available: 2m30s   nodefs.available: 2m30s   nodefs.inodesFree: 2m30s   imagefs.available: 2m30s   imagefs.inodesFree: 2m30s evictionHard:   memory.available: 2Gi   nodefs.available: 20%   nodefs.inodesFree: 10%   imagefs.available: 25%   imagefs.inodesFree: 15% evictionPressureTransitionPeriod: 5s  imageMinimumGCAge: 12h  imageGCHighThresholdPercent: 55 imageGCLowThresholdPercent: 50 EOF<\/code><\/pre>\n<p>** clusterDNS &#8212; \u043b\u0435\u0433\u043a\u043e \u043e\u0431\u0436\u0435\u0447\u044c\u0441\u044f, \u0435\u0441\u043b\u0438 \u0443\u043a\u0430\u0437\u0430\u043b \u043d\u0435\u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0435 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435.<\/p>\n<p>***  resolvConf &#8212; \u0432 Centos, Rhel, Almalinux \u043c\u043e\u0436\u0435\u0442 \u0440\u0443\u0433\u0430\u0442\u044c\u0441\u044f \u043d\u0430 \u043f\u0443\u0442\u044c, \u0440\u0435\u0448\u0430\u0435\u0442\u0441\u044f \u043a\u043e\u043c\u0430\u043d\u0434\u0430\u043c\u0438:<\/p>\n<p>systemctl daemon-reload<br \/>systemctl enable  systemd-resolved.service<br \/>systemctl start  systemd-resolved.service<\/p>\n<p>\u0414\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044f \u043e\u043f\u0438\u0441\u044b\u0432\u0430\u044e\u0449\u0430\u044f \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0443:<br \/><a href=\"https:\/\/kubernetes.io\/docs\/tasks\/administer-cluster\/dns-debugging-resolution\/#known-issues\" rel=\"noopener noreferrer nofollow\">https:\/\/kubernetes.io\/docs\/tasks\/administer-cluster\/dns-debugging-resolution\/#known-issues<\/a><\/p>\n<\/div>\n<\/details>\n<h2>System configs<\/h2>\n<p>\u041a \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0441\u044f:<\/p>\n<ol>\n<li>\n<p>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u0434\u0438\u0441\u043a\u043e\u0432\u043e\u0433\u043e \u043f\u0440\u043e\u0441\u0442\u0440\u0430\u043d\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \/var\/lib\/etcd <s>(\u0432 \u0434\u0430\u043d\u043d\u043e\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u043d\u0435 \u0440\u0430\u0441\u0441\u043c\u0430\u0442\u0440\u0438\u0432\u0430\u0435\u0442\u0441\u044f)<\/s><\/p>\n<\/li>\n<li>\n<p>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 sysctl<\/p>\n<\/li>\n<li>\n<p>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 modprobe<\/p>\n<\/li>\n<li>\n<p>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0431\u0430\u0437\u043e\u0432\u044b\u0445 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 (wget, tar)<\/p>\n<\/li>\n<\/ol>\n<details class=\"spoiler\">\n<summary>modprobe<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. cat &lt;&lt;EOF >> \/etc\/modules-load.d\/k8s.conf overlay br_netfilter EOF  sudo modprobe overlay sudo modprobe br_netfilter<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>sysctls<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. cat &lt;&lt;EOF >> \/etc\/sysctl.d\/99-network.conf net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 EOF  sysctl --system<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<h2>Kubeconfigs<\/h2>\n<p>\u0414\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0431\u0430\u0437\u043e\u0432\u044b\u0435 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u044b \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0438 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440 \u043c\u043e\u0433\u043b\u0438 \u043e\u0431\u0449\u0430\u0442\u044c\u0441\u044f \u0441               Kube-apiserver, \u043d\u0443\u0436\u043d\u043e \u0441\u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u0442\u044c <strong>kubeconfig <\/strong>\u0434\u043b\u044f \u043a\u0430\u0436\u0434\u043e\u0433\u043e \u0438\u0437 \u043d\u0438\u0445.<\/p>\n<p>** admin.conf <strong>Kubeconfig<\/strong> \u0441 \u043f\u0440\u0430\u0432\u0430\u043c\u0438 <strong>cluster-admin<\/strong> \u0434\u043b\u044f \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440\u043e\u043c.<\/p>\n<details class=\"spoiler\">\n<summary>admin.conf<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p ${BASE_K8S_PATH}  cat &lt;&lt;EOF >> ${BASE_K8S_PATH}\/admin.conf --- apiVersion: v1 clusters: - cluster:     certificate-authority: ${BASE_K8S_PATH}\/pki\/ca\/kubernetes-ca.pem     server: https:\/\/127.0.0.1:${KUBE_APISERVER_PORT}   name: kubernetes contexts: - context:     cluster: kubernetes     namespace: default     user: kubeadm   name: kubeadm@kubernetes current-context: kubeadm@kubernetes kind: Config preferences: {} users: - name: kubeadm   user:     client-certificate: ${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\/kubeadm-client.pem     client-key: ${BASE_K8S_PATH}\/pki\/certs\/kube-apiserver\/kubeadm-client-key.pem EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kube-scheduler<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p ${BASE_K8S_PATH}\/kube-scheduler\/  cat &lt;&lt;EOF >> ${BASE_K8S_PATH}\/kube-scheduler\/kubeconfig --- apiVersion: v1 clusters: - cluster:     certificate-authority: ${BASE_K8S_PATH}\/pki\/ca\/kubernetes-ca.pem     server: https:\/\/127.0.0.1:${KUBE_APISERVER_PORT}   name: kubernetes contexts: - context:     cluster: kubernetes     namespace: default     user: kube-scheduler   name: kube-scheduler@kubernetes current-context: kube-scheduler@kubernetes kind: Config preferences: {} users: - name: kube-scheduler   user:     client-certificate: ${BASE_K8S_PATH}\/pki\/certs\/kube-scheduler\/kube-scheduler-client.pem     client-key: ${BASE_K8S_PATH}\/pki\/certs\/kube-scheduler\/kube-scheduler-client-key.pem EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kube-controller-manager<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p ${BASE_K8S_PATH}\/kube-controller-manager  cat &lt;&lt;EOF >> ${BASE_K8S_PATH}\/kube-controller-manager\/kubeconfig --- apiVersion: v1 clusters: - cluster:     certificate-authority: ${BASE_K8S_PATH}\/pki\/ca\/kubernetes-ca.pem     server: https:\/\/127.0.0.1:${KUBE_APISERVER_PORT}   name: kubernetes contexts: - context:     cluster: kubernetes     namespace: default     user: kube-controller-manager   name: kube-controller-manager@kubernetes current-context: kube-controller-manager@kubernetes kind: Config preferences: {} users: - name: kube-controller-manager   user:     client-certificate: ${BASE_K8S_PATH}\/pki\/certs\/kube-controller-manager\/kube-controller-manager-client.pem     client-key: ${BASE_K8S_PATH}\/pki\/certs\/kube-controller-manager\/kube-controller-manager-client-key.pem EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. mkdir -p ${BASE_K8S_PATH}\/kubelet  cat &lt;&lt;EOF >> ${BASE_K8S_PATH}\/kubelet\/kubeconfig --- apiVersion: v1 clusters: - cluster:     certificate-authority: ${BASE_K8S_PATH}\/pki\/ca\/kubernetes-ca.pem     server: https:\/\/127.0.0.1:${KUBE_APISERVER_PORT}   name: kubernetes contexts: - context:     cluster: kubernetes     namespace: default     user: kubelet   name: kubelet@kubernetes current-context: kubelet@kubernetes kind: Config preferences: {} users: - name: kubelet   user:     client-certificate: ${BASE_K8S_PATH}\/pki\/certs\/kubelet\/kubelet-client.pem     client-key: ${BASE_K8S_PATH}\/pki\/certs\/kubelet\/kubelet-client-key.pem EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<h2>Static Pods<\/h2>\n<details class=\"spoiler\">\n<summary>kube-apiserver<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. export ADVERTISE_ADDRESS=$(ip route get 1.1.1.1 | grep -oP 'src \\K\\S+')  cat &lt;&lt;EOF > \/etc\/kubernetes\/manifests\/kube-apiserver.yaml apiVersion: v1 kind: Pod metadata:   annotations:     kubeadm.kubernetes.io\/kube-apiserver.advertise-address.endpoint: ${ADVERTISE_ADDRESS}:${KUBE_APISERVER_PORT}   creationTimestamp: null   labels:     component: kube-apiserver     tier: control-plane   name: kube-apiserver   namespace: kube-system spec:   containers:   - command:     - kube-apiserver     - --advertise-address=${ADVERTISE_ADDRESS}     - --allow-privileged=true     - --authorization-mode=Node,RBAC     - --bind-address=0.0.0.0     - --client-ca-file=\/etc\/kubernetes\/pki\/ca\/kubernetes-ca.pem     - --enable-admission-plugins=NodeRestriction     - --enable-bootstrap-token-auth=true     - --etcd-cafile=\/etc\/kubernetes\/pki\/ca\/etcd-ca.pem     - --etcd-certfile=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-etcd-client.pem     - --etcd-keyfile=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-etcd-client-key.pem     - --etcd-servers=${ETCD_SERVERS}     - --kubelet-client-certificate=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-kubelet-client.pem     - --kubelet-client-key=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-kubelet-client-key.pem     - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname     - --proxy-client-cert-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/front-proxy-client.pem     - --proxy-client-key-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/front-proxy-client-key.pem     - --requestheader-allowed-names=front-proxy-client     - --requestheader-client-ca-file=\/etc\/kubernetes\/pki\/ca\/front-proxy-ca.pem     - --requestheader-extra-headers-prefix=X-Remote-Extra-     - --requestheader-group-headers=X-Remote-Group     - --requestheader-username-headers=X-Remote-User     - --secure-port=${KUBE_APISERVER_PORT}     - --service-account-issuer=https:\/\/kubernetes.default.svc.cluster.local     - --service-account-key-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-sa.pub     - --service-account-signing-key-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-sa.pem     - --service-cluster-ip-range=${SERVICE_CIDR}     - --tls-cert-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver.pem     - --tls-private-key-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-key.pem     image: k8s.gcr.io\/kube-apiserver:${KUBERNETES_VERSION}     imagePullPolicy: IfNotPresent     livenessProbe:       failureThreshold: 8       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/livez         port: ${KUBE_APISERVER_PORT}         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     name: kube-apiserver     readinessProbe:       failureThreshold: 3       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/readyz         port: ${KUBE_APISERVER_PORT}         scheme: HTTPS       periodSeconds: 1       timeoutSeconds: 15     resources:       requests:         cpu: 250m     startupProbe:       failureThreshold: 24       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/livez         port: ${KUBE_APISERVER_PORT}         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     volumeMounts:     - mountPath: \/etc\/ssl\/certs       name: ca-certs       readOnly: true     - mountPath: \/etc\/ca-certificates       name: etc-ca-certificates       readOnly: true     - mountPath: \/var\/log\/kubernetes\/audit\/       name: k8s-audit     - mountPath: \/etc\/kubernetes\/pki\/ca       name: k8s-ca       readOnly: true     - mountPath: \/etc\/kubernetes\/pki\/certs       name: k8s-certs       readOnly: true     - mountPath: \/etc\/kubernetes\/kube-apiserver       name: k8s-kube-apiserver-configs       readOnly: true     - mountPath: \/usr\/local\/share\/ca-certificates       name: usr-local-share-ca-certificates       readOnly: true     - mountPath: \/usr\/share\/ca-certificates       name: usr-share-ca-certificates       readOnly: true   hostNetwork: true   priorityClassName: system-node-critical   securityContext:     seccompProfile:       type: RuntimeDefault   volumes:   - hostPath:       path: \/etc\/ssl\/certs       type: DirectoryOrCreate     name: ca-certs   - hostPath:       path: \/etc\/ca-certificates       type: DirectoryOrCreate     name: etc-ca-certificates   - hostPath:       path: \/var\/log\/kubernetes\/audit\/       type: DirectoryOrCreate     name: k8s-audit   - hostPath:       path: \/etc\/kubernetes\/pki\/ca       type: DirectoryOrCreate     name: k8s-ca   - hostPath:       path: \/etc\/kubernetes\/pki\/certs       type: DirectoryOrCreate     name: k8s-certs   - hostPath:       path: \/etc\/kubernetes\/kube-apiserver       type: DirectoryOrCreate     name: k8s-kube-apiserver-configs   - hostPath:       path: \/usr\/local\/share\/ca-certificates       type: DirectoryOrCreate     name: usr-local-share-ca-certificates   - hostPath:       path: \/usr\/share\/ca-certificates       type: DirectoryOrCreate     name: usr-share-ca-certificates status: {} EOF<\/code><\/pre>\n<p>** \u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u0430\u044f <strong>ADVERTISE_ADDRESS<\/strong> \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0438\u043d\u0442\u0435\u0440\u043d\u0435\u0442\u0430, \u0435\u0441\u043b\u0438 \u0435\u0433\u043e \u043d\u0435\u0442 \u043f\u0440\u043e\u0441\u0442\u043e \u0443\u043a\u0430\u0436\u0438\u0442\u0435 IP ADDRESS \u043d\u043e\u0434\u044b.<\/p>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kube-controller-manager<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. cat &lt;&lt;EOF > \/etc\/kubernetes\/manifests\/kube-controller-manager.yaml apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     component: kube-controller-manager     tier: control-plane   name: kube-controller-manager   namespace: kube-system spec:   containers:   - command:     - kube-controller-manager     - --authentication-kubeconfig=\/etc\/kubernetes\/kube-controller-manager\/kubeconfig     - --authorization-always-allow-paths=\/healthz,\/metrics     - --authorization-kubeconfig=\/etc\/kubernetes\/kube-controller-manager\/kubeconfig     - --bind-address=${ADVERTISE_ADDRESS}     - --client-ca-file=\/etc\/kubernetes\/pki\/ca\/kubernetes-ca.pem     - --cluster-cidr=${SERVICE_CIDR}     - --cluster-name=kubernetes     - --cluster-signing-cert-file=\/etc\/kubernetes\/pki\/ca\/kubernetes-ca.pem     - --cluster-signing-key-file=     - --controllers=*,bootstrapsigner,tokencleaner     - --kubeconfig=\/etc\/kubernetes\/kube-controller-manager\/kubeconfig     - --leader-elect=true     - --requestheader-client-ca-file=\/etc\/kubernetes\/pki\/ca\/front-proxy-ca.pem     - --root-ca-file=\/etc\/kubernetes\/pki\/ca\/kubernetes-ca.pem     - --secure-port=${KUBE_CONTROLLER_MANAGER_PORT}     - --service-account-private-key-file=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-sa.pem     - --tls-cert-file=\/etc\/kubernetes\/pki\/certs\/kube-controller-manager\/kube-controller-manager-server.pem     - --tls-private-key-file=\/etc\/kubernetes\/pki\/certs\/kube-controller-manager\/kube-controller-manager-server-key.pem     - --use-service-account-credentials=true     image: k8s.gcr.io\/kube-controller-manager:${KUBERNETES_VERSION}     imagePullPolicy: IfNotPresent     livenessProbe:       failureThreshold: 8       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/healthz         port: ${KUBE_CONTROLLER_MANAGER_PORT}         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     name: kube-controller-manager     resources:       requests:         cpu: 200m     startupProbe:       failureThreshold: 24       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/healthz         port: ${KUBE_CONTROLLER_MANAGER_PORT}         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     volumeMounts:     - mountPath: \/etc\/ssl\/certs       name: ca-certs       readOnly: true     - mountPath: \/etc\/ca-certificates       name: etc-ca-certificates       readOnly: true     - mountPath: \/usr\/libexec\/kubernetes\/kubelet-plugins\/volume\/exec       name: flexvolume-dir     - mountPath: \/etc\/kubernetes\/pki\/ca       name: k8s-ca       readOnly: true     - mountPath: \/etc\/kubernetes\/pki\/certs       name: k8s-certs       readOnly: true     - mountPath: \/etc\/kubernetes\/kube-controller-manager       name: k8s-kube-controller-manager-configs       readOnly: true     - mountPath: \/usr\/local\/share\/ca-certificates       name: usr-local-share-ca-certificates       readOnly: true     - mountPath: \/usr\/share\/ca-certificates       name: usr-share-ca-certificates       readOnly: true   hostNetwork: true   priorityClassName: system-node-critical   securityContext:     seccompProfile:       type: RuntimeDefault   volumes:   - hostPath:       path: \/etc\/ssl\/certs       type: DirectoryOrCreate     name: ca-certs   - hostPath:       path: \/etc\/ca-certificates       type: DirectoryOrCreate     name: etc-ca-certificates   - hostPath:       path: \/usr\/libexec\/kubernetes\/kubelet-plugins\/volume\/exec       type: DirectoryOrCreate     name: flexvolume-dir   - hostPath:       path: \/etc\/kubernetes\/pki\/ca       type: DirectoryOrCreate     name: k8s-ca   - hostPath:       path: \/etc\/kubernetes\/pki\/certs       type: DirectoryOrCreate     name: k8s-certs   - hostPath:       path: \/etc\/kubernetes\/kube-controller-manager       type: DirectoryOrCreate     name: k8s-kube-controller-manager-configs   - hostPath:       path: \/usr\/local\/share\/ca-certificates       type: DirectoryOrCreate     name: usr-local-share-ca-certificates   - hostPath:       path: \/usr\/share\/ca-certificates       type: DirectoryOrCreate     name: usr-share-ca-certificates status: {} EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kube-scheduler<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. cat &lt;&lt;EOF > \/etc\/kubernetes\/manifests\/kube-scheduler.yaml apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     component: kube-scheduler     tier: control-plane   name: kube-scheduler   namespace: kube-system spec:   containers:   - command:     - kube-scheduler     - --authentication-kubeconfig=\/etc\/kubernetes\/kube-scheduler\/kubeconfig     - --authorization-kubeconfig=\/etc\/kubernetes\/kube-scheduler\/kubeconfig     - --bind-address=${ADVERTISE_ADDRESS}     - --kubeconfig=\/etc\/kubernetes\/kube-scheduler\/kubeconfig     - --leader-elect=true     - --secure-port=${KUBE_SCHEDULER_PORT}     - --tls-cert-file=\/etc\/kubernetes\/pki\/certs\/kube-scheduler\/kube-scheduler-server.pem     - --tls-private-key-file=\/etc\/kubernetes\/pki\/certs\/kube-scheduler\/kube-scheduler-server-key.pem     image: k8s.gcr.io\/kube-scheduler:${KUBERNETES_VERSION}     imagePullPolicy: IfNotPresent     livenessProbe:       failureThreshold: 8       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/healthz         port: ${KUBE_SCHEDULER_PORT}         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     name: kube-scheduler     resources:       requests:         cpu: 100m     startupProbe:       failureThreshold: 24       httpGet:         host: ${ADVERTISE_ADDRESS}         path: \/healthz         port: ${KUBE_SCHEDULER_PORT}         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     volumeMounts:     - mountPath: \/etc\/kubernetes\/pki\/ca       name: k8s-ca       readOnly: true     - mountPath: \/etc\/kubernetes\/pki\/certs       name: k8s-certs       readOnly: true     - mountPath: \/etc\/kubernetes\/kube-scheduler       name: k8s-kube-scheduler-configs       readOnly: true   hostNetwork: true   priorityClassName: system-node-critical   securityContext:     seccompProfile:       type: RuntimeDefault   volumes:   - hostPath:       path: \/etc\/kubernetes\/pki\/ca       type: DirectoryOrCreate     name: k8s-ca   - hostPath:       path: \/etc\/kubernetes\/pki\/certs       type: DirectoryOrCreate     name: k8s-certs   - hostPath:       path: \/etc\/kubernetes\/kube-scheduler       type: DirectoryOrCreate     name: k8s-kube-scheduler-configs status: {} EOF <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>etcd<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. cat &lt;&lt;EOF > \/etc\/kubernetes\/manifests\/etcd.yaml --- apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     component: etcd     tier: control-plane   name: etcd   namespace: kube-system spec:   containers:   - name: etcd     command:       - etcd     args:       - --name=${MASTER_NAME}.${BASE_CLUSTER_DOMAIN}       - --initial-cluster=${ETCD_INITIAL_CLUSTER}       - --initial-advertise-peer-urls=https:\/\/${MASTER_NAME}.${BASE_CLUSTER_DOMAIN}:${ETCD_PEER_PORT}       - --advertise-client-urls=https:\/\/${MASTER_NAME}.${BASE_CLUSTER_DOMAIN}:${ETCD_SERVER_PORT}       - --peer-trusted-ca-file=\/etc\/kubernetes\/pki\/ca\/etcd-ca.pem       - --trusted-ca-file=\/etc\/kubernetes\/pki\/ca\/etcd-ca.pem       - --peer-cert-file=\/etc\/kubernetes\/pki\/certs\/etcd\/etcd-peer.pem       - --peer-key-file=\/etc\/kubernetes\/pki\/certs\/etcd\/etcd-peer-key.pem       - --cert-file=\/etc\/kubernetes\/pki\/certs\/etcd\/etcd-server.pem       - --key-file=\/etc\/kubernetes\/pki\/certs\/etcd\/etcd-server-key.pem       - --listen-client-urls=https:\/\/0.0.0.0:${ETCD_SERVER_PORT}       - --listen-peer-urls=https:\/\/0.0.0.0:${ETCD_PEER_PORT}       - --listen-metrics-urls=http:\/\/0.0.0.0:${ETCD_METRICS_PORT}       - --initial-cluster-token=etcd       - --initial-cluster-state=new       - --data-dir=\/var\/lib\/etcd       - --strict-reconfig-check=true       - --peer-client-cert-auth=true       - --peer-auto-tls=true       - --client-cert-auth=true       - --snapshot-count=10000       - --heartbeat-interval=250       - --election-timeout=1500       - --quota-backend-bytes=0       - --max-snapshots=10       - --max-wals=10       - --discovery-fallback=proxy       - --auto-compaction-retention=8       - --force-new-cluster=false       - --enable-v2=false       - --proxy=off       - --proxy-failure-wait=5000       - --proxy-refresh-interval=30000       - --proxy-dial-timeout=1000       - --proxy-write-timeout=5000       - --proxy-read-timeout=0       - --metrics=extensive       - --logger=zap     image: k8s.gcr.io\/etcd:${ETCD_VERSION}     imagePullPolicy: IfNotPresent     livenessProbe:       failureThreshold: 8       httpGet:         host: 127.0.0.1         path: \/health         port: ${ETCD_METRICS_PORT}         scheme: HTTP       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15     resources:       requests:         cpu: 100m         memory: 100Mi     startupProbe:       failureThreshold: 24       httpGet:         host: 127.0.0.1         path: \/health         port: ${ETCD_METRICS_PORT}         scheme: HTTP     volumeMounts:     - mountPath: \/var\/lib\/etcd       name: etcd-data     - mountPath: \/etc\/kubernetes\/pki\/certs\/etcd       name: etcd-certs     - mountPath: \/etc\/kubernetes\/pki\/ca       name: ca   hostNetwork: true   priorityClassName: system-node-critical   securityContext:       null   volumes:   - hostPath:       path: \/etc\/kubernetes\/pki\/certs\/etcd       type: DirectoryOrCreate     name: etcd-certs   - hostPath:       path: \/etc\/kubernetes\/pki\/ca       type: DirectoryOrCreate     name: ca   - hostPath:       path: \/var\/lib\/etcd       type: DirectoryOrCreate     name: etcd-data status: {} EOF <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<h2>Systemd<\/h2>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0434\u0435\u043b\u043e \u0437\u0430 \u043c\u0430\u043b\u044b\u043c, \u0432\u043a\u043b\u044e\u0447\u0430\u0435\u043c \u0432\u0441\u0435 \u0441\u0435\u0440\u0432\u0438\u0441\u044b \u0438 \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u0435\u043c \u0438\u0445 \u0432 \u0430\u0432\u0442\u043e\u0437\u0430\u043f\u0443\u0441\u043a.<\/p>\n<details class=\"spoiler\">\n<summary>services<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. systemctl daemon-reload systemctl enable --now \\ key-keeper.service \\ kubelet.service \\ containerd.service \\ systemd-resolved.service<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<h2>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430<\/h2>\n<p>\u0418\u0442\u0430\u043a, \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f \u0433\u043e\u0442\u043e\u0432\u0430, \u043c\u044b \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u043b\u0438 \u0432\u0441\u0435 \u044d\u0442\u0430\u043f\u044b \u043d\u0430 \u043a\u0430\u0436\u0434\u043e\u043c \u043c\u0430\u0441\u0442\u0435\u0440\u0435, \u0442\u0435\u043f\u0435\u0440\u044c \u043d\u0443\u0436\u043d\u043e \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u0442\u044c, \u0447\u0442\u043e \u0432\u0441\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e.<\/p>\n<p><strong>\u041f\u0435\u0440\u0432\u044b\u043c \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0435\u043c, \u0447\u0442\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b \u0437\u0430\u043a\u0430\u0437\u0430\u043d\u044b.<\/strong><\/p>\n<p><code>tree \/etc\/kubernetes\/pki\/ | grep -v key | grep pem | wc -l<\/code>    <br \/><u>\u041f\u0443\u043b\u0443\u0447\u0430\u0435\u043c 17 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432<\/u><\/p>\n<pre><code class=\"bash\">root@master-1-example:\/home\/dkot# tree  \/etc\/kubernetes\/pki\/ \u251c\u2500\u2500 ca \u2502\u00a0\u00a0 \u251c\u2500\u2500 etcd-ca.pem \u2502\u00a0\u00a0 \u251c\u2500\u2500 front-proxy-ca.pem \u2502\u00a0\u00a0 \u2514\u2500\u2500 kubernetes-ca.pem \u251c\u2500\u2500 certs \u2502\u00a0\u00a0 \u251c\u2500\u2500 etcd \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 etcd-peer-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 etcd-peer.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 etcd-server-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 etcd-server.pem \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 front-proxy-client-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 front-proxy-client.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kubeadm-client-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kubeadm-client.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver-etcd-client-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver-etcd-client.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver-kubelet-client-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver-kubelet-client.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-apiserver-sa.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 kube-apiserver-sa.pub \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-controller-manager \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-controller-manager-client-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-controller-manager-client.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kube-controller-manager-server-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 kube-controller-manager-server.pem \u2502\u00a0\u00a0 \u251c\u2500\u2500 kubelet \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kubelet-client-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kubelet-client.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 kubelet-server-key.pem \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 kubelet-server.pem \u2502\u00a0\u00a0 \u2514\u2500\u2500 kube-scheduler \u2502\u00a0\u00a0     \u251c\u2500\u2500 kube-scheduler-client-key.pem \u2502\u00a0\u00a0     \u251c\u2500\u2500 kube-scheduler-client.pem \u2502\u00a0\u00a0     \u251c\u2500\u2500 kube-scheduler-server-key.pem \u2502\u00a0\u00a0     \u2514\u2500\u2500 kube-scheduler-server.pem <\/code><\/pre>\n<p>\u0415\u0441\u043b\u0438 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432 \u043c\u0435\u043d\u044c\u0448\u0435 \u0438\u043b\u0438 \u0438\u0445 \u0432\u043e\u0432\u0441\u0435 \u043d\u0435\u0442, \u0447\u0438\u0442\u0430\u0435\u043c \u043b\u043e\u0433\u0438 \u0441\u0435\u0440\u0432\u0438\u0441\u0430 <strong>key-keeper.<\/strong><br \/><code>journalctl -xefu key-keeper<\/code>  \u0422\u0430\u043c \u043d\u0430\u0439\u0434\u0435\u0442\u0435 \u043e\u0442\u0432\u0435\u0442\u044b \u043d\u0430 \u0432\u0441\u0435 \u0432\u043e\u043f\u0440\u043e\u0441\u044b.<\/p>\n<p>\u0427\u0430\u0441\u0442\u044b\u0435 \u043e\u0448\u0438\u0431\u043a\u0438:<\/p>\n<ul>\n<li>\n<p>\u041d\u0435\u0432\u0430\u043b\u0438\u0434\u043d\u044b\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b.<\/p>\n<\/li>\n<li>\n<p>Key-keeper \u043d\u0435 \u043c\u043e\u0436\u0435\u0442 \u0430\u0432\u0442\u043e\u0440\u0438\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f.<\/p>\n<\/li>\n<li>\n<p>\u041d\u0435\u0442 \u043f\u043e\u043b\u0438\u0442\u0438\u043a \u0434\u043b\u044f \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0440\u043e\u043b\u0438 \u0443 token \u0438\u043b\u0438 approle.<\/p>\n<\/li>\n<li>\n<p>\u0417\u0430\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u043c\u044b\u0439 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442 \u0438\u043c\u0435\u0435\u0442 \u0430\u0440\u0433\u0443\u043c\u0435\u043d\u0442\u044b \u043d\u0435\u0440\u0430\u0437\u0440\u0435\u0448\u0435\u043d\u043d\u044b\u0435 \u0432 \u0440\u043e\u043b\u0438 vault.<\/p>\n<\/li>\n<\/ul>\n<p><strong>\u041f\u0440\u043e\u0432\u0435\u0440\u044f\u0435\u043c, \u0447\u0442\u043e \u0432\u0441\u0435 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u0437\u0430\u043f\u0443\u0449\u0435\u043d\u044b \u0438 \u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e \u0440\u0430\u0431\u043e\u0442\u0430\u044e\u0442<\/strong><\/p>\n<pre><code class=\"bash\">crictl  --runtime-endpoint unix:\/\/\/run\/containerd\/containerd.sock ps -a  CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID 08e2c895b4a20       23f16c2de4792       34 minutes ago      Running             kube-apiserver            4                   b89014de1d7d8 5f1f770280cc7       23f16c2de4792       35 minutes ago      Exited              kube-apiserver            3                   b89014de1d7d8 3313b1ec20e0a       aebe758cef4cd       35 minutes ago      Running             etcd                      2                   cb5b2ca15cc28 e91d3bbb55b97       aebe758cef4cd       37 minutes ago      Exited              etcd                      1                   cb5b2ca15cc28 b3b004e6896db       4bf8b96f38e3b       39 minutes ago      Running             kube-controller-manager   0                   9904b2d296bca 77d316d50693a       ea40e3ed8cf2f       39 minutes ago      Running             kube-scheduler            0                   24fac1b156ea4 <\/code><\/pre>\n<p>\u0415\u0441\u043b\u0438 \u043a\u0430\u043a\u043e\u0439-\u0442\u043e \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440 \u043d\u0430\u0445\u043e\u0434\u0438\u0442\u0441\u044f \u0432 \u0441\u0442\u0430\u0442\u0443\u0441\u0435 <strong>EXITED<\/strong> &#8212; \u0441\u043c\u043e\u0442\u0440\u0438\u043c \u043b\u043e\u0433\u0438.<\/p>\n<p><code>crictl  --runtime-endpoint unix:\/\/\/run\/containerd\/containerd.sock logs $CONTAINER_ID<\/code><\/p>\n<p><strong>\u041f\u0440\u043e\u0432\u0435\u0440\u044f\u0435\u043c, \u0441\u043e\u0431\u0440\u0430\u043d\u043d\u044b\u0439 \u043a\u043b\u0430\u0441\u0442\u0435\u0440 ETCD<\/strong><\/p>\n<details class=\"spoiler\">\n<summary>endpoint status<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. export ETCDCTL_CERT=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-etcd-client.pem \\ export ETCDCTL_KEY=\/etc\/kubernetes\/pki\/certs\/kube-apiserver\/kube-apiserver-etcd-client-key.pem \\ export ETCDCTL_CACERT=\/etc\/kubernetes\/pki\/ca\/etcd-ca.pem \\  etcd_endpoints () { export ENDPOINTS=$(echo $(ENDPOINTS=127.0.0.1:${ETCD_SERVER_PORT} etcdctl \\ --endpoints=$ENDPOINTS \\ member list | awk '{print $5}' | sed \"s\/,\/\/\") | sed \"s\/ \/,\/g\") }  etcd_endpoints  estat () { etcdctl \\ --write-out=table \\ --endpoints=$ENDPOINTS \\ endpoint status }  estat <\/code><\/pre>\n<p>\u041f\u043e\u043b\u0435\u0437\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0434\u0430\u043d\u043d\u044b\u0439 \u043a\u0443\u0441\u043e\u043a \u0432 bashrc, \u0434\u043b\u044f \u0443\u0434\u043e\u0431\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b, \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0441\u0442\u0430\u0442\u0443\u0441\u0430 \u0438\u043b\u0438 \u0434\u0435\u0431\u0430\u0433\u0430 etcd .<\/p>\n<\/div>\n<\/details>\n<p>\u041d\u0430 \u0432\u044b\u0445\u043e\u0434\u0435 \u0434\u043e\u043b\u0436\u043d\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u0442\u044c \u0430\u043d\u0430\u043b\u043e\u0433\u0438\u0447\u043d\u0443\u044e \u043a\u0430\u0440\u0442\u0438\u043d\u0443: (\u041a\u043e\u043b-\u0432\u043e \u0438\u043d\u0441\u0442\u0430\u043d\u0441\u043e\u0432 \u0434\u043e\u043b\u0436\u043d\u043e \u0431\u044b\u0442\u044c \u0440\u0430\u0432\u043d\u043e \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044e <code>MASTER_COUNT<\/code> )<\/p>\n<pre><code class=\"bash\">root@master-1-example:\/home\/dkot# estat +--------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ |                  ENDPOINT                  |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +--------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https:\/\/master-1.example.dobry-kot.ru:2379 | 530f4c34efefa4a2 |   3.5.3 |  8.3 MB |      true |      false |         2 |       6433 |               6433 |        | | https:\/\/master-2.example.dobry-kot.ru:2379 | 85281728dcb33e5f |   3.5.3 |  8.3 MB |     false |      false |         2 |       6433 |               6433 |        | | https:\/\/master-0.example.dobry-kot.ru:2379 | ae74003c0ad34ecd |   3.5.3 |  8.3 MB |     false |      false |         2 |       6433 |               6433 |        | +--------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ <\/code><\/pre>\n<p><strong>\u0418 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0435\u043c, \u0447\u0442\u043e Kubernetes API \u043e\u0442\u0432\u0435\u0447\u0430\u0435\u0442, \u0430 \u0432\u0441\u0435 \u043d\u043e\u0434\u044b \u0434\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u044b.<\/strong><\/p>\n<pre><code class=\"bash\">kubectl get nodes --kubeconfig=\/etc\/kubernetes\/admin.conf  NAME               STATUS     ROLES    AGE   VERSION master-0-example   NotReady   &lt;none>   30m   v1.23.12 master-1-example   NotReady   &lt;none>   29m   v1.23.12 master-2-example   NotReady   &lt;none>   25m   v1.23.12<\/code><\/pre>\n<p>\u0417\u0430\u043f\u0443\u0441\u043a\u0430\u0435\u043c \u0434\u0430\u043d\u043d\u0443\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u0443 \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u043c\u0430\u0441\u0442\u0435\u0440\u043e\u0432 \u0438 \u0432\u0438\u0434\u0438\u043c, \u0447\u0442\u043e \u0432\u0441\u0435 \u043d\u043e\u0434\u044b \u0434\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u044b \u0438 \u0432\u0441\u0435 \u0432 \u0441\u0442\u0430\u0442\u0443\u0441\u0435 <strong>NotReady<\/strong>, \u043d\u0435 \u043f\u0443\u0433\u0430\u0435\u043c\u0441\u044f \u044d\u0442\u043e \u0441\u0432\u044f\u0437\u0430\u043d\u043e \u0441 \u0442\u0435\u043c, \u0447\u0442\u043e \u043d\u0435 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d CNI Plugin. <\/p>\n<p>** \u041d\u0430\u0434\u0435\u044e\u0441\u044c \u043d\u0435 \u0437\u0430\u0431\u044b\u043b\u0438, \u0447\u0442\u043e \u043c\u044b \u043f\u0438\u0441\u0430\u043b\u0438 \u043f\u0440\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442 <a href=\"https:\/\/github.com\/fraima\/kubernetes\/blob\/f0e4c7bc8f8d2695c419b17fec4bacc2dd7c5f18\/modules\/k8s-config-vars\/locals.certs.tf#L234\" rel=\"noopener noreferrer nofollow\"><strong>kube-apiserver-kubelet-client<\/strong><\/a><strong>.<br \/><\/strong>\u0412 \u043d\u0430\u0448\u0435\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u0434\u0430\u043d\u043d\u044b\u0439 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442 \u043d\u0435 \u0431\u0443\u0434\u0435\u0442 \u0438\u043c\u0435\u0442\u044c \u043f\u0440\u0430\u0432 \u0438\u0437\u043d\u0430\u0447\u0430\u043b\u044c\u043d\u043e<strong>, <\/strong>\u043d\u043e                 <strong>kube-apiserver-(\u0443)<\/strong> \u0432\u0441\u0435 \u0435\u0449\u0435 \u043d\u0443\u0436\u043d\u044b \u043f\u0440\u0430\u0432\u0430 \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0443\u043f\u0430 \u043a <strong>Kubelet<\/strong> \u043d\u0430 \u043d\u043e\u0434\u0430\u0445, \u0442.\u043a. \u0438\u043c\u0435\u043d\u043d\u043e \u0441 \u044d\u0442\u0438\u043c \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u043c \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u044e\u0442\u0441\u044f \u0438 \u043f\u0440\u043e\u0445\u043e\u0434\u044f\u0442 \u0447\u0435\u0440\u0435\u0437 <strong>RBAC <\/strong>\u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0438 &#171;<strong>kubectl exec&#187;<\/strong> \u0438 &#171;<strong>kubectl logs&#187;.<br \/><\/strong>\u041a\u0430\u043a \u043d\u0438 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e, \u043d\u043e \u043e \u043d\u0430\u0441 \u0443\u0436\u0435 \u043f\u043e\u0437\u0430\u0431\u043e\u0442\u0438\u043b\u0438\u0441\u044c, \u0438 \u0432 \u0441\u0432\u0435\u0436\u0435\u043c \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435 \u0443\u0436\u0435 \u0435\u0441\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0430\u044f \u0440\u043e\u043b\u044c, \u0442\u0430\u043a \u0447\u0442\u043e \u043f\u0440\u043e\u0441\u0442\u043e \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0430\u043a\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0439 <strong>ClusterRolebinding<\/strong> \u0438 \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u043c \u043b\u043e\u0433\u0438.<\/p>\n<details class=\"spoiler\">\n<summary>ClusterRoleBinding<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">cat &lt;&lt;EOF | kubectl apply -f - --- apiVersion: rbac.authorization.k8s.io\/v1 kind: ClusterRoleBinding metadata:   name: custom:kube-apiserver-kubelet-client roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: system:kubelet-api-admin subjects: - kind: User   apiGroup: rbac.authorization.k8s.io   name: custom:kube-apiserver-kubelet-client EOF<\/code><\/pre>\n<p>** \u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442 \u0432\u044b\u043f\u0443\u0441\u043a\u0430\u0435\u0442\u0441\u044f \u0441 CN=custom:kube-apiserver-kubelet-client (\u0435\u0441\u043b\u0438 \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u043a\u0430\u0441\u0442\u043e\u043c\u0438\u0437\u0430\u0446\u0438\u044f \u0438\u043c\u0435\u043d\u0438, \u043d\u0443\u0436\u043d\u043e \u043f\u0440\u0430\u0432\u0438\u0442\u044c \u0432 \u043c\u043e\u0434\u0443\u043b\u0435 \u0442\u0435\u0440\u0440\u0430\u0444\u043e\u0440\u043c)<\/p>\n<\/div>\n<\/details>\n<h2>\u0418\u0422\u041e\u0413\u041e<\/h2>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e0b\/e7b\/0e4\/e0be7b0e4e1598f16ffe1bf0af99fa6d.png\" width=\"1280\" height=\"720\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e0b\/e7b\/0e4\/e0be7b0e4e1598f16ffe1bf0af99fa6d.png\"\/><figcaption><\/figcaption><\/figure>\n<p>\u0412 \u0434\u0430\u043d\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0440\u0435\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u0441\u0435 \u043f\u043e\u0441\u0442\u0430\u0432\u043b\u0435\u043d\u043d\u044b\u0435 \u0446\u0435\u043b\u0438:<\/p>\n<ol>\n<li>\n<p>\u0410\u043a\u0442\u0443\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b\u0438 \u044d\u0442\u0430\u043f\u044b \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f kubernetes \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430, \u0440\u0430\u0441\u0448\u0438\u0440\u0438\u0432 \u043e\u043f\u0438\u0441\u0430\u043d\u0438\u0435 \u0438 \u0434\u043e\u0431\u0430\u0432\u0438\u0432 \u0430\u043a\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438.<\/p>\n<\/li>\n<li>\n<p>\u041f\u043e\u043a\u0430\u0437\u0430\u043b\u0438, \u0447\u0442\u043e \u0434\u0430\u0436\u0435 \u0431\u0430\u0437\u043e\u0432\u0430\u044f \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f, \u0431\u0435\u0437 \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a \u043f\u043e\u0434 \u0432\u044b\u0441\u043e\u043a\u0443\u044e \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u044c, \u0438\u043d\u0442\u0435\u0433\u0440\u0430\u0446\u0438\u0439 \u0441 \u0432\u043d\u0435\u0448\u043d\u0438\u043c\u0438 \u0441\u0438\u0441\u0442\u0435\u043c\u0430\u043c\u0438 &#8212;  \u0442\u0440\u0443\u0434\u043e\u0435\u043c\u043a\u0438\u0439 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0438 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0445\u043e\u0440\u043e\u0448\u0435\u0433\u043e \u043f\u043e\u043d\u0438\u043c\u0430\u043d\u0438\u044f \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u0430.<\/p>\n<\/li>\n<li>\n<p>\u0412\u0441\u0435 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b \u0432\u044b\u043f\u0443\u0441\u043a\u0430\u044e\u0442\u0441\u044f \u0447\u0435\u0440\u0435\u0437 key-keeper (client) \u0432 \u0446\u0435\u043d\u0442\u0440\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d\u043d\u043e\u043c Vault \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0438 \u043f\u0435\u0440\u0435\u0432\u044b\u043f\u0443\u0441\u043a\u0430\u044e\u0442\u0441\u044f, \u0435\u0441\u043b\u0438 \u0438\u0441\u0442\u0435\u043a\u0430\u0435\u0442 \u0441\u0440\u043e\u043a \u0433\u043e\u0434\u043d\u043e\u0441\u0442\u0438.<\/p>\n<\/li>\n<\/ol>\n<p>\u0412 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0445\u043e\u0447\u0443 \u0437\u0430\u0442\u0440\u043e\u043d\u0443\u0442\u044c \u0432\u043e\u043f\u0440\u043e\u0441 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f Kubernetes \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0447\u0435\u0440\u0435\u0437 Terraform \u0438 \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u043f\u0435\u0440\u0432\u0443\u044e \u0432\u0435\u0440\u0441\u0438\u044e \u043e\u0431\u043b\u0430\u0447\u043d\u043e\u0433\u043e kubernetes \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441 \u041e\u0431\u043b\u0430\u043a\u0430, \u0438\u043c\u0435\u044e\u0449\u0443\u044e \u043f\u043e\u0447\u0442\u0438 \u0442\u0430\u043a\u043e\u0439 \u0436\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b, \u0447\u0442\u043e \u0438 <a href=\"https:\/\/cloud.yandex.ru\/docs\/managed-kubernetes\/?from=int-console-empty-state\" rel=\"noopener noreferrer nofollow\">Yandex Managed Service for Kubernetes<\/a>.<\/p>\n<p>\u041f\u043e\u0434\u043f\u0438\u0441\u044b\u0432\u0430\u0439\u0442\u0435\u0441\u044c, \u0441\u0442\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u043b\u0435\u0446 \u0432\u0432\u0435\u0440\u0445, \u0435\u0441\u043b\u0438 \u043f\u043e\u043d\u0440\u0430\u0432\u0438\u043b\u0430\u0441\u044c \u0441\u0442\u0430\u0442\u044c\u044f. <\/p>\n<p>\u0416\u0434\u0435\u043c \u0432\u0430\u0441 \u043d\u0430 \u043e\u0431\u0441\u0443\u0436\u0434\u0435\u043d\u0438\u044f \u043d\u0430\u0448\u0435\u0439 \u0440\u0430\u0431\u043e\u0442\u044b \u0432 <a href=\"https:\/\/t.me\/fraima_ru\" rel=\"noopener noreferrer nofollow\">https:\/\/t.me\/fraima_ru<\/a><\/p>\n<p>\u041f\u043e\u043b\u0435\u0437\u043d\u043e\u0435 \u0447\u0442\u0438\u0432\u043e:<\/p>\n<p><a href=\"https:\/\/habr.com\/ru\/post\/673730\/\" rel=\"noopener noreferrer nofollow\">\u0421\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b K8S \u0438\u043b\u0438 \u043a\u0430\u043a \u0440\u0430\u0441\u043f\u0443\u0442\u0430\u0442\u044c \u0432\u0435\u0440\u043c\u0438\u0448\u0435\u043b\u044c \u0427\u0430\u0441\u0442\u044c 1<\/a><\/p>\n<p><a href=\"https:\/\/habr.com\/ru\/post\/695344\/\" rel=\"noopener noreferrer nofollow\">\u0421\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b K8S \u0438\u043b\u0438 \u043a\u0430\u043a \u0440\u0430\u0441\u043f\u0443\u0442\u0430\u0442\u044c \u0432\u0435\u0440\u043c\u0438\u0448\u0435\u043b\u044c \u0427\u0430\u0441\u0442\u044c 2<\/a><\/p>\n<p><a href=\"https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way\" rel=\"noopener noreferrer nofollow\">https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way<\/a><\/p>\n<p><a href=\"https:\/\/github.com\/fraima\/key-keeper\" rel=\"noopener noreferrer nofollow\">Key-keeper Github<\/a><\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"v-portal\" style=\"display:none;\"><\/div>\n<\/div>\n<p> <!----> <!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/post\/699074\/\"> https:\/\/habr.com\/ru\/post\/699074\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0414\u043e\u0431\u0440\u044b\u0439 \u041a\u043e\u0442\u00a0<a href=\"https:\/\/t.me\/Dobry_kot\" rel=\"noopener noreferrer nofollow\">Telegram<\/a>.<\/p>\n<p>\u041e\u0442 \u043a\u043e\u043b\u043b\u0435\u043a\u0442\u0438\u0432\u0430 FR-Solutions \u0438 \u043f\u0440\u0438 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u043a\u0435\u00a0<a href=\"https:\/\/habr.com\/users\/irbgeo\" rel=\"noopener noreferrer nofollow\">@irbgeo<\/a>\u00a0<a href=\"https:\/\/t.me\/irbgeo\" rel=\"noopener noreferrer nofollow\">Telegram<\/a>\u00a0: \u041f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0441\u0435\u0440\u0438\u044e \u0441\u0442\u0430\u0442\u0435\u0439 \u043e  K8S.<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<p>\u0426\u0435\u043b\u0438 \u0434\u0430\u043d\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438:<\/p>\n<ol>\n<li>\n<p>\u0410\u043a\u0442\u0443\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043f\u043e\u0440\u044f\u0434\u043e\u043a \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f kubernetes, \u043e\u043f\u0438\u0441\u0430\u043d\u043d\u044b\u0439 \u0432\u0441\u0435\u043c\u0438 \u043b\u044e\u0431\u0438\u043c\u044b\u043c <a href=\"https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way\" rel=\"noopener noreferrer nofollow\">Kelsey Hightower<\/a>.<\/p>\n<\/li>\n<li>\n<p>\u0414\u043e\u043a\u0430\u0437\u0430\u0442\u044c \u0447\u0442\u043e &#171;kubernetes \u044d\u0442\u043e \u0432\u0441\u0435\u0433\u043e 5-\u0431\u0438\u043d\u0430\u0440\u0435\u0439&#187; \u0438 &#171;kubernetes \u044d\u0442\u043e \u043f\u0440\u043e\u0441\u0442\u043e&#187; &#8212; \u044d\u0442\u043e \u043d\u0435\u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0435 \u0441\u0443\u0436\u0434\u0435\u043d\u0438\u0435.<\/p>\n<\/li>\n<li>\n<p>\u0414\u043e\u0431\u0430\u0432\u0438\u0442\u044c <a href=\"https:\/\/github.com\/fraima\/key-keeper\" rel=\"noopener noreferrer nofollow\">Key-keeper<\/a> \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e kubernetes \u0434\u043b\u044f \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430\u043c\u0438.<\/p>\n<\/li>\n<\/ol>\n<h2>\u0418\u0437 \u0447\u0435\u0433\u043e \u0441\u043e\u0441\u0442\u043e\u0438\u0442 Kubernetes?<\/h2>\n<p>\u0412\u0441\u0435 \u043c\u044b \u043f\u043e\u043c\u043d\u0438\u043c \u0448\u0443\u0442\u043a\u0443 &#171;kubernetes \u044d\u0442\u043e \u0432\u0441\u0435\u0433\u043e 5-\u0431\u0438\u043d\u0430\u0440\u0435\u0439&#187;:<\/p>\n<ol>\n<li>\n<p>etcd<\/p>\n<\/li>\n<li>\n<p>kube-apiserver<\/p>\n<\/li>\n<li>\n<p>kube-controller-manager<\/p>\n<\/li>\n<li>\n<p>kube-scheduler<\/p>\n<\/li>\n<li>\n<p>kubelet<\/p>\n<\/li>\n<\/ol>\n<p>\u041d\u043e, \u0435\u0441\u043b\u0438 \u043c\u044b \u0431\u0443\u0434\u0435\u043c \u043e\u043f\u0435\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0442\u043e\u043b\u044c\u043a\u043e \u0438\u043c\u0438, \u0442\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440 \u0432\u044b \u043d\u0435 \u0441\u043e\u0431\u0435\u0440\u0435\u0442\u0435. <strong>\u041f\u043e\u0447\u0435\u043c\u0443 \u0436\u0435?<\/strong><\/p>\n<p><strong><u>kubelet -\u0443 \u0442\u0440\u0435\u0431\u0443\u044e\u0442\u0441\u044f \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0435 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u044b \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>Container Runtime Interface &#8212; CRI (containerd, cri-o, docker, etc.). <\/p>\n<\/li>\n<\/ol>\n<p><strong><u>CRI &#8212; \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>RUNC \u0431\u0438\u0431\u043b\u0438\u043e\u0442\u0435\u043a\u0430, \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430\u043c\u0438. <\/p>\n<\/li>\n<\/ol>\n<p><strong><u>Certificates:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>(cfssl, kubeadm, key-keeper) \u0442\u0440\u0435\u0431\u0443\u044e\u0442\u0441\u044f \u0434\u043b\u044f \u0432\u044b\u043f\u0443\u0441\u043a\u0430 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u043e\u0432.<\/p>\n<\/li>\n<\/ol>\n<p><strong><u>\u041f\u0440\u043e\u0447\u0435\u0435:<\/u><\/strong><\/p>\n<ol>\n<li>\n<p>kubectl (\u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 kubernetes)             &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<li>\n<p>crictl (\u0434\u043b\u044f \u0443\u0434\u043e\u0431\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b \u0441 CRI)              &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<li>\n<p>etcdctl (\u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 etcd \u043d\u0430 \u043c\u0430\u0441\u0442\u0435\u0440\u0430\u0445)   &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<li>\n<p>kubeadm (\u0434\u043b\u044f \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430)          &#8212; \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/li>\n<\/ol>\n<p><strong><u>\u0422\u0430\u043a\u0438\u043c \u043e\u0431\u0440\u0430\u0437\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442\u044c kubernetes, \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u043c\u0438\u043d\u0438\u043c\u0443\u043c 8 \u0431\u0438\u043d\u0430\u0440\u0435\u0439.<\/u><\/strong><\/p>\n<h2>\u042d\u0442\u0430\u043f\u044b \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 K8S<\/h2>\n<ol>\n<li>\n<p>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 linux \u043c\u0430\u0448\u0438\u043d, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u0431\u0443\u0434\u0435\u0442 \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442 control-plane \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<\/li>\n<li>\n<p>\u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043d\u0430 \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0445 linux \u043c\u0430\u0448\u0438\u043d\u0430\u0445:<\/p>\n<ol>\n<li>\n<p>\u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0431\u0430\u0437\u043e\u0432\u044b\u0445 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 (\u0434\u043b\u044f \u043e\u0431\u0441\u043b\u0443\u0436\u0438\u0432\u0430\u043d\u0438\u044f linux).<\/p>\n<\/li>\n<li>\n<p>\u0440\u0430\u0431\u043e\u0442\u0430 \u0441 modprobe.<\/p>\n<\/li>\n<li>\n<p>\u0440\u0430\u0431\u043e\u0442\u0430 \u0441 sysctls.<\/p>\n<\/li>\n<li>\n<p>\u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u044b\u0445 \u0434\u043b\u044f \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0431\u0438\u043d\u0430\u0440\u0435\u0439.<\/p>\n<\/li>\n<li>\n<p>\u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 \u0434\u043b\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0445 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432.<\/p>\n<\/li>\n<\/ol>\n<\/li>\n<li>\n<p>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 Vault \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0430.<\/p>\n<\/li>\n<li>\n<p>\u0413\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u044f static-pod \u043c\u0430\u043d\u0438\u0444\u0435\u0441\u0442\u043e\u0432.<\/p>\n<\/li>\n<li>\n<p>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430.<\/p>\n<\/li>\n<\/ol>\n<p>\u041a\u0430\u043a \u0432\u0438\u0434\u0438\u043c, \u0432\u0441\u0435\u0433\u043e 5-\u0442\u044c \u044d\u0442\u0430\u043f\u043e\u0432 &#8212; \u043d\u0438\u0447\u0435\u0433\u043e \u0441\u043b\u043e\u0436\u043d\u043e\u0433\u043e)<\/p>\n<h2>\u041d\u0443 \u0447\u0442\u043e, \u0434\u0430\u0432\u0430\u0439\u0442\u0435 \u043d\u0430\u0447\u043d\u0435\u043c!<\/h2>\n<p>1) <strong>\u0421\u043e\u0437\u0434\u0430\u0435\u043c 3 \u0423\u0437\u043b\u0430 \u043f\u043e\u0434 \u043c\u0430\u0441\u0442\u0435\u0440\u0430 \u0438 \u043f\u0440\u0438\u0432\u044f\u0437\u044b\u0432\u0430\u0435\u043c \u043a \u043d\u0438\u043c DNS \u0438\u043c\u0435\u043d\u0430 \u043f\u043e \u043c\u0430\u0441\u043a\u0435:<\/strong><\/p>\n<p><code>master-${INDEX}.${CLUSTER_NAME}.${BASE_DOMAIN}<\/code><\/p>\n<p>** <strong>\u0412\u0410\u0416\u041d\u041e<\/strong>: <code>${INDEX}<\/code> \u0434\u043e\u043b\u0436\u0435\u043d \u043d\u0430\u0447\u0438\u043d\u0430\u0442\u044c\u0441\u044f \u0441 <strong>0<\/strong> \u0438\u0437-\u0437\u0430 \u0440\u0435\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438 \u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0438\u043d\u0434\u0435\u043a\u0441\u043e\u0432 \u0432 \u043c\u043e\u0434\u0443\u043b\u0435 \u0442\u0435\u0440\u0440\u0430\u0444\u043e\u0440\u043c \u0434\u043b\u044f VAULT, \u043d\u043e \u043e \u043d\u0435\u043c \u043f\u043e\u0437\u0436\u0435.<\/p>\n<details class=\"spoiler\">\n<summary>environments<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## REQUIRED VARS:  export BASE_DOMAIN=dobry-kot.ru export CLUSTER_NAME=example export BASE_CLUSTER_DOMAIN=${CLUSTER_NAME}.${BASE_DOMAIN}  # \u041f\u043e\u0440\u0442\u044b \u0434\u043b\u044f ETCD export ETCD_SERVER_PORT=\"2379\" export ETCD_PEER_PORT=\"2380\" export ETCD_METRICS_PORT=\"2381\"  # \u041f\u043e\u0440\u0442\u044b \u0434\u043b\u044f KUBERNETES export KUBE_APISERVER_PORT=\"6443\" export KUBE_CONTROLLER_MANAGER_PORT=\"10257\" export KUBE_SCHEDULER_PORT=\"10259\"  # \u0423\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0435 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 1, 3, 5 export MASTER_COUNT=1  # \u0414\u043b\u044f Kube-apiserver  export ETCD_SERVERS=$(echo \\ $(for INDEX in `seq 0 $(($MASTER_COUNT-1))`; \\ do \\ echo https:\/\/master-${INDEX}.${BASE_CLUSTER_DOMAIN}:${ETCD_SERVER_PORT} ; \\ done) |  sed \"s\/,\/\/\" |  sed \"s\/ \/,\/g\")  # \u0414\u043b\u044f \u0444\u043e\u0440\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f ETCD \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 export ETCD_INITIAL_CLUSTER=$(echo \\ $(for INDEX in `seq 0 $(($MASTER_COUNT-1))`; \\ do \\ echo master-${INDEX}.${BASE_CLUSTER_DOMAIN}=https:\/\/master-${INDEX}.${BASE_CLUSTER_DOMAIN}:${ETCD_PEER_PORT} ; \\ done) |  sed \"s\/,\/\/\" |  sed \"s\/ \/,\/g\")   export KUBERNETES_VERSION=\"v1.23.12\" export ETCD_VERSION=\"3.5.3-0\" export ETCD_TOOL_VERSION=\"v3.5.5\" export RUNC_VERSION=\"v1.1.3\" export CONTAINERD_VERSION=\"1.6.8\" export CRICTL_VERSION=$(echo $KUBERNETES_VERSION |  sed -r 's\/^v([0-9]*).([0-9]*).([0-9]*)\/v\\1.\\2.0\/')  export BASE_K8S_PATH=\"\/etc\/kubernetes\"  export SERVICE_CIDR=\"29.64.0.0\/16\" # \u041d\u0435 \u043e\u0431\u0438\u0436\u0430\u0439\u0442\u0435\u0441\u044c - regexp \u0441\u0430\u043c\u0438 \u043d\u0430\u043f\u0438\u0448\u0438\u0442\u0435) export SERVICE_DNS=\"29.64.0.10\"  export VAULT_MASTER_TOKEN=\"hvs.vy0dqWuHkJpiwtYhw4yPT6cC\" export VAULT_SERVER=\"http:\/\/193.32.219.99:9200\/\"  export VAULT_MASTER_TOKEN=\"root\" export VAULT_SERVER=\"http:\/\/master-0.${CLUSTER_NAME}.${BASE_DOMAIN}:9200\/\"<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0415\u0441\u043b\u0438 \u0432\u044b \u0438\u0437\u0443\u0430\u0447\u0430\u043b\u0438 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044e \u043e\u0442 <a href=\"https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way\" rel=\"noopener noreferrer nofollow\">Kelsey Hightower<\/a>, \u0442\u043e \u0437\u0430\u043c\u0435\u0447\u0430\u043b\u0438, \u0447\u0442\u043e \u0432 \u043e\u0441\u043d\u043e\u0432\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 \u043b\u0435\u0436\u0430\u0442 ip \u0430\u0434\u0440\u0435\u0441\u0430 \u0443\u0437\u043b\u043e\u0432. \u0414\u0430\u043d\u043d\u044b\u0439 \u043f\u043e\u0434\u0445\u043e\u0434 \u0440\u0430\u0431\u043e\u0447\u0438\u0439, \u043d\u043e \u043c\u0435\u043d\u0435\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u044b\u0439, \u0434\u043b\u044f \u043f\u0440\u043e\u0441\u0442\u043e\u0442\u044b \u043e\u0431\u0441\u043b\u0443\u0436\u0438\u0432\u0430\u043d\u0438\u044f \u0438 \u0434\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0435\u0439 \u0448\u0430\u0431\u043b\u043e\u043d\u0438\u0437\u0430\u0446\u0438\u0438 \u043b\u0443\u0447\u0448\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0437\u0430\u0440\u0430\u043d\u0435\u0435 \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u044b\u0435 \u043d\u0430\u043c FQDN \u043c\u0430\u0441\u043a\u0438, \u043a\u0430\u043a \u044f \u0443\u043a\u0430\u0437\u044b\u0432\u0430\u043b \u0434\u043b\u044f \u043c\u0430\u0441\u0442\u0435\u0440\u043e\u0432 \u0432\u044b\u0448\u0435.<\/p>\n<p>2) <strong>\u0421\u043a\u0430\u0447\u0438\u0432\u0430\u0435\u043c \u0432\u0441\u0435 \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u044b\u0435 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c K8S \u0431\u0438\u043d\u0430\u0440\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b.<\/strong><\/p>\n<ul>\n<li>\n<p>\u0412 \u0434\u0430\u043d\u043d\u043e\u043c \u0441\u0435\u0442\u0430\u043f\u0435 \u044f \u043d\u0435 \u0431\u0443\u0434\u0443 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c RPM \u0438\u043b\u0438 DEB \u043f\u0430\u043a\u0435\u0442\u044b, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0441\u0442\u0430\u0440\u0430\u0442\u044c\u0441\u044f \u0434\u0435\u0442\u0430\u043b\u044c\u043d\u043e \u043f\u043e\u043a\u0430\u0437\u0430\u0442\u044c, \u0438\u0437 \u0447\u0435\u0433\u043e \u0441\u043e\u0441\u0442\u043e\u0438\u0442 \u0432\u0441\u044f \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u044f.<\/p>\n<\/li>\n<\/ul>\n<details class=\"spoiler\">\n<summary>download components<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. wget -O \/usr\/bin\/key-keeper   \"https:\/\/storage.yandexcloud.net\/m.images\/key-keeper-T2?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=YCAJEhOlYpv1GRY7hghCojNX5%2F20221020%2Fru-central1%2Fs3%2Faws4_request&amp;X-Amz-Date=20221020T123413Z&amp;X-Amz-Expires=2592000&amp;X-Amz-Signature=138701723B70343E38D82791A28AD1DB87040677F7C94D83610FF26ED9AF1954&amp;X-Amz-SignedHeaders=host\" wget -O \/usr\/bin\/kubectl       https:\/\/storage.googleapis.com\/kubernetes-release\/release\/${KUBERNETES_VERSION}\/bin\/linux\/amd64\/kubectl wget -O \/usr\/bin\/kubelet       https:\/\/storage.googleapis.com\/kubernetes-release\/release\/${KUBERNETES_VERSION}\/bin\/linux\/amd64\/kubelet wget -O \/usr\/bin\/kubeadm       https:\/\/storage.googleapis.com\/kubernetes-release\/release\/${KUBERNETES_VERSION}\/bin\/linux\/amd64\/kubeadm wget -O \/usr\/bin\/runc          https:\/\/github.com\/opencontainers\/runc\/releases\/download\/${RUNC_VERSION}\/runc.amd64 wget -O \/tmp\/etcd.tar.gz       https:\/\/github.com\/etcd-io\/etcd\/releases\/download\/${ETCD_TOOL_VERSION}\/etcd-${ETCD_TOOL_VERSION}-linux-amd64.tar.gz wget -O \/tmp\/containerd.tar.gz https:\/\/github.com\/containerd\/containerd\/releases\/download\/v${CONTAINERD_VERSION}\/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz wget -O \/tmp\/crictl.tar.gz     https:\/\/github.com\/kubernetes-sigs\/cri-tools\/releases\/download\/${CRICTL_VERSION}\/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz  chmod +x \/usr\/bin\/key-keeper  chmod +x \/usr\/bin\/kubelet  chmod +x \/usr\/bin\/kubectl  chmod +x \/usr\/bin\/kubeadm chmod +x \/usr\/bin\/runc  mkdir -p \/tmp\/containerd mkdir -p \/tmp\/etcd  tar -C \"\/tmp\/etcd\"        -xvf \/tmp\/etcd.tar.gz tar -C \"\/tmp\/containerd\"  -xvf \/tmp\/containerd.tar.gz tar -C \"\/usr\/bin\"         -xvf \/tmp\/crictl.tar.gz  cp \/tmp\/etcd\/etcd*\/etcdctl \/usr\/bin\/ cp \/tmp\/containerd\/bin\/*   \/usr\/bin\/<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>3) <strong>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432:<\/strong><\/p>\n<p>\u0421\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0432 \u043d\u0430\u0448\u0435\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0438 \u0432\u0441\u0435\u0433\u043e 3 (key-keeper, kubelet, containerd)<\/p>\n<details class=\"spoiler\">\n<summary>containerd.service<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE FOR CONTAINERD  cat &lt;&lt;EOF > \/etc\/systemd\/system\/containerd.service [Unit] Description=containerd container runtime Documentation=https:\/\/containerd.io After=network.target  [Service] ExecStartPre=\/sbin\/modprobe overlay ExecStart=\/usr\/bin\/containerd Restart=always RestartSec=5 Delegate=yes KillMode=process OOMScoreAdjust=-999 LimitNOFILE=1048576 LimitNPROC=infinity LimitCORE=infinity  [Install] WantedBy=multi-user.target EOF <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>key-keeper.service<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE FOR KEY-KEEPER cat &lt;&lt;EOF > \/etc\/systemd\/system\/key-keeper.service [Unit] Description=key-keeper-agent  Wants=network-online.target After=network-online.target  [Service] ExecStart=\/usr\/bin\/key-keeper -config-dir ${BASE_K8S_PATH}\/pki -config-regexp .*vault-config   Restart=always StartLimitInterval=0 RestartSec=10  [Install] WantedBy=multi-user.target EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet.service<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE FOR KUBELET cat &lt;&lt;EOF > \/etc\/systemd\/system\/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=https:\/\/kubernetes.io\/docs\/home\/ Wants=network-online.target After=network-online.target   [Service] ExecStart=\/usr\/bin\/kubelet  Restart=always StartLimitInterval=0 RestartSec=10  [Install] WantedBy=multi-user.target EOF <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet.d\/conf<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE-CONFIG FOR KUBELET  mkdir -p \/etc\/systemd\/system\/kubelet.service.d  cat &lt;&lt;EOF > \/etc\/systemd\/system\/kubelet.service.d\/10-fraima.conf [Service] EnvironmentFile=-${BASE_K8S_PATH}\/kubelet\/service\/kubelet-args.env  ExecStart= ExecStart=\/usr\/bin\/kubelet \\ \\$KUBELET_HOSTNAME \\ \\$KUBELET_CNI_ARGS \\ \\$KUBELET_RUNTIME_ARGS \\ \\$KUBELET_AUTH_ARGS \\ \\$KUBELET_CONFIGS_ARGS \\ \\$KUBELET_BASIC_ARGS \\ \\$KUBELET_KUBECONFIG_ARGS EOF<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<details class=\"spoiler\">\n<summary>kubelet-args.env<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON EACH MASTER. ## SETUP SERVICE-CONFIG FOR KUBELET  mkdir -p  ${BASE_K8S_PATH}\/kubelet\/service\/  cat &lt;&lt;EOF > ${BASE_K8S_PATH}\/kubelet\/service\/kubelet-args.env KUBELET_HOSTNAME=\"\" KUBELET_BASIC_ARGS=\"     --register-node=true     --cloud-provider=external     --image-pull-progress-deadline=2m     --feature-gates=RotateKubeletServerCertificate=true     --cert-dir=\/etc\/kubernetes\/pki\/certs\/kubelet     --authorization-mode=Webhook     --v=2 \" KUBELET_AUTH_ARGS=\"     --anonymous-auth=\"false\" \" KUBELET_CNI_ARGS=\"     --cni-bin-dir=\/opt\/cni\/bin     --cni-conf-dir=\/etc\/cni\/net.d     --network-plugin=cni \" KUBELET_CONFIGS_ARGS=\"     --config=${BASE_K8S_PATH}\/kubelet\/config.yaml     --root-dir=\/var\/lib\/kubelet     --register-node=true     --image-pull-progress-deadline=2m     --v=2 \" KUBELET_KUBECONFIG_ARGS=\"     --kubeconfig=${BASE_K8S_PATH}\/kubelet\/kubeconfig \" KUBELET_RUNTIME_ARGS=\"     --container-runtime=remote     --container-runtime-endpoint=\/run\/containerd\/containerd.sock     --pod-infra-container-image=k8s.gcr.io\/pause:3.6 \" EOF<\/code><\/pre>\n<p>** \u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0435\u0441\u043b\u0438 \u0432\u044b \u0432 \u043f\u0435\u0440\u0441\u043f\u0435\u043a\u0442\u0438\u0432\u0435 \u0431\u0443\u0434\u0435\u0442\u0435 \u0440\u0430\u0437\u0432\u043e\u0440\u0430\u0447\u0438\u0432\u0430\u0442\u044c K8S \u0432 \u043e\u0431\u043b\u0430\u043a\u0435 \u0438 \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0435\u0433\u043e \u0441 \u043d\u0438\u043c, \u0442\u043e \u0441\u0442\u0430\u0432\u044c\u0442\u0435  &#8212;cloud-provider=external<\/p>\n<p>*** \u041f\u043e\u043b\u0435\u0437\u043d\u043e\u0439 \u0444\u0438\u0447\u0435\u0439 \u043c\u043e\u0436\u0435\u0442 \u043e\u043a\u0430\u0437\u0430\u0442\u044c\u0441\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043b\u0435\u0439\u0431\u043b\u0438\u043d\u0433 \u043d\u043e\u0434\u044b \u043f\u0440\u0438 \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0430\u0446\u0438\u0438 \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435<br \/><code>--node-labels=node.kubernetes.io\/master,foo=bar<br \/><\/code><br \/><strong><u>\u041d\u0438\u0436\u0435 \u043f\u0440\u0438\u0432\u0435\u0434\u0435\u043d \u0441\u043f\u0438\u0441\u043e\u043a \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u0445 \u043c\u0435\u0442\u043e\u043a, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u043e\u0436\u043d\u043e \u043c\u0435\u043d\u044f\u0442\u044c:<\/u><\/strong><br \/><a href=\"http:\/\/kubelet.kubernetes.io\" rel=\"noopener noreferrer nofollow\">kubelet.kubernetes.io<\/a><br \/><a href=\"http:\/\/node.kubernetes.io\" rel=\"noopener noreferrer nofollow\">node.kubernetes.io<\/a><br \/><a href=\"http:\/\/beta.kubernetes.io\/arch\" rel=\"noopener noreferrer nofollow\">beta.kubernetes.io\/arch<\/a>,<br \/><a href=\"http:\/\/beta.kubernetes.io\/instance-type\" rel=\"noopener noreferrer nofollow\">beta.kubernetes.io\/instance-type<\/a>,<br \/><a href=\"http:\/\/beta.kubernetes.io\/os\" rel=\"noopener noreferrer nofollow\">beta.kubernetes.io\/os<\/a>,<br \/><a href=\"http:\/\/failure-domain.beta.kubernetes.io\/region\" rel=\"noopener noreferrer nofollow\">failure-domain.beta.kubernetes.io\/region<\/a>,<br \/><a href=\"http:\/\/failure-domain.beta.kubernetes.io\/zone\" rel=\"noopener noreferrer nofollow\">failure-domain.beta.kubernetes.io\/zone<\/a>,<br \/><a href=\"http:\/\/kubernetes.io\/arch\" rel=\"noopener noreferrer nofollow\">kubernetes.io\/arch<\/a>,<br \/><a href=\"http:\/\/kubernetes.io\/hostname\" rel=\"noopener noreferrer nofollow\">kubernetes.io\/hostname<\/a>,<br \/><a href=\"http:\/\/kubernetes.io\/os\" rel=\"noopener noreferrer nofollow\">kubernetes.io\/os<\/a>,<br \/><a href=\"http:\/\/node.kubernetes.io\/instance-type\" rel=\"noopener noreferrer nofollow\">node.kubernetes.io\/instance-type<\/a>,<br \/><a href=\"http:\/\/topology.kubernetes.io\/region\" rel=\"noopener noreferrer nofollow\">topology.kubernetes.io\/region<\/a>,<br \/><a href=\"http:\/\/topology.kubernetes.io\/zone\" rel=\"noopener noreferrer nofollow\">topology.kubernetes.io\/zone<\/a><\/p>\n<p><strong><u>\u0414\u043b\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u0430, \u043d\u0435\u043b\u044c\u0437\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u0435 \u043b\u0435\u0439\u0431\u043b \u043d\u0435 \u0438\u0437 \u0441\u043f\u0438\u0441\u043a\u0430:<br \/><\/u><\/strong><code>--node-labels=node-role.kubernetes.io\/master<\/code><\/p>\n<\/div>\n<\/details>\n<p>4) <strong>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 Vault.<\/strong><\/p>\n<p>\u041a\u0430\u043a \u043c\u044b \u0440\u0430\u043d\u0435\u0435 \u043f\u0438\u0441\u0430\u043b\u0438, \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u044b \u0431\u0443\u0434\u0435\u043c \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0447\u0435\u0440\u0435\u0437 \u0446\u0435\u043d\u0442\u0440\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d\u043d\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u0449\u0435 Vault.<\/p>\n<p>\u0414\u043b\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u0430 \u043c\u044b \u0440\u0430\u0437\u043c\u0435\u0441\u0442\u0438\u043c \u043e\u043f\u043e\u0440\u043d\u044b\u0439 <strong>Vault server<\/strong> \u043d\u0430 <code>master-0<\/code> \u0432 \u0440\u0435\u0436\u0438\u043c\u0435 <code>dev<\/code> \u0441 \u0443\u0436\u0435 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u043c \u0441\u0442\u043e\u0440\u0435\u0434\u0436\u043e\u043c \u0438 \u0434\u0435\u0444\u043e\u043b\u0442\u043d\u044b\u043c \u0442\u043e\u043a\u0435\u043d\u043e\u043c \u0434\u043b\u044f \u0443\u0434\u043e\u0431\u0441\u0442\u0432\u0430.<\/p>\n<details class=\"spoiler\">\n<summary>Vault<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">## RUN ON MASTER-0. export VAULT_VERSION=\"1.12.1\" export VAULT_ADDR=${VAULT_SERVER} export VAULT_TOKEN=${VAULT_MASTER_TOKEN}  wget -O<\/code><\/pre>\n<\/div>\n<\/details>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-341394","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/341394","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=341394"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/341394\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=341394"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=341394"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=341394"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}