{"id":483249,"date":"2026-06-11T05:20:51","date_gmt":"2026-06-11T05:20:51","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=483249"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=483249","title":{"rendered":"\u041a\u0430\u043a \u0441\u043e\u0437\u0434\u0430\u0442\u044c \u0441\u043e\u0431\u0441\u0442\u0432\u0435\u043d\u043d\u044b\u0439 \u0434\u0438\u0441\u0442\u0440\u0438\u0431\u0443\u0442\u0438\u0432 Kubernetes I"},"content":{"rendered":"<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p><a href=\"https:\/\/github.com\/ingresslabs\/k8s-release\" rel=\"noopener noreferrer nofollow\">k8s-release<\/a> \u2014 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u0432\u043e\u0441\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u043c\u044b\u0445 DEB\/RPM-\u043f\u0430\u043a\u0435\u0442\u043e\u0432 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432 Kubernetes (kubelet, kube-apiserver, etcd, Flannel, Istio \u0438 \u0434\u0440.) \u0432 \u0433\u0435\u0440\u043c\u0435\u0442\u0438\u0447\u043d\u043e\u043c Docker-\u043e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u0438. \u0421\u043e\u0437\u0434\u0430\u0451\u0442 \u043f\u043e\u0434\u043f\u0438\u0441\u0430\u043d\u043d\u044b\u0435 apt\/yum-\u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438, SBOM, \u0445\u0435\u0448-\u0441\u0443\u043c\u043c\u044b \u0438 airgap-\u0431\u0430\u043d\u0434\u043b\u044b \u0434\u043b\u044f \u0437\u0430\u043a\u0440\u044b\u0442\u044b\u0445, \u0440\u0435\u0433\u0443\u043b\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0438\u0437\u043e\u043b\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0445 \u0441\u0440\u0435\u0434. \u0412 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438 \u0442\u0430\u043a\u0436\u0435 \u043b\u0435\u0436\u0438\u0442 <a href=\"https:\/\/github.com\/ingresslabs\/k8s-release\/tree\/main\/terraform\/k2vm-ha-lab\" rel=\"noopener noreferrer nofollow\">Terraform-\u043c\u043e\u0434\u0443\u043b\u044c<\/a> \u0434\u043b\u044f \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f HA Kubernetes \u043d\u0430 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c Firecracker: 3 control-plane, 1+ workers, Flannel, Istio, SELinux enforcing.<\/p>\n<p>k8s-release \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442 \u043b\u044e\u0431\u044b\u0435 \u043a\u043e\u043c\u0431\u0438\u043d\u0430\u0446\u0438\u0438 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432 \u0447\u0435\u0440\u0435\u0437 \u0444\u043b\u0430\u0433 &#8212;component:<\/p>\n<pre><code class=\"bash\">.\/k8s-release build v1.36.1 --component kubelet,kubectl --format deb,rpm.\/k8s-release build v1.36.1 --component all --format deb.\/k8s-release build 3.5.22 --component etcd --format rpm.\/k8s-release build 1.30.0 --component istio --format deb,rpm<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:87px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>\u041c\u043e\u0436\u043d\u043e \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0435 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u044b (kubelet, etcd, flannel, istio), all \u0434\u043b\u044f \u0432\u0441\u0435\u0433\u043e \u0441\u0440\u0430\u0437\u0443, \u0438 \u0444\u043e\u0440\u043c\u0430\u0442 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 \u2014 deb, rpm \u0438\u043b\u0438 \u043e\u0431\u0430.<\/p>\n<h3>\u041a\u0430\u043a \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c<\/h3>\n<h4>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430<\/h4>\n<pre><code class=\"css\">git clone https:\/\/github.com\/ingresslabs\/k8s-release.gitcd k8s-release\/terraform\/k2vm-ha-labcp terraform.tfvars.example terraform.tfvars<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>\u041d\u0430 \u0443\u0434\u0430\u043b\u0451\u043d\u043d\u043e\u043c \u0445\u043e\u0441\u0442\u0435 \u0434\u043e\u043b\u0436\u043d\u044b \u0431\u044b\u0442\u044c: Firecracker, Docker, \u044f\u0434\u0440\u043e, initrd, \u043c\u043e\u0434\u0443\u043b\u0438 \u0438 \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 rootfs.ext4 \u043f\u043e \u043f\u0443\u0442\u044f\u043c \u0438\u0437 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445.<\/p>\n<pre><code class=\"bash\">terraform initterraform planterraform applyterraform destroy<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<pre><code class=\"bash\">variable \"name\"                      { type = string; default = \"k8s-ha-lab\" }variable \"target_host\"               { type = string; default = \"\" }variable \"target_user\"               { type = string; default = \"root\" }variable \"github_repo\"               { type = string; default = \"ingresslabs\/k8s-release\" }variable \"github_run_id\"             { type = number; default = 26680027183 }variable \"control_plane_count\"       { type = number; default = 3 }variable \"worker_count\"              { type = number; default = 2 }variable \"kubernetes_version\"        { type = string; default = \"v1.36.1\" }variable \"subnet_prefix\"             { type = string; default = \"198.19.2\" }variable \"network_plugin\"            { type = string; default = \"flannel\" }variable \"firecracker_binary\"        { type = string; default = \"\/usr\/local\/bin\/firecracker\" }variable \"kernel_path\"               { type = string; default = \"\/opt\/fc-lab\/vmlinux-5.15-generic\" }variable \"initrd_path\"               { type = string; default = \"\/opt\/fc-lab\/initrd-5.15-generic.img\" }variable \"kernel_modules_tar_path\"   { type = string; default = \"\/opt\/fc-lab\/modules-5.15-generic.tar.gz\" }variable \"base_rootfs_path\"          { type = string; default = \"\/opt\/fc-lab\/rootfs.ext4\" }variable \"vcpu_count\"                { type = number; default = 2 }variable \"guest_selinux_mode\"        { type = string; default = \"permissive\" }variable \"enable_istio\"              { type = bool;   default = false }variable \"redeploy_token\"            { type = string; default = \"\" }<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<pre><code class=\"bash\"># main.tf \u2014 \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 Kubernetes HA \u043d\u0430 Firecracker \u0447\u0435\u0440\u0435\u0437 k8s-releaseterraform {  required_version = \"&gt;= 1.5.0\"  required_providers {    local = { source = \"hashicorp\/local\", version = \"~&gt; 2.5\" }    null  = { source = \"hashicorp\/null\", version = \"~&gt; 3.2\" }  }}<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>\u041e\u0431\u0430 \u043f\u0440\u043e\u0435\u043a\u0442\u0430 \u2014 \u0447\u0430\u0441\u0442\u0438 \u043e\u0434\u043d\u043e\u0439 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u044b. \u041f\u0435\u0440\u0432\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u043e\u043f\u0438\u0441\u044b\u0432\u0430\u0435\u0442 \u0441\u0430\u043c\u043e\u0434\u0435\u043b\u044c\u043d\u0443\u044e Firecracker CI-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443 (\u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u0437\u0430 10 \u043c\u0441, \u043a\u0435\u0448\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435, \u0441\u043d\u0430\u043f\u0448\u043e\u0442\u044b). \u0412\u0442\u043e\u0440\u043e\u0439 \u043f\u0440\u043e\u0435\u043a\u0442, k8s-release, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u0442\u0443 \u0436\u0435 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u0438 \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f Kubernetes \u043d\u0430 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c Firecracker. Terraform-\u043c\u043e\u0434\u0443\u043b\u044c \u0432 k8s-release \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0438\u0440\u0443\u0435\u0442 \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 HA-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u043f\u043e \u0442\u0435\u043c \u0436\u0435 \u043f\u0440\u0438\u043d\u0446\u0438\u043f\u0430\u043c, \u0447\u0442\u043e \u043e\u043f\u0438\u0441\u0430\u043d\u044b \u0432 \u0441\u0442\u0430\u0442\u044c\u0435.<\/p>\n<hr\/>\n<p>\u042d\u0442\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u043d\u0435\u043f\u043e\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e \u043d\u0430 \u0445\u043e\u0441\u0442\u0435 Linux. \u041e\u043d\u043e \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442 \u043e\u0434\u043d\u0443 \u043f\u0435\u0440\u0435\u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c\u0443\u044e \u0433\u043e\u0441\u0442\u0435\u0432\u0443\u044e rootfs, \u043a\u043b\u043e\u043d\u0438\u0440\u0443\u0435\u0442 \u0435\u0451 \u043d\u0430 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0434\u0438\u0441\u043a\u043e\u0432 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c Firecracker, \u0437\u0430\u0433\u0440\u0443\u0436\u0430\u0435\u0442:<\/p>\n<ul>\n<li>\n<p>3 \u0443\u0437\u043b\u0430 control-plane kubeadm<\/p>\n<\/li>\n<li>\n<p>\u0432\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u044b\u0439 etcd, \u043f\u043e \u043e\u0434\u043d\u043e\u043c\u0443 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u0443 \u043d\u0430 \u0443\u0437\u0435\u043b control-plane<\/p>\n<\/li>\n<li>\n<p>2 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u0430 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e<\/p>\n<\/li>\n<li>\n<p>\u0438 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442 Kubernetes API \u0447\u0435\u0440\u0435\u0437 \u0445\u043e\u0441\u0442-\u043f\u0440\u043e\u043a\u0441\u0438 HAProxy.<\/p>\n<\/li>\n<\/ul>\n<h3>\u041f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0435 \u0442\u0440\u0435\u0431\u043e\u0432\u0430\u043d\u0438\u044f<\/h3>\n<p>\u0412\u044b\u043f\u043e\u043b\u043d\u044f\u0439\u0442\u0435 \u0432\u0441\u0451 \u043d\u0438\u0436\u0435 \u043e\u0442 root \u043d\u0430 \u0445\u043e\u0441\u0442\u0435 Linux, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u0431\u0443\u0434\u0435\u0442 \u0437\u0430\u043f\u0443\u0449\u0435\u043d Firecracker. \u041d\u0430 \u0445\u043e\u0441\u0442\u0435 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u044b:<\/p>\n<ul>\n<li>\n<p>Docker<\/p>\n<\/li>\n<li>\n<p>Firecracker<\/p>\n<\/li>\n<li>\n<p>curl, ip, iptables, ssh, scp, ssh-keygen<\/p>\n<\/li>\n<li>\n<p>unsquashfs \u0438\u0437 squashfs-tools<\/p>\n<\/li>\n<li>\n<p>mkfs.ext4, e2fsck, resize2fs \u0438\u0437 e2fsprogs<\/p>\n<\/li>\n<li>\n<p>mount, umount, truncate, chroot<\/p>\n<\/li>\n<li>\n<p>\u0438\u0441\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0434\u043e\u0441\u0442\u0443\u043f \u0432 \u0438\u043d\u0442\u0435\u0440\u043d\u0435\u0442 \u0434\u043b\u044f:<\/p>\n<\/li>\n<li>\n<p>\u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0438 \u043e\u0431\u0440\u0430\u0437\u0430 \u044f\u0434\u0440\u0430 LinuxKit<\/p>\n<\/li>\n<li>\n<p>\u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044f Ubuntu rootfs \u0438\u0437 Firecracker CI<\/p>\n<\/li>\n<li>\n<p>\u043f\u0430\u043a\u0435\u0442\u043e\u0432 Kubernetes \u0441 pkgs.k8s.io<\/p>\n<\/li>\n<li>\n<p>\u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u044f Docker containerd<\/p>\n<\/li>\n<li>\n<p>\u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044f \u043c\u0430\u043d\u0438\u0444\u0435\u0441\u0442\u0430 Flannel<\/p>\n<\/li>\n<li>\n<p>\u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0438 \u043e\u0431\u0440\u0430\u0437\u0430 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430 HAProxy \u0412\u044b\u0431\u0435\u0440\u0438\u0442\u0435 \u043f\u043e\u0434\u0441\u0435\u0442\u044c, \u043d\u0435 \u043f\u0435\u0440\u0435\u0441\u0435\u043a\u0430\u044e\u0449\u0443\u044e\u0441\u044f \u0441 \u0434\u0440\u0443\u0433\u0438\u043c\u0438 \u043c\u043e\u0441\u0442\u0430\u043c\u0438 \u043d\u0430 \u0445\u043e\u0441\u0442\u0435.<\/p>\n<\/li>\n<\/ul>\n<h3>\u042d\u043a\u0441\u043f\u043e\u0440\u0442 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445 \u043b\u0430\u0431\u043e\u0440\u0430\u0442\u043e\u0440\u0438\u0438<\/h3>\n<p>\u042d\u0442\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u043e \u0441\u043e\u0437\u0434\u0430\u0451\u0442 3 \u0443\u0437\u043b\u0430 control-plane \u0438 2 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u0430 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e. \u0418\u0437\u043c\u0435\u043d\u0438\u0442\u0435 WORKER_COUNT, \u0435\u0441\u043b\u0438 \u043d\u0443\u0436\u043d\u043e \u043c\u0435\u043d\u044c\u0448\u0435 \u0438\u043b\u0438 \u0431\u043e\u043b\u044c\u0448\u0435 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u043e\u0432.<\/p>\n<pre><code class=\"bash\">export RUN_ROOT=\/var\/lib\/firecracker-kubeadm-haexport CACHE_ROOT=\/var\/cache\/firecracker-kubeadm-haexport CONTROL_PLANE_COUNT=3export WORKER_COUNT=2export NODE_COUNT=\"$((CONTROL_PLANE_COUNT + WORKER_COUNT))\"export SUBNET_PREFIX=198.19.0export GATEWAY=\"${SUBNET_PREFIX}.1\"export CIDR=\"${SUBNET_PREFIX}.0\/24\"export API_LB_IP=\"${SUBNET_PREFIX}.5\"export API_LB_PORT=6443export PRIMARY_CONTROL_PLANE_IP=\"${SUBNET_PREFIX}.10\"export BRIDGE_NAME=k8sha198export TAP_PREFIX=k8sha198export FIRECRACKER_BIN=\/usr\/local\/bin\/firecrackerexport FIRECRACKER_ARCH=\"$(uname -m)\"export LINUXKIT_KERNEL_IMAGE=linuxkit\/kernel:6.12.59export ROOTFS_SIZE_GIB=12export CONTROL_PLANE_MEM_MIB=2048export WORKER_MEM_MIB=1536export VCPU_COUNT=2export KUBERNETES_MINOR=v1.36export KUBERNETES_VERSION=v1.36.1export POD_CIDR=10.244.0.0\/16export SERVICE_CIDR=10.96.0.0\/12export CNI_PLUGINS_VERSION=v1.3.0export NETWORK_PLUGIN=flannelexport FLANNEL_MANIFEST_URL=\"https:\/\/github.com\/flannel-io\/flannel\/releases\/latest\/download\/kube-flannel.yml\"export HAPROXY_IMAGE=haproxy:3.2.19-alpineexport API_LB_CONTAINER_NAME=\"kubeadm-ha-api-lb-${BRIDGE_NAME}\"export GUEST_SSH_KEY=\"${CACHE_ROOT}\/lab_ssh_key\"export GUEST_SSH_PUB=\"${GUEST_SSH_KEY}.pub\"export ARTIFACT_PREFIX=\/var\/log\/kubeadm-ha-labexport KERNEL_BOOT_ARGS=\"console=ttyS0 reboot=k panic=1 pci=off root=\/dev\/vda rw random.trust_cpu=on systemd.mask=serial-getty@ttyS0.service systemd.mask=systemd-random-seed.service\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0435\u0439 \u0445\u043e\u0441\u0442\u0430 \u0438 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0433\u043e SSH-\u043a\u043b\u044e\u0447\u0430<\/h3>\n<pre><code class=\"bash\">command -v awk chroot curl docker e2fsck grep ip iptables mkfs.ext4 mount \\  resize2fs sha256sum sort ssh scp ssh-keygen truncate umount unsquashfs &gt;\/dev\/nulltest -x \"${FIRECRACKER_BIN}\"mkdir -p \"${RUN_ROOT}\" \"${CACHE_ROOT}\"if [[ ! -f \"${GUEST_SSH_KEY}\" || ! -f \"${GUEST_SSH_PUB}\" ]]; then  ssh-keygen -q -t ed25519 -N \"\" -f \"${GUEST_SSH_KEY}\"fi<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u0435 \u0432\u0441\u043f\u043e\u043c\u043e\u0433\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u0444\u0443\u043d\u043a\u0446\u0438\u0439<\/h3>\n<pre><code class=\"bash\">node_role() {  if (( $1 &lt; CONTROL_PLANE_COUNT )); then    echo control-plane  else    echo worker  fi}node_ip() {  printf '%s.%d\\n' \"${SUBNET_PREFIX}\" \"$((10 + $1))\"}node_name() {  printf 'k8s-%02d\\n' \"$1\"}node_tap() {  printf '%s%d\\n' \"${TAP_PREFIX}\" \"$1\"}node_mac() {  printf '06:36:00:00:00:%02x\\n' \"$((16 + $1))\"}node_mem() {  if [[ \"$(node_role \"$1\")\" == \"control-plane\" ]]; then    echo \"${CONTROL_PLANE_MEM_MIB}\"  else    echo \"${WORKER_MEM_MIB}\"  fi}SSH_OPTS=(-i \"${GUEST_SSH_KEY}\" -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=\/dev\/null -o LogLevel=ERROR -o ConnectTimeout=5)node_ssh() {  local idx=\"$1\"  shift  ssh \"${SSH_OPTS[@]}\" \"root@$(node_ip \"${idx}\")\" \"$@\"}server_ssh() {  node_ssh 0 \"$@\"}copy_guest_file() {  local remote_path=\"$1\"  local local_path=\"$2\"  if server_ssh \"test -f $(printf '%q' \"${remote_path}\")\" &gt;\/dev\/null 2&gt;&amp;1; then    server_ssh \"cat $(printf '%q' \"${remote_path}\")\" &gt; \"${local_path}\"  fi}wait_for_ssh() {  local ip=\"$1\"  for _ in $(seq 1 120); do    if ssh \"${SSH_OPTS[@]}\" \"root@${ip}\" true &gt;\/dev\/null 2&gt;&amp;1; then      return 0    fi    sleep 2  done  return 1}wait_for_node_ready() {  local name=\"$1\"  for _ in $(seq 1 240); do    if server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf get node ${name} --no-headers 2&gt;\/dev\/null | awk '\\$2==\\\"Ready\\\"{ok=1} END{exit(ok?0:1)}'\"; then      return 0    fi    sleep 2  done  return 1}<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0421\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u0435 \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0433\u043e \u044f\u0434\u0440\u0430 \u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 rootfs<\/h3>\n<p>\u0412 \u044d\u0442\u043e\u043c \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f:<\/p>\n<ul>\n<li>\n<p>\u043e\u0431\u0440\u0430\u0437 \u044f\u0434\u0440\u0430 LinuxKit<\/p>\n<\/li>\n<li>\n<p>\u043f\u043e\u0441\u043b\u0435\u0434\u043d\u044f\u044f Ubuntu squashfs \u0438\u0437 Firecracker CI \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0439 \u0432\u0435\u0440\u0441\u0438\u0438 Firecracker major\/minor<\/p>\n<\/li>\n<\/ul>\n<h3>\u0417\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u044f\u0434\u0440\u0430 LinuxKit \u0438 \u0438\u0437\u0432\u043b\u0435\u0447\u0435\u043d\u0438\u0435 vmlinux<\/h3>\n<pre><code class=\"bash\">export LINUXKIT_KEY=\"$(printf '%s\\n' \"${LINUXKIT_KERNEL_IMAGE}\" | sha256sum | awk '{print substr($1,1,16)}')\"export LINUXKIT_CACHE_DIR=\"${CACHE_ROOT}\/downloads\/linuxkit-${LINUXKIT_KEY}\"export KERNEL_PATH=\"${LINUXKIT_CACHE_DIR}\/vmlinux\"export KERNEL_BZIMAGE_PATH=\"${LINUXKIT_CACHE_DIR}\/kernel\"export KERNEL_MODULES_TAR_PATH=\"${LINUXKIT_CACHE_DIR}\/kernel.tar\"export KERNEL_DEV_TAR_PATH=\"${LINUXKIT_CACHE_DIR}\/kernel-dev.tar\"mkdir -p \"${LINUXKIT_CACHE_DIR}\"if [[ ! -f \"${KERNEL_PATH}\" || ! -f \"${KERNEL_MODULES_TAR_PATH}\" ]]; then  rm -rf \"${LINUXKIT_CACHE_DIR}.tmp\"  mkdir -p \"${LINUXKIT_CACHE_DIR}.tmp\"  docker pull \"${LINUXKIT_KERNEL_IMAGE}\" &gt;\/dev\/null  cid=\"$(docker create \"${LINUXKIT_KERNEL_IMAGE}\" \/bin\/sh)\"  docker cp \"${cid}:\/kernel\" \"${LINUXKIT_CACHE_DIR}.tmp\/kernel\"  docker cp \"${cid}:\/kernel.tar\" \"${LINUXKIT_CACHE_DIR}.tmp\/kernel.tar\"  docker cp \"${cid}:\/kernel-dev.tar\" \"${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar\"  docker rm -f \"${cid}\" &gt;\/dev\/null  export LINUXKIT_HEADERS_DIR=\"$(    tar -tf \"${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar\" \\      | sed -n 's#^\\(usr\/src\/linux-headers-[^\/]*\\)\/scripts\/extract-vmlinux$#\\1#p' \\      | head -n 1  )\"  tar -xOf \"${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar\" \"${LINUXKIT_HEADERS_DIR}\/scripts\/extract-vmlinux\" &gt; \"${LINUXKIT_CACHE_DIR}.tmp\/extract-vmlinux\"  chmod +x \"${LINUXKIT_CACHE_DIR}.tmp\/extract-vmlinux\"  \"${LINUXKIT_CACHE_DIR}.tmp\/extract-vmlinux\" \"${LINUXKIT_CACHE_DIR}.tmp\/kernel\" &gt; \"${LINUXKIT_CACHE_DIR}.tmp\/vmlinux\"  mv \"${LINUXKIT_CACHE_DIR}.tmp\/kernel\" \"${KERNEL_BZIMAGE_PATH}\"  mv \"${LINUXKIT_CACHE_DIR}.tmp\/vmlinux\" \"${KERNEL_PATH}\"  mv \"${LINUXKIT_CACHE_DIR}.tmp\/kernel.tar\" \"${KERNEL_MODULES_TAR_PATH}\"  mv \"${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar\" \"${KERNEL_DEV_TAR_PATH}\"  rm -rf \"${LINUXKIT_CACHE_DIR}.tmp\"fi<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0421\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u0435 Ubuntu rootfs squashfs \u0438\u0437 Firecracker CI<\/h3>\n<pre><code class=\"bash\">export FIRECRACKER_CI_VERSION=\"$(\"${FIRECRACKER_BIN}\" --version | awk '{print $2}' | sed 's\/\\.[0-9]*$\/\/')\"export UBUNTU_KEY=\"$(  curl -fsSL \"https:\/\/s3.amazonaws.com\/spec.ccfc.min?prefix=firecracker-ci\/${FIRECRACKER_CI_VERSION}\/${FIRECRACKER_ARCH}\/ubuntu-&amp;list-type=2\" \\    | grep -oP \"(?&lt;=&lt;Key&gt;)(firecracker-ci\/${FIRECRACKER_CI_VERSION}\/${FIRECRACKER_ARCH}\/ubuntu-[0-9]+\\.[0-9]+\\.squashfs)(?=&lt;\/Key&gt;)\" \\    | sort -V | tail -1)\"export ROOTFS_SQUASHFS_PATH=\"${CACHE_ROOT}\/downloads\/$(basename \"${UBUNTU_KEY}\")\"mkdir -p \"${CACHE_ROOT}\/downloads\"if [[ ! -f \"${ROOTFS_SQUASHFS_PATH}\" ]]; then  curl -fsSL \"https:\/\/s3.amazonaws.com\/spec.ccfc.min\/${UBUNTU_KEY}\" -o \"${ROOTFS_SQUASHFS_PATH}\"fi<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 ext4 rootfs<\/h3>\n<p>\u041f\u0440\u0435\u043e\u0431\u0440\u0430\u0437\u0443\u0435\u0442 squashfs \u0432 \u043e\u0431\u0440\u0430\u0437 ext4 \u0438 \u0432\u043d\u0435\u0434\u0440\u044f\u0435\u0442 \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0439 SSH-\u043a\u043b\u044e\u0447.<\/p>\n<pre><code class=\"bash\">export BASE_KEY=\"$(  {    sha256sum \"${ROOTFS_SQUASHFS_PATH}\"    sha256sum \"${GUEST_SSH_PUB}\"  } | sha256sum | awk '{print substr($1,1,16)}')\"export BASE_ROOTFS_PATH=\"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.ext4\"mkdir -p \"${CACHE_ROOT}\/base\"if [[ ! -s \"${BASE_ROOTFS_PATH}\" ]]; then  rm -rf \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\"  rm -f \"${BASE_ROOTFS_PATH}.tmp\"  mkdir -p \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\"  unsquashfs -d \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\" \"${ROOTFS_SQUASHFS_PATH}\" &gt;\/dev\/null  mkdir -p \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/root\/.ssh\"  chmod 700 \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/root\/.ssh\"  cp \"${GUEST_SSH_PUB}\" \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/root\/.ssh\/authorized_keys\"  chmod 600 \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/root\/.ssh\/authorized_keys\"  if [[ -f \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/etc\/ssh\/sshd_config\" ]]; then    sed -i 's\/^#\\?PermitRootLogin.*\/PermitRootLogin yes\/' \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/etc\/ssh\/sshd_config\" || true    sed -i 's\/^#\\?PubkeyAuthentication.*\/PubkeyAuthentication yes\/' \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/etc\/ssh\/sshd_config\" || true    sed -i 's\/^#\\?PasswordAuthentication.*\/PasswordAuthentication no\/' \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\/etc\/ssh\/sshd_config\" || true  fi  chown -R root:root \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\"  truncate -s 4G \"${BASE_ROOTFS_PATH}.tmp\"  mkfs.ext4 -d \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\/rootfs\" -F \"${BASE_ROOTFS_PATH}.tmp\" &gt;\/dev\/null  mv \"${BASE_ROOTFS_PATH}.tmp\" \"${BASE_ROOTFS_PATH}\"  rm -rf \"${CACHE_ROOT}\/base\/ubuntu-${BASE_KEY}.rootfs\"fi<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u043f\u0435\u0440\u0435\u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c\u043e\u0433\u043e \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0433\u043e \u043e\u0431\u0440\u0430\u0437\u0430 \u0441 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u043a\u043e\u0439 kubeadm<\/h3>\n<p>\u0423\u0441\u0442\u0430\u043d\u0430\u0432\u043b\u0438\u0432\u0430\u0435\u0442:<\/p>\n<ul>\n<li>\n<p>\u043c\u043e\u0434\u0443\u043b\u0438 \u044f\u0434\u0440\u0430 \u0438\u0437 \u0430\u0440\u0445\u0438\u0432\u0430 \u044f\u0434\u0440\u0430 LinuxKit<\/p>\n<\/li>\n<li>\n<p>containerd<\/p>\n<\/li>\n<li>\n<p>kubelet, kubeadm, kubectl<\/p>\n<\/li>\n<li>\n<p>\u043f\u043b\u0430\u0433\u0438\u043d\u044b CNI<\/p>\n<\/li>\n<li>\n<p>\u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u044b Kubernetes \u0438 \u0431\u0430\u0437\u043e\u0432\u0443\u044e \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0443 \u0441\u043b\u0443\u0436\u0431<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"bash\">export PREPARED_KEY=\"$(  {    sha256sum \"${BASE_ROOTFS_PATH}\" \"${KERNEL_PATH}\" \"${GUEST_SSH_PUB}\" \"${KERNEL_MODULES_TAR_PATH}\"    printf 'kubernetes_minor=%s\\n' \"${KUBERNETES_MINOR}\"    printf 'kubernetes_version=%s\\n' \"${KUBERNETES_VERSION}\"    printf 'cni_plugins=%s\\n' \"${CNI_PLUGINS_VERSION}\"    printf 'rootfs_size_gib=%s\\n' \"${ROOTFS_SIZE_GIB}\"    printf 'generation=manual-kubeadm-firecracker-ha-v1\\n'  } | sha256sum | awk '{print substr($1,1,16)}')\"export PREPARED_ROOTFS_PATH=\"${CACHE_ROOT}\/prepared-${PREPARED_KEY}.ext4\"if [[ ! -s \"${PREPARED_ROOTFS_PATH}\" ]]; then  cp --reflink=auto \"${BASE_ROOTFS_PATH}\" \"${PREPARED_ROOTFS_PATH}.tmp\" 2&gt;\/dev\/null || cp \"${BASE_ROOTFS_PATH}\" \"${PREPARED_ROOTFS_PATH}.tmp\"  e2fsck -fy \"${PREPARED_ROOTFS_PATH}.tmp\" &gt;\/dev\/null 2&gt;&amp;1 || true  truncate -s \"${ROOTFS_SIZE_GIB}G\" \"${PREPARED_ROOTFS_PATH}.tmp\"  resize2fs \"${PREPARED_ROOTFS_PATH}.tmp\" &gt;\/dev\/null  export MNT=\"${CACHE_ROOT}\/mnt-${PREPARED_KEY}\"  mkdir -p \"${MNT}\"  mount -o loop \"${PREPARED_ROOTFS_PATH}.tmp\" \"${MNT}\"  mount -t proc proc \"${MNT}\/proc\"  mount -t sysfs sysfs \"${MNT}\/sys\"  mount --bind \/dev \"${MNT}\/dev\"  mount -t devpts devpts \"${MNT}\/dev\/pts\"  mount --bind \/run \"${MNT}\/run\"  rm -f \"${MNT}\/etc\/resolv.conf\"  printf 'nameserver 1.1.1.1\\nnameserver 8.8.8.8\\noptions timeout:2 attempts:3\\n' &gt; \"${MNT}\/etc\/resolv.conf\"  rm -rf \"${MNT}\/lib\/modules\"  mkdir -p \"${MNT}\/lib\"  tar -xf \"${KERNEL_MODULES_TAR_PATH}\" -C \"${MNT}\" .\/lib\/modules  chmod 1777 \"${MNT}\/tmp\"  mkdir -p \"${MNT}\/dev\/pts\" \"${MNT}\/var\/cache\/apt\/archives\/partial\" \"${MNT}\/var\/lib\/apt\/lists\/partial\" \"${MNT}\/var\/log\/apt\"  touch \"${MNT}\/var\/log\/dpkg.log\"  mkdir -p \"${MNT}\/root\/.ssh\"  chmod 700 \"${MNT}\/root\/.ssh\"  cp \"${GUEST_SSH_PUB}\" \"${MNT}\/root\/.ssh\/authorized_keys\"  chmod 600 \"${MNT}\/root\/.ssh\/authorized_keys\"  if [[ -f \"${MNT}\/etc\/ssh\/sshd_config\" ]]; then    sed -i 's\/^#\\?PermitRootLogin.*\/PermitRootLogin yes\/' \"${MNT}\/etc\/ssh\/sshd_config\" || true    sed -i 's\/^#\\?PubkeyAuthentication.*\/PubkeyAuthentication yes\/' \"${MNT}\/etc\/ssh\/sshd_config\" || true    sed -i 's\/^#\\?PasswordAuthentication.*\/PasswordAuthentication no\/' \"${MNT}\/etc\/ssh\/sshd_config\" || true  fi  chroot \"${MNT}\" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 update  DEBIAN_FRONTEND=noninteractive chroot \"${MNT}\" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 \\    -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install -y --no-install-recommends \\    apt-transport-https ca-certificates conntrack curl ebtables ethtool gpg iptables ipset jq openssh-server socat tar xz-utils  mkdir -p \"${MNT}\/etc\/apt\/keyrings\" \"${MNT}\/etc\/apt\/sources.list.d\"  docker_arch=\"$(chroot \"${MNT}\" dpkg --print-architecture)\"  docker_codename=\"$(awk -F= '\/^VERSION_CODENAME=\/{print $2}' \"${MNT}\/etc\/os-release\")\"  chroot \"${MNT}\" curl -fsSL \"https:\/\/download.docker.com\/linux\/ubuntu\/gpg\" | chroot \"${MNT}\" gpg --dearmor -o \/etc\/apt\/keyrings\/docker.gpg  printf 'deb [arch=%s signed-by=\/etc\/apt\/keyrings\/docker.gpg] https:\/\/download.docker.com\/linux\/ubuntu %s stable\\n' \"${docker_arch}\" \"${docker_codename}\" &gt; \"${MNT}\/etc\/apt\/sources.list.d\/docker.list\"  chroot \"${MNT}\" curl -fsSL \"https:\/\/pkgs.k8s.io\/core:\/stable:\/${KUBERNETES_MINOR}\/deb\/Release.key\" | chroot \"${MNT}\" gpg --dearmor -o \/etc\/apt\/keyrings\/kubernetes-apt-keyring.gpg  printf 'deb [signed-by=\/etc\/apt\/keyrings\/kubernetes-apt-keyring.gpg] https:\/\/pkgs.k8s.io\/core:\/stable:\/%s\/deb\/ \/\\n' \"${KUBERNETES_MINOR}\" &gt; \"${MNT}\/etc\/apt\/sources.list.d\/kubernetes.list\"  chroot \"${MNT}\" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 update  DEBIAN_FRONTEND=noninteractive chroot \"${MNT}\" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 \\    -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install -y containerd.io \\      \"kubelet=${KUBERNETES_VERSION#v}-*\" \"kubeadm=${KUBERNETES_VERSION#v}-*\" \"kubectl=${KUBERNETES_VERSION#v}-*\"  chroot \"${MNT}\" apt-mark hold kubelet kubeadm kubectl &gt;\/dev\/null 2&gt;&amp;1 || true  case \"$(uname -m)\" in    x86_64) cni_arch=amd64 ;;    aarch64) cni_arch=arm64 ;;    *) echo \"\u043d\u0435\u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u043c\u0430\u044f \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 CNI: $(uname -m)\" &gt;&amp;2; exit 1 ;;  esac  export CNI_ARCHIVE=\"${CACHE_ROOT}\/downloads\/cni-plugins-linux-${cni_arch}-${CNI_PLUGINS_VERSION}.tgz\"  if [[ ! -f \"${CNI_ARCHIVE}\" ]]; then    curl -fsSL \"https:\/\/github.com\/containernetworking\/plugins\/releases\/download\/${CNI_PLUGINS_VERSION}\/cni-plugins-linux-${cni_arch}-${CNI_PLUGINS_VERSION}.tgz\" -o \"${CNI_ARCHIVE}\"  fi  mkdir -p \"${MNT}\/opt\/cni\/bin\"  tar -C \"${MNT}\/opt\/cni\/bin\" -xzf \"${CNI_ARCHIVE}\"  pause_image=\"$(chroot \"${MNT}\" kubeadm config images list 2&gt;\/dev\/null | awk '\/pause\/{print $1; exit}')\"  mkdir -p \"${MNT}\/etc\/containerd\"  chroot \"${MNT}\" containerd config default &gt; \"${MNT}\/etc\/containerd\/config.toml\"  sed -i 's\/SystemdCgroup = false\/SystemdCgroup = true\/' \"${MNT}\/etc\/containerd\/config.toml\"  if [[ -n \"${pause_image}\" ]]; then    sed -i \"s#sandbox_image = \\\".*\\\"#sandbox_image = \\\"${pause_image}\\\"#\" \"${MNT}\/etc\/containerd\/config.toml\"  fi  mkdir -p \"${MNT}\/etc\/modules-load.d\" \"${MNT}\/etc\/sysctl.d\" \"${MNT}\/etc\/systemd\/system\/multi-user.target.wants\"  cat &gt; \"${MNT}\/etc\/modules-load.d\/kubernetes.conf\" &lt;&lt;'EOF'overlaybr_netfilterEOF  cat &gt; \"${MNT}\/etc\/sysctl.d\/99-kubernetes.conf\" &lt;&lt;'EOF'net.bridge.bridge-nf-call-iptables=1net.bridge.bridge-nf-call-ip6tables=1net.ipv4.ip_forward=1EOF  touch \"${MNT}\/etc\/cloud\/cloud-init.disabled\" 2&gt;\/dev\/null || true  chroot \"${MNT}\" update-alternatives --set iptables \/usr\/sbin\/iptables-legacy &gt;\/dev\/null 2&gt;&amp;1 || true  chroot \"${MNT}\" update-alternatives --set ip6tables \/usr\/sbin\/ip6tables-legacy &gt;\/dev\/null 2&gt;&amp;1 || true  ln -sf \/lib\/systemd\/system\/ssh.service \"${MNT}\/etc\/systemd\/system\/multi-user.target.wants\/ssh.service\"  ln -sf \/lib\/systemd\/system\/systemd-networkd.service \"${MNT}\/etc\/systemd\/system\/multi-user.target.wants\/systemd-networkd.service\"  umount \"${MNT}\/dev\/pts\" || true  umount \"${MNT}\/proc\" || true  umount \"${MNT}\/sys\" || true  umount \"${MNT}\/dev\" || true  umount \"${MNT}\/run\" || true  umount \"${MNT}\" || true  mv \"${PREPARED_ROOTFS_PATH}.tmp\" \"${PREPARED_ROOTFS_PATH}\"fi<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u043c\u043e\u0441\u0442\u0430, NAT \u0438 \u0431\u0430\u043b\u0430\u043d\u0441\u0438\u0440\u043e\u0432\u0449\u0438\u043a\u0430 API \u043d\u0430 \u0441\u0442\u043e\u0440\u043e\u043d\u0435 \u0445\u043e\u0441\u0442\u0430<\/h3>\n<p>\u042f\u0432\u043d\u044b\u0435 \u043f\u0440\u0430\u0432\u0438\u043b\u0430 FORWARD \u0432\u0430\u0436\u043d\u044b \u043d\u0430 \u0445\u043e\u0441\u0442\u0430\u0445, \u0433\u0434\u0435 \u043f\u043e\u043b\u0438\u0442\u0438\u043a\u0430 FORWARD \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e \u2014 DROP.<\/p>\n<pre><code class=\"bash\">docker rm -f \"${API_LB_CONTAINER_NAME}\" &gt;\/dev\/null 2&gt;&amp;1 || trueip link add name \"${BRIDGE_NAME}\" type bridge 2&gt;\/dev\/null || trueip addr add \"${GATEWAY}\/24\" dev \"${BRIDGE_NAME}\" 2&gt;\/dev\/null || trueip addr add \"${API_LB_IP}\/24\" dev \"${BRIDGE_NAME}\" 2&gt;\/dev\/null || trueip link set \"${BRIDGE_NAME}\" upsysctl -w net.ipv4.ip_forward=1 &gt;\/dev\/nulliptables -C FORWARD -i \"${BRIDGE_NAME}\" -j ACCEPT 2&gt;\/dev\/null || iptables -A FORWARD -i \"${BRIDGE_NAME}\" -j ACCEPTiptables -C FORWARD -o \"${BRIDGE_NAME}\" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2&gt;\/dev\/null || \\  iptables -A FORWARD -o \"${BRIDGE_NAME}\" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPTiptables -t nat -C POSTROUTING -s \"${CIDR}\" ! -o \"${BRIDGE_NAME}\" -j MASQUERADE 2&gt;\/dev\/null || \\  iptables -t nat -A POSTROUTING -s \"${CIDR}\" ! -o \"${BRIDGE_NAME}\" -j MASQUERADEcat &gt; \"${RUN_ROOT}\/haproxy.cfg\" &lt;&lt;EOFglobal  maxconn 2048defaults  mode tcp  timeout connect 5s  timeout client 60s  timeout server 60sfrontend kube_api  bind ${API_LB_IP}:${API_LB_PORT}  default_backend kube_apisbackend kube_apis  balance roundrobin  option tcp-check  default-server inter 2s fall 3 rise 2EOFfor i in $(seq 0 \"$((CONTROL_PLANE_COUNT - 1))\"); do  printf '  server cp%s %s:6443 check\\n' \"${i}\" \"$(node_ip \"${i}\")\" &gt;&gt; \"${RUN_ROOT}\/haproxy.cfg\"donedocker run -d --name \"${API_LB_CONTAINER_NAME}\" --network host \\  -v \"${RUN_ROOT}\/haproxy.cfg:\/usr\/local\/etc\/haproxy\/haproxy.cfg:ro\" \\  \"${HAPROXY_IMAGE}\" &gt;\/dev\/null<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0438 \u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c<\/h3>\n<p>\u041a\u0430\u0436\u0434\u0430\u044f \u0412\u041c \u043f\u043e\u043b\u0443\u0447\u0430\u0435\u0442 \u0441\u043e\u0431\u0441\u0442\u0432\u0435\u043d\u043d\u044b\u0439 \u043a\u043b\u043e\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0439 \u0434\u0438\u0441\u043a ext4, \u0438\u043c\u044f \u0445\u043e\u0441\u0442\u0430, \/etc\/hosts \u0438 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e systemd-networkd.<\/p>\n<pre><code class=\"bash\">for i in $(seq 0 \"$((NODE_COUNT - 1))\"); do  vm_dir=\"${RUN_ROOT}\/nodes\/node${i}\"  mkdir -p \"${vm_dir}\"  cp --reflink=auto \"${PREPARED_ROOTFS_PATH}\" \"${vm_dir}\/rootfs.ext4\" 2&gt;\/dev\/null || cp \"${PREPARED_ROOTFS_PATH}\" \"${vm_dir}\/rootfs.ext4\"  e2fsck -fy \"${vm_dir}\/rootfs.ext4\" &gt;\/dev\/null 2&gt;&amp;1 || true  mkdir -p \"${vm_dir}\/mnt\"  mount -o loop \"${vm_dir}\/rootfs.ext4\" \"${vm_dir}\/mnt\"  printf '%s\\n' \"$(node_name \"${i}\")\" &gt; \"${vm_dir}\/mnt\/etc\/hostname\"  {    printf '127.0.0.1 localhost\\n'    printf '127.0.1.1 %s\\n' \"$(node_name \"${i}\")\"    printf '%s api-lb\\n' \"${API_LB_IP}\"    for j in $(seq 0 \"$((NODE_COUNT - 1))\"); do      printf '%s %s\\n' \"$(node_ip \"${j}\")\" \"$(node_name \"${j}\")\"    done  } &gt; \"${vm_dir}\/mnt\/etc\/hosts\"  mkdir -p \"${vm_dir}\/mnt\/etc\/systemd\/network\" \"${vm_dir}\/mnt\/etc\/systemd\/system\/multi-user.target.wants\"  cat &gt; \"${vm_dir}\/mnt\/etc\/systemd\/network\/20-eth0.network\" &lt;&lt;EOF[Match]Name=eth0[Network]Address=$(node_ip \"${i}\")\/24Gateway=${GATEWAY}DNS=1.1.1.1DNS=8.8.8.8EOF  rm -f \"${vm_dir}\/mnt\/etc\/resolv.conf\"  printf 'nameserver 1.1.1.1\\nnameserver 8.8.8.8\\n' &gt; \"${vm_dir}\/mnt\/etc\/resolv.conf\"  rm -f \"${vm_dir}\/mnt\/etc\/machine-id\" \"${vm_dir}\/mnt\/var\/lib\/dbus\/machine-id\" 2&gt;\/dev\/null || true  touch \"${vm_dir}\/mnt\/etc\/machine-id\"  rm -rf \"${vm_dir}\/mnt\/etc\/kubernetes\" \"${vm_dir}\/mnt\/var\/lib\/etcd\" \"${vm_dir}\/mnt\/var\/lib\/cni\" \"${vm_dir}\/mnt\/var\/lib\/kubelet\" \"${vm_dir}\/mnt\/var\/lib\/containerd\" \"${vm_dir}\/mnt\/etc\/cni\/net.d\"  mkdir -p \"${vm_dir}\/mnt\/var\/lib\/containerd\" \"${vm_dir}\/mnt\/etc\/cni\/net.d\"  ln -sf \/lib\/systemd\/system\/ssh.service \"${vm_dir}\/mnt\/etc\/systemd\/system\/multi-user.target.wants\/ssh.service\"  ln -sf \/lib\/systemd\/system\/systemd-networkd.service \"${vm_dir}\/mnt\/etc\/systemd\/system\/multi-user.target.wants\/systemd-networkd.service\"  umount \"${vm_dir}\/mnt\"  tap=\"$(node_tap \"${i}\")\"  mac=\"$(node_mac \"${i}\")\"  mem=\"$(node_mem \"${i}\")\"  ip tuntap add dev \"${tap}\" mode tap 2&gt;\/dev\/null || true  ip link set \"${tap}\" master \"${BRIDGE_NAME}\"  ip link set \"${tap}\" up  cat &gt; \"${vm_dir}\/vm.json\" &lt;&lt;EOF{\"boot-source\":{\"kernel_image_path\":\"${KERNEL_PATH}\",\"boot_args\":\"${KERNEL_BOOT_ARGS}\"},\"drives\":[{\"drive_id\":\"rootfs\",\"path_on_host\":\"${vm_dir}\/rootfs.ext4\",\"is_root_device\":true,\"is_read_only\":false}],\"machine-config\":{\"vcpu_count\":${VCPU_COUNT},\"mem_size_mib\":${mem}},\"network-interfaces\":[{\"iface_id\":\"eth0\",\"host_dev_name\":\"${tap}\",\"guest_mac\":\"${mac}\"}],\"logger\":{\"log_path\":\"${vm_dir}\/firecracker.log\",\"level\":\"Info\",\"show_level\":true,\"show_log_origin\":true}}EOF  \"${FIRECRACKER_BIN}\" --api-sock \"${vm_dir}\/fc.sock\" --config-file \"${vm_dir}\/vm.json\" &gt; \"${vm_dir}\/console.log\" 2&gt;&amp;1 &amp;  echo $! &gt; \"${vm_dir}\/pid\"done<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041e\u0436\u0438\u0434\u0430\u043d\u0438\u0435 SSH \u0438 \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430 \u0443\u0437\u043b\u043e\u0432<\/h3>\n<pre><code class=\"bash\">for i in $(seq 0 \"$((NODE_COUNT - 1))\"); do  wait_for_ssh \"$(node_ip \"${i}\")\"donefor i in $(seq 0 \"$((NODE_COUNT - 1))\"); do  node_ssh \"${i}\" \"cat &gt;\/root\/prepare-node.sh\" &lt;&lt;'EOF'#!\/usr\/bin\/env bashset -euo pipefailswapoff -a || truesed -i '\/ swap \/ s\/^\/#\/' \/etc\/fstab 2&gt;\/dev\/null || truemodprobe overlay || truemodprobe br_netfilter || truesysctl --system &gt;\/dev\/nullsystemctl daemon-reload || truesystemctl enable --now ssh containerd kubelet &gt;\/dev\/null 2&gt;&amp;1 || truesystemctl restart containerdkubeadm reset -f &gt;\/dev\/null 2&gt;&amp;1 || truerm -rf \/etc\/kubernetes \/var\/lib\/etcd \/var\/lib\/cni \/var\/lib\/kubelet\/* \/etc\/cni\/net.d\/*mkdir -p \/etc\/cni\/net.dkubeadm config images pull --kubernetes-version=\"$(kubeadm version -o short)\" &gt;\/dev\/null 2&gt;&amp;1 || trueEOF  node_ssh \"${i}\" \"chmod +x \/root\/prepare-node.sh &amp;&amp; \/root\/prepare-node.sh\"done<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0418\u043d\u0438\u0446\u0438\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u044f \u043f\u0435\u0440\u0432\u043e\u0433\u043e \u0443\u0437\u043b\u0430 control-plane<\/h3>\n<pre><code class=\"bash\">server_ssh \"cat &gt;\/root\/init-primary.sh\" &lt;&lt;EOF#!\/usr\/bin\/env bashset -euo pipefailversion=\"${KUBERNETES_VERSION}\"kubeadm init \\  --kubernetes-version \"\\${version}\" \\  --control-plane-endpoint \"${API_LB_IP}:${API_LB_PORT}\" \\  --apiserver-advertise-address \"${PRIMARY_CONTROL_PLANE_IP}\" \\  --pod-network-cidr \"${POD_CIDR}\" \\  --service-cidr \"${SERVICE_CIDR}\" \\  --upload-certs \\  --ignore-preflight-errors=allmkdir -p \/root\/.kubecp \/etc\/kubernetes\/admin.conf \/root\/.kube\/configcertificate_key=\"\\$(kubeadm init phase upload-certs --upload-certs 2&gt;\/dev\/null | tail -n 1)\"join_cmd=\"\\$(kubeadm token create --print-join-command)\"printf '%s\\n' \"\\${certificate_key}\" &gt;\/root\/certificate-key.txtprintf '%s\\n' \"\\${join_cmd}\" &gt;\/root\/join-command.txtEOFserver_ssh \"chmod +x \/root\/init-primary.sh &amp;&amp; \/root\/init-primary.sh\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 Flannel<\/h3>\n<pre><code class=\"bash\">server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf apply -f $(printf '%q' \"${FLANNEL_MANIFEST_URL}\")\"for _ in $(seq 1 180); do  flannel_ns=\"$(server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf get daemonset -A --no-headers 2&gt;\/dev\/null | awk '\\$2==\\\"kube-flannel-ds\\\"{print \\$1; exit}'\" || true)\"  if [[ -n \"${flannel_ns}\" ]]; then    if server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n ${flannel_ns} rollout status daemonset\/kube-flannel-ds --timeout=5s\" &gt;\/dev\/null 2&gt;&amp;1; then      break    fi  fi  sleep 2done<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 \u0434\u0432\u0443\u0445 \u0434\u0440\u0443\u0433\u0438\u0445 \u0443\u0437\u043b\u043e\u0432 control-plane<\/h3>\n<pre><code class=\"bash\">join_cmd=\"$(server_ssh \"cat \/root\/join-command.txt\")\"certificate_key=\"$(server_ssh \"cat \/root\/certificate-key.txt\")\"for i in 1 2; do  ip=\"$(node_ip \"${i}\")\"  node_ssh \"${i}\" \"cat &gt;\/root\/join-control-plane.sh\" &lt;&lt;EOF#!\/usr\/bin\/env bashset -euo pipefail${join_cmd} --control-plane --certificate-key ${certificate_key} --apiserver-advertise-address ${ip} --ignore-preflight-errors=allEOF  node_ssh \"${i}\" \"chmod +x \/root\/join-control-plane.sh &amp;&amp; \/root\/join-control-plane.sh\"  wait_for_node_ready \"$(node_name \"${i}\")\"done<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u043e\u0432<\/h3>\n<pre><code class=\"bash\">for i in $(seq \"${CONTROL_PLANE_COUNT}\" \"$((NODE_COUNT - 1))\"); do  node_ssh \"${i}\" \"cat &gt;\/root\/join-worker.sh\" &lt;&lt;EOF#!\/usr\/bin\/env bashset -euo pipefail${join_cmd} --ignore-preflight-errors=allEOF  node_ssh \"${i}\" \"chmod +x \/root\/join-worker.sh &amp;&amp; \/root\/join-worker.sh\"  wait_for_node_ready \"$(node_name \"${i}\")\"done<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041e\u0436\u0438\u0434\u0430\u043d\u0438\u0435 \u043f\u043e\u043b\u043d\u043e\u0439 \u0433\u043e\u0442\u043e\u0432\u043d\u043e\u0441\u0442\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430<\/h3>\n<pre><code class=\"bash\">for _ in $(seq 1 240); do  server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf get nodes -o wide &gt; ${ARTIFACT_PREFIX}.nodes 2&gt;&amp;1 || truekubectl --kubeconfig \/etc\/kubernetes\/admin.conf get nodes --no-headers &gt; ${ARTIFACT_PREFIX}.nodes-plain 2&gt;&amp;1 || truekubectl --kubeconfig \/etc\/kubernetes\/admin.conf get endpoints kubernetes -o wide &gt; ${ARTIFACT_PREFIX}.apiserver-endpoints 2&gt;&amp;1 || true\" &gt;\/dev\/null 2&gt;&amp;1 || true  ready_count=\"$(server_ssh \"awk '\\$2==\\\"Ready\\\"{c++} END{print c+0}' ${ARTIFACT_PREFIX}.nodes-plain 2&gt;\/dev\/null || echo 0\")\"  control_plane_ready=\"$(server_ssh \"awk '\\$2==\\\"Ready\\\" &amp;&amp; \\$3 ~ \/control-plane\/ {c++} END{print c+0}' ${ARTIFACT_PREFIX}.nodes-plain 2&gt;\/dev\/null || echo 0\")\"  endpoint_count=\"$(server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf get endpoints kubernetes -o jsonpath='{range .subsets[*].addresses[*]}{.ip}{\\\"\\\\n\\\"}{end}' 2&gt;\/dev\/null | awk 'NF{c++} END{print c+0}' || echo 0\")\"  if [[ \"${ready_count}\" -ge \"${NODE_COUNT}\" &amp;&amp; \"${control_plane_ready}\" -ge \"${CONTROL_PLANE_COUNT}\" &amp;&amp; \"${endpoint_count}\" -ge \"${CONTROL_PLANE_COUNT}\" ]]; then    break  fi  sleep 3doneserver_ssh \"cat ${ARTIFACT_PREFIX}.nodes\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>\u041e\u0436\u0438\u0434\u0430\u0435\u043c\u044b\u0439 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442:<\/p>\n<pre><code class=\"bash\">k8s-00   Ready   control-planek8s-01   Ready   control-planek8s-02   Ready   control-planek8s-03   Ready   &lt;none&gt;k8s-04   Ready   &lt;none&gt;<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u041f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u0435 \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0436\u0434\u0435\u043d\u0438\u044f \u0447\u043b\u0435\u043d\u0441\u0442\u0432\u0430 etcd<\/h3>\n<pre><code class=\"bash\">etcd_pod=\"etcd-$(node_name 0)\"server_ssh \"kubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n kube-system exec ${etcd_pod} -- etcdctl --endpoints=https:\/\/127.0.0.1:2379 --cacert=\/etc\/kubernetes\/pki\/etcd\/ca.crt --cert=\/etc\/kubernetes\/pki\/etcd\/healthcheck-client.crt --key=\/etc\/kubernetes\/pki\/etcd\/healthcheck-client.key member list -w table &gt; ${ARTIFACT_PREFIX}.etcd-members 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n kube-system exec ${etcd_pod} -- etcdctl --endpoints=https:\/\/127.0.0.1:2379 --cacert=\/etc\/kubernetes\/pki\/etcd\/ca.crt --cert=\/etc\/kubernetes\/pki\/etcd\/healthcheck-client.crt --key=\/etc\/kubernetes\/pki\/etcd\/healthcheck-client.key endpoint status -w table &gt; ${ARTIFACT_PREFIX}.etcd-endpoints 2&gt;&amp;1\"server_ssh \"cat ${ARTIFACT_PREFIX}.etcd-members\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>\u0414\u043e\u043b\u0436\u043d\u043e \u043e\u0442\u043e\u0431\u0440\u0430\u0437\u0438\u0442\u044c\u0441\u044f \u0442\u0440\u0438 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u0430 etcd.<\/p>\n<h3>\u0417\u0430\u043f\u0443\u0441\u043a \u043f\u0440\u043e\u0432\u0435\u0440\u043e\u0447\u043d\u043e\u0433\u043e \u0442\u0435\u0441\u0442\u0430<\/h3>\n<pre><code class=\"bash\">server_ssh \"cat &gt;\/root\/run-smoke.sh\" &lt;&lt;'EOF'#!\/usr\/bin\/env bashset -euo pipefailkubectl --kubeconfig \/etc\/kubernetes\/admin.conf create namespace smoke --dry-run=client -o yaml | kubectl --kubeconfig \/etc\/kubernetes\/admin.conf apply -f -cat &lt;&lt;'YAML' | kubectl --kubeconfig \/etc\/kubernetes\/admin.conf apply -f -apiVersion: apps\/v1kind: DaemonSetmetadata:  name: node-smoke  namespace: smokespec:  selector:    matchLabels:      app: node-smoke  template:    metadata:      labels:        app: node-smoke    spec:      tolerations:        - operator: Exists      containers:        - name: node-smoke          image: busybox:1.36          command: [\"sh\", \"-lc\", \"sleep 3600\"]---apiVersion: apps\/v1kind: Deploymentmetadata:  name: echo  namespace: smokespec:  replicas: 2  selector:    matchLabels:      app: echo  template:    metadata:      labels:        app: echo    spec:      tolerations:        - operator: Exists      containers:        - name: echo          image: nginx:1.27-alpine          ports:            - containerPort: 80---apiVersion: v1kind: Servicemetadata:  name: echo  namespace: smokespec:  selector:    app: echo  ports:    - port: 80      targetPort: 80YAMLkubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n smoke rollout status daemonset\/node-smoke --timeout=240skubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n smoke rollout status deployment\/echo --timeout=240skubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n kube-system rollout status deployment\/coredns --timeout=240scat &lt;&lt;'YAML' | kubectl --kubeconfig \/etc\/kubernetes\/admin.conf apply -f -apiVersion: batch\/v1kind: Jobmetadata:  name: dns-http  namespace: smokespec:  backoffLimit: 0  template:    spec:      restartPolicy: Never      tolerations:        - operator: Exists      containers:        - name: dns-http          image: busybox:1.36          command:            - sh            - -lc            - |              for _ in $(seq 1 30); do                nslookup echo.smoke.svc.cluster.local &amp;&amp; \\                wget -qO- http:\/\/echo.smoke.svc.cluster.local &gt;\/tmp\/echo.html &amp;&amp; \\                test -s \/tmp\/echo.html &amp;&amp; exit 0                sleep 2              done              exit 1YAMLkubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n smoke wait --for=condition=complete job\/dns-http --timeout=240skubectl --kubeconfig \/etc\/kubernetes\/admin.conf get nodes -o wide &gt; \/var\/log\/kubeadm-ha-lab.nodes 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf get pods -A -o wide &gt; \/var\/log\/kubeadm-ha-lab.pods 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf get svc -A -o wide &gt; \/var\/log\/kubeadm-ha-lab.services 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n smoke get pods -o wide &gt; \/var\/log\/kubeadm-ha-lab.smoke-pods 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n smoke get svc echo -o wide &gt; \/var\/log\/kubeadm-ha-lab.smoke-service 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf -n smoke logs job\/dns-http &gt; \/var\/log\/kubeadm-ha-lab.smoke-job.log 2&gt;&amp;1kubectl --kubeconfig \/etc\/kubernetes\/admin.conf config view --raw &gt; \/var\/log\/kubeadm-ha-lab.kubeconfig 2&gt;&amp;1EOFserver_ssh \"chmod +x \/root\/run-smoke.sh &amp;&amp; \/root\/run-smoke.sh\"server_ssh \"cat ${ARTIFACT_PREFIX}.smoke-pods\"server_ssh \"cat ${ARTIFACT_PREFIX}.smoke-job.log\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0421\u043e\u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0435 \u0430\u0440\u0442\u0435\u0444\u0430\u043a\u0442\u043e\u0432<\/h3>\n<pre><code class=\"bash\">mkdir -p \"${RUN_ROOT}\/artifacts\"copy_guest_file \"${ARTIFACT_PREFIX}.nodes\" \"${RUN_ROOT}\/artifacts\/nodes.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.pods\" \"${RUN_ROOT}\/artifacts\/pods.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.services\" \"${RUN_ROOT}\/artifacts\/services.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.apiserver-endpoints\" \"${RUN_ROOT}\/artifacts\/apiserver-endpoints.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.etcd-members\" \"${RUN_ROOT}\/artifacts\/etcd-members.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.etcd-endpoints\" \"${RUN_ROOT}\/artifacts\/etcd-endpoints.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.smoke-pods\" \"${RUN_ROOT}\/artifacts\/smoke-pods.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.smoke-service\" \"${RUN_ROOT}\/artifacts\/smoke-service.txt\"copy_guest_file \"${ARTIFACT_PREFIX}.smoke-job.log\" \"${RUN_ROOT}\/artifacts\/smoke-job.log\"copy_guest_file \"${ARTIFACT_PREFIX}.kubeconfig\" \"${RUN_ROOT}\/artifacts\/kubeconfig.yaml\"curl -ksSf --max-time 5 \"https:\/\/${API_LB_IP}:${API_LB_PORT}\/version\" &gt; \"${RUN_ROOT}\/artifacts\/api-lb-version.json\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>\u041f\u043e\u0441\u043b\u0435 \u044d\u0442\u043e\u0433\u043e:<\/p>\n<pre><code class=\"bash\">export KUBECONFIG=\"${RUN_ROOT}\/artifacts\/kubeconfig.yaml\"kubectl get nodes -o widekubectl get pods -A -o widekubectl -n smoke get pods -o wide<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h3>\u0414\u0435\u043c\u043e\u043d\u0442\u0430\u0436<\/h3>\n<pre><code class=\"bash\">docker rm -f \"${API_LB_CONTAINER_NAME}\" &gt;\/dev\/null 2&gt;&amp;1 || truefor pid_file in \"${RUN_ROOT}\"\/nodes\/*\/pid; do  [[ -f \"${pid_file}\" ]] || continue  kill \"$(cat \"${pid_file}\")\" 2&gt;\/dev\/null || truedonesleep 1for pid_file in \"${RUN_ROOT}\"\/nodes\/*\/pid; do  [[ -f \"${pid_file}\" ]] || continue  kill -9 \"$(cat \"${pid_file}\")\" 2&gt;\/dev\/null || truedonefor i in $(seq 0 \"$((NODE_COUNT - 1))\"); do  ip link del \"$(node_tap \"${i}\")\" 2&gt;\/dev\/null || truedoneiptables -D FORWARD -i \"${BRIDGE_NAME}\" -j ACCEPT 2&gt;\/dev\/null || trueiptables -D FORWARD -o \"${BRIDGE_NAME}\" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2&gt;\/dev\/null || trueiptables -t nat -D POSTROUTING -s \"${CIDR}\" ! -o \"${BRIDGE_NAME}\" -j MASQUERADE 2&gt;\/dev\/null || trueip link set \"${BRIDGE_NAME}\" down 2&gt;\/dev\/null || trueip link del \"${BRIDGE_NAME}\" type bridge 2&gt;\/dev\/null || truerm -rf \"${RUN_ROOT}\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>CACHE_ROOT \u043d\u0430\u043c\u0435\u0440\u0435\u043d\u043d\u043e \u043d\u0435 \u0443\u0434\u0430\u043b\u044f\u0435\u0442\u0441\u044f, \u0447\u0442\u043e\u0431\u044b \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u043b\u043e \u043f\u0435\u0440\u0435\u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0441\u043a\u0430\u0447\u0430\u043d\u043d\u043e\u0435 \u044f\u0434\u0440\u043e, Ubuntu squashfs, \u0431\u0430\u0437\u043e\u0432\u044b\u0439 \u043e\u0431\u0440\u0430\u0437 ext4 \u0438 \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0439 \u043e\u0431\u0440\u0430\u0437 \u043f\u0440\u0438 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u043c \u0437\u0430\u043f\u0443\u0441\u043a\u0435.<\/p>\n<\/div>\n<p>\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/1046307\/\">https:\/\/habr.com\/ru\/articles\/1046307\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>k8s-release \u2014 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u0432\u043e\u0441\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u043c\u044b\u0445 DEB\/RPM-\u043f\u0430\u043a\u0435\u0442\u043e\u0432 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432 Kubernetes (kubelet, kube-apiserver, etcd, Flannel, Istio \u0438 \u0434\u0440.) \u0432 \u0433\u0435\u0440\u043c\u0435\u0442\u0438\u0447\u043d\u043e\u043c Docker-\u043e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u0438. \u0421\u043e\u0437\u0434\u0430\u0451\u0442 \u043f\u043e\u0434\u043f\u0438\u0441\u0430\u043d\u043d\u044b\u0435 apt\/yum-\u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438, SBOM, \u0445\u0435\u0448-\u0441\u0443\u043c\u043c\u044b \u0438 airgap-\u0431\u0430\u043d\u0434\u043b\u044b \u0434\u043b\u044f \u0437\u0430\u043a\u0440\u044b\u0442\u044b\u0445, \u0440\u0435\u0433\u0443\u043b\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0438\u0437\u043e\u043b\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0445 \u0441\u0440\u0435\u0434. \u0412 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438 \u0442\u0430\u043a\u0436\u0435 \u043b\u0435\u0436\u0438\u0442 Terraform-\u043c\u043e\u0434\u0443\u043b\u044c \u0434\u043b\u044f \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f HA Kubernetes \u043d\u0430 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c Firecracker: 3 control-plane, 1+ workers, Flannel, Istio, SELinux enforcing.k8s-release \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442 \u043b\u044e\u0431\u044b\u0435 \u043a\u043e\u043c\u0431\u0438\u043d\u0430\u0446\u0438\u0438 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432 \u0447\u0435\u0440\u0435\u0437 \u0444\u043b\u0430\u0433 &#8212;component:.\/k8s-release build v1.36.1 &#8212;component kubelet,kubectl &#8212;format deb,rpm.\/k8s-release build v1.36.1 &#8212;component all &#8212;format deb.\/k8s-release build 3.5.22 &#8212;component etcd &#8212;format rpm.\/k8s-release build 1.30.0 &#8212;component istio &#8212;format deb,rpm\u041c\u043e\u0436\u043d\u043e \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0435 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u044b (kubelet, etcd, flannel, istio), all \u0434\u043b\u044f \u0432\u0441\u0435\u0433\u043e \u0441\u0440\u0430\u0437\u0443, \u0438 \u0444\u043e\u0440\u043c\u0430\u0442 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 \u2014 deb, rpm \u0438\u043b\u0438 \u043e\u0431\u0430.\u041a\u0430\u043a \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430git clone https:\/\/github.com\/ingresslabs\/k8s-release.gitcd k8s-release\/terraform\/k2vm-ha-labcp terraform.tfvars.example terraform.tfvars\u041d\u0430 \u0443\u0434\u0430\u043b\u0451\u043d\u043d\u043e\u043c \u0445\u043e\u0441\u0442\u0435 \u0434\u043e\u043b\u0436\u043d\u044b \u0431\u044b\u0442\u044c: Firecracker, Docker, \u044f\u0434\u0440\u043e, initrd, \u043c\u043e\u0434\u0443\u043b\u0438 \u0438 \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0439 rootfs.ext4 \u043f\u043e \u043f\u0443\u0442\u044f\u043c \u0438\u0437 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445.terraform initterraform planterraform applyterraform destroyvariable &#171;name&#187;                      { type = string; default = &#171;k8s-ha-lab&#187; }variable &#171;target_host&#187;               { type = string; default = &#171;&#187; }variable &#171;target_user&#187;               { type = string; default = &#171;root&#187; }variable &#171;github_repo&#187;               { type = string; default = &#171;ingresslabs\/k8s-release&#187; }variable &#171;github_run_id&#187;             { type = number; default = 26680027183 }variable &#171;control_plane_count&#187;       { type = number; default = 3 }variable &#171;worker_count&#187;              { type = number; default = 2 }variable &#171;kubernetes_version&#187;        { type = string; default = &#171;v1.36.1&#187; }variable &#171;subnet_prefix&#187;             { type = string; default = &#171;198.19.2&#187; }variable &#171;network_plugin&#187;            { type = string; default = &#171;flannel&#187; }variable &#171;firecracker_binary&#187;        { type = string; default = &#171;\/usr\/local\/bin\/firecracker&#187; }variable &#171;kernel_path&#187;               { type = string; default = &#171;\/opt\/fc-lab\/vmlinux-5.15-generic&#187; }variable &#171;initrd_path&#187;               { type = string; default = &#171;\/opt\/fc-lab\/initrd-5.15-generic.img&#187; }variable &#171;kernel_modules_tar_path&#187;   { type = string; default = &#171;\/opt\/fc-lab\/modules-5.15-generic.tar.gz&#187; }variable &#171;base_rootfs_path&#187;          { type = string; default = &#171;\/opt\/fc-lab\/rootfs.ext4&#187; }variable &#171;vcpu_count&#187;                { type = number; default = 2 }variable &#171;guest_selinux_mode&#187;        { type = string; default = &#171;permissive&#187; }variable &#171;enable_istio&#187;              { type = bool;   default = false }variable &#171;redeploy_token&#187;            { type = string; default = &#171;&#187; }# main.tf \u2014 \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 Kubernetes HA \u043d\u0430 Firecracker \u0447\u0435\u0440\u0435\u0437 k8s-releaseterraform {  required_version = &#171;&gt;= 1.5.0&#187;  required_providers {    local = { source = &#171;hashicorp\/local&#187;, version = &#171;~&gt; 2.5&#187; }    null  = { source = &#171;hashicorp\/null&#187;, version = &#171;~&gt; 3.2&#187; }  }}\u041e\u0431\u0430 \u043f\u0440\u043e\u0435\u043a\u0442\u0430 \u2014 \u0447\u0430\u0441\u0442\u0438 \u043e\u0434\u043d\u043e\u0439 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u044b. \u041f\u0435\u0440\u0432\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u043e\u043f\u0438\u0441\u044b\u0432\u0430\u0435\u0442 \u0441\u0430\u043c\u043e\u0434\u0435\u043b\u044c\u043d\u0443\u044e Firecracker CI-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443 (\u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u0437\u0430 10 \u043c\u0441, \u043a\u0435\u0448\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435, \u0441\u043d\u0430\u043f\u0448\u043e\u0442\u044b). \u0412\u0442\u043e\u0440\u043e\u0439 \u043f\u0440\u043e\u0435\u043a\u0442, k8s-release, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u0442\u0443 \u0436\u0435 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u0438 \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u044f Kubernetes \u043d\u0430 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c Firecracker. Terraform-\u043c\u043e\u0434\u0443\u043b\u044c \u0432 k8s-release \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0438\u0440\u0443\u0435\u0442 \u0440\u0430\u0437\u0432\u0451\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 HA-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u043f\u043e \u0442\u0435\u043c \u0436\u0435 \u043f\u0440\u0438\u043d\u0446\u0438\u043f\u0430\u043c, \u0447\u0442\u043e \u043e\u043f\u0438\u0441\u0430\u043d\u044b \u0432 \u0441\u0442\u0430\u0442\u044c\u0435.\u042d\u0442\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u043d\u0435\u043f\u043e\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e \u043d\u0430 \u0445\u043e\u0441\u0442\u0435 Linux. \u041e\u043d\u043e \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442 \u043e\u0434\u043d\u0443 \u043f\u0435\u0440\u0435\u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c\u0443\u044e \u0433\u043e\u0441\u0442\u0435\u0432\u0443\u044e rootfs, \u043a\u043b\u043e\u043d\u0438\u0440\u0443\u0435\u0442 \u0435\u0451 \u043d\u0430 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0434\u0438\u0441\u043a\u043e\u0432 \u043c\u0438\u043a\u0440\u043e-\u0412\u041c Firecracker, \u0437\u0430\u0433\u0440\u0443\u0436\u0430\u0435\u0442:3 \u0443\u0437\u043b\u0430 control-plane kubeadm\u0432\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u044b\u0439 etcd, \u043f\u043e \u043e\u0434\u043d\u043e\u043c\u0443 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u0443 \u043d\u0430 \u0443\u0437\u0435\u043b control-plane2 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u0430 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e\u0438 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442 Kubernetes API \u0447\u0435\u0440\u0435\u0437 \u0445\u043e\u0441\u0442-\u043f\u0440\u043e\u043a\u0441\u0438 HAProxy.\u041f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0435 \u0442\u0440\u0435\u0431\u043e\u0432\u0430\u043d\u0438\u044f\u0412\u044b\u043f\u043e\u043b\u043d\u044f\u0439\u0442\u0435 \u0432\u0441\u0451 \u043d\u0438\u0436\u0435 \u043e\u0442 root \u043d\u0430 \u0445\u043e\u0441\u0442\u0435 Linux, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u0431\u0443\u0434\u0435\u0442 \u0437\u0430\u043f\u0443\u0449\u0435\u043d Firecracker. \u041d\u0430 \u0445\u043e\u0441\u0442\u0435 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u044b:DockerFirecrackercurl, ip, iptables, ssh, scp, ssh-keygenunsquashfs \u0438\u0437 squashfs-toolsmkfs.ext4, e2fsck, resize2fs \u0438\u0437 e2fsprogsmount, umount, truncate, chroot\u0438\u0441\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0434\u043e\u0441\u0442\u0443\u043f \u0432 \u0438\u043d\u0442\u0435\u0440\u043d\u0435\u0442 \u0434\u043b\u044f:\u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0438 \u043e\u0431\u0440\u0430\u0437\u0430 \u044f\u0434\u0440\u0430 LinuxKit\u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044f Ubuntu rootfs \u0438\u0437 Firecracker CI\u043f\u0430\u043a\u0435\u0442\u043e\u0432 Kubernetes \u0441 pkgs.k8s.io\u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u044f Docker containerd\u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044f \u043c\u0430\u043d\u0438\u0444\u0435\u0441\u0442\u0430 Flannel\u0437\u0430\u0433\u0440\u0443\u0437\u043a\u0438 \u043e\u0431\u0440\u0430\u0437\u0430 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430 HAProxy \u0412\u044b\u0431\u0435\u0440\u0438\u0442\u0435 \u043f\u043e\u0434\u0441\u0435\u0442\u044c, \u043d\u0435 \u043f\u0435\u0440\u0435\u0441\u0435\u043a\u0430\u044e\u0449\u0443\u044e\u0441\u044f \u0441 \u0434\u0440\u0443\u0433\u0438\u043c\u0438 \u043c\u043e\u0441\u0442\u0430\u043c\u0438 \u043d\u0430 \u0445\u043e\u0441\u0442\u0435.\u042d\u043a\u0441\u043f\u043e\u0440\u0442 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445 \u043b\u0430\u0431\u043e\u0440\u0430\u0442\u043e\u0440\u0438\u0438\u042d\u0442\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u043e \u0441\u043e\u0437\u0434\u0430\u0451\u0442 3 \u0443\u0437\u043b\u0430 control-plane \u0438 2 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u0430 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e. \u0418\u0437\u043c\u0435\u043d\u0438\u0442\u0435 WORKER_COUNT, \u0435\u0441\u043b\u0438 \u043d\u0443\u0436\u043d\u043e \u043c\u0435\u043d\u044c\u0448\u0435 \u0438\u043b\u0438 \u0431\u043e\u043b\u044c\u0448\u0435 \u0440\u0430\u0431\u043e\u0447\u0438\u0445 \u0443\u0437\u043b\u043e\u0432.export RUN_ROOT=\/var\/lib\/firecracker-kubeadm-haexport CACHE_ROOT=\/var\/cache\/firecracker-kubeadm-haexport CONTROL_PLANE_COUNT=3export WORKER_COUNT=2export NODE_COUNT=&#187;$((CONTROL_PLANE_COUNT + WORKER_COUNT))&#187;export SUBNET_PREFIX=198.19.0export GATEWAY=&#187;${SUBNET_PREFIX}.1&#8243;export CIDR=&#187;${SUBNET_PREFIX}.0\/24&#8243;export API_LB_IP=&#187;${SUBNET_PREFIX}.5&#8243;export API_LB_PORT=6443export PRIMARY_CONTROL_PLANE_IP=&#187;${SUBNET_PREFIX}.10&#8243;export BRIDGE_NAME=k8sha198export TAP_PREFIX=k8sha198export FIRECRACKER_BIN=\/usr\/local\/bin\/firecrackerexport FIRECRACKER_ARCH=&#187;$(uname -m)&#187;export LINUXKIT_KERNEL_IMAGE=linuxkit\/kernel:6.12.59export ROOTFS_SIZE_GIB=12export CONTROL_PLANE_MEM_MIB=2048export WORKER_MEM_MIB=1536export VCPU_COUNT=2export KUBERNETES_MINOR=v1.36export KUBERNETES_VERSION=v1.36.1export POD_CIDR=10.244.0.0\/16export SERVICE_CIDR=10.96.0.0\/12export CNI_PLUGINS_VERSION=v1.3.0export NETWORK_PLUGIN=flannelexport FLANNEL_MANIFEST_URL=&#187;https:\/\/github.com\/flannel-io\/flannel\/releases\/latest\/download\/kube-flannel.yml&#187;export HAPROXY_IMAGE=haproxy:3.2.19-alpineexport API_LB_CONTAINER_NAME=&#187;kubeadm-ha-api-lb-${BRIDGE_NAME}&#187;export GUEST_SSH_KEY=&#187;${CACHE_ROOT}\/lab_ssh_key&#187;export GUEST_SSH_PUB=&#187;${GUEST_SSH_KEY}.pub&#187;export ARTIFACT_PREFIX=\/var\/log\/kubeadm-ha-labexport KERNEL_BOOT_ARGS=&#187;console=ttyS0 reboot=k panic=1 pci=off root=\/dev\/vda rw random.trust_cpu=on systemd.mask=serial-getty@ttyS0.service systemd.mask=systemd-random-seed.service&#187;\u041f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0435\u0439 \u0445\u043e\u0441\u0442\u0430 \u0438 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0433\u043e SSH-\u043a\u043b\u044e\u0447\u0430command -v awk chroot curl docker e2fsck grep ip iptables mkfs.ext4 mount \\  resize2fs sha256sum sort ssh scp ssh-keygen truncate umount unsquashfs &gt;\/dev\/nulltest -x &#171;${FIRECRACKER_BIN}&#187;mkdir -p &#171;${RUN_ROOT}&#187; &#171;${CACHE_ROOT}&#187;if [[ ! -f &#171;${GUEST_SSH_KEY}&#187; || ! -f &#171;${GUEST_SSH_PUB}&#187; ]]; then  ssh-keygen -q -t ed25519 -N &#171;&#187; -f &#171;${GUEST_SSH_KEY}&#187;fi\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u0435 \u0432\u0441\u043f\u043e\u043c\u043e\u0433\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u0444\u0443\u043d\u043a\u0446\u0438\u0439node_role() {  if (( $1 &lt; CONTROL_PLANE_COUNT )); then    echo control-plane  else    echo worker  fi}node_ip() {  printf &#8216;%s.%d\\n&#8217; &#171;${SUBNET_PREFIX}&#187; &#171;$((10 + $1))&#187;}node_name() {  printf &#8216;k8s-%02d\\n&#8217; &#171;$1&#187;}node_tap() {  printf &#8216;%s%d\\n&#8217; &#171;${TAP_PREFIX}&#187; &#171;$1&#187;}node_mac() {  printf &#8217;06:36:00:00:00:%02x\\n&#8217; &#171;$((16 + $1))&#187;}node_mem() {  if [[ &#171;$(node_role &#171;$1&#8243;)&#187; == &#171;control-plane&#187; ]]; then    echo &#171;${CONTROL_PLANE_MEM_MIB}&#187;  else    echo &#171;${WORKER_MEM_MIB}&#187;  fi}SSH_OPTS=(-i &#171;${GUEST_SSH_KEY}&#187; -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=\/dev\/null -o LogLevel=ERROR -o ConnectTimeout=5)node_ssh() {  local idx=&#187;$1&#8243;  shift  ssh &#171;${SSH_OPTS[@]}&#187; &#171;root@$(node_ip &#171;${idx}&#187;)&#187; &#171;$@&#187;}server_ssh() {  node_ssh 0 &#171;$@&#187;}copy_guest_file() {  local remote_path=&#187;$1&#8243;  local local_path=&#187;$2&#8243;  if server_ssh &#171;test -f $(printf &#8216;%q&#8217; &#171;${remote_path}&#187;)&#187; &gt;\/dev\/null 2&gt;&amp;1; then    server_ssh &#171;cat $(printf &#8216;%q&#8217; &#171;${remote_path}&#187;)&#187; &gt; &#171;${local_path}&#187;  fi}wait_for_ssh() {  local ip=&#187;$1&#8243;  for _ in $(seq 1 120); do    if ssh &#171;${SSH_OPTS[@]}&#187; &#171;root@${ip}&#187; true &gt;\/dev\/null 2&gt;&amp;1; then      return 0    fi    sleep 2  done  return 1}wait_for_node_ready() {  local name=&#187;$1&#8243;  for _ in $(seq 1 240); do    if server_ssh &#171;kubectl &#8212;kubeconfig \/etc\/kubernetes\/admin.conf get node ${name} &#8212;no-headers 2&gt;\/dev\/null | awk &#8216;\\$2==\\&#187;Ready\\&#187;{ok=1} END{exit(ok?0:1)}'&#187;; then      return 0    fi    sleep 2  done  return 1}\u0421\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u0435 \u0433\u043e\u0441\u0442\u0435\u0432\u043e\u0433\u043e \u044f\u0434\u0440\u0430 \u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 rootfs\u0412 \u044d\u0442\u043e\u043c \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f:\u043e\u0431\u0440\u0430\u0437 \u044f\u0434\u0440\u0430 LinuxKit\u043f\u043e\u0441\u043b\u0435\u0434\u043d\u044f\u044f Ubuntu squashfs \u0438\u0437 Firecracker CI \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0439 \u0432\u0435\u0440\u0441\u0438\u0438 Firecracker major\/minor\u0417\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u044f\u0434\u0440\u0430 LinuxKit \u0438 \u0438\u0437\u0432\u043b\u0435\u0447\u0435\u043d\u0438\u0435 vmlinuxexport LINUXKIT_KEY=&#187;$(printf &#8216;%s\\n&#8217; &#171;${LINUXKIT_KERNEL_IMAGE}&#187; | sha256sum | awk &#8216;{print substr($1,1,16)}&#8217;)&#187;export LINUXKIT_CACHE_DIR=&#187;${CACHE_ROOT}\/downloads\/linuxkit-${LINUXKIT_KEY}&#187;export KERNEL_PATH=&#187;${LINUXKIT_CACHE_DIR}\/vmlinux&#187;export KERNEL_BZIMAGE_PATH=&#187;${LINUXKIT_CACHE_DIR}\/kernel&#187;export KERNEL_MODULES_TAR_PATH=&#187;${LINUXKIT_CACHE_DIR}\/kernel.tar&#187;export KERNEL_DEV_TAR_PATH=&#187;${LINUXKIT_CACHE_DIR}\/kernel-dev.tar&#187;mkdir -p &#171;${LINUXKIT_CACHE_DIR}&#187;if [[ ! -f &#171;${KERNEL_PATH}&#187; || ! -f &#171;${KERNEL_MODULES_TAR_PATH}&#187; ]]; then  rm -rf &#171;${LINUXKIT_CACHE_DIR}.tmp&#187;  mkdir -p &#171;${LINUXKIT_CACHE_DIR}.tmp&#187;  docker pull &#171;${LINUXKIT_KERNEL_IMAGE}&#187; &gt;\/dev\/null  cid=&#187;$(docker create &#171;${LINUXKIT_KERNEL_IMAGE}&#187; \/bin\/sh)&#187;  docker cp &#171;${cid}:\/kernel&#187; &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel&#187;  docker cp &#171;${cid}:\/kernel.tar&#187; &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel.tar&#187;  docker cp &#171;${cid}:\/kernel-dev.tar&#187; &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar&#187;  docker rm -f &#171;${cid}&#187; &gt;\/dev\/null  export LINUXKIT_HEADERS_DIR=&#187;$(    tar -tf &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar&#187; \\      | sed -n &#8216;s#^\\(usr\/src\/linux-headers-[^\/]*\\)\/scripts\/extract-vmlinux$#\\1#p&#8217; \\      | head -n 1  )&#187;  tar -xOf &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar&#187; &#171;${LINUXKIT_HEADERS_DIR}\/scripts\/extract-vmlinux&#187; &gt; &#171;${LINUXKIT_CACHE_DIR}.tmp\/extract-vmlinux&#187;  chmod +x &#171;${LINUXKIT_CACHE_DIR}.tmp\/extract-vmlinux&#187;  &#171;${LINUXKIT_CACHE_DIR}.tmp\/extract-vmlinux&#187; &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel&#187; &gt; &#171;${LINUXKIT_CACHE_DIR}.tmp\/vmlinux&#187;  mv &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel&#187; &#171;${KERNEL_BZIMAGE_PATH}&#187;  mv &#171;${LINUXKIT_CACHE_DIR}.tmp\/vmlinux&#187; &#171;${KERNEL_PATH}&#187;  mv &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel.tar&#187; &#171;${KERNEL_MODULES_TAR_PATH}&#187;  mv &#171;${LINUXKIT_CACHE_DIR}.tmp\/kernel-dev.tar&#187; &#171;${KERNEL_DEV_TAR_PATH}&#187;  rm -rf &#171;${LINUXKIT_CACHE_DIR}.tmp&#187;fi\u0421\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u0435 Ubuntu rootfs squashfs \u0438\u0437 Firecracker CIexport FIRECRACKER_CI_VERSION=&#187;$(&#171;${FIRECRACKER_BIN}&#187; &#8212;version | awk &#8216;{print $2}&#8217; | sed &#8216;s\/\\.[0-9]*$\/\/&#8217;)&#187;export&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-483249","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/483249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=483249"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/483249\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=483249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=483249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=483249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}