{"id":380845,"date":"2024-06-29T03:19:49","date_gmt":"2024-06-29T03:19:49","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=380845"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=380845","title":{"rendered":"<span>An Antidote to Absent-Mindedness, or How I Gained Access to an OpenShift Node without an SSH Key<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p> Taking a brief glance at OpenShift, I noticed that applications took longer to start and ran slower. Further research revealed that one of the Nodes had fallen out of the OS cluster. I attempted to try and fix the problem via SSH, but then I remembered that I left my SSH key on the office computer. The message &#171;Permission denied (publickey,gssapi-keyex,gssapi-with-mic)&#187; only confirmed my unfortunate situation. There I was, looking at the VM screen on the Hypervisor console, unable to get inside. Oh well!<\/p>\n<p> Below is a retrospective on how I restored the Node\u2019s functionality. As the saying goes, any resemblance to actual events, locales or persons is purely coincidental.<\/p>\n<ul>\n<li>\n<p><a href=\"#emergency\"><em>Emergency boot and gaining access to the terminal<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#password\"><em>Changing the core user password<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#no_password\"><em>Solving the SSH login problem<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#network\"><em>Configuring network interfaces<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#tuning\"><em>Modifying the system kernel<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#next\"><em>What&#8217;s next?<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#help\"><em>If nothing helps<\/em><\/a><\/p>\n<\/li>\n<\/ul>\n<h2> Emergency boot and gaining access to the terminal<\/h2>\n<p><a class=\"anchor\" name=\"emergency\" id=\"emergency\"><\/a><\/p>\n<p> It all started with an emergency boot. In order to obtain access to the emergency shell when the system starts, we have to append a certain kernel parameter\u00a0\u2014 <em>rd.break<\/em>.<\/p>\n<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/f-\/4k\/ru\/f-4kruqfjrvosyivimpn-kskxgu.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/f-\/4k\/ru\/f-4kruqfjrvosyivimpn-kskxgu.png\"\/><figcaption><\/figcaption><\/figure>\n<p> As the system starts, we\u2019re presented with the grub boot menu. We press E and add rd.break to the linux string: <\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/et\/rw\/cr\/etrwcrhlwbru07qjopdatqqfmey.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/et\/rw\/cr\/etrwcrhlwbru07qjopdatqqfmey.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Press Ctrl+X and wait for the system to boot.<\/p>\n<p>The system boots, but the terminal does not appear. What&#8217;s wrong?<\/p>\n<p> As it turns out, it was necessary to disable the interactive terminal. To do so, remove the following strings: \u201cconsole=tty0 console=ttyS0,115200n8\u201d:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/wv\/eh\/o8\/wveho8ln26emnx4hytbkclgmgro.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/wv\/eh\/o8\/wveho8ln26emnx4hytbkclgmgro.png\"\/><figcaption><\/figcaption><\/figure>\n<p> The final code will look like this:<\/p>\n<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/98\/bw\/mr\/98bwmrnzhilbvy3c9jgecynaz58.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/98\/bw\/mr\/98bwmrnzhilbvy3c9jgecynaz58.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Press Ctrl+X for the system to boot.<\/p>\n<p>Press Enter to gain access to the vital shell:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/1a\/2c\/x-\/1a2cx-j4byrzf1ppon3kx8m2yz4.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/1a\/2c\/x-\/1a2cx-j4byrzf1ppon3kx8m2yz4.png\"\/><figcaption><\/figcaption><\/figure>\n<h2> Changing the core user password<\/h2>\n<p><a class=\"anchor\" name=\"password\" id=\"password\"><\/a><\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/lk\/uq\/np\/lkuqnpy7fs3muo_imcbmthsolgo.png\" alt=\"pa\" title=\"pa\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/lk\/uq\/np\/lkuqnpy7fs3muo_imcbmthsolgo.png\"\/><figcaption>pa<\/figcaption><\/figure>\n<p> To change the password, I temporarily changed the root directory using the chroot command:<\/p>\n<pre><code class=\"bash\">mount -o remount,rw \/sysroot chroot \/sysroot passwd core<\/code><\/pre>\n<p>OpenShift runs with SELinux enabled; therefore, the file must have the corresponding attributes:<\/p>\n<pre><code class=\"bash\">ls -Za \/etc\/shadow ? \/etc\/shadow\/<\/code><\/pre>\n<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/43\/zl\/kq\/43zlkqf1suwkm6ubjlubfnvogka.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/43\/zl\/kq\/43zlkqf1suwkm6ubjlubfnvogka.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Add the relevant attribute:<\/p>\n<pre><code class=\"bash\">chcon -h system_u:object_r:shadow_t:s0 \/etc\/shadow*<\/code><\/pre>\n<p>Ensure that:<\/p>\n<pre><code class=\"bash\">ls -Za \/etc\/shadow system_u:object_r:shadow_t:s0 \/etc\/shadow<\/code><\/pre>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/vu\/s_\/f1\/vus_f1ign_hffqeb4q-b59sc6qa.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/vu\/s_\/f1\/vus_f1ign_hffqeb4q-b59sc6qa.png\"\/><figcaption><\/figcaption><\/figure>\n<p> Now reboot the system using the following command:<\/p>\n<pre><code class=\"bash\">\/sbin\/reboot -f<\/code><\/pre>\n<p> Alternatively, click the button on the control panel for the virtual machine\/server to reboot.<\/p>\n<\/p>\n<p>Solving the SSH login problem<\/p>\n<p> By default, the system can only be accessed with SSH keys which are stored in .ssh\/authorized_keys<\/p>\n<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/e0\/gq\/cb\/e0gqcbs9gcaqkqd2cwadxgx28qk.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/e0\/gq\/cb\/e0gqcbs9gcaqkqd2cwadxgx28qk.png\"\/><figcaption><\/figcaption><\/figure>\n<p> If you left your SSH key at work or lost it in the usual routine or, if you need to add another SSH key, this is the file you\u2019re after. We\u2019ll have to add a new key from your id_rsa.pub. Before that, however, check if the file has the relevant rights and SELinux attributes:<\/p>\n<pre><code class=\"bash\">chmod 600 -Rv ~\/.ssh\/ chown core:core -Rv ~\/.ssh\/ chcon -t unconfined_u:object_r:ssh_home_t:s0 ~\/.ssh\/authorized_keys<\/code><\/pre>\n<p> <em>or<\/em><\/p>\n<pre><code class=\"bash\">restorecon -v ~\/.ssh\/authorized_keys<\/code><\/pre>\n<h2>Enabling login using username and password without SSH keys<\/h2>\n<p><a class=\"anchor\" name=\"no_password\" id=\"no_password\"><\/a><\/p>\n<p> <strong><em>Attention! This action compromises system security! You must log in as the core user, using your SSH key.<\/em><\/strong><\/p>\n<p> You can solve the problem of forgotten SSH keys radically by enabling system login using username and password. To do this, use one of two options, depending on the available system functionality:<\/p>\n<p><strong><em>Option 1. You are able to log into the system as the core user<\/em><\/strong><\/p>\n<p> Obtain admin privileges and gain access to the file system:<\/p>\n<pre><code class=\"bash\">sudo su mount -o remount,rw vi \/etc\/ssh\/sshd_config<\/code><\/pre>\n<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/kk\/v9\/th\/kkv9thffwskrlrglv64cxvgjx-g.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/kk\/v9\/th\/kkv9thffwskrlrglv64cxvgjx-g.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Edit the file to enable username\/password login. Change the <em>PasswordAuthentication<\/em> parameter to <em>yes<\/em>.<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/4h\/ys\/qp\/4hysqpivs5awgcp9dkmtddmtxto.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/4h\/ys\/qp\/4hysqpivs5awgcp9dkmtddmtxto.png\"\/><figcaption><\/figcaption><\/figure>\n<p> Log out of root and restart the service.<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/ei\/3r\/mk\/ei3rmkuuum4gynrbhmgw9uoddmo.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/ei\/3r\/mk\/ei3rmkuuum4gynrbhmgw9uoddmo.png\"\/><figcaption><\/figcaption><\/figure>\n<p>Additionally, verify that the file retains its SELinux attribute. This will come in handy for option 2.<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/qp\/01\/w_\/qp01w_o_9x8u2g4ewl3ljjke-s0.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/qp\/01\/w_\/qp01w_o_9x8u2g4ewl3ljjke-s0.png\"\/><figcaption><\/figcaption><\/figure>\n<p> <strong><em>Option 2. Booting in rd.break mode<\/em><\/strong><\/p>\n<p> Edit \/etc\/ssh\/sshd_config in the same manner, by setting<\/p>\n<p> <em>PasswordAuthentication yes<\/em><\/p>\n<p> Save and check the SELinux attributes:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/sf\/k5\/84\/sfk584iobg5yimhfnm_dhicwjvo.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/sf\/k5\/84\/sfk584iobg5yimhfnm_dhicwjvo.png\"\/><figcaption><\/figcaption><\/figure>\n<p> As we can see, the SELinux attributes were lost. Restore them:<\/p>\n<pre><code class=\"bash\">chcon -h system_u:object_r:etc_t:s0 \/etc\/ssh\/sshd_config<\/code><\/pre>\n<h2>Configuring network interfaces<\/h2>\n<p><a class=\"anchor\" name=\"network\" id=\"network\"><\/a><\/p>\n<p> If you need to restore or configure network interfaces for a virtual machine, use the nmcli utility.<\/p>\n<p> You can display network connections with this simple command:<\/p>\n<pre><code class=\"bash\">nmcli connection show<\/code><\/pre>\n<p> Use nmcli connection mod \u201cconnection_name\u201d to edit the network settings and then restart the network interface.<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/3_\/ky\/fi\/3_kyfinkgkccc6n6ev9rzz4dvr4.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/3_\/ky\/fi\/3_kyfinkgkccc6n6ev9rzz4dvr4.png\"\/><figcaption><\/figcaption><\/figure>\n<h2>Tuning the system core<\/h2>\n<p><a class=\"anchor\" name=\"tuning\" id=\"tuning\"><\/a><\/p>\n<p> As we are actively using the command line, we can take this chance to modify the kernel.\u00a0On systems that use rpm-ostree, you can add or remove kernel parameters that are set when the system boots.<\/p>\n<p> For example, we can improve system performance by disabling protection against the Spectre v1 and Spectre v2 vulnerabilities. To do this, add the following parameters:<\/p>\n<pre><code class=\"bash\"> \"nospectre_v1,nospectre_v2,nospec_store_bypass_disable\"<\/code><\/pre>\n<p>This is accomplished by the following commands:<\/p>\n<pre><code class=\"bash\">rpm-ostree kargs --append=\"nospectre_v1\" --append=\"nospectre_v2\" --append=\"nospec_store_bypass_disable\" Staging deployment... done  Kernel arguments updated.  Run \"systemctl reboot\" to start a reboot  [root@localhost core]# rpm-ostree kargs mitigations=auto,nosmt console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal $ignition_firstboot ostree=\/ostree\/boot.1\/fedora-coreos\/0045ea4dd22400fe745be6b98741225cd831069a635d08100f5d25f1c77a13ac\/0 root=UUID=de3da0d6-308e-4f4e-b60a-04db75452575 rw rootflags=prjquota boot=UUID=02e3df85-38cd-4173-a672-3b6fb5dfe4b0 nospectre_v1 nospectre_v2 nospec_store_bypass_disable<\/code><\/pre>\n<p> After the system is rebooted, you can check if the kernel parameters have been applied by running the command:<\/p>\n<pre><code class=\"bash\">rpm-ostree kargs<\/code><\/pre>\n<p>Conversely, if you want to remove kernel parameters, you can run the following command:<\/p>\n<pre><code class=\"bash\">sudo rpm-ostree kargs --delete=\"nospectre_v1\" --delete=\"nospectre_v2\" \u2014delete=\"nospec_store_bypass_disable\"<\/code><\/pre>\n<p>You can view help for this command by running:<\/p>\n<pre><code class=\"bash\">rpm-ostree kargs \u2013help<\/code><\/pre>\n<p> Additional parameters that you might need:<\/p>\n<pre><code class=\"bash\">--append=\"ipv6.disable=1\" --append=\"quiet\"<\/code><\/pre>\n<p> The quiet mode allows you to delete these messages from the virtual console of the first system screen:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/rd\/2i\/pz\/rd2ipzmsf5th-82eg1plhxk4apo.png\" alt=\"\" title=\"\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/rd\/2i\/pz\/rd2ipzmsf5th-82eg1plhxk4apo.png\"\/><figcaption><\/figcaption><\/figure>\n<pre><code class=\"bash\">sudo rpm-ostree kargs --append=\"quiet\" Staging deployment... done  Kernel arguments updated.  Run \"systemctl reboot\" to start a reboot<\/code><\/pre>\n<h2>Mounting a new drive to the system<\/h2>\n<p><a class=\"anchor\" name=\"mount\" id=\"mount\"><\/a><\/p>\n<p> Additionally, we\u2019ll take a look at how to mount a second hard drive. The drive name may differ, depending on the type of the hard drive controller in the system\/virtual machine. In my example, the disks are called \/dev\/sd*.<\/p>\n<p>First, we\u2019ll partition the disk:<\/p>\n<pre><code class=\"bash\">fdisk \/dev\/sdb mkfs.ext4 \/dev\/sdb1 -L disk2_test<\/code><\/pre>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/va\/lj\/kp\/valjkpohxxtie5yquuvfl4gabce.png\" alt=\"\" title=\"\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/va\/lj\/kp\/valjkpohxxtie5yquuvfl4gabce.png\"\/><figcaption><\/figcaption><\/figure>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/mb\/k2\/nf\/mbk2nfycjsdv-h9fm2kbifg-etu.png\" alt=\"\" title=\"\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/mb\/k2\/nf\/mbk2nfycjsdv-h9fm2kbifg-etu.png\"\/><figcaption><\/figcaption><\/figure>\n<p>In order to mount the drive to \/usr\/local\/1, create the following Unit file systemd:<\/p>\n<pre><code class=\"bash\">mkdir -pv \/usr\/local\/1 cat &lt;&lt; EOF >\/etc\/systemd\/system\/var-usrlocal-1.mount [Unit] Description = Mount for Container Storage  [Mount] #What=\/dev\/sdb1 What=\/dev\/disk\/by-uuid\/7fb1139c-b9c9-404c-91fa-9b2f251fb11c Where=\/var\/usrlocal\/1 Type=ext4  [Install] WantedBy = multi-user.target  EOF<\/code><\/pre>\n<p>Enable auto-start and run it:<\/p>\n<pre><code class=\"bash\">systemctl enable var-usrlocal-1.mount Created symlink \/etc\/systemd\/system\/multi-user.target.wants\/var-usrlocal-1.mount \u2192 \/etc\/systemd\/system\/var-usrlocal-1.mount.  systemctl start var-usrlocal-1.mount<\/code><\/pre>\n<p> Check if the partition (df -h) has been mounted:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/77\/ex\/qy\/77exqyubn-z1nxxqfnjmm80ynyo.png\" alt=\" \" title=\" \" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/77\/ex\/qy\/77exqyubn-z1nxxqfnjmm80ynyo.png\"\/><figcaption> <\/figcaption><\/figure>\n<p> Critically, pay attention to the SELinux attributes required for the systemd unit file (system_u:object_r:systemd_unit_file_t:s0).<\/p>\n<p> If you\u2019re mounting a drive to the \/var\/lib\/containers folder, review the following information:<\/p>\n<p><a href=\"https:\/\/bugzilla.redhat.com\/show_bug.cgi?id=1692513\"><u>https:\/\/bugzilla.redhat.com\/show_bug.cgi?id=1692513<\/u><\/a><\/p>\n<h2>What&#8217;s next?<\/h2>\n<p><a class=\"anchor\" name=\"next\" id=\"next\"><\/a><\/p>\n<p> After gaining access to the system, read through the service log files as usual. After concluding diagnostics and reparing faults, which should hopefully see the offending Node return to the OpenShift cluster, we can use the standard utilities for collecting log files from all OpenShift Nodes:<\/p>\n<p><code>oc adm must-gather<\/code><\/p>\n<p> If your Node\u2019s data (stored on its Persistent Volume) connected to its HostPath has been saved, and there are further plans to install the system again on a clean virtual machine\/server, then you can use this concise cheat-sheet:<\/p>\n<p><a href=\"https:\/\/docs.openshift.com\/container-platform\/4.6\/installing\/installing_bare_metal\/installing-bare-metal.html\"><u>https:\/\/docs.openshift.com\/container-platform\/4.6\/installing\/installing_bare_metal\/installing-bare-metal.html<\/u><\/a><\/p>\n<h2>If nothing helps<\/h2>\n<p><a class=\"anchor\" name=\"help\" id=\"help\"><\/a><\/p>\n<p> If the above methods of restoring access to the Node do not bear results, then your only recourse is to re-install the system.\u00a0You can use one of the two most popular images for installing OpenShift:<\/p>\n<ul>\n<li>\n<p> RHCOS (Red Hat Enterprise\u00a0Linux CoreOS)<\/p>\n<\/li>\n<\/ul>\n<p> <a href=\"https:\/\/mirror.openshift.com\/pub\/openshift-v4\/dependencies\/rhcos\/4.6\/4.6.8\/\"><u>https:\/\/mirror.openshift.com\/pub\/openshift-v4\/dependencies\/rhcos\/4.6\/4.6.8\/<\/u><\/a><\/p>\n<ul>\n<li>\n<p> Fedora CoreOS<\/p>\n<\/li>\n<\/ul>\n<p> <a href=\"https:\/\/getfedora.org\/ru\/coreos?stream=stable\"><u>https:\/\/getfedora.org\/ru\/coreos?stream=stable<\/u><\/a><\/p>\n<p> After you start using the <a href=\"https:\/\/mirror.openshift.com\/pub\/openshift-v4\/dependencies\/rhcos\/4.6\/4.6.8\/rhcos-4.6.8-x86_64-live.x86_64.iso\"><u>rhcos-4.6.8-x86_64-live.x86_64.iso<\/u><\/a> image, the welcome screen and the core-installer sample information will appear.<\/p>\n<p> In the case of a manual install and image preparation, the installation is started using the coreos-installer, it will necessitate the input of the necessary installation parameters, a URL to the ignition files (for bootstrap, compute node and worker node) files and the like.<\/p>\n<p> This may also require additional configuration, such as setting a proper IP address for the pertinent virtual machine or server, DNS names, addresses to where the images are located, etc. Here&#8217;s one of my examples for KVM virtualisation:<\/p>\n<pre><code class=\"bash\">sudo coreos-installer install \/dev\/vda --ignition-url http:\/\/192.168.122.239:8080\/okd4\/master.ign --insecure-ignition --append-karg rd.neednet=1 --append-karg ip=192.168.122.16::192.168.122.1:255.255.255.0:okd4-control-plane-1.lab.okd.local:ens3:none --append-karg nameserver=192.168.122.1 --append-karg nameserver=192.168.122.239 --image-url http:\/\/192.168.122.239:8080\/okd4\/fcos.raw.xz<\/code><\/pre>\n<p>P.\u00a0S. If you want to train &#171;for fun&#187;, to always be ready for a situation when you have no SSH key on hand, you can install the ISO on a virtual machine. For example, on VirtualBox:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/webt\/ka\/ic\/en\/kaicenlq0umaodpum99wnuskll8.png\" width=\"auto\" height=\"auto\" data-src=\"https:\/\/habrastorage.org\/webt\/ka\/ic\/en\/kaicenlq0umaodpum99wnuskll8.png\"\/><figcaption><\/figcaption><\/figure>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/670302\/\"> https:\/\/habr.com\/ru\/articles\/670302\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p> Taking a brief glance at OpenShift, I noticed that applications took longer to start and ran slower. Further research revealed that one of the Nodes had fallen out of the OS cluster. I attempted to try and fix the problem via SSH, but then I remembered that I left my SSH key on the office computer. The message &#171;Permission denied (publickey,gssapi-keyex,gssapi-with-mic)&#187; only confirmed my unfortunate situation. There I was, looking at the VM screen on the Hypervisor console, unable to get inside. Oh well!<\/p>\n<p> Below is a retrospective on how I restored the Node\u2019s functionality. As the saying goes, any resemblance to actual events, locales or persons is purely coincidental.<\/p>\n<ul>\n<li>\n<p><a href=\"#emergency\"><em>Emergency boot and gaining access to the terminal<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#password\"><em>Changing the core user password<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#no_password\"><em>Solving the SSH login problem<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#network\"><em>Configuring network interfaces<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#tuning\"><em>Modifying the system kernel<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#next\"><em>What&#8217;s next?<\/em><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#help\"><em>If nothing helps<\/em><\/a><\/p>\n<\/li>\n<\/ul>\n<h2> Emergency boot and gaining access to the terminal<\/h2>\n<p><a class=\"anchor\" name=\"emergency\" id=\"emergency\"><\/a><\/p>\n<p> It all started with an emergency boot. In order to obtain access to the emergency shell when the system starts, we have to append a certain kernel parameter\u00a0\u2014 <em>rd.break<\/em>.<\/p>\n<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> As the system starts, we\u2019re presented with the grub boot menu. We press E and add rd.break to the linux string: <\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>Press Ctrl+X and wait for the system to boot.<\/p>\n<p>The system boots, but the terminal does not appear. What&#8217;s wrong?<\/p>\n<p> As it turns out, it was necessary to disable the interactive terminal. To do so, remove the following strings: \u201cconsole=tty0 console=ttyS0,115200n8\u201d:<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> The final code will look like this:<\/p>\n<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>Press Ctrl+X for the system to boot.<\/p>\n<p>Press Enter to gain access to the vital shell:<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<h2> Changing the core user password<\/h2>\n<p><a class=\"anchor\" name=\"password\" id=\"password\"><\/a><\/p>\n<figure class=\"\"><figcaption>pa<\/figcaption><\/figure>\n<p> To change the password, I temporarily changed the root directory using the chroot command:<\/p>\n<pre><code class=\"bash\">mount -o remount,rw \/sysroot chroot \/sysroot passwd core<\/code><\/pre>\n<p>OpenShift runs with SELinux enabled; therefore, the file must have the corresponding attributes:<\/p>\n<pre><code class=\"bash\">ls -Za \/etc\/shadow ? \/etc\/shadow\/<\/code><\/pre>\n<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>Add the relevant attribute:<\/p>\n<pre><code class=\"bash\">chcon -h system_u:object_r:shadow_t:s0 \/etc\/shadow*<\/code><\/pre>\n<p>Ensure that:<\/p>\n<pre><code class=\"bash\">ls -Za \/etc\/shadow system_u:object_r:shadow_t:s0 \/etc\/shadow<\/code><\/pre>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> Now reboot the system using the following command:<\/p>\n<pre><code class=\"bash\">\/sbin\/reboot -f<\/code><\/pre>\n<p> Alternatively, click the button on the control panel for the virtual machine\/server to reboot.<\/p>\n<\/p>\n<p>Solving the SSH login problem<\/p>\n<p> By default, the system can only be accessed with SSH keys which are stored in .ssh\/authorized_keys<\/p>\n<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> If you left your SSH key at work or lost it in the usual routine or, if you need to add another SSH key, this is the file you\u2019re after. We\u2019ll have to add a new key from your id_rsa.pub. Before that, however, check if the file has the relevant rights and SELinux attributes:<\/p>\n<pre><code class=\"bash\">chmod 600 -Rv ~\/.ssh\/ chown core:core -Rv ~\/.ssh\/ chcon -t unconfined_u:object_r:ssh_home_t:s0 ~\/.ssh\/authorized_keys<\/code><\/pre>\n<p> <em>or<\/em><\/p>\n<pre><code class=\"bash\">restorecon -v ~\/.ssh\/authorized_keys<\/code><\/pre>\n<h2>Enabling login using username and password without SSH keys<\/h2>\n<p><a class=\"anchor\" name=\"no_password\" id=\"no_password\"><\/a><\/p>\n<p> <strong><em>Attention! This action compromises system security! You must log in as the core user, using your SSH key.<\/em><\/strong><\/p>\n<p> You can solve the problem of forgotten SSH keys radically by enabling system login using username and password. To do this, use one of two options, depending on the available system functionality:<\/p>\n<p><strong><em>Option 1. You are able to log into the system as the core user<\/em><\/strong><\/p>\n<p> Obtain admin privileges and gain access to the file system:<\/p>\n<pre><code class=\"bash\">sudo su mount -o remount,rw vi \/etc\/ssh\/sshd_config<\/code><\/pre>\n<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>Edit the file to enable username\/password login. Change the <em>PasswordAuthentication<\/em> parameter to <em>yes<\/em>.<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> Log out of root and restart the service.<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>Additionally, verify that the file retains its SELinux attribute. This will come in handy for option 2.<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> <strong><em>Option 2. Booting in rd.break mode<\/em><\/strong><\/p>\n<p> Edit \/etc\/ssh\/sshd_config in the same manner, by setting<\/p>\n<p> <em>PasswordAuthentication yes<\/em><\/p>\n<p> Save and check the SELinux attributes:<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p> As we can see, the SELinux attributes were lost. Restore them:<\/p>\n<pre><code class=\"bash\">chcon -h system_u:object_r:etc_t:s0 \/etc\/ssh\/sshd_config<\/code><\/pre>\n<h2>Configuring network interfaces<\/h2>\n<p><a class=\"anchor\" name=\"network\" id=\"network\"><\/a><\/p>\n<p> If you need to restore or configure network interfaces for a virtual machine, use the nmcli utility.<\/p>\n<p> You can display network connections with this simple command:<\/p>\n<pre><code class=\"bash\">nmcli connection show<\/code><\/pre>\n<p> Use nmcli connection mod \u201cconnection_name\u201d to edit the network settings and then restart the network interface.<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<h2>Tuning the system core<\/h2>\n<p><a class=\"anchor\" name=\"tuning\" id=\"tuning\"><\/a><\/p>\n<p> As we are actively using the command line, we can take this chance to modify the kernel.\u00a0On systems that use rpm-ostree, you can add or remove kernel parameters that are set when the system boots.<\/p>\n<p> For example, we can improve system performance by disabling protection against the Spectre v1 and Spectre v2 vulnerabilities. To do this, add the following parameters:<\/p>\n<pre><code class=\"bash\"> \"nospectre_v1,nospectre_v2,nospec_store_bypass_disable\"<\/code><\/pre>\n<p>This is accomplished by the following commands:<\/p>\n<pre><code class=\"bash\">rpm-ostree kargs --append=\"nospectre_v1\" --append=\"nospectre_v2\" --append=\"nospec_store_bypass_disable\" Staging deployment... done  Kernel arguments updated.  Run \"systemctl reboot\" to start a reboot  [root@localhost core]# rpm-ostree kargs mitigations=auto,nosmt console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal $ignition_firstboot ostree=\/ostree\/boot.1\/fedora-coreos\/0045ea4dd22400fe745be6b98741225cd831069a635d08100f5d25f1c77a13ac\/0 root=UUID=de3da0d6-308e-4f4e-b60a-04db75452575 rw rootflags=prjquota boot=UUID=02e3df85-38cd-4173-a672-3b6fb5dfe4b0 nospectre_v1 nospectre_v2 nospec_store_bypass_disable<\/code><\/pre>\n<p> After the system is rebooted, you can check if the kernel parameters have been applied by running the command:<\/p>\n<pre><code class=\"bash\">rpm-ostree kargs<\/code><\/pre>\n<p>Conversely, if you want to remove kernel parameters, you can run the following command:<\/p>\n<pre><code class=\"bash\">sudo rpm-ostree kargs --delete=\"nospectre_v1\" --delete=\"nospectre_v2\" \u2014delete=\"nospec_store_bypass_disable\"<\/code><\/pre>\n<p>You can view help for this command by running:<\/p>\n<pre><code class=\"bash\">rpm-ostree kargs \u2013help<\/code><\/pre>\n<p> Additional parameters that you might need:<\/p>\n<pre><code class=\"bash\">--append=\"ipv6.disable=1\" --append=\"quiet\"<\/code><\/pre>\n<p> The quiet mode allows you to delete these messages from the virtual console of the first system screen:<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<pre><code class=\"bash\">sudo rpm-ostree kargs --append=\"quiet\" Staging deployment... done  Kernel arguments updated.  Run \"systemctl reboot\" to start a reboot<\/code><\/pre>\n<h2>Mounting a new drive to the system<\/h2>\n<p><a class=\"anchor\" name=\"mount\" id=\"mount\"><\/a><\/p>\n<p> Additionally, we\u2019ll take a look at how to mount a second hard drive. The drive name may differ, depending on the type of the hard drive controller in the system\/virtual machine. In my example, the disks are called \/dev\/sd*.<\/p>\n<p>First, we\u2019ll partition the disk:<\/p>\n<pre><code class=\"bash\">fdisk \/dev\/sdb mkfs.ext4 \/dev\/sdb1 -L disk2_test<\/code><\/pre>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<p>In order to mount the drive to \/usr\/local\/1, create the following Unit file systemd:<\/p>\n<pre><code class=\"bash\">mkdir -pv \/usr\/local\/1 cat &lt;&lt; EOF >\/etc\/systemd\/system\/var-usrlocal-1.mount [Unit] Description = Mount for Container Storage  [Mount] #What=\/dev\/sdb1 What=\/dev\/disk\/by-uuid\/7fb1139c-b9c9-404c-91fa-9b2f251fb11c Where=\/var\/usrlocal\/1 Type=ext4  [Install] WantedBy = multi-user.target  EOF<\/code><\/pre>\n<p>Enable auto-start and run it:<\/p>\n<pre><code class=\"bash\">systemctl enable var-usrlocal-1.mount Created symlink \/etc\/systemd\/system\/multi-user.target.wants\/var-usrlocal-1.mount \u2192 \/etc\/systemd\/system\/var-usrlocal-1.mount.  systemctl start var-usrlocal-1.mount<\/code><\/pre>\n<p> Check if the partition (df -h) has been mounted:<\/p>\n<figure class=\"\"><figcaption> <\/figcaption><\/figure>\n<p> Critically, pay attention to the SELinux attributes required for the systemd unit file (system_u:object_r:systemd_unit_file_t:s0).<\/p>\n<p> If you\u2019re mounting a drive to the \/var\/lib\/containers folder, review the following information:<\/p>\n<p><a href=\"https:\/\/bugzilla.redhat.com\/show_bug.cgi?id=1692513\"><u>https:\/\/bugzilla.redhat.com\/show_bug.cgi?id=1692513<\/u><\/a><\/p>\n<h2>What&#8217;s next?<\/h2>\n<p><a class=\"anchor\" name=\"next\" id=\"next\"><\/a><\/p>\n<p> After gaining access to the system, read through the service log files as usual. After concluding diagnostics and reparing faults, which should hopefully see the offending Node return to the OpenShift cluster, we can use the standard utilities for collecting log files from all OpenShift Nodes:<\/p>\n<p><code>oc adm must-gather<\/code><\/p>\n<p> If your Node\u2019s data (stored on its Persistent Volume) connected to its HostPath has been saved, and there are further plans to install the system again on a clean virtual machine\/server, then you can use this concise cheat-sheet:<\/p>\n<p><a href=\"https:\/\/docs.openshift.com\/container-platform\/4.6\/installing\/installing_bare_metal\/installing-bare-metal.html\"><u>https:\/\/docs.openshift.com\/container-platform\/4.6\/installing\/installing_bare_metal\/installing-bare-metal.html<\/u><\/a><\/p>\n<h2>If nothing helps<\/h2>\n<p><a class=\"anchor\" name=\"help\" id=\"help\"><\/a><\/p>\n<p> If the above methods of restoring access to the Node do not bear results, then your only recourse is to re-install the system.\u00a0You can use one of the two most popular images for installing OpenShift:<\/p>\n<ul>\n<li>\n<p> RHCOS (Red Hat Enterprise\u00a0Linux CoreOS)<\/p>\n<\/li>\n<\/ul>\n<p> <a href=\"https:\/\/mirror.openshift.com\/pub\/openshift-v4\/dependencies\/rhcos\/4.6\/4.6.8\/\"><u>https:\/\/mirror.openshift.com\/pub\/openshift-v4\/dependencies\/rhcos\/4.6\/4.6.8\/<\/u><\/a><\/p>\n<ul>\n<li>\n<p> Fedora CoreOS<\/p>\n<\/li>\n<\/ul>\n<p> <a href=\"https:\/\/getfedora.org\/ru\/coreos?stream=stable\"><u>https:\/\/getfedora.org\/ru\/coreos?stream=stable<\/u><\/a><\/p>\n<p> After you start using the <a href=\"https:\/\/mirror.openshift.com\/pub\/openshift-v4\/dependencies\/rhcos\/4.6\/4.6.8\/rhcos-4.6.8-x86_64-live.x86_64.iso\"><u>rhcos-4.6.8-x86_64-live.x86_64.iso<\/u><\/a> image, the welcome screen and the core-installer sample information will appear.<\/p>\n<p> In the case of a manual install and image preparation, the installation is started using the coreos-installer, it will necessitate the input of the necessary installation parameters, a URL to the ignition files (for bootstrap, compute node and worker node) files and the like.<\/p>\n<p> This may also require additional configuration, such as setting a proper IP address for the pertinent virtual machine or server, DNS names, addresses to where the images are located, etc. Here&#8217;s one of my examples for KVM virtualisation:<\/p>\n<pre><code class=\"bash\">sudo coreos-installer install \/dev\/vda --ignition-url http:\/\/192.168.122.239:8080\/okd4\/master.ign --insecure-ignition --append-karg rd.neednet=1 --append-karg ip=192.168.122.16::192.168.122.1:255.255.255.0:okd4-control-plane-1.lab.okd.local:ens3:none --append-karg nameserver=192.168.122.1 --append-karg nameserver=192.168.122.239 --image-url http:\/\/192.168.122.239:8080\/okd4\/fcos.raw.xz<\/code><\/pre>\n<p>P.\u00a0S. If you want to train &#171;for fun&#187;, to always be ready for a situation when you have no SSH key on hand, you can install the ISO on a virtual machine. For example, on VirtualBox:<\/p>\n<figure class=\"\"><figcaption><\/figcaption><\/figure>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/670302\/\"> https:\/\/habr.com\/ru\/articles\/670302\/<\/a><br \/><\/br><\/br><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-380845","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/380845","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=380845"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/380845\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=380845"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=380845"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=380845"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}