{"id":493218,"date":"2026-09-02T10:49:17","date_gmt":"2026-09-02T10:49:17","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=493218"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=493218","title":{"rendered":"I Cloned the Same VM 100\u00a0Times and Found Five Identities That Were No Longer Unique"},"content":{"rendered":"
\n

The original plan was simple. I prepared one Ubuntu Server VM under KVM, installed the packages needed for the lab, enabled SSH, configured cloud\u2011init, created a WireGuard interface for the private test network and installed a small monitoring agent. Once the system looked clean, I shut it down and used it as a template for 100\u00a0full clones.<\/p>\n

The hypervisor did its part perfectly. Every clone received its own virtual NIC and MAC address. DHCP assigned different addresses, and provisioning changed the hostname according to the VM number. The resulting environment looked exactly as expected.<\/p>\n

vm-001    10.20.0.101vm-002    10.20.0.102vm-003    10.20.0.103...vm-100    10.20.0.200<\/code>
<\/a><\/div><\/pre>\n
Ping worked, SSH worked, the monitoring dashboard showed 100\u00a0hosts, and every VM produced its own CPU and memory graphs. For a few minutes I considered the job finished.<\/p>\n
The problem appeared when I started writing a small inventory script for an unrelated test. I wanted every result to contain something more stable than an IP address, so I added \/etc\/machine-id<\/code> to the collected data. The first three rows were identical.<\/p>\n
At first I thought the script was reading the local file instead of the remote one. That would have been an embarrassingly simple bug. I logged into several VMs manually and ran the same command.<\/p>\n
cat \/etc\/machine-id<\/code>
<\/a><\/div><\/pre>\n
vm-001\u00a0returned:<\/p>\n
8f75c476869c4b1cbcc8f03a714e1285<\/code>
<\/a><\/div><\/pre>\n
vm-018\u00a0returned exactly the same value. So did vm-064\u00a0and vm-100.<\/p>\n
That was the point where the model in my head started to change. The network had created 100\u00a0distinct endpoints, but that did not mean the operating systems inside those endpoints had become 100\u00a0distinct installations.<\/p>\n
The template had already been booted before cloning. systemd had generated an identity, SSH had created host keys, cloud\u2011init had finished initialization, the filesystem had been created, and WireGuard already had its private key. Shutting down the source VM did not return it to some neutral pre\u2011identity state. It simply froze all of that persistent state inside the disk image.<\/p>\n
The clones were not being born from a blank template. They were waking up from copies of a machine that had already decided who it was.<\/p>\n
The first collision was machine\u2011id, but SSH was much worse<\/h3>\n
The duplicated \/etc\/machine-id<\/code> was the easiest problem to understand. On a systemd\u2011based Linux installation, this file contains a persistent identifier for the local system. It is small, boring and easy to forget, which is probably why it is such a good example of this entire problem.<\/p>\n
Nothing dramatic happens when two machines share it. The kernel still boots, applications still start and the server remains reachable. That makes the collision easy to miss. At the same time, software can use machine\u2011id directly or derive stable identifiers from it, so duplicated values can confuse inventory tools, agents and services that assume the value distinguishes installations.<\/p>\n
The SSH check was less harmless.<\/p>\n
Each clone contained the same files under \/etc\/ssh<\/code> that had been generated on the original VM:<\/p>\n
ls -l \/etc\/ssh\/ssh_host_*<\/code>
<\/a><\/div><\/pre>\n
The relevant files included the RSA, ECDSA and Ed25519\u00a0host keys. I compared the Ed25519\u00a0fingerprints on several machines.<\/p>\n
ssh-keygen -lf \/etc\/ssh\/ssh_host_ed25519_key.pub<\/code>
<\/a><\/div><\/pre>\n
Every host returned the same fingerprint.<\/p>\n
To make sure this was not just some formatting oddity, I hashed the private key itself.<\/p>\n
sudo sha256sum \/etc\/ssh\/ssh_host_ed25519_key<\/code>
<\/a><\/div><\/pre>\n
Again, identical.<\/p>\n
This changed the seriousness of the experiment. A duplicated machine\u2011id is mainly an identity\u2011management problem. A duplicated SSH host private key is a duplicated secret.<\/p>\n
Suppose vm-023\u00a0is destroyed and another clone later receives the same network address. With unique SSH host keys, a client that previously connected to that address can notice that the server identity has changed. With a shared host key, the replacement presents exactly the key the client already trusts.<\/p>\n
There is a second consequence that I initially overlooked. If one VM is compromised and its SSH host private key is stolen, the attacker does not obtain the identity of one server. They obtain the identity currently used by the entire cloned fleet.<\/p>\n
That was enough to stop treating the problem as a funny lab artifact. The template had silently turned per\u2011host credentials into shared credentials.<\/p>\n
The virtual disk was unique, but the filesystem inside it was not<\/h3>\n
The third collision was the root filesystem UUID. Once I thought about it properly, this one was almost embarrassingly predictable.<\/p>\n
Each clone had its own virtual disk object. From the hypervisor point of view those disks were different. But a virtual disk is only a container for bytes, and the ext4\u00a0filesystem inside it had been copied byte for byte from the source machine.<\/p>\n
I checked the root device first:<\/p>\n
findmnt -n -o SOURCE \/<\/code>
<\/a><\/div><\/pre>\n
Then the filesystem metadata:<\/p>\n
sudo blkid<\/code>
<\/a><\/div><\/pre>\n
Every clone reported the same UUID for the root ext4\u00a0filesystem.<\/p>\n
\/dev\/vda2: UUID=8e157db1-d42a-4d2c-a85d-d21db08353ad TYPE=ext4<\/code>
<\/a><\/div><\/pre>\n
While every disk remained isolated inside its own VM, nothing obviously failed. That made it tempting to ignore the duplication. The problem became much easier to see when I attached two cloned system disks to the same recovery machine.<\/p>\n
Now one Linux system could see two filesystems claiming exactly the same UUID.<\/p>\n
That is where the difference between disk identity and filesystem identity stops being theoretical. Recovery commands, mount configuration and tooling that expects a UUID to identify one filesystem can become ambiguous. The hypervisor had created another virtual disk, but it had not created another ext4\u00a0filesystem. It had copied an existing one.<\/p>\n
I briefly considered adding a command to regenerate the UUID during first boot, but that felt too casual for the root filesystem. Boot configuration may refer to the UUID, \/etc\/fstab<\/code> may refer to it, and changing it after the system is already assembled can create a much more annoying failure than the one being fixed.<\/p>\n
So filesystem identity became a provisioning concern rather than a generic first\u2011boot cleanup step. In my setup, the disk image is now prepared so the filesystem identity is handled deliberately during image creation or disk provisioning.<\/p>\n
That was the first useful design lesson from the experiment: not every identity should be reset in the same place. Different components own different identities, and they should ideally be allowed to create those identities at the correct stage of the machine lifecycle.<\/p>\n
cloud\u2011init was not broken; my image lifecycle was<\/h3>\n
cloud\u2011init produced the strangest symptoms because initially it looked like the component that should have prevented exactly this problem.<\/p>\n
Some per\u2011instance initialization did not behave consistently after cloning, so I started with the obvious checks.<\/p>\n
cloud-init status --longcloud-init query instance_idls -la \/var\/lib\/cloud\/<\/code>
<\/a><\/div><\/pre>\n
The answer was sitting in \/var\/lib\/cloud<\/code>.<\/p>\n
The source VM had already completed a normal boot before it became a template. That meant cloud\u2011init had already stored local state describing the instance it had initialized. In the first version of the lab I had also preserved instance\u2011specific NoCloud seed data longer than I should have.<\/p>\n
The clones therefore inherited not only software configuration but evidence that first\u2011boot initialization had already happened.<\/p>\n
For a while I was mentally blaming cloud\u2011init because I expected it to make the cloned systems unique. That expectation was backwards. cloud\u2011init can initialize a new instance, but it cannot magically know that a filesystem containing completed initialization state is supposed to represent a completely different machine unless the image is prepared for that transition.<\/p>\n
The source VM had already crossed the boundary between image and instance.<\/p>\n
That distinction became the most useful way to think about the whole setup. Packages belong to the image. Base operating system configuration can belong to the image. A host identity belongs to the instance. Cloud metadata belongs to the instance. A secret generated to identify one particular server belongs to the instance.<\/p>\n
Once that boundary is clear, a lot of cleanup decisions stop looking arbitrary.<\/p>\n
The fifth duplicate was a WireGuard private identity<\/h3>\n
WireGuard was the most obvious mistake in retrospect.<\/p>\n
The source VM already had a private key under its WireGuard configuration when it was converted into a template. Naturally, the private key was copied to every clone.<\/p>\n
Checking the public identity was enough to prove it:<\/p>\n
sudo wg show wg0 public-key<\/code>
<\/a><\/div><\/pre>\n
The result was identical on every VM.<\/p>\n
At the network layer, the environment looked like 100\u00a0hosts. At the WireGuard layer, it looked like one cryptographic peer copied 100\u00a0times.<\/p>\n
This one was entirely self\u2011inflicted. WireGuard had no way to know that its private key was sitting inside a reusable image, and the hypervisor had no way to know that one particular sequence of bytes inside the virtual disk represented a credential that should become unique.<\/p>\n
That realization made me widen the search beyond the original five fields. What about a monitoring agent that generates an installation UUID on its first start? What about a backup client that enrolls once and stores a local identity? What about a service discovery agent that writes its node ID under \/var\/lib<\/code>? What about locally generated TLS keys?<\/p>\n
A VM image has no concept of identity semantics. It simply contains files and blocks. If some software generates a persistent identifier before the image is sealed, that identifier becomes part of the image unless something explicitly removes it.<\/p>\n
The practical question I now ask for every service is simple: if I copy this state to another machine, can that machine appear to be the original one?<\/p>\n
If the answer is yes, that state deserves attention before the image is reused.<\/p>\n
Checking 100\u00a0machines manually was not going to happen<\/h3>\n
After finding the first few collisions manually, I wanted to know whether the problem existed on a handful of VMs or across the whole fleet. Logging into 100\u00a0systems and running five commands on each was not an attractive plan, so I wrote a collector.<\/p>\n
The script connects to every VM, extracts the identities I cared about and stores them in a CSV file. It is intentionally simple because the interesting part here is not SSH automation. The interesting part is having a repeatable way to ask whether the supposedly independent machines are actually unique at the layers being tested.<\/p>\n
#!\/usr\/bin\/env bashset -uOUTPUT=\"vm-identities.csv\"printf '%s\\n' \\  \"host,machine_id,ssh_ed25519,root_uuid,cloud_instance_id,wireguard_public_key\" \\  > \"$OUTPUT\"for i in $(seq -w 1 100); do    HOST=\"vm-$i\"    echo \"Scanning $HOST\" >&2    SSH_OPTIONS=(        -o BatchMode=yes        -o ConnectTimeout=3        -o StrictHostKeyChecking=accept-new    )    MACHINE_ID=$(        ssh \"${SSH_OPTIONS[@]}\" \"$HOST\" \\            \"cat \/etc\/machine-id 2>\/dev\/null\" \\            2>\/dev\/null    )    SSH_ED25519=$(        ssh \"${SSH_OPTIONS[@]}\" \"$HOST\" \\            \"ssh-keygen -lf \/etc\/ssh\/ssh_host_ed25519_key.pub 2>\/dev\/null |             awk '{print \\$2}'\" \\            2>\/dev\/null    )    ROOT_UUID=$(        ssh \"${SSH_OPTIONS[@]}\" \"$HOST\" '            ROOT=$(findmnt -n -o SOURCE \/)            blkid -s UUID -o value \"$ROOT\" 2>\/dev\/null        ' 2>\/dev\/null    )    CLOUD_INSTANCE_ID=$(        ssh \"${SSH_OPTIONS[@]}\" \"$HOST\" '            cloud-init query instance_id 2>\/dev\/null ||            cat \/var\/lib\/cloud\/data\/instance-id 2>\/dev\/null        ' 2>\/dev\/null    )    WG_PUBLIC_KEY=$(        ssh \"${SSH_OPTIONS[@]}\" \"$HOST\" \\            \"sudo wg show wg0 public-key 2>\/dev\/null\" \\            2>\/dev\/null    )    MACHINE_ID=${MACHINE_ID\/\/$'\\n'\/}    SSH_ED25519=${SSH_ED25519\/\/$'\\n'\/}    ROOT_UUID=${ROOT_UUID\/\/$'\\n'\/}    CLOUD_INSTANCE_ID=${CLOUD_INSTANCE_ID\/\/$'\\n'\/}    WG_PUBLIC_KEY=${WG_PUBLIC_KEY\/\/$'\\n'\/}    printf '%s,%s,%s,%s,%s,%s\\n' \\        \"$HOST\" \\        \"$MACHINE_ID\" \\        \"$SSH_ED25519\" \\        \"$ROOT_UUID\" \\        \"$CLOUD_INSTANCE_ID\" \\        \"$WG_PUBLIC_KEY\" \\        >> \"$OUTPUT\"doneechoecho \"Inventory written to $OUTPUT\"for column in 2 3 4 5 6; do    count=$(        cut -d, -f\"$column\" \"$OUTPUT\" |        tail -n +2 |        grep -v '^$' |        sort -u |        wc -l    )    echo \"column $column: $count\"done<\/code>
<\/a><\/div><\/pre>\n
The first complete run produced a wonderfully bad result.<\/p>\n
Hosts scanned:                    100Unique machine IDs:                 1Unique SSH Ed25519 host keys:        1Unique root filesystem UUIDs:        1Unique cloud-init instance IDs:      1Unique WireGuard public keys:        1<\/code>
<\/a><\/div><\/pre>\n
That output changed the problem from a collection of suspicious files into something measurable. The environment contained 100\u00a0running operating systems, but five independent mechanisms still saw essentially one identity.<\/p>\n
I turned the collisions into a test that could fail the image build<\/h3>\n
A CSV file was useful while investigating the problem, but I wanted the check to survive after the experiment. If the template was rebuilt six months later, I did not want uniqueness to depend on someone remembering to inspect several files manually.<\/p>\n
The next step was a small Python validator. It groups hosts by every collected identity, prints collisions and returns a non\u2011zero exit code when duplicated values are found. That makes the check useful not only for a lab but also as a gate in a template validation pipeline.<\/p>\n
#!\/usr\/bin\/env python3from __future__ import annotationsimport csvimport sysfrom collections import defaultdictfrom pathlib import PathINPUT_FILE = Path(\"vm-identities.csv\")FIELDS = {    \"machine_id\": \"Linux machine ID\",    \"ssh_ed25519\": \"SSH Ed25519 host key\",    \"root_uuid\": \"Root filesystem UUID\",    \"cloud_instance_id\": \"cloud-init instance ID\",    \"wireguard_public_key\": \"WireGuard public key\",}def load_inventory(path: Path) -> list[dict[str, str]]:    if not path.exists():        print(f\"Inventory file not found: {path}\", file=sys.stderr)        raise SystemExit(2)    with path.open(\"r\", encoding=\"utf-8\", newline=\"\") as file:        rows = list(csv.DictReader(file))    if not rows:        print(\"Inventory is empty\", file=sys.stderr)        raise SystemExit(2)    return rowsdef group_hosts(    rows: list[dict[str, str]],    field: str,) -> dict[str, list[str]]:    groups: dict[str, list[str]] = defaultdict(list)    for row in rows:        host = row.get(\"host\", \"\").strip()        value = row.get(field, \"\").strip()        if not host:            continue        groups[value or \"<missing>\"].append(host)    return groupsdef check_field(    rows: list[dict[str, str]],    field: str,    label: str,) -> bool:    groups = group_hosts(rows, field)    missing = groups.pop(\"<missing>\", [])    collisions = {        value: hosts        for value, hosts in groups.items()        if len(hosts) > 1    }    print(\"=\" * 72)    print(label)    print(\"=\" * 72)    print(f\"Unique values: {len(groups)}\")    print(f\"Missing values: {len(missing)}\")    if not collisions:        print(\"Status: OK\")        print()        return False    print(f\"Collisions: {len(collisions)}\")    print()    for value, hosts in sorted(        collisions.items(),        key=lambda item: len(item[1]),        reverse=True,    ):        print(f\"Value: {value}\")        print(f\"Used by: {len(hosts)} hosts\")        for host in hosts[:8]:            print(f\"  {host}\")        if len(hosts) > 8:            print(f\"  ... {len(hosts) - 8} more\")        print()    return Truedef main() -> int:    rows = load_inventory(INPUT_FILE)    print(f\"Hosts scanned: {len(rows)}\")    print()    failed = False    for field, label in FIELDS.items():        if check_field(rows, field, label):            failed = True    if failed:        print(\"RESULT: identity collisions detected\")        return 1    print(\"RESULT: no identity collisions detected\")    return 0if __name__ == \"__main__\":    raise SystemExit(main())<\/code>
<\/a><\/div><\/pre>\n
After rebuilding the template, I ran the same validation again. This time the important host\u2011level identities were unique.<\/p>\n
Hosts scanned:                    100Unique machine IDs:               100Unique SSH Ed25519 host keys:      100Unique cloud-init instance IDs:    100Unique WireGuard public keys:      100<\/code>
<\/a><\/div><\/pre>\n
Filesystem identity remained a separate validation because I had moved that part into disk provisioning. That separation was intentional rather than an exception hidden in the script.<\/p>\n
The rule that came out of this was useful far beyond this particular lab: duplicated values are not automatically bugs, but duplicated identities should always be deliberate.<\/p>\n
Fixing the image was cleaner than repairing every clone<\/h3>\n
My first instinct was to create one large post\u2011clone script that removed IDs, deleted keys, reset cloud\u2011init and restarted services. It would probably have worked for this exact image.<\/p>\n
It would also have become a trap.<\/p>\n
A better solution was to change the template lifecycle so that the image was sealed before instance\u2011specific identity became permanent.<\/p>\n
For systemd, the reusable image should not retain the finished machine ID of the source VM. Before sealing, the machine ID can be cleared so the new instance creates its own value during initialization.<\/p>\n
sudo truncate -s 0 \/etc\/machine-id<\/code>
<\/a><\/div><\/pre>\n
I also check \/var\/lib\/dbus\/machine-id<\/code> rather than assuming its layout, because depending on the system it may be linked or managed differently.<\/p>\n
ls -l \/etc\/machine-id \/var\/lib\/dbus\/machine-id<\/code>
<\/a><\/div><\/pre>\n
For OpenSSH, the host keys are removed before sealing.<\/p>\n
sudo rm -f \/etc\/ssh\/ssh_host_*<\/code>
<\/a><\/div><\/pre>\n
The important part is not the deletion command itself. The important part is making sure the boot process generates new host keys before SSH is used.<\/p>\n
cloud\u2011init is reset as part of the image sealing process rather than treated as something that will somehow repair an already initialized clone.<\/p>\n
sudo cloud-init clean --logs --machine-id<\/code>
<\/a><\/div><\/pre>\n
Instance metadata is then supplied separately to each new VM.<\/p>\n
WireGuard follows the same principle. Its private key does not belong in a generic template if the key is intended to identify one host. The key is generated during per\u2011instance provisioning.<\/p>\n
sudo install -d -m 700 \/etc\/wireguardsudo sh -c '    umask 077    wg genkey > \/etc\/wireguard\/private.key    wg pubkey < \/etc\/wireguard\/private.key \\        > \/etc\/wireguard\/public.key'<\/code>
<\/a><\/div><\/pre>\n
The resulting lifecycle is now much easier to understand.<\/p>\n
build base image        \u2193install software        \u2193remove instance-specific identity        \u2193seal image        \u2193clone        \u2193supply instance metadata        \u2193first boot        \u2193generate machine identity and credentials<\/code>
<\/a><\/div><\/pre>\n
Previously I had accidentally built this instead:<\/p>\n
build machine        \u2193boot it        \u2193generate identity and credentials        \u2193configure everything        \u2193seal finished machine        \u2193copy its identity 100 times<\/code>
<\/a><\/div><\/pre>\n
Those two pipelines look similar until something actually depends on the copied identity.<\/p>\n
There is no single identity of a Linux VM<\/h3>\n
The experiment became more interesting once I stopped asking whether a clone was unique and started asking who considered it unique.<\/p>\n
The hypervisor saw 100\u00a0VM objects. The virtual switch saw 100\u00a0MAC addresses. DHCP saw 100\u00a0clients. DNS saw 100\u00a0hostnames. Monitoring saw 100\u00a0metric sources.<\/p>\n
systemd originally saw one machine identity copied 100\u00a0times. SSH saw one host key copied 100\u00a0times. ext4\u00a0saw 100\u00a0copies of one filesystem. WireGuard saw one cryptographic peer. cloud\u2011init inherited state from one initialized source.<\/p>\n
None of those layers were behaving randomly. Each one was internally consistent. They simply used different definitions of machine identity.<\/p>\n
That explains why this class of problem can stay invisible for a surprisingly long time. Normal health checks tend to focus on reachability, process state, CPU, memory, HTTP responses and network addresses. A VM can pass all of those checks while still carrying an identity copied from another machine.<\/p>\n
The failure is not necessarily a crash. Sometimes it is two monitoring agents overwriting each other. Sometimes it is a strange SSH trust relationship. Sometimes it appears only during recovery, when two disks with the same filesystem UUID are attached to one host. Sometimes it is a security issue caused by a private key that was never supposed to leave the original machine.<\/p>\n
The most dangerous cloned identity is often the one nobody remembered existed.<\/p>\n
What I check before sealing a VM image now<\/h3>\n
The template process now has a small identity review before an image is considered reusable:<\/p>\n