{"id":494307,"date":"2026-09-11T05:07:26","date_gmt":"2026-09-11T05:07:26","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=494307"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=494307","title":{"rendered":"Looking for lateral movement with a neural network trained on synthetic data"},"content":{"rendered":"
\n

Can you train a cyberattack detector without ever showing it a real cyberattack?<\/p>\n

It sounds like a contradiction. If you want a neural network to detect lateral movement, you would expect to show it lateral movement. I did the opposite: I generated an entire corporate network with its login history, staged an attack inside that artificial world, and trained networks on it. Not a single real row in the training data. The whole world is a 135-line config; each network has four thousand parameters and trains in seconds on a laptop, and the best result came from six of them, trained on six different invented worlds.<\/p>\n

Then I pointed them at real data: the authentication logs of Los Alamos National Laboratory, 1.65 billion events, with red-team exercises labelled in them.<\/p>\n

And it worked.<\/strong> The networks rank 3.6 million windows by suspicion, and the top twenty-three rows of that list hold sixteen real attacks and seven false alarms: all the analyst has to do is open those rows. A threshold counter on the same data needs a hundred and sixty-one thousand false alarms to reach the sixteenth attack. By AUC the synthetic training landed inside the range of published research trained on real labelled data, although the two cannot be compared head-on, and I will explain why.<\/p>\n

No miracle happened. Something else did: twice I wrote down an elegant conclusion, and twice the experiment made me take it back. And I found the loop that makes a data generator worth writing at all: the detector\u2019s error points at one specific machine, you work out which phenomenon your invented world lacks, you add two lines to the config, and the error is gone.<\/p>\n

The rest, in order: who I am and why I needed this, what lateral movement is, how the invented world is built, what the exam showed, and how much of it is honest. If you do not need the theory, skip ahead to \u201cThe first result on real data\u201d<\/a>.<\/p>\n

I write TDCV2, an open-source constructor for test data, and I wanted to know what it is actually good for. Filling a table with fake Smiths is one thing; producing data that something working can be trained on is another. You can\u2019t check that on a toy, so I took a real information-security problem, trained a classifier on invented data only, and turned it loose on the logs of a real network at Los Alamos National Laboratory.<\/em><\/p>\n

Three caveats, so the rest reads the way it should.<\/p>\n

I am not a machine-learning specialist<\/strong>. Everything below is the work of a person who writes a data generator and decided to test it on a serious problem. If it looks to you like I\u2019m reinventing a wheel somewhere, or reaching for the wrong tool — you\u2019re probably right.<\/p>\n

I am not competing with anyone<\/strong>. The results from research groups further down are there as a coordinate system, so the reader has a sense of scale, not because I\u2019m claiming a place in that row.<\/p>\n

I wasn\u2019t counting on a result<\/strong>. Honestly, I expected the synthetic data to do worse! It went differently, and that\u2019s the interesting part. There was no miracle anyway, but two things happened that are more interesting than a miracle: a measured boundary of a whole class of methods, and a limit on training from generated data that I then retracted myself.<\/p>\n

What lateral movement is<\/h3>\n

Since the headline opens with a bit of jargon, let me explain what this beast is first. All the more so because the phenomenon is built in a non-obvious way, and half of your intuition about \u201chacking\u201d doesn\u2019t apply.<\/p>\n

There\u2019s a common picture in which the attacker cracks a password and that\u2019s the end of it. In reality, getting a login somewhere is only the beginning, and it is almost never the place they wanted. The first thing to fall is somebody\u2019s workstation: a person opened an attachment, or a service with an old hole was exposed to the outside. There is nothing valuable on that machine. The valuable things are in the database, on the file server, in the domain controller — and those still have to be reached.<\/p>\n

Reaching them is what\u2019s called lateral movement<\/strong>. In the MITRE ATT&CK classification it is a tactic of its own, TA0008, and its most troublesome technique is T1078, Valid Accounts.<\/p>\n

How it spreads<\/h4>\n

The mechanism is simple. A captured machine almost always has somebody else\u2019s credentials lying on it: a cached domain login, a token in memory, a password in a maintenance script, a saved connection. The attacker takes them and logs in to the neighbouring machine — an ordinary login, done exactly the way a real employee does it. There they collect that machine\u2019s credentials. And round it goes.<\/p>\n

After that it\u2019s arithmetic: every step grows both the number of machines and the number of available accounts, and every new account opens new machines. One workstation becomes ten, ten become a hundred.<\/p>\n

\"How<\/p>\n
How this spreads. In our data it was exactly like that: 94% of the labelled events came from a single foothold, and the exercise reached 301 machines in all<\/em><\/figcaption><\/div>\n<\/figure>\n

The speed is nothing like the movies. Each step takes hours, the whole picture unfolds over days and weeks. That, broadly, is the one piece of good news — there is time to notice.<\/p>\n

Who this affects, and who it doesn\u2019t<\/h4>\n

Lateral movement needs somewhere to move, laterally. It lives in large corporate networks: hundreds and thousands of machines, a shared domain, service accounts with access to many places at once, administrators who legitimately go everywhere.<\/p>\n

A small business is not threatened by this phenomenon as such — not because it is safer there, but because there is nowhere to \u201cmove\u201d. If a company is five people, three laptops and all the work in cloud services, then an attacker who has taken the accountant\u2019s laptop is already where the money is. They need no \u201ctactic\u201d, it\u2019s all at hand. Such a company will have other problems, and those are caught differently.<\/p>\n

So everything below is about a network with somewhere to go.<\/p>\n

Why it is hard to catch<\/h3>\n

An awkward property of this phenomenon. Every single lateral-movement event is flawless<\/strong>: the password is correct, the answer is \u201csuccess\u201d, the account is genuine, the domain is our own, the time is business hours. There is nothing anomalous in it, because it really is an ordinary login — just made by the wrong person.<\/p>\n

To a signature detector, and to any threshold counter, it is indistinguishable from an employee doing their job.<\/p>\n

\"An<\/p>\n
An ordinary login on the left, lateral movement on the right. Every field is valid in both cases; a single event cannot tell them apart<\/em><\/figcaption><\/div>\n<\/figure>\n

Why I needed this particular problem<\/h3>\n

Now I can come back to where I started — the generator.<\/p>\n

Testing a data constructor on a problem whose answer is known in advance isn\u2019t interesting. You want one where invented data will either work or fail honestly, and where the outcome can be measured rather than eyeballed. Lateral movement fitted for three reasons, and each of them matters.<\/p>\n

    \n
  1. \n

    Nobody has labelled examples.<\/strong> To train a detector the usual way, you need examples of the attack in your own network. And to get them, you first have to be compromised, notice it, and label it. Synthetic data here isn\u2019t a shortcut, it\u2019s the only way — if it works at all.<\/p>\n<\/li>\n

  2. \n

    The phenomenon has a computable definition.<\/strong> \u201cAn account logs in where it has never been, from a machine where it has never been\u201d is a rule, not a picture. And a rule can be written down as a config without a single real example in front of you.<\/p>\n<\/li>\n

  3. \n

    There is open data from a real network, with labelled ground truth in it.<\/strong> Which means the exam can be taken honestly: train on the invented, check on the live, and not grade your own homework.<\/p>\n<\/li>\n<\/ol>\n

    The plan followed from that, and it was simple. Show the network not one real row — not in training, not in validation, not in threshold picking. Write the entire training world as a generator config. And then turn it loose on real logs and see what it finds there.<\/p>\n

    The exam data here is the open Los Alamos set Comprehensive, Multi-Source Cyber-Security Events<\/a>: 58 days, 17,684 machines, 12,425 users, 1.65 billion events. What interests me is the authentication log.<\/p>\n

    The labelled truth in it is the record of a red team exercise. And it looks like this:<\/p>\n

    \n

    749 exercise events: 104 compromised accounts, 301 destination machines and a total of 4 sources<\/strong>. 94% of the events come from a single foothold.<\/p>\n<\/blockquote>\n

    One machine. A hundred stolen logins. Three hundred places.<\/p>\n

    The exercise activity is spread across the days extremely unevenly, and from here on I refer to three sets constantly. Everything below is counted in windows<\/strong>: a window is one machine for one hour, everything it did in that hour. A labelled window is one that caught at least one exercise event. To keep it straight, let\u2019s fix the names right away:<\/p>\n

    \n
    \n\n\n\n\n\n\n
    \n

    set<\/p>\n<\/th>\n

    		\n

    what it is<\/p>\n<\/th>\n

    		\n

    windows<\/p>\n<\/th>\n

    		\n

    labelled<\/p>\n<\/th>\n<\/tr>\n

    \n

    day 8<\/strong><\/p>\n<\/td>\n

    		\n

    the densest activity; debugging was done on it<\/p>\n<\/td>\n

    		\n

    231,787<\/p>\n<\/td>\n

    		\n

    15<\/p>\n<\/td>\n<\/tr>\n

    \n

    day 12<\/strong><\/p>\n<\/td>\n

    		\n

    second densest, also a working day<\/p>\n<\/td>\n

    		\n

    223,987<\/p>\n<\/td>\n

    		\n

    12<\/p>\n<\/td>\n<\/tr>\n

    \n

    held-out set<\/strong><\/p>\n<\/td>\n

    		\n

    the other sixteen days, never touched<\/p>\n<\/td>\n

    		\n

    3,600,398<\/p>\n<\/td>\n

    		\n

    64<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

    Day 8 turned out to be easy — almost any model solves it. Day 12 is hard: on it the models finally differ. The held-out set is the only honest measure, and it will become clear why.<\/p>\n

    The rule that saved the project: the exam first<\/h3>\n

    My first impulse was to sit down and write the generator config. Good thing I didn\u2019t.<\/p>\n

    I spent half a day opening up and measuring the real data before<\/em> the first line of config. And it turned out the problem was not the one I had pictured: I was thinking about password guessing, and what\u2019s labelled is an entirely different phenomenon. Everything I would have written in advance would have gone in the bin, all of it.<\/p>\n

    The reconnaissance produced three facts that determined absolutely everything that followed.<\/p>\n

    The account name carries no useful signal<\/h4>\n

    The main \u201cvictim\u201d of the exercise — the account through which the most labelled events passed — appears 30,497 times<\/strong> in a clean four-hour slice. That is the busiest regular administrator, and catching them as an anomaly is pointless.<\/p>\n

    Which means the feature has to be not about whose<\/em> account it is, but about where it came from<\/em>. Not a property, but a relationship.<\/p>\n

    The naive counter is fooled by the shape of the network<\/h4>\n

    The hypothesis \u201cmany different accounts from one machine is suspicious\u201d breaks against the topology:<\/p>\n

    2996  2752  2686  2590   domain controllers 252   118   110          file servers   1     2     1          workstations<\/code>
    <\/a><\/div><\/pre>\n
    The foothold with its hundred accounts falls between<\/em> the controllers and the workstations, and is indistinguishable from a small server. With a counter over destination machines it is exactly the same story.<\/p>\n
    \"A<\/p>\n
    A threshold that catches the foothold covers half the servers. A threshold that leaves the servers alone misses the foothold<\/em><\/figcaption><\/div>\n<\/figure>\n
    The real signal is the novelty of the connection<\/h4>\n
    Lateral movement produces new edges<\/strong> in the login graph: the pair \u201caccount — machine it has never logged in from\u201d, or \u201cmachine — place it has never reached before\u201d. And here is the measurement that made the reconnaissance worth doing. The share of new edges in normal activity:<\/p>\n
    \n
    \n\n\n\n\n\n\n
    \n
    share of new edges in normal activity<\/p>\n<\/th>\n
    		\n
    value<\/p>\n<\/th>\n<\/tr>\n
    \n
    median<\/p>\n<\/td>\n
    		\n
    0.0000<\/strong><\/p>\n<\/td>\n<\/tr>\n
    \n
    90th percentile<\/p>\n<\/td>\n
    		\n
    0.0000<\/strong><\/p>\n<\/td>\n<\/tr>\n
    \n
    99th percentile<\/p>\n<\/td>\n
    		\n
    0.5000<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
    For those new to this.<\/strong> A percentile is \u201cthe share of windows whose value is no larger than this\u201d. The 90th percentile being zero means nine windows out of ten have no new edges at all; and only one window in a hundred gets a share of new edges as high as one half.<\/p>\n
    People walk well-worn routes. The background is almost perfectly clean, and that is exactly what makes the problem solvable in principle.<\/p>\n
    \"An<\/p>\n
    An ordinary day adds zero new edges to the login graph. Lateral movement consists of almost nothing else<\/em><\/figcaption><\/div>\n<\/figure>\n
    A world in a hundred and thirty lines<\/h3>\n
    Only now can the generator be written — already knowing exactly what to produce.<\/p>\n
    The key idea: don\u2019t imitate lateral movement, produce its definition<\/strong>. The phenomenon has a computable formulation — \u201can account logs in where it has never been, from a machine where it has never been\u201d. Since the definition is algorithmic, it is synthesizable without losing the essence.<\/p>\n
    The heart of the config looks like this:<\/p>\n
    <pool name=\"Work\" count=\"290\">  <sequence name=\"hid\"><gen type=\"increment\" value=\"1\"\/><\/sequence>  <sequence name=\"role\"><gen type=\"text\" value=\"0,2,3,4\"                             percent=\"78.8,0.6,20,0.6\"\/><\/sequence>  <sequence name=\"userBase\"><gen type=\"formula\" expr=\"hid * 30\"\/><\/sequence>  <sequence name=\"userSpan\"><gen type=\"number\" value=\"1..4\"\/><\/sequence>  <sequence name=\"fgBase\"><gen type=\"number\" value=\"1..290\"\/><\/sequence>  <sequence name=\"fgSpan\"><gen type=\"number\" value=\"2..60\"\/><\/sequence>  ...<\/pool><\/code>
    <\/a><\/div><\/pre>\n
    The machines live in a lookup table, each with its own number, its own role and its own circle of accounts: the machine with number hid<\/code> owns accounts from hid*30<\/code> to hid*30+span<\/code>. Separately it has a compromise circle<\/strong> — the numbers of other<\/em> machines whose accounts travel through it. The width of that circle runs from 2 to 60, and that isn\u2019t sloppiness but a deliberate spread: the network has to learn the class, not one particular preset.<\/p>\n
    The account number for an event is then assembled by a formula:<\/p>\n
    <sequence name=\"User\">  <gen type=\"formula\" expr=\"Foreign == 1     ? ((W.fgBase + floor(hash(N, 1) * W.fgSpan)) % 290) * 30       + floor(hash(N, 18) * 2)     : UserBase + floor(hash(N, 1) * UserSpan)\"\/><\/sequence><\/code>
    <\/a><\/div><\/pre>\n
    The formula is abridged here: the service-account and churn branches are left out, the full one is in the spoiler below. Here hash(N, salt)<\/code> is a deterministic function of the row number and a salt, giving a reproducible number between zero and one. The whole set is deterministic by seed and rebuilds byte for byte.<\/p>\n
    For those new to this.<\/strong> A seed is the number the generator counts all its \u201crandomness\u201d from. The same seed gives the same world byte for byte, a different seed gives a different world under the same rules. I\u2019ll use the term in one more sense later: a training seed<\/em> is the same kind of number, but for the network\u2019s random initial state. Different training seeds mean the same network trained several times from different starting points.<\/p>\n
    The full world is a hundred and thirty lines and 400 thousand events in 9 seconds<\/strong>. There are five kinds of machine in it, and each one earns its place:<\/p>\n