{"id":493404,"date":"2026-09-03T17:30:04","date_gmt":"2026-09-03T17:30:04","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=493404"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=493404","title":{"rendered":"I Kept Average Test Time at 10 Seconds but Changed Only Its Variance: Why Unpredictable Tests Break Developer Focus"},"content":{"rendered":"
\n

Two test suites can have exactly the same average runtime and still create completely different development workflows. I modeled 300 code \u2192 test \u2192 fix cycles while keeping mean feedback latency close to ten seconds and changing only its variance. One runner behaved predictably. The other was usually fast but occasionally became extremely slow. The average stayed almost the same. The cost of waiting did not.<\/p>\n

<\/figure>\n

Introduction<\/h3>\n

A test that always takes ten seconds is annoying, but after a few runs it becomes predictable. I press the shortcut, know roughly how much time I have, and usually stay inside the same task.<\/p>\n

A test that sometimes finishes in two seconds and sometimes takes thirty is different. Even if its average duration is also ten seconds, I cannot build the same expectation around it. After three seconds I do not know whether the result is about to arrive or whether I have another half minute to kill.<\/p>\n

That uncertainty changes the decision to wait.<\/p>\n

Most discussions about build and test performance focus on average execution time. Sometimes p95 and p99 appear in CI dashboards, but even then they are usually treated as infrastructure metrics. I wanted to look at the same distribution from another angle: what does latency variance do to the development loop itself?<\/p>\n

So I built a simple simulation of 300 code \u2192 test \u2192 fix cycles.<\/p>\n

The goal was not to estimate a universal human attention threshold. The model is intentionally synthetic. I wanted to isolate one variable that is usually mixed together with everything else: predictability.<\/p>\n

Both simulated test runners have approximately the same average runtime.<\/p>\n

One is boring.<\/p>\n

The other has a long tail.<\/p>\n

That difference turns out to matter much more than the average suggests.<\/p>\n

1. Same Average, Different Systems<\/h3>\n

The first runner is almost deterministic. Most executions finish near ten seconds.<\/p>\n

A typical sequence might look like this:<\/p>\n

9.410.29.810.110.49.710.09.9<\/code>
<\/a><\/div><\/pre>\n
The second runner is usually much faster, but occasionally enters a slow path:<\/p>\n
2.13.02.74.22.431.82.930.9<\/code>
<\/a><\/div><\/pre>\n
The averages can be made almost identical.<\/p>\n
If I reduce both systems to one metric, they look equivalent.<\/p>\n
That is exactly the problem.<\/p>\n
The stable runner has a narrow distribution. Its median, p95 and p99 remain relatively close.<\/p>\n
The unstable runner can have a much better median while producing an ugly tail.<\/p>\n
That means a developer can experience the second system as fast most of the time and still trust it much less.<\/p>\n
This is not only about comfort.<\/p>\n
Interactive development is a control loop:<\/p>\n
change code\u2193run test\u2193receive result\u2193update hypothesis\u2193make next change<\/code>
<\/a><\/div><\/pre>\n
The usefulness of the loop depends not only on how quickly feedback arrives on average, but also on whether I can predict when it will arrive.<\/p>\n
A predictable delay lets me stay attached to the current hypothesis.<\/p>\n
An unpredictable delay creates a decision point.<\/p>\n
Do I keep waiting?<\/p>\n
Do I open documentation?<\/p>\n
Do I answer a message?<\/p>\n
Do I start another small task?<\/p>\n
Once that decision appears, test runtime stops being the full cost of feedback.<\/p>\n
2. The Important Metric Is Remaining Wait, Not Average Wait<\/h3>\n
Suppose a test has already been running for eight seconds.<\/p>\n
Knowing that average runtime is ten seconds is not enough.<\/p>\n
What I actually care about is:<\/p>\n
How much longer is this particular run likely to take,given that it has already lasted eight seconds?<\/code>
<\/a><\/div><\/pre>\n
Mathematically, this is the expected residual lifetime:<\/p>\n
E[T - t | T > t]<\/code>
<\/a><\/div><\/pre>\n
T is total execution time.<\/p>\n
t is elapsed waiting time.<\/p>\n
This quantity behaves very differently for narrow and heavy-tailed distributions.<\/p>\n
For a nearly deterministic ten-second process, waiting longer usually means I am getting closer to completion.<\/p>\n
If the test has already survived nine seconds, the expected remaining wait is tiny.<\/p>\n
That makes waiting rational.<\/p>\n
Now consider a runner where most tests finish in three seconds but some take thirty.<\/p>\n
If the test is still running after eight seconds, I have learned something important.<\/p>\n
This is probably not one of the fast runs.<\/p>\n
The fact that I have already waited a long time does not necessarily mean completion is near.<\/p>\n
It may mean I have entered the slow population.<\/p>\n
That changes the economics of waiting completely.<\/p>\n
The same effect can be described through the survival function:<\/p>\n
S(t) = P(T > t)<\/code>
<\/a><\/div><\/pre>\n
This is the probability that the process is still running after time t.<\/p>\n
From it, we can derive the hazard function:<\/p>\n
h(t) = f(t) \/ S(t)<\/code>
<\/a><\/div><\/pre>\n
where f(t) is the probability density of completion time.<\/p>\n
The name sounds more dramatic than the idea.<\/p>\n
The hazard rate tells me how likely completion is around the current moment, assuming the test has not finished yet.<\/p>\n
For a predictable runner, the hazard becomes high near the usual completion point.<\/p>\n
For a long-tailed runner, surviving beyond the normal fast region may actually be bad news.<\/p>\n
That is the first reason average latency is a weak metric for interactive tooling.<\/p>\n
Two systems can share the same mean and create opposite incentives to wait.<\/p>\n
3. Modeling 300 Feedback Cycles<\/h3>\n
I split the simulation into two groups of 300 runs.<\/p>\n
The first distribution stays tightly clustered around ten seconds.<\/p>\n
The second is a mixture.<\/p>\n
Most runs are fast, but a smaller group enters a much slower path.<\/p>\n
Conceptually:<\/p>\n
Stable runnermean \u2248 10 slow variancesmall distance between p50 and p99<\/code>
<\/a><\/div><\/pre>\n
and:<\/p>\n
Unstable runnermean \u2248 10 shigh variancefast medianlarge p95very large p99<\/code>
<\/a><\/div><\/pre>\n
This is the important part of the experiment.<\/p>\n
I am not changing average runtime deliberately.<\/p>\n
I am changing uncertainty.<\/p>\n
For the stable runner, I can approximate execution time with something close to a narrow normal distribution centered near ten seconds.<\/p>\n
For the unstable runner, a mixture model works better:<\/p>\n
T =    fast distribution, with high probability    slow distribution, with low probability<\/code>
<\/a><\/div><\/pre>\n
For example:<\/p>\n
P(fast) = 0.8P(slow) = 0.2<\/code>
<\/a><\/div><\/pre>\n
The exact coefficients are not the conclusion.<\/p>\n
They are knobs.<\/p>\n
What matters is that the second distribution has the same approximate mean but a much longer right tail.<\/p>\n
Once I compare them through p50, p95 and p99 rather than mean alone, the difference becomes obvious.<\/p>\n
The stable runner is mediocre every time.<\/p>\n
The unstable runner is excellent most of the time and terrible just often enough to destroy confidence.<\/p>\n
That trade-off appears constantly in real systems.<\/p>\n
A build step usually finishes quickly, except when dependency resolution behaves differently.<\/p>\n
An integration test normally takes a few seconds, except when a container cold-starts.<\/p>\n
A local database query is fast until something invalidates a cache.<\/p>\n
A compiler is predictable until incremental state disappears.<\/p>\n
From the perspective of the average, these events are noise.<\/p>\n
From the perspective of the developer, they are remembered.<\/p>\n
4. Developer Behavior Has Memory<\/h3>\n
A simple latency model assumes every test run is independent.<\/p>\n
I do not think developer behavior works that way.<\/p>\n
Suppose five recent runs took:<\/p>\n
91011109<\/code>
<\/a><\/div><\/pre>\n
Now compare that with:<\/p>\n
2182189<\/code>
<\/a><\/div><\/pre>\n
The average is similar.<\/p>\n
My expectation is not.<\/p>\n
After the first sequence, I trust the tool.<\/p>\n
After the second, I know that another long delay can appear without warning.<\/p>\n
That memory affects the next run.<\/p>\n
Even if the sixth test completes in three seconds, I may leave after two because the previous history has trained me not to wait.<\/p>\n
Now a fast test can produce a slow feedback loop.<\/p>\n
The test itself did nothing wrong.<\/p>\n
Its predecessors changed my behavior.<\/p>\n
A simple way to represent recent expectation is an exponentially weighted estimate:<\/p>\n
E_n = \u03b1T_n-1 + (1 - \u03b1)E_n-1<\/code>
<\/a><\/div><\/pre>\n
Recent runs receive more weight than old ones.<\/p>\n
But expected duration alone is still not enough.<\/p>\n
The developer also learns variance.<\/p>\n
A more useful internal state would contain at least:<\/p>\n
expected durationuncertaintyrecent extreme events<\/code>
<\/a><\/div><\/pre>\n
This matters because outliers can have a delayed cost.<\/p>\n
A thirty-second test does not necessarily waste only thirty seconds.<\/p>\n
It can make me switch earlier during the next several runs.<\/p>\n
The latency penalty leaks forward.<\/p>\n
That is difficult to see in CI dashboards because test systems measure executions independently.<\/p>\n
Humans do not experience them independently.<\/p>\n
5. Adding Context Switching to the Model<\/h3>\n
I then added a simple decision rule.<\/p>\n
At each point during a wait, the simulated developer can remain in the task or leave the current context.<\/p>\n
The switching probability depends on three variables:<\/p>\n
elapsed timeexpected remaining timeuncertainty<\/code>
<\/a><\/div><\/pre>\n
Conceptually:<\/p>\n
P(switch) =    sigmoid(        a        + b1 \u00d7 elapsed        + b2 \u00d7 expected_remaining        + b3 \u00d7 uncertainty    )<\/code>
<\/a><\/div><\/pre>\n
The coefficients are not empirical claims.<\/p>\n
They only provide a mechanism for comparing distributions.<\/p>\n
The result I care about is qualitative.<\/p>\n
With stable tests, uncertainty remains low. Even when execution is not especially fast, waiting is relatively easy to justify.<\/p>\n
With unstable tests, a few long runs increase the perceived uncertainty of future runs.<\/p>\n
Switching probability rises earlier.<\/p>\n
This creates situations like:<\/p>\n
run 1   3 srun 2   4 srun 3   32 srun 4   3 srun 5   4 s<\/code>
<\/a><\/div><\/pre>\n
Run 4 is technically fast.<\/p>\n
But after run 3, I may leave the IDE after two seconds.<\/p>\n
Run 4 completes one second later.<\/p>\n
I do not notice immediately because I am already somewhere else.<\/p>\n
Then I return.<\/p>\n
Then I reconstruct the original code context.<\/p>\n
A three-second test has just produced a feedback loop lasting much longer than three seconds.<\/p>\n
The test duration is not the problem.<\/p>\n
Trust in the timing model is.<\/p>\n
6. Effective Feedback Latency Is Larger Than Test Latency<\/h3>\n
This suggests separating two metrics.<\/p>\n
The first is obvious:<\/p>\n
T_test<\/code>
<\/a><\/div><\/pre>\n
The second is what I actually experience:<\/p>\n
T_effective<\/code>
<\/a><\/div><\/pre>\n
A simple decomposition is:<\/p>\n
T_effective =    T_test    + T_detection    + T_reconstruction<\/code>
<\/a><\/div><\/pre>\n
T_detection is the delay between test completion and me noticing it.<\/p>\n
T_reconstruction is the time required to resume meaningful work on the original problem.<\/p>\n
If I stay inside the IDE, both values may be close to zero.<\/p>\n
If I switch context, both can become substantial.<\/p>\n
Now variance becomes expensive in a second way.<\/p>\n
A long-tailed runner creates more context switches.<\/p>\n
More context switches increase detection delay.<\/p>\n
More detection delay increases effective feedback latency.<\/p>\n
Higher effective latency makes future waiting less attractive.<\/p>\n
That creates a feedback loop:<\/p>\n
unpredictable latency\u2193more context switching\u2193more missed completions\u2193higher effective latency\u2193less trust in waiting\u2193more context switching<\/code>
<\/a><\/div><\/pre>\n
The tool develops a reputation.<\/p>\n
Once that reputation becomes bad, even fast runs stop receiving the benefit of patience.<\/p>\n
This is why I think performance consistency can matter more than average performance for interactive tooling.<\/p>\n
7. Slow Outliers Also Change How Code Is Written<\/h3>\n
There is another effect that has nothing to do with returning to the IDE.<\/p>\n
Slow feedback changes batch size.<\/p>\n
When tests are predictable and cheap, this workflow feels natural:<\/p>\n
changetestchangetestchangetest<\/code>
<\/a><\/div><\/pre>\n
When test duration becomes unpredictable, I start wanting to amortize the risk.<\/p>\n
The workflow changes:<\/p>\n
changechangechangechangetest<\/code>
<\/a><\/div><\/pre>\n
That reduces the number of waits.<\/p>\n
It also increases the number of possible causes when a test fails.<\/p>\n
With one change between passing and failing states, the search space is small.<\/p>\n
With four changes, it is larger.<\/p>\n
Now the test may not have become slower at all.<\/p>\n
The development process around it became harder.<\/p>\n
This produces another loop:<\/p>\n
unpredictable tests\u2193less frequent runs\u2193larger edit batches\u2193larger failure search space\u2193more expensive debugging\u2193less willingness to run tests frequently<\/code>
<\/a><\/div><\/pre>\n
This is where a performance problem starts changing engineering behavior.<\/p>\n
At that point test latency is no longer just infrastructure latency.<\/p>\n
It affects the granularity of development.<\/p>\n
8. Why Removing the Worst 5 Percent May Be More Valuable Than Improving the Median<\/h3>\n
Imagine two optimization projects.<\/p>\n
The first improves the common case:<\/p>\n
beforep50 = 8 sp95 = 10 sp99 = 12 s<\/code>
<\/a><\/div><\/pre>\n
After optimization:<\/p>\n
p50 = 5 sp95 = 9 sp99 = 12 s<\/code>
<\/a><\/div><\/pre>\n
The median becomes clearly better.<\/p>\n
Now consider a different system:<\/p>\n
p50 = 8 sp95 = 14 sp99 = 35 s<\/code>
<\/a><\/div><\/pre>\n
Suppose the second optimization barely changes the median but removes the tail:<\/p>\n
p50 = 8 sp95 = 10 sp99 = 12 s<\/code>
<\/a><\/div><\/pre>\n
Which optimization improves development more?<\/p>\n
If I only count average seconds saved, the first one may look better.<\/p>\n
If I care about predictability, the second may have a larger behavioral effect.<\/p>\n
It removes the pathological runs that teach developers not to trust the tool.<\/p>\n
This is similar to tail-latency engineering in distributed systems.<\/p>\n
A tiny percentage of extremely slow requests can consume resources, hold queues and create cascading effects far beyond their frequency.<\/p>\n
In developer tooling, part of the cascade happens inside the human side of the loop.<\/p>\n
The outlier interrupts work.<\/p>\n
The interruption changes future waiting behavior.<\/p>\n
That behavior changes how often tests are run.<\/p>\n
The new test frequency changes the size of code batches.<\/p>\n
One outlier can therefore affect several later cycles.<\/p>\n
9. What I Would Measure in a Real Development Environment<\/h3>\n
A real experiment does not require recording code, keystrokes, browser history or message contents.<\/p>\n
Timing metadata is enough.<\/p>\n
From the test runner I would collect:<\/p>\n
run startedrun completedtest targetresult<\/code>
<\/a><\/div><\/pre>\n
From the IDE or operating system:<\/p>\n
IDE lost focusIDE regained focusfirst editor actionnext test launch<\/code>
<\/a><\/div><\/pre>\n
Then a single cycle could look like this:<\/p>\n
12:00:00.000   test started12:00:04.200   IDE lost focus12:00:08.600   test completed12:00:14.100   IDE regained focus12:00:16.400   first meaningful action<\/code>
<\/a><\/div><\/pre>\n
Raw test latency:<\/p>\n
8.6 s<\/code>
<\/a><\/div><\/pre>\n
Effective feedback latency:<\/p>\n
16.4 s<\/code>
<\/a><\/div><\/pre>\n
The test runner sees 8.6 seconds.<\/p>\n
The developer experiences more than sixteen.<\/p>\n
Across hundreds of runs, I would compare not only duration but also:<\/p>\n