Two test suites can have exactly the same average runtime and still create completely different development workflows. I modeled 300 code → test → fix cycles while keeping mean feedback latency close to ten seconds and changing only its variance. One runner behaved predictably. The other was usually fast but occasionally became extremely slow. The average stayed almost the same. The cost of waiting did not.
Introduction
A test that always takes ten seconds is annoying, but after a few runs it becomes predictable. I press the shortcut, know roughly how much time I have, and usually stay inside the same task.
A test that sometimes finishes in two seconds and sometimes takes thirty is different. Even if its average duration is also ten seconds, I cannot build the same expectation around it. After three seconds I do not know whether the result is about to arrive or whether I have another half minute to kill.
That uncertainty changes the decision to wait.
Most discussions about build and test performance focus on average execution time. Sometimes p95 and p99 appear in CI dashboards, but even then they are usually treated as infrastructure metrics. I wanted to look at the same distribution from another angle: what does latency variance do to the development loop itself?
So I built a simple simulation of 300 code → test → fix cycles.
The goal was not to estimate a universal human attention threshold. The model is intentionally synthetic. I wanted to isolate one variable that is usually mixed together with everything else: predictability.
Both simulated test runners have approximately the same average runtime.
One is boring.
The other has a long tail.
That difference turns out to matter much more than the average suggests.
1. Same Average, Different Systems
The first runner is almost deterministic. Most executions finish near ten seconds.
A typical sequence might look like this:
9.410.29.810.110.49.710.09.9
The second runner is usually much faster, but occasionally enters a slow path:
2.13.02.74.22.431.82.930.9
The averages can be made almost identical.
If I reduce both systems to one metric, they look equivalent.
That is exactly the problem.
The stable runner has a narrow distribution. Its median, p95 and p99 remain relatively close.
The unstable runner can have a much better median while producing an ugly tail.
That means a developer can experience the second system as fast most of the time and still trust it much less.
This is not only about comfort.
Interactive development is a control loop:
change code↓run test↓receive result↓update hypothesis↓make next change
The usefulness of the loop depends not only on how quickly feedback arrives on average, but also on whether I can predict when it will arrive.
A predictable delay lets me stay attached to the current hypothesis.
An unpredictable delay creates a decision point.
Do I keep waiting?
Do I open documentation?
Do I answer a message?
Do I start another small task?
Once that decision appears, test runtime stops being the full cost of feedback.
2. The Important Metric Is Remaining Wait, Not Average Wait
Suppose a test has already been running for eight seconds.
Knowing that average runtime is ten seconds is not enough.
What I actually care about is:
How much longer is this particular run likely to take,given that it has already lasted eight seconds?
Mathematically, this is the expected residual lifetime:
E[T - t | T > t]
T is total execution time.
t is elapsed waiting time.
This quantity behaves very differently for narrow and heavy-tailed distributions.
For a nearly deterministic ten-second process, waiting longer usually means I am getting closer to completion.
If the test has already survived nine seconds, the expected remaining wait is tiny.
That makes waiting rational.
Now consider a runner where most tests finish in three seconds but some take thirty.
If the test is still running after eight seconds, I have learned something important.
This is probably not one of the fast runs.
The fact that I have already waited a long time does not necessarily mean completion is near.
It may mean I have entered the slow population.
That changes the economics of waiting completely.
The same effect can be described through the survival function:
S(t) = P(T > t)
This is the probability that the process is still running after time t.
From it, we can derive the hazard function:
h(t) = f(t) / S(t)
where f(t) is the probability density of completion time.
The name sounds more dramatic than the idea.
The hazard rate tells me how likely completion is around the current moment, assuming the test has not finished yet.
For a predictable runner, the hazard becomes high near the usual completion point.
For a long-tailed runner, surviving beyond the normal fast region may actually be bad news.
That is the first reason average latency is a weak metric for interactive tooling.
Two systems can share the same mean and create opposite incentives to wait.
3. Modeling 300 Feedback Cycles
I split the simulation into two groups of 300 runs.
The first distribution stays tightly clustered around ten seconds.
The second is a mixture.
Most runs are fast, but a smaller group enters a much slower path.
Conceptually:
Stable runnermean ≈ 10 slow variancesmall distance between p50 and p99
and:
Unstable runnermean ≈ 10 shigh variancefast medianlarge p95very large p99
This is the important part of the experiment.
I am not changing average runtime deliberately.
I am changing uncertainty.
For the stable runner, I can approximate execution time with something close to a narrow normal distribution centered near ten seconds.
For the unstable runner, a mixture model works better:
T = fast distribution, with high probability slow distribution, with low probability
For example:
P(fast) = 0.8P(slow) = 0.2
The exact coefficients are not the conclusion.
They are knobs.
What matters is that the second distribution has the same approximate mean but a much longer right tail.
Once I compare them through p50, p95 and p99 rather than mean alone, the difference becomes obvious.
The stable runner is mediocre every time.
The unstable runner is excellent most of the time and terrible just often enough to destroy confidence.
That trade-off appears constantly in real systems.
A build step usually finishes quickly, except when dependency resolution behaves differently.
An integration test normally takes a few seconds, except when a container cold-starts.
A local database query is fast until something invalidates a cache.
A compiler is predictable until incremental state disappears.
From the perspective of the average, these events are noise.
From the perspective of the developer, they are remembered.
4. Developer Behavior Has Memory
A simple latency model assumes every test run is independent.
I do not think developer behavior works that way.
Suppose five recent runs took:
91011109
Now compare that with:
2182189
The average is similar.
My expectation is not.
After the first sequence, I trust the tool.
After the second, I know that another long delay can appear without warning.
That memory affects the next run.
Even if the sixth test completes in three seconds, I may leave after two because the previous history has trained me not to wait.
Now a fast test can produce a slow feedback loop.
The test itself did nothing wrong.
Its predecessors changed my behavior.
A simple way to represent recent expectation is an exponentially weighted estimate:
E_n = αT_n-1 + (1 - α)E_n-1
Recent runs receive more weight than old ones.
But expected duration alone is still not enough.
The developer also learns variance.
A more useful internal state would contain at least:
expected durationuncertaintyrecent extreme events
This matters because outliers can have a delayed cost.
A thirty-second test does not necessarily waste only thirty seconds.
It can make me switch earlier during the next several runs.
The latency penalty leaks forward.
That is difficult to see in CI dashboards because test systems measure executions independently.
Humans do not experience them independently.
5. Adding Context Switching to the Model
I then added a simple decision rule.
At each point during a wait, the simulated developer can remain in the task or leave the current context.
The switching probability depends on three variables:
elapsed timeexpected remaining timeuncertainty
Conceptually:
P(switch) = sigmoid( a + b1 × elapsed + b2 × expected_remaining + b3 × uncertainty )
The coefficients are not empirical claims.
They only provide a mechanism for comparing distributions.
The result I care about is qualitative.
With stable tests, uncertainty remains low. Even when execution is not especially fast, waiting is relatively easy to justify.
With unstable tests, a few long runs increase the perceived uncertainty of future runs.
Switching probability rises earlier.
This creates situations like:
run 1 3 srun 2 4 srun 3 32 srun 4 3 srun 5 4 s
Run 4 is technically fast.
But after run 3, I may leave the IDE after two seconds.
Run 4 completes one second later.
I do not notice immediately because I am already somewhere else.
Then I return.
Then I reconstruct the original code context.
A three-second test has just produced a feedback loop lasting much longer than three seconds.
The test duration is not the problem.
Trust in the timing model is.
6. Effective Feedback Latency Is Larger Than Test Latency
This suggests separating two metrics.
The first is obvious:
T_test
The second is what I actually experience:
T_effective
A simple decomposition is:
T_effective = T_test + T_detection + T_reconstruction
T_detection is the delay between test completion and me noticing it.
T_reconstruction is the time required to resume meaningful work on the original problem.
If I stay inside the IDE, both values may be close to zero.
If I switch context, both can become substantial.
Now variance becomes expensive in a second way.
A long-tailed runner creates more context switches.
More context switches increase detection delay.
More detection delay increases effective feedback latency.
Higher effective latency makes future waiting less attractive.
That creates a feedback loop:
unpredictable latency↓more context switching↓more missed completions↓higher effective latency↓less trust in waiting↓more context switching
The tool develops a reputation.
Once that reputation becomes bad, even fast runs stop receiving the benefit of patience.
This is why I think performance consistency can matter more than average performance for interactive tooling.
7. Slow Outliers Also Change How Code Is Written
There is another effect that has nothing to do with returning to the IDE.
Slow feedback changes batch size.
When tests are predictable and cheap, this workflow feels natural:
changetestchangetestchangetest
When test duration becomes unpredictable, I start wanting to amortize the risk.
The workflow changes:
changechangechangechangetest
That reduces the number of waits.
It also increases the number of possible causes when a test fails.
With one change between passing and failing states, the search space is small.
With four changes, it is larger.
Now the test may not have become slower at all.
The development process around it became harder.
This produces another loop:
unpredictable tests↓less frequent runs↓larger edit batches↓larger failure search space↓more expensive debugging↓less willingness to run tests frequently
This is where a performance problem starts changing engineering behavior.
At that point test latency is no longer just infrastructure latency.
It affects the granularity of development.
8. Why Removing the Worst 5 Percent May Be More Valuable Than Improving the Median
Imagine two optimization projects.
The first improves the common case:
beforep50 = 8 sp95 = 10 sp99 = 12 s
After optimization:
p50 = 5 sp95 = 9 sp99 = 12 s
The median becomes clearly better.
Now consider a different system:
p50 = 8 sp95 = 14 sp99 = 35 s
Suppose the second optimization barely changes the median but removes the tail:
p50 = 8 sp95 = 10 sp99 = 12 s
Which optimization improves development more?
If I only count average seconds saved, the first one may look better.
If I care about predictability, the second may have a larger behavioral effect.
It removes the pathological runs that teach developers not to trust the tool.
This is similar to tail-latency engineering in distributed systems.
A tiny percentage of extremely slow requests can consume resources, hold queues and create cascading effects far beyond their frequency.
In developer tooling, part of the cascade happens inside the human side of the loop.
The outlier interrupts work.
The interruption changes future waiting behavior.
That behavior changes how often tests are run.
The new test frequency changes the size of code batches.
One outlier can therefore affect several later cycles.
9. What I Would Measure in a Real Development Environment
A real experiment does not require recording code, keystrokes, browser history or message contents.
Timing metadata is enough.
From the test runner I would collect:
run startedrun completedtest targetresult
From the IDE or operating system:
IDE lost focusIDE regained focusfirst editor actionnext test launch
Then a single cycle could look like this:
12:00:00.000 test started12:00:04.200 IDE lost focus12:00:08.600 test completed12:00:14.100 IDE regained focus12:00:16.400 first meaningful action
Raw test latency:
8.6 s
Effective feedback latency:
16.4 s
The test runner sees 8.6 seconds.
The developer experiences more than sixteen.
Across hundreds of runs, I would compare not only duration but also:
-
p50
-
p95
-
p99
-
coefficient of variation
-
extreme outlier frequency
-
probability of leaving the IDE
-
completion-to-return delay
-
return-to-first-action delay
-
edit batch size between test runs
That last metric may be especially useful.
If latency variance really changes development strategy, the number of edits between test runs should increase as timing becomes less predictable.
That gives a behavioral signal that can be measured without interpreting subjective focus.
10. Average Runtime May Be the Wrong Optimization Target
Performance teams naturally want a single number.
Average test duration is convenient.
Median duration is better.
p95 and p99 reveal more.
For interactive development, I think even that may be incomplete.
The metric I actually care about is closer to:
probability that feedback arrivesbefore the developer abandons the current context
That quantity combines the tool and the user.
It depends on runtime distribution, recent history, task complexity and context-switch cost.
This also means two test suites with the same p50 and p99 could still behave differently if one has a more predictable structure between those points.
A narrow distribution creates a reliable expectation.
A multimodal distribution can create uncertainty even when common percentile values look acceptable.
For example, a test runner can have two distinct modes:
fast modeslow mode
That often happens for concrete technical reasons:
warm cache vs cold cacheincremental build vs full rebuildrunning container vs container startuplocal dependency vs remote dependencyJIT warm state vs cold state
If those modes are not visible to the developer, the tool feels random.
Sometimes the best performance improvement may therefore be exposing the mode rather than reducing runtime.
If the IDE tells me that a full rebuild is happening and will probably take thirty seconds, uncertainty falls even though latency does not.
Predictability can be improved through information as well as speed.
That is an interesting design implication.
Conclusion
Two development tools can have exactly the same average latency and still create completely different workflows.
A stable ten-second test is slow but understandable.
A test that averages ten seconds by alternating between two and thirty seconds creates uncertainty.
That uncertainty matters because waiting is a prediction problem.
The developer is continuously estimating whether staying inside the current task is still rational.
A predictable tool makes that estimate easy.
An unpredictable one teaches the developer to leave.
Once that happens, the cost spreads beyond the slow run itself. Context switches create detection delay and reconstruction cost. A pathological run changes expectations for later runs. Testing becomes less frequent. Edit batches become larger. Failed tests become harder to localize.
This is why I no longer think average test runtime is enough to describe developer feedback quality.
The distribution matters.
Its tail matters.
Its recent history matters.
And its predictability may matter as much as its speed.
If I had to choose between making every test slightly faster and removing the handful of pathological runs that make the system unpredictable, I would no longer automatically choose the first option.
The worst test may not be the one that takes the longest.
It may be the one that makes me stop trusting all the tests that come after it.
ссылка на оригинал статьи https://habr.com/ru/articles/1078302/