After my previous experiments with test latency, I started wondering whether I was still looking at the wrong statistic.
Mean latency is obviously incomplete.
p95 is better.
p99 is useful when rare slow runs matter.
But all of these measurements have one property that is easy to overlook:
They do not care about order.
If I take 300 test durations and randomly rearrange them, the mean remains identical.
So do p50, p95 and p99.
The total waiting time is identical too.
Yet from a developer perspective, ten slow runs scattered across an afternoon do not necessarily feel like ten slow runs arriving almost back to back.
That gave me a very specific experiment.
I generated one set of 300 test durations.
Then I created two timelines from exactly the same values.
In the first timeline, durations were randomly ordered.
In the second, slow runs were deliberately clustered.
Nothing else changed.
The result surprised me more than changing the latency distribution itself.
Both timelines had:
mean: 7.65 sp50: 5.49 sp95: 17.20 sp99: 42.66 smax: 49.24 s
Both contained exactly the same total waiting time:
2295.83 seconds
That is about 38.26 minutes.
A latency dashboard based only on percentiles would describe the two workloads as identical.
But one timeline contained an 18-run streak of tests slower than 12 seconds.
The other never exceeded two.
That is the part I wanted to understand.
First I needed the same data, not merely similar data
I did not want to compare two independently generated distributions.
If I generated one fast workload and one bursty workload separately, a difference in the result could always be explained by slightly different samples.
So I created the durations once.
The synthetic workload contains mostly short test runs, a smaller group of medium ones, and a few expensive runs.
The generator looked roughly like this:
import numpy as nprng = np.random.default_rng(42)n = 300u = rng.random(n)durations = np.empty(n)for i, x in enumerate(u): if x < 0.78: durations[i] = np.clip( rng.lognormal(np.log(5), 0.25), 2.5, 9 ) elif x < 0.95: durations[i] = np.clip( rng.lognormal(np.log(12), 0.25), 8, 22 ) else: durations[i] = np.clip( rng.lognormal(np.log(38), 0.22), 25, 60 )
This is not production telemetry.
It is a synthetic workload designed to give me a realistic‑looking mixture of ordinary test runs and occasional expensive ones.
The important part is what happens afterward.
I never regenerate those 300 numbers.
Every comparison uses the exact same multiset of durations.
That means every ordinary distribution statistic is guaranteed to remain unchanged when I rearrange them.
The random timeline
For the first sequence I simply shuffled the 300 durations.
rng = np.random.default_rng(123)random_order = durations[ rng.permutation(len(durations))]
This produced the kind of timeline I normally imagine when looking at a histogram.
Fast run.
Fast run.
Slow one.
A few fast runs.
Medium run.
Another fast run.
Occasional large spike.
Nothing particularly interesting happens temporally.
There are 37 runs above 12 seconds in the dataset.
In this ordering, the longest consecutive streak above 12 seconds was only two runs.
A developer might see something like:
5s4s13s16s6s5s7s4s41s6s
There are unpleasant waits, but they are separated by ordinary feedback cycles.
Then I rearranged exactly the same numbers.
Creating a clustered timeline without changing the distribution
I needed a way to introduce temporal persistence.
Simply sorting the durations from fastest to slowest would be too artificial.
Real degradation usually does not look like that.
A CI worker gets overloaded for some period.
A dependency becomes slow.
Disk contention appears.
A shared runner gets noisy.
A cache becomes cold.
Then the system recovers.
So I generated a latent time series with positive autocorrelation.
The model was:
z[t] = 0.9 × z[t-1] + ε[t]
where ε is random Gaussian noise.
The coefficient 0.9 creates persistence.
When the latent state becomes high, it tends to remain high for a while.
When it becomes low, it tends to remain low.
Then I ranked the 300 timeline positions by this latent value and assigned the slowest test durations to the highest positions.
In simplified Python:
rng = np.random.default_rng(7)z = np.zeros(300)eps = rng.normal(size=300)for t in range(1, 300): z[t] = 0.9 * z[t - 1] + eps[t]positions = np.argsort(z)sorted_durations = np.sort(durations)clustered = np.empty(300)clustered[positions] = sorted_durations
This operation does something useful.
It changes temporal structure without changing a single duration.
The 49.24-second test is still there.
The 17-second tests are still there.
Every 5-second test is still there.
Nothing has been added or removed.
Only their positions changed.
And every normal percentile remained identical
This part is mathematically trivial, but I think it is the most important part of the experiment.
Quantiles depend on the sorted values.
A permutation does not change the sorted values.
Therefore:
mean(random) = mean(clustered)p50(random) = p50(clustered)p95(random) = p95(clustered)p99(random) = p99(clustered)sum(random) = sum(clustered)
For both sequences I got:
mean = 7.65 sp50 = 5.49 sp95 = 17.20 sp99 = 42.66 smax = 49.24 s
If these were two CI pipelines and my monitoring showed only these numbers, I would conclude that their test latency was effectively identical.
That conclusion would be technically correct.
It would also miss something fairly large.
The first number that exposed the difference was autocorrelation
I calculated lag-1 autocorrelation.
In this case I am asking a simple question:
Does knowing the duration of the current test run tell me anything about the duration of the next one?
For the randomly ordered sequence, lag-1 autocorrelation was about:
0.10
For the clustered sequence:
0.83
That is a completely different system.
With low serial correlation, a slow run tells me little about what happens next.
With strong positive serial correlation, a slow run makes another slow run much more likely.
This is something p99 cannot express.
p99 tells me how large the upper tail is.
It does not tell me whether those tail events are isolated or arrive together.
Then I counted slow streaks
I picked 12 seconds as an analytical threshold.
This is not meant to be a universal threshold for human attention.
I just needed a fixed line above which a test run would be classified as slow for this experiment.
There were 37 such runs in both sequences.
Again, exactly the same number.
In the random sequence, the longest consecutive streak above 12 seconds was:
2
In the clustered sequence:
18
That difference is hard to see in a percentile.
The two datasets have the same number of slow tests.
But one can produce something like this:
slowslowfastfastfastslowfastfast
while the other can produce:
slowslowslowslowslowslowslowslow...
By the eighteenth slow feedback cycle, I am no longer looking at a rare latency spike.
I am experiencing a slow period.
That distinction seems important.
Five‑cycle windows made the difference even clearer
Individual latency is not always the most useful unit.
When I am coding, I usually care about a sequence of feedback loops.
Write something.
Run tests.
Fix something.
Run them again.
Change another line.
Run them again.
So I calculated the average latency inside every consecutive five‑run window.
For each timeline I then found the worst five‑run period.
In the random sequence, the worst five‑run average was approximately:
22.23 s
In the clustered sequence:
42.10 s
Remember that the global mean in both cases is still:
7.65 s
Nothing about the distribution changed.
Yet at the local level, the clustered timeline produced a five‑cycle period where the average feedback delay was almost twice as high.
I repeated the calculation with ten‑run windows.
The worst ten‑run average in the random ordering was approximately:
14.40 s
For the clustered ordering:
35.88 s
This is where the global average becomes almost misleading.
A pipeline can have a perfectly acceptable daily mean while still producing terrible local periods.
I did not want the result to depend on one lucky shuffle
At this point I had one random sequence and one clustered sequence.
That is not enough.
The random sequence might simply have been unusually well behaved.
So I took the same 300 durations and performed 10,000 independent random permutations.
For each permutation I calculated the longest streak above 12 seconds.
The median longest streak was:
2
The 95th percentile was:
4
The largest streak I saw across all 10,000 random permutations was:
6
The clustered sequence produced:
18
I also repeated the analysis for the worst five‑run average.
Across 10,000 random orderings, the median maximum five‑run average was about:
19.63 s
The 95th percentile was:
24.55 s
The 99th percentile was:
27.77 s
The clustered sequence reached:
42.10 s
The same thing happened with ten‑run windows.
For random permutations, the median worst ten‑run average was about 14.56 seconds.
The 95th percentile was 17.85 seconds.
The 99th percentile was 19.64 seconds.
The clustered timeline reached 35.88 seconds.
At that point I stopped thinking of this as a weird permutation.
The temporal structure was producing a property the ordinary latency distribution simply did not contain.
Percentiles deliberately throw away order
This is not a criticism of percentiles.
p95 is doing exactly what it is supposed to do.
Take 300 measurements.
Sort them.
Look near the upper end.
Once I sort the observations, time disappears.
The test that happened at 10:01 and the test that happened at 16:47 are now just two numbers in an ordered array.
That is useful when I want to know how slow the slowest portion of requests tends to be.
It is useless when the question is whether slow events arrive in clusters.
Mathematically, any metric that is invariant under permutation cannot detect temporal clustering.
Mean is permutation‑invariant.
Variance is permutation‑invariant.
Median is permutation‑invariant.
p95 is permutation‑invariant.
p99 is permutation‑invariant.
A histogram is permutation‑invariant.
I can completely rearrange the experience while leaving every one of those measurements untouched.
That was the part I had underestimated.
Variance was not enough either
This was particularly interesting after my previous experiment.
I had already been looking at latency variance and how unpredictable test times can change the feedback loop.
But variance has the same problem here.
Because the underlying 300 durations are identical, the variance is identical too.
I can preserve:
meanvariancep50p95p99minimummaximumtotal waiting time
and still create radically different slow streaks.
So there are really two separate dimensions.
The first is marginal variability.
How different are individual test durations from each other?
The second is temporal dependence.
Does a slow test tend to be followed by another slow test?
A normal latency histogram can describe the first.
It says almost nothing about the second.
A simple example outside testing made this obvious to me
Imagine two services.
Both have a 1 percent error rate.
Service A returns one failed request roughly every hundred requests.
Service B works perfectly for hours and then fails one hundred requests in a row.
Same total request count.
Same number of errors.
Same 1 percent error rate.
Very different operational behaviour.
Latency can have the same problem.
Ten 40-second test runs spread across a day are one thing.
Ten 40-second runs arriving in one development session are something else.
Aggregation can make both look identical.
Where could these clusters come from in a real CI system?
I can think of several mechanisms that could create serial dependence without dramatically changing the long‑term histogram.
A shared runner can become temporarily saturated.
Several jobs can compete for disk bandwidth at the same time.
A remote dependency can enter a slow period.
A container image or dependency cache can repeatedly miss after an invalidation.
CPU throttling can persist for several runs.
A noisy neighbour can affect the same worker for minutes rather than milliseconds.
Garbage collection or memory pressure can become correlated with workload phases.
Test ordering can move expensive integration tests into the same region of the suite.
The exact mechanism is not important for this experiment.
What matters is that many real performance problems have state.
They do not independently reroll themselves from scratch for every execution.
Once a machine becomes overloaded, the next execution is more likely to run on an overloaded machine too.
That produces memory in the latency process.
This changes what I would put on a test‑latency dashboard
I would still keep mean, p50, p95 and p99.
They answer useful questions.
But for a feedback loop used hundreds of times per day, I would also want some measure of temporal structure.
Lag autocorrelation is one option.
Rolling latency is another.
Maximum rolling averages can expose bad local periods.
Run‑length statistics can show how often slow executions arrive consecutively.
Even a simple chart of latency against execution order can reveal things that a histogram hides immediately.
I do not think every engineering dashboard needs a full time‑series analysis section.
But if developers complain that tests are sometimes unusably slow while the daily p95 looks fine, I would no longer assume that one of them is wrong.
They may simply be measuring different properties.
There is a trap in daily aggregation
Suppose I run tests 300 times during a working period.
For 250 runs everything feels normal.
Then the CI environment enters a bad state.
The next 20 runs become slow.
Later it recovers.
At the end of the day I calculate one p95 value.
I have compressed the entire sequence into a single number.
The temporary degradation becomes part of the tail, but the fact that it happened as one contiguous event disappears.
This matters operationally too.
If I only investigate the slowest individual runs, I may look for a problem inside particular tests.
But if slow runs are highly autocorrelated, the cause may live outside the tests completely.
The runner matters.
The machine matters.
The network matters.
The current load matters.
The previous run may suddenly become relevant evidence.
I would now ask one more question when looking at latency
Previously, if somebody showed me:
mean 7.6 sp95 17.2 sp99 42.7 s
I would immediately start reasoning about the distribution.
How heavy is the tail?
Are rare integration tests responsible?
Would reducing p99 improve the feedback loop?
Now I would ask something else first:
Where are those slow runs in time?
If they are independent spikes, one optimization strategy may make sense.
If they arrive in persistent clusters, I may be looking at a completely different failure mode.
The distinction cannot be recovered from the percentiles afterward.
Once I throw away the ordering, the information is gone.
What this experiment does not prove
There is one limitation I want to make explicit.
I did not measure developers.
I did not put two groups of programmers in front of these timelines and measure concentration, task completion, eye movement or context switching.
So this experiment does not prove that an 18-run slow streak reduces human productivity by some particular percentage.
That would require a different study.
What I demonstrated is narrower.
Two test‑latency sequences can have exactly the same durations, the same mean, the same variance, the same percentiles and the same total waiting time while having radically different temporal behaviour.
Whether that temporal behaviour harms a particular developer is a separate question.
But if I want to study developer feedback loops, ignoring it now seems like a fairly large omission.
The surprising part was how little I had to change
I originally expected to discover something only after changing the latency distribution.
Increase variance.
Add a heavier tail.
Raise p99.
Make a few tests much slower.
None of that was necessary.
I kept every latency value.
I kept every percentile.
I kept all 2295.83 seconds of waiting.
Then I moved the numbers around.
That alone changed the longest slow streak from two runs to eighteen and pushed the worst five‑cycle average from roughly 22 seconds to 42 seconds.
The global statistics did not move at all.
The timeline did.
And for a developer sitting in front of the test runner, the timeline is the thing that actually happens.
ссылка на оригинал статьи https://habr.com/ru/articles/1081454/