The experiment started with a fairly common kind of infrastructure problem: users clearly experience something broken, but the server refuses to look broken. CPU usage stays low, there is no swap activity, disks are not saturated, network throughput is nowhere near the interface limit, and the application continues passing health checks. At the same time, some clients wait several seconds before establishing a connection, while others connect normally. Problems like this are unpleasant because the obvious indicators provide almost no direction. They encourage people to stare at application logs, restart services, increase timeouts, and blame the network in a very general way without having any solid evidence.
The lab was intentionally simple. One machine acted as a client, another Linux host worked as a NAT gateway, and a third machine ran a small HTTP backend. Every connection from the client had to pass through the gateway. Under normal conditions the backend handled roughly 900 requests per second, median latency stayed around 6 ms, p95 remained close to 11 ms, and p99 usually stayed below 20 ms. Nothing about the application itself was particularly interesting, and that was the point. The failure had to happen below the application layer so that restarting the service or reading its logs would not magically reveal the answer.
The hidden limitation was the Linux Netfilter connection tracking table. The gateway kept state for flows passing through it so that NAT and stateful firewall rules could understand which packets belonged to which connection. Normally that table can contain far more entries than this small lab needed, so the configured limit was deliberately reduced to 1024. Once the table filled, existing connections often continued to work, but new connection attempts began to disappear. CPU usage still remained below 25 percent, memory consumption barely moved, disk activity stayed almost flat, and the backend itself did not become overloaded. From the operating system resource graphs, the gateway still looked surprisingly healthy.
sudo sysctl -w net.ipv4.ip_forward=1sudo iptables -t nat -A POSTROUTING \ -s 10.10.0.0/24 \ -o eth1 \ -j MASQUERADEsudo sysctl -w net.netfilter.nf_conntrack_max=1024while true; do count=$(cat /proc/sys/net/netfilter/nf_conntrack_count) max=$(cat /proc/sys/net/netfilter/nf_conntrack_max) printf "%s conntrack=%s/%s\n" \ "$(date +%H:%M:%S)" "$count" "$max" sleep 1done
The low limit was used only to make the failure repeatable. Setting such a value on a real production gateway would be an excellent way to create an outage rather than study one. The useful part was not the exact number 1024, but the shape of the failure. The machine did not slowly become overloaded in an obvious way. Instead, one specific kernel resource approached a hard limit while most familiar system metrics remained calm.
First investigation: working as a Linux sysadmin would have in 2008
The first diagnostic run deliberately ignored Prometheus, Grafana, centralized logging, tracing, and dashboards. The only available path was SSH and the tools installed directly on the machine. The first checks were predictable: top showed plenty of idle CPU, vmstat did not reveal a long run queue or swap activity, and iostat showed that the block devices had almost nothing to do. Network throughput was also low enough that interface saturation made little sense. Within a few minutes the usual suspects had already been cleared.
This style of troubleshooting feels quite different from working from dashboards. A dashboard usually encourages the question of which metric is abnormal. A terminal session often encourages another question: which subsystem has not been eliminated yet? Once CPU, memory, storage, and the application itself looked healthy, attention moved toward the path between the client and the backend. That change in thinking mattered more than any individual command.
The first genuinely useful clue came from dmesg. Among otherwise uninteresting kernel messages appeared nf_conntrack: table full, dropping packet. That line was enough to change the entire investigation. Reading /proc/sys/net/netfilter/nf_conntrack_count showed 1024. Reading /proc/sys/net/netfilter/nf_conntrack_max showed the same number. The machine was not generally overloaded at all. One state table had simply reached its configured limit.
This is where the failure became interesting. Many infrastructure limits are discussed as percentages: CPU at 90 percent, memory at 85 percent, disk at 95 percent. Conntrack behaves differently. At 1023 entries everything can still look normal. At 1024, the next new flow may fail to obtain state and the behaviour of the system changes immediately. A client may see a timeout even though the backend process never receives the first packet. How many dashboards would clearly show that if nobody had thought to export this specific counter?
Packet capture turned the theory into evidence
Finding the full conntrack table gave a strong explanation, but it was still useful to prove that the observed client problem matched the kernel state. tcpdump was started on both sides of the gateway. On the client-facing interface, SYN packets appeared normally and some were retransmitted after receiving no response. On the backend-facing interface, a portion of those same SYN packets never appeared at all. The backend therefore had no opportunity to accept the connection, reject it, log an error, or respond slowly. From the application point of view, those requests simply did not exist.
That observation eliminates a surprising amount of useless debugging. If the client sends a SYN and the packet never reaches the server-facing interface, tuning application workers or increasing HTTP timeouts is pointless. The problem exists before the application becomes involved. This is one reason packet capture still has such a strong place in Linux troubleshooting. A monitoring system usually presents an interpretation of exported state. A capture shows what actually crossed an interface at a particular moment.
The old-style investigation reached the root cause in a little under seven minutes. Most of that time was not spent entering commands. It was spent ruling out healthy subsystems and forming the next hypothesis. That distinction is important because experienced administrators often look unusually fast when working from a shell. The speed rarely comes from typing commands faster. It comes from knowing which questions to stop asking.
To make the failure reproducible, the load generator did not attempt to simulate realistic users or produce an impressive benchmark number. Its only job was to maintain enough simultaneous TCP connections to fill the deliberately small conntrack table while a separate stream of ordinary requests measured the visible effect.
import asyncioHOST = "10.20.0.20"PORT = 8080CONNECTIONS = 1800HOLD_SECONDS = 20async def open_connection(): try: reader, writer = await asyncio.open_connection(HOST, PORT) writer.write( b"GET /health HTTP/1.1\r\n" b"Host: lab\r\n" b"Connection: keep-alive\r\n\r\n" ) await writer.drain() await asyncio.sleep(HOLD_SECONDS) writer.close() await writer.wait_closed() except Exception: passasync def main(): await asyncio.gather( *(open_connection() for _ in range(CONNECTIONS)) )asyncio.run(main())
The generator itself was intentionally primitive. Once enough sessions accumulated, the conntrack count moved toward the limit and the separate client traffic began showing long connection attempts and failures. This kept the experiment reasonably clean: one workload created the fault, another observed it.
Second investigation: the same failure with a 2026 DevOps stack
After the gateway was reset and the same failure was reproduced again, the diagnostic method changed completely. This time the first stop was not SSH but the monitoring stack. Request latency increased almost immediately. Median latency barely moved, but the tail became ugly: p99 rose from roughly 17 ms to several seconds, and a portion of requests failed while establishing the connection. Application error metrics were less useful because some failed connections never reached the application process in the first place.
The ordinary node dashboard was surprisingly unhelpful. CPU looked fine, memory looked fine, disk latency looked fine, load average looked fine, and overall network throughput did not suggest congestion. A person relying only on the common four or five host graphs could easily have concluded that the issue lived somewhere else. The decisive metric was node_nf_conntrack_entries, viewed together with node_nf_conntrack_entries_limit. The number of tracked flows had climbed until both values were nearly identical.
At that point the modern investigation reached the likely cause much faster than the shell-only approach. More importantly, Prometheus preserved the timeline. It was possible to move backward and watch the conntrack table grow through 60 percent, 75 percent, 90 percent, and eventually 100 percent. Only after the table reached the limit did connection failures begin to rise. Instead of discovering only the current state, the monitoring system showed how the incident developed.
That historical view is something classic command-line tools cannot recreate after the event unless somebody was already collecting the information. A shell can answer what the kernel is doing now. Time-series monitoring can answer what it was doing twenty minutes ago, when the problem began, which hosts changed together, and whether the same pattern happened yesterday. For real production environments, that difference is enormous.
The modern stack was faster, but only because the right metric existed
The second pass found the likely root cause in roughly two minutes. It would be tempting to end the comparison there and declare modern observability the winner. The experiment did not support such a simple conclusion. Prometheus was useful because the relevant conntrack counters had already been exported. If they had not been collected, Grafana would have shown several attractive graphs describing symptoms without showing the resource that was actually exhausted.
That is one of the less comfortable properties of observability systems. They can store millions of samples and still miss the single state variable that matters during a particular failure. A green panel does not mean the machine is healthy. It only means the conditions represented by that panel have not crossed their configured thresholds. Linux contains considerably more state than any practical dashboard can display.
Once the monitoring data pointed toward conntrack, the investigation returned to the same old tools anyway. dmesg confirmed that the kernel was dropping packets because the table was full. proc exposed the current count and maximum. tcpdump confirmed that some new connection attempts disappeared before reaching the backend. Modern tooling was excellent at locating the incident in time and narrowing the search to one subsystem. Classic tools were excellent at proving what the kernel was actually doing.
This combination turned out to be more useful than either method alone. The modern stack shortened the search. The old tools shortened the distance between the hypothesis and the operating system.
Why experienced sysadmins sometimes appear to diagnose problems by instinct
Watching an experienced Linux administrator investigate a broken server can look almost unfair. A handful of commands are executed, one file under proc is checked, a short packet capture is taken, and suddenly the problem has a very specific explanation. The useful skill is usually not memorizing hundreds of commands. It is maintaining a rough model of the path work takes through the system: scheduler, memory, filesystem, block layer, sockets, routing, firewall, connection tracking, DNS, process, application. Each command simply tests one piece of that model.
The 2008-style method depended heavily on this internal map. Without it, a shell session can turn into random command execution very quickly. top gets checked, then free, then df, then logs, then something is restarted because nothing obvious appeared. With a decent model, the process becomes elimination rather than guessing. If the application is healthy and the packet never reaches it, the search naturally moves down the stack.
Modern observability changes this process by performing many of those checks continuously. That is a huge improvement. It can compare hundreds of machines, retain history, expose trends, send alerts, and make correlations visible before an engineer even connects to the host. At the same time, dashboards can weaken the habit of asking what happens outside the dashboard. If CPU, memory, disk, and HTTP latency are the four panels everyone watches, those four metrics can quietly become the entire perceived system.
Conntrack exhaustion is a good example of the gap. The gateway had available CPU, free memory, quiet disks, and a working application. Yet it was failing at its job because a stateful networking resource had reached a hard limit.
Where the old approach clearly loses
There is no reason to romanticize classic system administration. SSHing into machines one by one does not scale well, especially when an environment contains dozens or hundreds of hosts. Transient events can disappear before anyone connects. Current counters say nothing about what happened an hour earlier. Comparing multiple nodes manually becomes slow and error-prone, and an engineer can accidentally change the system while trying to observe it.
In this experiment, the terminal-based pass found the cause but could not reconstruct exactly when the conntrack count started increasing. It did not immediately show whether another gateway behaved the same way. It provided no warning when utilization passed 70 or 80 percent. Had the table filled briefly and recovered before anyone logged in, the strongest evidence might have disappeared completely.
A modern telemetry system solves these problems extremely well. It can record the state continuously, compare hosts, alert before the limit is reached, and show whether an incident is new or repeating. When a production environment grows beyond a handful of servers, these are not nice extras. They are basic operational requirements.
The real weakness of the old approach is therefore not that the tools are primitive. tcpdump and proc are still extremely powerful. The weakness is that they mostly observe the present, while real incidents often require history.
Where the modern approach still loses
The weakness of the 2026 approach appeared in the opposite place. Monitoring only understands what somebody chose to collect. If the important resource has no exporter, no dashboard, and no alert, a very expensive observability stack can still produce almost no useful explanation. The system may show that latency rose, connection errors increased, and traffic dropped, while remaining silent about why.
This creates a subtle dependency on dashboard design. Engineers may begin debugging the representation rather than the machine. If the answer is not visible in Grafana, the assumption becomes that more dashboards are needed, even though the operating system may already expose the necessary evidence through a simple counter or packet capture.
There is another problem: abstractions can hide causal distance. A panel might show connection errors, another might show latency, and a third might show service availability. All three can be generated by a packet being dropped before it reaches the application. Without understanding that path, three symptoms may look like three separate problems.
The most useful modern metric in this experiment was not complicated. It was simply the number of conntrack entries divided by their configured maximum. No anomaly detection was required. No machine learning model was required. The difficult part was knowing that this counter mattered enough to collect in the first place.
The useful answer is not old sysadmin versus new DevOps
The experiment started as a comparison between two generations of operational tooling, but the result made that framing less convincing. The older method was slower at reconstructing the timeline but strong at getting close to real kernel behaviour. The modern method was dramatically better at history, correlation, and early detection but depended on the right telemetry already existing.
The fastest practical workflow used both. Monitoring identified the time window, the affected host, and the subsystem that changed. Direct Linux tools then verified the current state. dmesg provided the kernel message, proc showed the resource limit, and tcpdump showed what happened to packets on the wire. Each layer reduced uncertainty left by the previous one.
After this test, the useful definition of server resources also became broader. CPU, memory, disks, and interface bandwidth are only the obvious ones. Network-facing Linux hosts also depend on conntrack entries, socket queues, file descriptors, ephemeral ports, TCP state, application pools, DNS behaviour, and several other resources that can hit limits while the machine still looks mostly idle.
The lesson from the old sysadmin is not to abandon Grafana and return to a terminal for everything. The lesson from the modern DevOps engineer is not to replace understanding with dashboards. The useful combination is much simpler: use telemetry to know where and when to look, then move closer to the machine until the explanation matches what the kernel and the packets are actually doing.
A modern monitoring stack provides a much better map than administrators had fifteen or twenty years ago. The map is faster, searchable, historical, and capable of watching thousands of systems at once. But it is still a map. When the symptoms and the map disagree, the operating system remains the final source of truth.
ссылка на оригинал статьи https://habr.com/ru/articles/1086398/