{"id":494610,"date":"2026-09-14T13:34:30","date_gmt":"2026-09-14T13:34:30","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=494610"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=494610","title":{"rendered":"How I Built a Diagnostic Tool for a Legacy System With No Dev Budget, and Cut Incident Triage From 6\u00a0Hours to 20\u00a0Minutes"},"content":{"rendered":"
\n

The problem<\/h2>\n

I’m part of a systems support team, where alongside diagnostics and incident troubleshooting I also do development work. One of the systems we support ingests telemetry from a large fleet of IoT trackers installed on vehicles. Every few seconds each device reports its coordinates and status parameters. An internal processing service turns that raw stream into higher level business objects: events, incidents, trips, that the rest of the platform consumes.<\/p>\n

The system has been in pure maintenance mode for years. No development budget, no vendor to call. It still has to be supported though, people use it every day.<\/p>\n

Every so often one device develops a hardware fault and starts \u201cspamming\u201d: emitting an abnormal volume of points or malformed events in a short window. That degrades the processing pipeline not only for that device, but for everyone sharing it.<\/p>\n

Finding the culprit used to mean a first line engineer manually pulling several raw tables for the relevant day (events, incidents, raw points), cross referencing them by timestamp and device ID, and eyeballing the result for anomalous patterns.<\/p>\n

With the events table alone running millions of rows a day, that was a 6\u00a0to 7\u00a0hour job, basically an entire shift, with no guarantee of actually finding the source.<\/p>\n

The solution<\/h2>\n

I designed and built a small internal web tool that automates this triage.<\/p>\n

Stack: a Node.js\/Fastify service handles uploads, job orchestration and progress reporting. A set of Python scripts do the actual line by line parsing and aggregation. The frontend is React. There is no live database connection: the tool consumes the same flat file dumps engineers already know how to export, and that was a deliberate choice. Adding zero new infrastructure or DB access mattered as much as the analysis itself.<\/p>\n

Here’s the flow:<\/p>\n

    \n
  1. \n

    A first line engineer exports a dump exactly the way they already do it manually, no new export format to learn.<\/p>\n<\/li>\n

  2. \n

    They drop the file onto the portal through a simple web interface.<\/p>\n<\/li>\n

  3. \n

    The backend streams it line by line instead of loading it into memory, because a multi\u2011million\u2011row dump has to run on a support engineer’s laptop, not a data processing cluster.<\/p>\n<\/li>\n

  4. \n

    The data gets aggregated down to what actually matters: which device or devices show abnormal patterns, over what window, with what signature (burst frequency, stale timestamps, invalid GPS fixes, frozen coordinates, and so on).<\/p>\n<\/li>\n

  5. \n

    The output shows the specific problem with all the data needed to understand it, so a first line engineer immediately knows what to do next: temporarily disable the device, move it to a test loop, or initiate an on\u2011site repair, instead of a wall of raw rows.<\/p>\n<\/li>\n<\/ol>\n

    The tool has grown into a small family of report types (data freshness scoring, burst detection for two different entity types, log analysis, a couple of cross dump reconciliation reports), and I’m actively expanding it, aiming for more than twenty report types that cover the recurring diagnostic scenarios our first line runs into. Reports show a live progress bar computed from actual row counts instead of a spinner, and for point level data there’s an animated map view that redraws a device’s GPS trace so you can visually confirm the anomaly instead of just trusting a number.<\/p>\n

    Why it was harder than it sounds<\/h2>\n

    The interesting engineering problems weren’t in the UI.<\/p>\n

    Streaming instead of batch processing. Multi million row dumps had to be processed without loading them wholesale into memory, on modest hardware, in a reasonable amount of time.<\/p>\n

    Aggregating without losing the signal. The easy version just counts rows per device. The useful version buckets by time window, tracks run length of repeated or frozen coordinates, classifies write lag into severity tiers, and merges adjacent bad windows into readable ranges, so the output points at why a device is misbehaving, not just that it is.<\/p>\n

    Zero new dependencies. The target environment had no route to install third party packages, so the Excel export is generated by hand assembling a valid.xlsx file as a zip of XML parts, no spreadsheet library involved at all.<\/p>\n

    Zero onboarding cost. Any first line engineer had to be able to use it right away, with the export they already knew how to produce. Anything that required learning a new format or understanding the schema would have defeated the whole point.<\/p>\n

    Results<\/h2>\n
    \n
    \n\n\n\n\n\n\n
    \n

    \n<\/th>\n

    		\n

    Before<\/p>\n<\/th>\n

    		\n

    After<\/p>\n<\/th>\n<\/tr>\n

    \n

    Time to triage one incident<\/p>\n<\/td>\n

    		\n

    6\u00a0to 7\u00a0hours, a full shift<\/p>\n<\/td>\n

    		\n

    ~15\u00a0min to export plus 10\u00a0to 20\u00a0min to diagnose<\/p>\n<\/td>\n<\/tr>\n

    \n

    Who can do it<\/p>\n<\/td>\n

    		\n

    An engineer who knows the DB schema in depth<\/p>\n<\/td>\n

    		\n

    Any first line engineer<\/p>\n<\/td>\n<\/tr>\n

    \n

    Dependency on vendor or budget<\/p>\n<\/td>\n

    		\n

    Full, no changes possible<\/p>\n<\/td>\n

    		\n

    None<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

    The time saved isn’t just convenience. It’s hours of first line capacity freed up per incident, and a whole category of case (device level event spam) that used to get escalated to the second line now gets resolved without leaving the first line at all.<\/p>\n

    What’s next<\/h2>\n

    I’m planning to open source a generic version of the core, stripped of anything domain or infrastructure specific, since the streaming aggregation approach and the time series anomaly detection logic seem generally useful for anyone maintaining a legacy system with high volume telemetry and no budget for a proper rewrite.<\/p>\n

    Happy to go deeper on any part of this in the comments, especially interested in hearing from anyone who’s had to diagnose systems they weren’t allowed to touch.<\/p>\n<\/div>\n

    \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1082092\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

    The problemI’m part of a systems support team, where alongside diagnostics and incident troubleshooting I also do development work. One of the systems we support ingests telemetry from a large fleet of IoT trackers installed on vehicles. Every few seconds each device reports its coordinates and status parameters. An internal processing service turns that raw stream into higher level business objects: events, incidents, trips, that the rest of the platform consumes.The system has been in pure maintenance mode for years. No development budget, no vendor to call. It still has to be supported though, people use it every day.Every so often one device develops a hardware fault and starts \u201cspamming\u201d: emitting an abnormal volume of points or malformed events in a short window. That degrades the processing pipeline not only for that device, but for everyone sharing it.Finding the culprit used to mean a first line engineer manually pulling several raw tables for the relevant day (events, incidents, raw points), cross referencing them by timestamp and device ID, and eyeballing the result for anomalous patterns.With the events table alone running millions of rows a day, that was a 6\u00a0to 7\u00a0hour job, basically an entire shift, with no guarantee of actually finding the source.The solutionI designed and built a small internal web tool that automates this triage.Stack: a Node.js\/Fastify service handles uploads, job orchestration and progress reporting. A set of Python scripts do the actual line by line parsing and aggregation. The frontend is React. There is no live database connection: the tool consumes the same flat file dumps engineers already know how to export, and that was a deliberate choice. Adding zero new infrastructure or DB access mattered as much as the analysis itself.Here’s the flow:A first line engineer exports a dump exactly the way they already do it manually, no new export format to learn.They drop the file onto the portal through a simple web interface.The backend streams it line by line instead of loading it into memory, because a multi\u2011million\u2011row dump has to run on a support engineer’s laptop, not a data processing cluster.The data gets aggregated down to what actually matters: which device or devices show abnormal patterns, over what window, with what signature (burst frequency, stale timestamps, invalid GPS fixes, frozen coordinates, and so on).The output shows the specific problem with all the data needed to understand it, so a first line engineer immediately knows what to do next: temporarily disable the device, move it to a test loop, or initiate an on\u2011site repair, instead of a wall of raw rows.The tool has grown into a small family of report types (data freshness scoring, burst detection for two different entity types, log analysis, a couple of cross dump reconciliation reports), and I’m actively expanding it, aiming for more than twenty report types that cover the recurring diagnostic scenarios our first line runs into. Reports show a live progress bar computed from actual row counts instead of a spinner, and for point level data there’s an animated map view that redraws a device’s GPS trace so you can visually confirm the anomaly instead of just trusting a number.Why it was harder than it soundsThe interesting engineering problems weren’t in the UI.Streaming instead of batch processing. Multi million row dumps had to be processed without loading them wholesale into memory, on modest hardware, in a reasonable amount of time.Aggregating without losing the signal. The easy version just counts rows per device. The useful version buckets by time window, tracks run length of repeated or frozen coordinates, classifies write lag into severity tiers, and merges adjacent bad windows into readable ranges, so the output points at why a device is misbehaving, not just that it is.Zero new dependencies. The target environment had no route to install third party packages, so the Excel export is generated by hand assembling a valid.xlsx file as a zip of XML parts, no spreadsheet library involved at all.Zero onboarding cost. Any first line engineer had to be able to use it right away, with the export they already knew how to produce. Anything that required learning a new format or understanding the schema would have defeated the whole point.ResultsBeforeAfterTime to triage one incident6\u00a0to 7\u00a0hours, a full shift~15\u00a0min to export plus 10\u00a0to 20\u00a0min to diagnoseWho can do itAn engineer who knows the DB schema in depthAny first line engineerDependency on vendor or budgetFull, no changes possibleNoneThe time saved isn’t just convenience. It’s hours of first line capacity freed up per incident, and a whole category of case (device level event spam) that used to get escalated to the second line now gets resolved without leaving the first line at all.What’s nextI’m planning to open source a generic version of the core, stripped of anything domain or infrastructure specific, since the streaming aggregation approach and the time series anomaly detection logic seem generally useful for anyone maintaining a legacy system with high volume telemetry and no budget for a proper rewrite.Happy to go deeper on any part of this in the comments, especially interested in hearing from anyone who’s had to diagnose systems they weren’t allowed to touch.\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1082092\/<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-494610","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/494610","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=494610"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/494610\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=494610"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=494610"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=494610"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}