How we test the backend

Getting started

Hello, my name is Vladimir Smirnov, and I am responsible for testing the trading backend at EXANTE.

Our domain is both complex and challenging. On one hand, it involves a high variability of scenarios and significant risks from potential errors. On the other hand, we work with short development iterations and frequent delivery cycles.

I joined the company at the end of 2017 as a technical support specialist, maintaining Python scripts for some time. In May 2019, I transitioned to a tester role.

From here on, I will use the term «tester» because I agree with the opinion expressed in the book Agile Testing that the effort to label specialists as QC or QA often stems more from a desire to escape negative associations with low-skilled positions than from actual competency expansion. Nevertheless, testers in Agile are generally more T-shaped and influence the entire software development process. We will not discuss dedicated QC and QA roles that do not directly engage in testing.

In May 2021, I was promoted to test lead, a position that eventually evolved into a test manager role. Over the past three and a half years, we’ve refined our testing processes multiple times—initially driven by our initiatives and later aligned with the company’s adoption of the SCRUM methodology. At first, we wanted to do everything at once, only to realise that this approach was unfeasible. Through trial and error, we discovered more effective methods. Interestingly, we later found that many of our approaches were already considered best practices in the industry. However, without our experimentation, we might not have understood their importance in our specific context. So, let’s begin.

What Do We Test?

Our product is a brokerage platform available in desktop, mobile, and web interfaces. Underneath these three faces lies a multitude of microservices, almost always strictly separated and highly interconnected. Just as the front end has a website, CRM, and internal interfaces, the back end has various post-processing scripts and data migration services for which we (the trading backend) are not responsible. However, we do handle the following (each point has its team):

1. User and account management

It sounds simple, but in reality, this includes three services, one of which is not a microservice and operates in three modes. In addition, the system includes embedded logic for authentication and authorisation, access control, roles, margin risks, commissions, and reports—and this is just the tip of the iceberg. There’s also trade processing, transaction handling, rollbacks, and mirroring, among others (by now, your brain might have already stopped processing the jargon and started waiting for the next bullet point).

2. Management of financial instruments and historical market data

Users have access to over 2 million instruments, all related information, settings required for their use by other services, and historical data. This massive dataset must be quickly collected, filtered, and delivered according to specific requests.

3. Counterparty integrations and client order processing

We gather data from numerous sources that use various protocols and APIs, process it, and transmit it internally within our backend in a standardized format. Additionally, specialized services independently calculate data based on available indicators, filling in gaps that external suppliers may not fully provide.

4. Proxies between the above services, trading platforms, and APIs for clients (HTTP and FIX).

It may sound as straightforward as «take and give,» but the process is complicated by numerous interdependencies and the need to prevent excessive strain on individual services.

Typically, consumers access data services via REST APIs, HTTP streams, or binary sockets using protobuf (for which we have our Python clients for testing purposes).

How Many Testers Do We Need?

In Scrum, it’s often said there are no distinct roles like programmers or testers—everyone is considered a developer. The idea is that anyone can step in for someone else, such as testing. While this sounds great in theory, it’s not entirely practical in reality.

The perennial debate about how many testers are needed to release on time never disappears. Resource allocation and the elimination of bottlenecks depend on several factors:

• Complexity of the logic being tested.
  I like the DDD approach, which divides subdomains into core, supporting, and generic. This approach helps assess the complexity of the logic to be tested. Business logic in the core domain deserves more attention and thorough testing.

• Testing debt.
  Some areas have well-established regression test suites, while others struggle to keep up with testing new features.

• Additional responsibilities.
  Without dedicated business analysts, their tasks don’t disappear—they’re simply performed less professionally by others. Similarly, when production incidents occur, someone has to investigate them or at least explain how things work.

Depending on the circumstances, the team decides how many testers are sufficient, whether reinforcement is needed, or if resources can be reallocated elsewhere.

In Team 1, our primary subdomain, the ratio tends toward one developer per tester. This is due to the high variability of tests, the likelihood of regressions, and the need to increase test automation coverage.

In Team 2, where functionality is pretty predictable, there is little complex logic and already good test automation coverage; the ratio is two developers to 1 tester.

In Team 3, the ratio of developers to testers is approximately 1.3 to 1, driven by the need to manage a large number of diverse integrations that require expertise in various protocols. These integrations also need to be thoroughly tested and supported in the staging environment, where they may be used by other teams (let’s call this internal beta testing).

Unexpectedly, the ratio in Team 4 is one developer to one tester because of numerous dependencies and legacy tests that require maintenance and rewriting for current realities (more on this later).

Onboarding

Every great experience begins with an introduction: a theatre visit starts with the coat check, ice skating begins with renting skates and starting at a new company kicks off with onboarding. Since everyone goes through this phase, it’s worth paying proper attention to it.

The goal of onboarding is to familiarise newcomers with the essentials and prepare them to collaborate effectively with their teams as quickly as possible.

We allocate two weeks for this process, during which the test manager supervises the new hire as they complete a checklist. The checklist includes steps for gaining access, installing necessary software, and reading essential policies and guides to gain an overview of the system’s architecture and functionality.

The main goal of onboarding is to learn to navigate the massive amount of information and scattered documentation. Deeper learning happens during tasks related to specific functionality being tested. For instance, attending internal meetups about individual services becomes meaningful only when directly working with those services.

At the end of onboarding, the test manager sets criteria for passing the probation period. The newcomer is then introduced to the team and begins fully engaging in their work. Of course, tasks are simpler in the beginning, with generous time estimates. Over the remaining two and a half months, the complexity increases and timeframes decrease.

How Do Teams Test?

New Features

At the moment, we follow this model:

1. Development and testing begin simultaneously. Testers create a plan that developers can review. What does this achieve?

   • Different perspectives on requirements are discussed early on, revealing discrepancies before actual work begins.

   • Developers understand how the functionality will be tested and used, allowing them to implement it more conveniently (ATDD).

   • Developers’ unit tests and testers’ integration tests are designed to cover a predefined set of agreed-upon cases, ensuring comprehensive coverage while avoiding unnecessary duplication.

2. Testers start writing automated tests for the feature before the code is written (progression testing). A small set of automated tests is ready when the feature is deployed to the testing environment. This allows:

   • Quick feedback is provided immediately after deployment without requiring developers to switch contexts.

   • Reduced retesting time.

   • Confidence in the feature’s stability post-release, as it enriches the regression test suite.

Writing at least one automated test is a mandatory part of testing a task—it’s impossible to move a task to «ready for release» status without it. The whole team agrees on this. Of course, some tasks may not lend themselves to automation or may be too costly, but this is the exception rather than the rule.

Automated tests, moved to a separate task, will rarely be written 99% of the time. Therefore, it’s better to have one working test that is already functioning (see benefits above) than a mythical complete coverage, especially since the coverage value is directly proportional to the number of expected fixes, which sharply decreases after the active development phase of the service ends.

3. After successfully launching automated tests (both new and regression), it is time to review the prominent cases that could not be automated manually. At this stage, exploratory testing also occurs, and communication with everyone who could be affected by this feature is conducted to check its integration into the system.

   • This helps reduce the task’s downtime while waiting (during the transition of the card from development status to testing and back) and the costs of context switching.

Old tests

Each new test adds to the old tests and needs to be kept in working order. Some fail due to reasons unrelated to the service, such as infrastructure issues or changes in adjacent teams (which were either forgotten or not communicated). These issues need to be addressed immediately. Some tests start behaving unreliably, failing intermittently without any apparent reason (flaky tests) – these also need attention (turn off/refactor/investigate). A designated person is assigned to handle this during the sprint, who:

• Draws information from notifications with links to Allure reports, analyses the causes of failures during nightly runs of all the team’s tests, and takes necessary actions for known issues (questions in chats, Jira tickets, quick fixes).

• Prioritises tasks for independent investigation and reproduction for handover to development (new failures for unknown reasons, the most frequently failing flaky tests). The biggest challenge for debugging is tests dependent on other teams’ services – in such cases, consultations with colleagues are required. To avoid distracting them from their sprints, we have a reserved time slot where interested parties can be invited to discuss the issues that have arisen.

• May be involved in other automation tasks: preparing for use in other environments (beyond the test environment), increasing coverage, and refactoring tests to meet new standards.

Time spent on analysis is not counted in the sprint’s capacity. However, researching and finding overlooked factors that cause periodic test failures is a task that cannot be delayed. Maintaining the current set of automated tests is part of the work in the sprint, so the need for it should be considered during planning. The time allocated to this activity is an excellent indicator of the team’s balance. Typically, when trying to meet the business’s interests and deliver new functionality, this kind of work tends to receive low priority and gets postponed, signalling a dev/qa imbalance in the team.

Release

The developer compiles the approved set of tasks into a version, while the tester supports the deployment to the staging environment, either independently or with the help of the operations team. As the release responsible person, the tester:

• Ensures all libraries have been pulled and all expected functionality is included in the release (specific tools are available for this).

• Performs acceptance testing (both manually and with automated tests).

• Creates a request to deploy the release version to production. A separate production support team handles this. They provide step-by-step instructions for migration or configuration changes.

• Monitors the release version on the staging environment for the remaining days coming up to the production release (in our case – two days).

Shift-left

During the current sprint, testers analyse tasks for the upcoming sprint.

• Acceptance criteria are defined and automated first.

• Tasks are broken down into smaller components whenever possible. Fewer requirements mean fewer variations, allowing for faster and less risky testing.

• A decision table is created with the developers, considering all parameters that may influence the functionality’s logic.

• Whilst familiarising themselves with the selected tasks, testers bring up questions. The product owner must provide the answers by the time sprint planning begins.

• Testers can reject a task from the sprint if they do not fully understand what and how to test.

Yes, this is the most tedious part of the job. It requires detailed preparation, but in practice, this prevents many problems and surprises during the sprint.

Documentation

Amidst all this hustle (everything described above happens within two weeks), there’s little room for documentation, but that’s where the concept of mandatory automated tests comes in handy – tests are written in such a way that they document themselves (a touch of BDD). In the TMS, only the following are added:

• Hard-to-automate but essential cases,

• The main flows for newcomers should preferably be tested manually to understand how everything is structured and connected.

Help for Teams

Automation engineers

All our testers carry the «Automation QA Engineer» label, as writing new and maintaining old automated tests is part of everyone’s job responsibilities. However, besides the testers, we also have a small, separate team of automators, which can even be called an SDET team.

What is the biggest challenge for testing in Agile teams? Tight deadlines.
What is the key ally in overcoming this challenge? Automated tests.
What’s the worst-case scenario? When automated tests consume time instead of saving it.

One reason for this can be the difficulty of writing and maintaining them. Due to the countless connections between services and the inability to test each independently, we have a single repository with a shared framework for automated tests using Python and pytest for all the trading backend services. This simplifies the work because each team can reuse the code when interacting with neighbouring services for data setup or result validation.

We aim to minimise the effort testers require to write automated tests. Our goal is for tests to be easy to write and read. Ideally, a test they wrote looks like a set of step functions. The automation team works on the following:

• Implements wrappers and helper classes (e.g., setup tools – special tools for setting up and monitoring necessary data integrity).

• Creates services and libraries that facilitate access to test data (e.g., a proxy server that connects to streams, to which tests make HTTP requests when data needs to be checked in the stream).

• Refactors and optimises code to solve emerging technical problems.

• Configures CI (example of an implemented pipeline in CI: habr or medium).

• Configures and writes linters.

• Implements informative messages about test run results.

• Conducts code reviews from the perspective of general standards and agreements. The automation framework is a living entity. New standards and tools are constantly created, and we use them to achieve our goals. Code review is one of these tools. Approval from automation confirms adherence to standards, and approval from a project colleague confirms the validity of test cases.

In short, it significantly saves time for colleagues in project teams.

Test Manager as a PMA (Project Manager in Agile)

In addition to hiring and training employees, supporting colleagues, and solving emerging problems, I focus on what is overlooked in day-to-day work, develop the overall testing strategy, and ensure its implementation.
It’s essential to avoid a conflict of interest here: testers should not feel caught between two fires, where on one side, the product owner asks to «test faster,» while the test manager expects the implementation of their instructions. Therefore, as the test manager, I act as the customer.

• Based on my communication with the testers in the teams, I define the overall strategy and, from it, create the necessary tasks for the teams.

• During sprint planning, the product owner decides when and in which sprint they will be implemented.

Thus, the tasks are the team’s overall responsibility, with the product owner determining resources, time, and priorities within the teams. The testing strategy is gradually implemented. Just like the capacity allocated to support the current set of automated tests, the movement of these tasks and the speed of their implementation are excellent indicators of the dev/QA balance.

Book Club

Policies, regulations, and written procedures are not the defining documents – they are just tools for recording the established understanding of what, how, and why we do things. There are different ways to synchronise this understanding: training organised by the company or partial compensation for completing educational courses outside the company. In a small group of testers working on the trading backend, we created a «book club» for this purpose:

• We select a book on the most relevant or «painful» issue.

• We set a reading schedule for chapters for weekly optional calls.

• We discuss what we’ve read through the lens of our experience and situations in current projects.

Besides organizing scattered knowledge on theory, this activity also serves as a source of new ideas.

What’s Great?

Let’s summarise our preferences.

First and foremost, my primary focus is on my work. I actively participate in feature discussions from the requirements stage, quickly develop test code, identify particularly tricky cases, and ensure that no serious bugs are overlooked. Additionally, we set specific, shared goals and monitor their gradual achievement. For example, fast (less than seven minutes) regression test runs or acceptance of release versions using automated tests (and the ability to deploy them on any environment).
Professional growth is another rewarding aspect. It’s exciting to see how testing becomes more accessible and understandable to other team members and how testing approaches and tools evolve into a shared responsibility across the team.

What’s Not So Great?

The vision described above represents an ideal approach to setting up testing processes. However, challenges still exist and manifest differently across teams, depending on their unique circumstances. Among them:

• Prevailing business tasks over technical ones.
  This will always be an issue, but it cannot be ignored. For example, maintaining the current set of automated tests is often seen as something that can be neglected during planning. However, we continue instilling in the team that tests are a living entity that need to be watched and maintained during the sprint. If left unsatisfactory, they could result in missing important information about the product.

• Lack of team balance.
  People are people—there are always beliefs, setups, vacations, illnesses, and resignations. Testing sometimes becomes a bottleneck, slowing down the delivery of product functionality. But the solution is not always, nor only, about hiring. Redistributing responsibilities within and between teams, identifying time leaks (e.g., unnecessary meetings, duplicated work), and improving or automating processes (e.g., parts of the release procedure) can help.

• Monitoring the test environments.
  The desire to discuss necessary metrics, identify them, monitor them regularly, and maintain a staging environment load comparable to production permanently is something for which we currently lack the resources. It may be a collaboration with support or a separate role of a system tester.

• Weak cross-team collaboration.
  The company tackles this problem in different ways. Testing, on its part, tries two ways: targeted monitoring of those affected by changes in functionality and directly engaging with them, as well as creating test documentation for our platforms’ most likely usage paths. Creating a new role for a tester who performs end-to-end tests (backend-only) for old and new features passing through all components will act as a barrier that identifies all shortcomings at the early stages.

A Minute of Advertising:
You may be the one to help us solve these problems. We have open positions for backend testers, including in two new teams! (Career page)

Time to Wrap Up

While reading this article, you may realise there’s much more that could be said—but expanding it further would make it overly lengthy. It’s important to stop at the right point, so consider this a brief introduction. For any details that pique your interest, feel free to explore further in the comments or follow up directly. Ask questions, share your thoughts, and see you again soon!