in recent years, microservices architectures have become increasingly popular and are gradually being adopted by enterprises. as the software architecture changes, how does the corresponding software testing strategy need to be adjusted? this article will introduce testing strategies under a microservices architecture and share the evolution of a nine-year project testing strategy over the course of business and architectural evolution.

01 about microservices

A microservices architecture is an architectural pattern that advocates dividing a single application into a small set of services, each running in its own process, and the services communicate with each other using lightweight communication mechanisms (usually RESTful APIs based on the HTTP protocol). Each service is built around a specific business and can be deployed independently to production and pre-production environments.

as can be seen from the concept of microservices, it has the following benefits:

• each service can be developed independently
• the units processed are finer-grained
• a single service supports standalone deployment and publishing
• more conducive to the expansion of the business

At the same time, independent development leads to technical separation, HTTP communication and the mechanism of Queue increase the complexity of problem diagnosis, which brings great challenges to the quality assurance of the system’s functionality, performance and security. In addition, the complex dependencies between services bring a lot of uncertainty, and to achieve independent deployment, it also puts forward higher requirements for operation and maintenance. Systems with microservices architectures should pay special attention to these aspects:

• dependencies and connectivity between services
• fault tolerance and availability of services
• the eventual consistency of the data
• stand-alone deployment
• uncertain

02 selection of testing strategies

When it comes to the testing strategy of microservices, it is easy to think of the article “Microservices Testing” on the Lao Ma website Toby Clemson, which recommends the testing strategy under the microservices framework as follows:

the strategy model of the classical testing strategy emphasizes the test layering and the appropriate coverage of each layer, and the overall structure is in line with the pyramid structure. is it optimal?

someone questioned this… think that the strategy model should be hive-shaped (see article):

THIS MODEL FOCUSES ON INTEGRATION TESTING BETWEEN SERVICES, WITH FEWER UNIT TESTS AND UI LAYER E2E TESTING AT BOTH ENDS.

some colleagues have also proposed that the test structure under microservices should be diamond-shaped, the integration between services is still the focus, unit testing is less, and non-functional tests such as security and performance are added to the top level.

it seems to make sense, what kind of strategy model is good to choose? can’t help but get stuck… what to do? let’s hear the story of our project first!

03 the story of the project

1. test the evolution of the strategy

still that blue whale project, unconsciously entering its ninth year. in the past nine years, as the business has continued to develop, the system architecture has also undergone many evolutions and adjustments. correspondingly, the testing strategy has undergone interesting evolutions.

in the initial single-user system and monolithic architecture, automated testing at each level was organized in strict accordance with the test pyramid. with the expansion of the function, a large number of mock unit tests have brought great inconvenience to refactoring.

WHEN THE ENTERPRISE SYSTEM BEGAN TO DEVELOP, WE ADJUSTED THE STRATEGY, REDUCED THE WRITING OF UNIT TESTS, INCREASED THE COVERAGE OF UI LAYER E2E TESTS, AND THE TEST STRUCTURE EVOLVED FROM THE ORIGINAL PYRAMID TO THE FORM OF AN INVERTED TRIANGLE BELOW THE TRAPEZOID ABOVE.

LATER, THE ARCHITECTURE WAS ADJUSTED AND THE SERVICE BEGAN. AT THIS POINT, A LARGE NUMBER OF E2E TESTS GRADUALLY EXPOSED THE PROBLEM:

• THE TEST EXECUTION TIME ON CI IS GETTING LONGER AND LONGER, AND THE ABILITY TO LOCATE THE PROBLEM IS VERY WEAK, ONCE THE TEST FAILS, IT TAKES A LONG TIME TO FIX, THE TESTER CAN NOT GET THE VERSION THAT CAN BE TESTED FOR SEVERAL DAYS, AND THE FEEDBACK CYCLE IS TOO LONG;
• DUE TO THE INCREASE IN INSTABILITY CAUSED BY SERVITIZATION, E2E TESTING CANNOT COVER THE REQUIRED SCENARIOS WELL, AND TESTERS WILL ALWAYS HAVE VARIOUS DEFECTS EVEN IF THEY GET A MEASURABLE VERSION.

THEREFORE, THE PROJECT INTRODUCED CONTRACT TESTS, STOPPED WRITING NEW E2E TESTS, AND MOVED THE TESTS DOWN, REPLACING THEM WITH API TESTS AND CONTRACT TESTS, RESPECTIVELY.

With the continuous increase of functions, although the amount of E2E testing does not increase, but its instability, difficult maintenance, difficult positioning problems continue to increase, at this time it is difficult to ensure the quality of the product by automated testing. To balance costs and benefits, the project considered removing most of the E2E tests and keeping only a small number of Smoke tests, moving more tests down.

AT THE SAME TIME, A NEW TECHNOLOGY “QA IN THE PRODUCTION ENVIRONMENT” APPEARED ON THE TECHNOLOGY RADAR, AND THE PROJECT BEGAN TO CARE ABOUT THE PRODUCTION ENVIRONMENT, AND COMBINED WITH THE CHARACTERISTICS OF MICROSERVICES IN THE QA TESTING STAGE TO CARRY OUT CORRESPONDING EXPLORATORY TESTS.

2. address the challenges of microservices

having mentioned the challenges posed by microservices earlier, let’s take a look at how the project is addressing them.

• dependencies and connectivity between services

UNDER THE MICROSERVICES ARCHITECTURE, THE INTEGRATION OF INDEPENDENTLY DEVELOPED SERVICES IS THE MOST CHALLENGING, AND HOW TO ENSURE THE DEPENDENCY AND CONNECTIVITY BETWEEN SERVICES IS CRITICAL. AS MENTIONED EARLIER, E2E INTEGRATION TESTING HAS GREAT CHALLENGES AND IS NOT SUITABLE, AND CONSUMER-DRIVEN CONTRACT TESTING IS A GOOD CHOICE. THE PROJECT USES CONTRACT TESTING TO ENSURE CONNECTIVITY BETWEEN SERVICES AND REPLACE SOME E2E INTEGRATION TESTS.

• fault tolerance and availability of services

when a system load reaches a certain level or a service fails, the microservices architecture has two techniques to ensure the availability of the system: fuse and downgrade of the service. service circuit breaking refers to when a service fails, in order to ensure the availability of the system as a whole, the failed service will be shut down; the downgrade of the service is to consider shutting down some peripheral services to ensure the overall availability of the system when the overall load of the system is overloaded.

the corresponding tests include:

1. fuse: from the performance point of view, when the system load reaches a certain fuse state, whether the service can be correctly fused; at the same time, from the functional point of view, verify whether the behavior of the system after fusing is consistent with expectations;
2. downgrade: from the business point of view, it is necessary to be able to distinguish between the core business and the peripheral business, and not to affect the core business when it is necessary to downgrade; when a service is downgraded, verify whether the system behavior is consistent with expectations from the functional point of view.

• the eventual consistency of the data

data consistency is a particular concern for microservices. for example, after the successful payment of an order on the e-commerce platform, it is necessary to update the points and order status, and when one of the order services or the integration service fails, it will lead to the final data inconsistencies.

test this situation, analyze which services will cause data inconsistencies from a business perspective, and create corresponding anomalies to test the final consistency of the data.

• stand-alone deployment

The independent deployment of microservices requires CI and CD support, which is inseparable from DevOps practice. At the same time, it is more critical that contract testing is required to verify the correctness of service behavior after independent deployment. For the work of the project in this regard, please refer to Wang Jian’s article: Do your microservices dare to deliver independently?

• uncertain

microservices architecture makes the complexity of the system increase a lot, and many things happen unpredictably, and the reasons for their occurrence can only be found after they occur. therefore, it is impossible to test in the pre-production environment to find the issue that only occurs in the real production environment, and we need to turn our attention to the production environment and use the uncertainty of the production environment and the unpredictability of microservices to build anti-fragile systems.

The main technology used in this regard of the project is QA in the production environment, please refer to the article: QA in the production environment

3. project testing strategy

as can be seen from the evolution described earlier, the project testing strategy combines references to different strategy models at different stages: pyramid – > approximate diamond (unless functional testing, similar to the diamond model) – > hive. in the later stage of full service, we believe that the hive model is more suitable.

Of course, automated testing that conforms to this strategy model is not enough, our project also uses exploratory testing for the characteristics of microservices, maintaining a continuous delivery rhythm, practicing DevOps practices, and combining QA and other technologies in the production environment to move the focus right to the production environment.

now, the overall testing strategy of the project has evolved into the following diagram:

1. the project uses an agile iterative development and continuous delivery model, with a release cycle every four weeks.
2. The automated tests implemented during development are implemented hierarchically: a small number of unit tests at the bottom, and the most intermediate amount is API testing (similar to the component tests in the policy model recommended by Lao Ma), with a part of the contract test and a small number of smoke tests to ensure contracts and integration between services. In addition to this, QA has manual exploratory testing, which includes some tests on microservices features. The entire test structure is similar to the honeycomb model.
3. Using QA technology in the production environment, using the production environment, error monitoring, user behavior analysis, and user feedback collection, so as to influence and guide the development and testing of the pre-production environment.
4. Using DevOps practice, efficient deployment and monitoring, combined with QA in the production environment, form a benign loop to ensure the normal delivery of the project.

04 testing strategies rethinking

the adjustment of the test strategy on the project seems very simple, in fact, each adjustment is not an easy process, it is a decision to balance the pros and cons and consider multiple factors.

analyzing the entire adjustment process, it suddenly found that when we were faced with multiple strategy models and did not know how to choose, we actually fell into a misunderstanding of paying too much attention to the test structure and forgetting what the original goal was.

1. factors affecting the testing strategy

step out of the misunderstanding, go back to square one, and rethink the goal of testing the strategy. the most critical factors influencing the strategy are business value, quality requirements, and pain points.

• business value

bringing greater business value and helping enterprises win more profits is the goal of software systems; software testing is one of the guarantees of the success of software systems, and business value is also the ultimate goal of testing strategies. all testing activities should be carried out around this goal, considering business priorities and effectively avoiding business risks.

• quality requirements

different systems, different stakeholders in the same system (different roles involved) may have different definitions and requirements for quality, which is undoubtedly a key factor influencing the testing strategy.

for systems with only internal users, the focus may be on the function of the system; while the products released to the outside world are more demanding, and the inappropriate position of a button may bring about the loss of a large number of users.

• pain point

The real pain points are often the highest priority and urgently need to be solved. Pain points that can be addressed through the adjustment of testing strategies have naturally become one of the key influencing factors. For example, CI Pipeline is too slow to send packages, in order to improve the efficiency of package delivery, on the one hand, In Pipeline itself, on the other hand, adjusting the proportion of automated testing, execution frequency, etc. is also one of the solutions.

2. evolved testing strategy

projects at different stages, under the big goal of business value, other influencing factors will be different, and like the evolution of technical architecture, the testing strategy should also be evolved.

starting from the goal, the influencing factors of various aspects of the stage are integrated to develop a testing strategy suitable for the time. over time, the strategy is evaluated and measured, and further improved and enhanced to better meet the needs. this is the evolved, goal-driven testing strategy.

05 summary

the integration of multiple services under a microservices architecture is the most challenging, and the most important thing is contract testing. contract testing effectively ensures that the contractual relationship between services is not broken, ensures the connectivity of services, and helps to achieve true independent deployment and independent delivery.

THE UNCERTAINTIES INTRODUCED BY MICROSERVICES ARCHITECTURES ARE NOT A BAD THING, AND THEY CAN BE USED TO BENEFIT FROM THE USE OF TECHNOLOGIES SUCH AS QA IN THE PRODUCTION ENVIRONMENT TO ENHANCE THE ANTIFRAGILITY OF THE SYSTEM.

the influencing factors of the testing strategy are not the only ones, and the technical architecture is not the most critical factor. the testing strategy under the microservices architecture is not fundamentally different from that under other architectures.

business value is always our ultimate goal. driven by this ultimate goal, the test strategy is not completed and can be shelved, and it needs to be continuously measured and improved throughout the software system construction process, which is evolved.