what is your test-to-development ratio? test full stage participation, how can it be busy?
the whole stage is tested, so what tests are needed to ensure quality?
automated test coverage is required to be 99%, including functionality, performance, and even ease of use…
the first and second questions are often asked, which are of greater concern to everyone, while the third is the real requirements of a project. the questions about how testers test, how testing activities are carried out, and how much test coverage should be, are related to the lean testing that will be discussed next.
01. lean production
before introducing lean testing, let’s first understand the concept of lean manufacturing.
Lean production can be seen from Wikipedia as an explanation of “lean production“: lean production is mainly derived from the production philosophy of the Toyota Way of Production (TPS), so it is also called Toyotaism (Toyotism), which was not called lean production until 1990. The goal of lean production is to control the amount of inventory and reduce waste in the production process, the core of which is “Just in time”, which aims to produce the required products according to the amount needed when needed, that is, to create value with the least amount of work.
02. lean testing
agile testing requires testing to shift left and test the whole process, and its purpose is also to quickly feedback and reduce costs to deliver high-quality software. if the concept of lean is introduced into agile testing, it will help reduce waste in the testing process and maximize the value of testing.
what is lean testing
to achieve lean testing, it needs to be understood: you can’t blindly pursue test coverage, large and complete test coverage is a waste, and effective testing is more valuable.
whether it is manual testing or automated testing, it is necessary to first clarify the business value and quality objectives, perform tests according to business risks, and focus on testing for high priority, while low priority can reduce test coverage.
According to the “Two-Eight Principle”, 80% of the business priorities may only be on 20% of the functional modules, while the other 80% of the functional modules only occupy 20% of the business. If you treat everyone equally, pursue comprehensive coverage, and spend a lot of energy on those 80% of low-priority modules, it will inevitably cause a lot of waste. On the contrary, not pursuing test coverage and pursuing the effectiveness of tests will double the results with less and bring higher ROI. Therefore, many times, it is crucial to test just right, and going online with bugs may be a good strategy.
note that the quality goals here are key, and for some software systems that are critical to life, the quality requirements will be particularly high, and the comprehensive test coverage is effective and just right.
this is the idea of lean testing.
therefore, lean testing can be defined as: to the business value as the goal, to deliver high-quality software at the lowest possible cost, test and test at the point where the value can be reflected, to achieve effective coverage, reduce waste.
the essence of lean testing
Based on the definition of lean testing, the essence of lean testing can be summarized as TAT: timely (Time), appropriate amount (Amount), and precision (Target).
- JUST IN TIME (T)
agile testing requires testing to be fully involved, so that testing activities occur at every step of the agile software development lifecycle, and each type of testing occurs at the moment when it should most happen, which is the concept of “just in time”. for example, the confirmation of requirements before development, the implementation and verification of automated tests before the submission of development code, the smoke testing of the system after deployment, and so on.
- APPROPRIATE AMOUNT (A)
for test coverage, some people will think that the higher the better, such as the previously mentioned team requires 99% of automated test coverage, even if these test coverage are effective, but spend too much energy to test some modules that are not so important or not so easy to problem, may also be more than worth the loss, resulting in waste. we recommend that test coverage, whether manual or automated, is just the right amount, according to the risk to determine the need to strengthen the business priority of testing and response to the test coverage, must not blindly pursue high coverage. there are trade-offs and trade-offs and time spent on things that are truly valuable, which is also a reflection of lean.
FOR EXAMPLE, WHEN DOING USER STORY ACCEPTANCE, HOW MUCH DETAIL NEEDS TO BE MEASURED HAS ALWAYS BEEN A CONTROVERSIAL ISSUE. I THINK THE NORMAL USE CASES OF THE MAIN PATH, PLUS SOME POINTS THAT REQUIRE SPECIAL ATTENTION, INCLUDE ERROR-PRONE ABNORMAL PATHS, SPECIAL BROWSERS LIKE IE, HIGH-RISK SECURITY ISSUES, ETC. YOU CAN’T AND DON’T NEED TO COVER ALL THE CORNERS OF THE STORY WHEN IT COMES TO ACCEPTANCE, AFTER ALL, IT’S NOT EASY TO GET TOGETHER, AND IT’S BETTER TO KEEP THE TIME AS SHORT AS POSSIBLE. OF COURSE, THERE MAY BE TEAMS THAT ALSO NEED TO ACCEPT LOGS, PERFORMANCE, ETC. AT THE TIME OF STORY ACCEPTANCE, AS LONG AS THEY ARE AS EFFICIENT AS POSSIBLE AND ACCORDING TO THE NEEDS OF THE TEAM, AND THERE IS NO COOKIE-CUTTER ANSWER.
- PRECISION (T)
precision testing usually refers to automated testing that is targeted according to the scope of the code changes. the scope of precision testing mentioned here is broader, which can be understood as risk-based testing, and the risk may come from the business and architectural level, may also come from code changes, and may also be related to system characteristics or other project factors. it is more important to analyze and design before performing tests, and not to blindly test.
03. guidance framework for lean testing
there are two common guiding frameworks for lean testing: the test four-quadrant and the test layering, which are described below.
test the four quadrants
The Four Quadrant for Agile Testing was first proposed by Brian Marick, and Lisa Crispin and Janet Gregory cite this four-quadrant framework in the book Agile Software Testing: A Practical Guide for Testers and Agile Teams. Since the role of this quadrant framework is not limited to agile environments, I call it the test quadrant here.
testing the four quadrants testing the left and right sides of the four-quadrant matrix refer to the role of the test, supporting the team and evaluating the product, respectively, while the upper and lower parts refer to the objects of the test, which are business-oriented and technology-oriented, respectively.
support team testing
the tests of the support team on the left are used to tell the team what code to write, and play a role in clarifying the requirements and assisting the design.
AMONG THEM, THE FIRST QUADRANT IS THE TESTING OF THE TECHNICAL SUPPORT TEAM TO HELP BUILD THE INTERNAL QUALITY OF THE PRODUCT, THAT IS, THE GUARANTEE OF CODE QUALITY, SUCH AS UNIT TESTING AND API TESTING, ETC.; THE SECOND QUADRANT IS THE TESTING OF THE BUSINESS-ORIENTED SUPPORT TEAM, FROM A HIGHER LEVEL IN A WAY THAT THE BUSINESS EXPERT CAN UNDERSTAND THE DESIRED BEHAVIOR OF THE SYSTEM, TO HELP THE TEAM CLARIFY THE BUSINESS TO BETTER UNDERSTAND THE REAL BUSINESS VALUE.
tests in these two quadrants provide feedback quickly and ensure rapid resolution of issues, both guiding feature development and providing a safety net against refactoring and the introduction of new code that leads to undesirable behavior.
evaluate the tests of the product
programmers write code that allows the business-oriented tests on the left to pass, but it may also not produce what the customer really wants, so testing of the evaluation product in the third and fourth quadrants is also required.
the third quadrant is a test of a business-oriented evaluation product that helps confirm whether the product is really needed by mimicking the way real users use the application; the fourth quadrant is a test for the technical evaluation product, which mainly uses tools and corresponding technologies to evaluate non-functional features such as the performance, robustness and safety of the product, and considers the development of these tests at each step of the development cycle.
the information generated in the tests of these two quadrants should be fed back to the left side of the quadrant matrix and used to create new tests to drive the next step in development, forming a benign reinforcement loop.
test the use of quadrants
the order of the quadrants is independent of the order in which the tests are executed, and agile development often begins with customer testing (business-oriented testing). factors related to the timing of test execution are typically:
- risk of product launch
- customer-side requirements for product objectives
- is it based on the development of legacy systems or new systems built from scratch
- available testing resources, etc
The test quadrant provides a framework for thinking about what tests are needed to ensure quality, and can be considered according to the specific situation of the project to carry out the corresponding tests. The test type shown in the test quadrant of the Blue Whale project shown above is not the same as that introduced in the Lisa book, which is based on the current cooperation method of the project with the customer, business requirements, quality requirements, etc. To determine the current tests that need to be performed, such as the safety tests in which are divided into business parts and technical parts.
The idea of test layering is mainly aimed at automated testing, which is divided into different layers according to the scope that the test can cover. The following is a borrowing of several test hierarchy diagrams of the microservices architecture on the Martin Fowler website to explain the concept of test layering:
test hierarchy as you can see from the diagram, the size of the range that can be covered by several different types of tests is not the same.
regarding the concept of test stratification, you may be more familiar with the test pyramid, the test pyramid presents the proportion of different types of tests, the underlying unit test is more, the higher the upper level test the smaller the proportion, presented as a pyramid structure.
in fact, with the technical architecture, system characteristics, quality requirements, team skill level and other factors, the proportion of each test is not the same, not necessarily the pyramid structure, as shown in the following figure, the honeycomb structure or the cone ice cream structure shown in the following figure is possible.
regardless of the proportions of hive and cone ice cream, each layer of testing has its own characteristics. the closer the test to the bottom, the closer to the code, the lower the cost of writing, the faster the execution speed, and the more accurate the positioning problem, but the farther away from the business, it cannot well reflect the business value; the higher the test, the closer to the business, the more reflective the business value, but there is not enough stability, slow execution, higher realization cost, and difficult problem positioning. therefore, there is a trade-off that needs to be weighed against the specifics of the project, with realistic goals to determine the proportion of tests at each level.
the idea of lean testing is mainly to help the team develop a suitable testing strategy, not a specific testing method. the essence of lean testing is to make the test timely, appropriate and accurate, that is, to make the test just right to reduce waste.
testing strategies are influenced by many factors, need to be goal-driven, and evolve over time depending on the situation. the test four quadrants and test layering are only a guiding framework, not a specification that must be followed, and can be used as a reference model for test strategy development, and the strategy for specific projects needs to be adjusted according to the specific situation of the project.