The first exposure to Exploratory Testing was at the first QA community event when I joined ThoughtWorks three years ago. Colleague Nizi talked about a long PPT and carefully listed all the exploratory test methods. I just thought it was funny, but I don’t remember it exactly. But I learned that the key to exploratory testing is to break the conventional routine, first design a part of the test, then execute, and then diverge some new test ideas based on the results of the execution. Isn’t that exactly how I usually work? I was secretly pleased that my “wild way” had such a good theoretical basis.
later, i gradually participated in a lot of agile projects, from simple monolithic applications to large complex systems with dozens of microservice interactions, and the idea of exploratory testing helped me upgrade all the way to fight monsters, and i gradually gained a deeper understanding of it.
what is exploratory testing
“Exploratory Testing” was proposed as a technical term by testing expert Dr. Cem Kaner in 1983. Some call it a “testing style,” some call it a “testing method,” and some equate it with manual testing, but I prefer to define it as a “testing idea.” It is different from a specific test technology (equivalence class division, boundary value testing, automated testing, etc.), emphasizing the selection of appropriate test methods according to the current context, adapting to local conditions and avoiding southern oranges and northern oranges. It can be used to help testers analyze test scenarios, develop test strategies, and even guide automated tests.
our conventional understanding of “testing” is to know what the expected result is, and to verify that the actual result is consistent with the expectation through some testing methods. it’s a bit like the “mona lisa puzzle” that you played as a child: the goal of the game is very clear, you just need to design the steps of the puzzle, follow the steps, and then you can see the famous mona lisa smile. the difference is that experienced players can complete the puzzle with the fastest speed and the fewest number of steps, while beginner players need more costs.
the traditional test method is “design first, test before testing”. that is to say, the test points are analyzed first, then the test cases are designed for the test points, and finally the tests are executed. this approach is feasible in cases where the test is targeted. in a real project, however, the testing activities we face are much more complex. it’s more of a “gopher” game, with the overall goal of knocking back the gopher that has popped its head back, but you don’t know which hole its little head will poke out the next second.
in contrast to traditional testing methods, exploratory testing advocates learning, emphasizing simultaneous test design, execution, and feedback from the results to continuously optimize testing. it’s a test mindset that advocates improvisation, quick experimentation, rapid learning, and dynamically adjusted. it’s also an iterative process, which doesn’t need to be considered absolutely enough at the beginning, quickly designing some scenarios, and then getting feedback from the results, and then further optimizing the test, so as to make the test more abundant. this coincides with the concept of “small-step iteration and rapid feedback” in agile development.
in my opinion, traditional testing and exploratory testing are not good or bad, but complement each other. while traditional testing methods emphasize planned conduct, exploratory testing is designed to roam a space with a mission, but without a pre-set route, to uncover more problems that were not foreseen in advance.
how to use exploratory testing in agile projects
i have been involved in several agile projects over the years, and i have been experimenting with learning and applying exploratory testing, which not only helps me sort out my testing strategies, but also guides me through the testing process, and plays a crucial role in the design of automated testing.
1.use exploratory testing to sort out test strategies
the so-called testing strategy, in simple terms, is to comprehensively solve the problems of “what to measure” and “how to measure” from the three angles of “system-borne business”, “platform involved in the system” and “system architecture”.
different products often have different characteristics, how to reasonably plan and decompose the tested products, is to solve the problem of “what to measure”.
the book “the road to exploratory testing practice” mentions a “roaming map model” that compares testing to tourists roaming in cities, helping testers divide the tested products into different areas, and then adopt targeted testing schemes for the characteristics of each area.
as shown in the picture above, for tourists, travel guides help tourists understand where a city is worth going and what is not. for testers, then, the software can be logically divided using the “roaming map model”:
- business districts: users spend money on software because the characteristics of the software allow their business to be completed. the business district test type focuses on testing the main features of the software, so regression testing needs to be performed frequently to continuously guarantee the availability of these key features. due to the high repeatability, testing in the “business district” will be the focus of automated testing.
- tourist areas: for software, some features are more attractive to new users. this also involves testing some user rights.
- history area: the software also has the legacy code of the previous version, and the purpose of testing in this area is to test the legacy code and legacy defects. this is particularly evident in legacy system retrofit projects. “new development features do not affect the original features” will become the focus of testing.
- hotel zone: when the software “rests”, it is actually still very busy. the hotel district model focuses on some accessibility features, such as software running in the background.
- Entertainment area: For software, such as a shopping website, the business district is to search for goods, add to the shopping cart, generate orders, payment, etc., and its entertainment area refers to a beautiful and beautiful UI, friendly user interface, etc. This requires a focus on user experience and compatibility testing.
- dilapidated areas: unlike travel, dilapidated areas of the software may have serious defects, security and performance issues. this part can be the hardest hit area of the software, with a focus on anomaly testing, performance testing, and security testing.
as for “how to measure”, the actual situation of each project is different, and the various stages of the project are not the same. according to the current actual situation, we must use the test layering theory, test design, and tool selection. more importantly, it is necessary to adjust the strategy and control the risk in a timely manner according to the feedback given by the test execution.
2. use exploratory testing to guide the agile testing process
In agile development, we work through product development in small steps and iterations. For example, we are developing a small system with 4 Features, each of which is broken down into several Stories, each of which is the smallest business unit in the system.
- Feature 1: Contains Story A, B, C, C, D, E, F
- Feature 2: Contains Story G, H, I, J, K
- Feature 3: Contains Story L, M, N, O, P, Q
- Feature 4: Contains Story R, S, T, U
Among them, Story A and B in Feature 1 interact with each other; Story E in Feature 1 is interdependent with Story K in Feature 2; Story I in Feature 2 is related to Story N in Face 3; and Story L in Feature 3 is interdependent with Story R in Fact 4.
Suppose, according to the product plan, we are about to complete development in 4 iterations, and release every two iterations. Then, according to the requirements priority, demand dependencies, team rate and other factors, the story is reasonably arranged to iteration 1 to iteration 4, as shown in the following figure.
so how can we effectively use exploratory testing to help us implement testing?
three common testing methods are mentioned in exploratory testing: single feature testing, interactive feature testing, and system interaction testing. when applied to agile development, it goes like this:
- First, since each Story is the smallest business unit, each Story needs to complete the tests independently in the current iteration, which is called a single feature test;
- Second, since some stories have dependencies between them, such as Story A and Story B in Iteration 1, when A and B complete the tests independently, we have to focus on the tests of the interaction between the two Stories. A typical example is that A is the UI of a page and B is the backend implementation. When testing Story A, we care about front-end display, validation, interaction, etc.; when testing Story B, it is likely that A has not yet completed development, and we need to pass API testing to complete the verification of the end-end logic artificially by controlling the input parameters. However, how do we ensure that the input parameters passed to the backend are correct when the frontend interacts with the backend? For development students, “joint tuning” is to solve this problem, and for testers, this is the minimum interaction characteristic test.
- In addition, through iteration 1 and iteration 2, we basically completed the development and testing of all the stories of Nature 1 and Property 2, and we can complete the first Release. Before Release, we need to complete a round of regression testing in the range of Nature 1 and Nature 2. Assuming that Feature 1 is “Register” and Feature 2 is “Login”, we have completed the acceptance of the registration and login functions respectively, but the “whether the newly registered account can log in normally” has not been fully verified. This is interaction feature testing, which aims to discover potential defects in features when interacting with each feature.
- Then, in Iteration 3, Story N correlates with Story I, which has already been released in production, and in order to better fit, it is likely that Story N needs to be refactored. Then when we test Story I, we should also pay attention to the interaction with Story N.
- Finally, after Iteration 4, we completed the development and testing of all 4 Features, and we had to go live again. Similarly, before going live, we have another round of regression testing. However, the scope of this regression test includes not only the newly developed content of Iteration 3 and Iteration 4, but also all the features that have been released online before, that is, the testing of the entire system so far. This is system interaction testing in order to discover potential flaws in the interaction between business units throughout the system.
of course, this is just an ideal example, in true agile development, we need to focus not only on single-point and regression testing of functionality, but also on performance, security, and compatibility testing. but regardless of the type of test, the scope of testing evolves from “single feature – interaction feature – system interaction”.
3. use exploratory testing to improve automated testing
Due to the fast pace of agile development, almost every iteration and every version needs to be tested for regression, large or small. The system is evolving larger and larger, and the amount of regression testing is increasing day by day. In the agile full-featured team, often four or five Dev equipped with a QA, even if it is a thousand hands Kannon, I am afraid it is difficult to complete all the regression tests purely by hand, then the automated tests should be on the stage.
- automate the parts that require clear requirements, stable business, and repeated regression testing;
- the parts with unclear requirements, frequent changes, no need to repeat tests, or special forms of verification results such as logs are combined with exploratory test ideas to complete through manual testing;
in the process of manual testing, you can also gradually extract some new test points to complete into automated testing, which will greatly improve test efficiency.
- unit tests: designed to test the smallest measurable unit in a program, with high isolation, no additional dependencies, and fast execution, it is recommended to complete “unit tests” after each code commit and through compilation, and before deploying the test environment.
- API testing: Mainly focusing on the business logic layer of the system architecture, we call a specific API through tools or code, give inputs, get outputs, and verify the response results. API testing should not only focus on normal scenarios, but also on abnormal scenarios. API test execution efficiency is relatively high, according to the stability of the project environment, set up each deployment environment after the start of the test, you can also perform the test at a fixed time every day.
- UI testing: UI testing focuses on the user interface and user experience. Code or tools simulate the user interacting with the system through behaviors such as keyboard input or mouse actions. Since UI test execution efficiency is relatively low, it is recommended to perform tests during periods of time when the environment is relatively stable every day.
- E2E testing: End-to-end testing should cover those paths with the highest business value and not focus on anomalous scenarios. To do this, you need to design the test, from the end user’s point of view, to determine the test scenario through the user portrait and user journey. E2E testing is a good helper for regression testing and may be performed frequently for some time before the release of the version.
- Smoke testing: Smoke testing is a quick validation strategy for software version packages that eliminates the need for more in-depth testing if the smoke test does not pass. Therefore, we can extract two or three of the most core scenarios from the E2E test and set them up as smoke tests, starting tests after each environment deployment to quickly verify the availability of this release.
no matter what kind of automated testing, it is necessary to set specific inputs and have clear outputs. but agile development is an evolutionary development process, and some acceptance conditions are like this in this iteration, and it is likely to be very different in the next iteration; in addition, our understanding of the system will also undergo a gradual deepening process. so, some of the methods in exploratory testing can help us make up for this process.
there are many exploratory testing methods, but the one that has helped me the most here is the four categories defined in the book exploratory software testing:
1). free-style exploratory testing
Freestyle Exploratory Testing randomly tests an application in any order and method, without setting the scope and rules of the test in advance, and without extensive preparation.
ABOVE WE MENTIONED THAT THE SMOKE TEST IS A BASIC SCENARIO EXTRACTED FROM THE E2E TEST. HOWEVER, E2E TESTING IS ACTUALLY THE MOST DIFFICULT OF THE WHOLE TEST, IT DEPENDS HEAVILY ON THE ENVIRONMENT, AND THE CREATION AND MAINTENANCE OF A COMPLETE ENVIRONMENT CAN TAKE A LOT OF EFFORT, ESPECIALLY WHEN THERE ARE MULTIPLE DIFFERENT TEAMS DEVELOPING INDEPENDENTLY. SO, WHEN AUTOMATED SMOKE TESTING CAN’T BE DONE, WE CAN GUIDE MANUAL SMOKE TESTING WITH THIS FREE-STYLE EXPLORATORY TEST TO QUICKLY VERIFY THAT THE PACKAGE IS FREE TO HAVE MAJOR CRASHES OR CRITICAL FLAWS.
2). scenario-based exploratory testing
Scenario-based exploratory testing is somewhat similar to traditional testing methods, with clear start and end points for tests. For example, in E2E testing, it may be based on user journals, user stories, the life cycle of the program, etc. The difference is that it does not restrict the route. For example, the scene we set is from Xi’an to Beijing, but there is no restriction on the way and route of travel, you can take a car, high-speed rail, plane, you can even circle the earth, as long as you can finally get there.
3). exploratory testing based on testing strategies
Strategy-based exploratory testing refers to the combination of freestyle exploration with experience, methods, skills and the sixth sense, which belongs to but is not exactly equivalent to freestyle exploration, it is completed under the guidance of experience and skills, and is more suitable for experienced players.
The existing testing strategy is the key to the success of exploratory testing based on the testing strategy, and the more extensive the testing knowledge and experience of the tester, the higher the test efficiency and the better the effect. Test newcomers can learn and extract some new test scenarios from the test path of test veterans, and supplement them into automated tests.
4). feedback-based exploratory testing
Feedback-based exploratory testing refers to testers dynamically adjusting their tests based on the results of the “last” or “previous” test.
for example, if you go to the hospital for a check-up because of a stomachache, the doctor gently presses you on a certain part of the abdomen, and if you do not respond, he will try to adjust the part; if you make a slight “ah” sound, the doctor knows that there is a suspicion in this area, and he is likely to press it again, and with a little more force. an “ah” is your feedback on the doctor’s test.
for another example, when team building, everyone often plays a “number guessing game”. the judge writes down a number of 0-100 on the paper, and the remaining people guess the number from left to right, and the person who guesses will be punished as the “lucky one”. this is fully in line with feedback-based exploratory testing.
Case 1: The first person guesses “66” and the judge replies “Big.”
the second person then speculates that the number is between 0-66. so he adjusted his numbers.
Case 2: The second person guesses “63” and the judge replies “small”.
then the test range is narrowed down to between 64-66, obviously, the answer is in 64 and 65, then the third or fourth place will be the one who “won the lottery”, and the best part will come.
we often encounter similar situations in testing, where we don’t know what the expected result should be at first, so we can’t automate the test. so we first conduct a set of tests according to our own experience or speculation, then analyze and set the next set of tests through the test results, and cycle through them until we find the final knot. once all the steps and results are clear, it can be done through an automated script.
finally, since exploratory testing is only a way of thinking, not a specific test method, there is no fixed routine in practical application, and everyone has a different understanding of it. but i hope that while we complete the test work, we can stop and summarize and think from time to time, so as to make the test more perfect and efficient.
- Exploratory Software Testing by James A. Whittaker
- “the road to exploratory test practice” shi liang and gao xiang