Inside the out … er … outside the in?
We have already highlighted the fact that there are different, sometimes contradictory perspectives on automated software tests. The fact that the view of white and black box tests can be at least as ambivalent is perhaps surprising. And shows that the view of white/black box tests depends heavily on the context. We have decided not to bend reality any further but to accept this and engage in the resulting vicious circle. So, we have examined white and black box tests from two angles. The latter of course with black light. Enjoy reading Are you ready to think outside the box? und Testing in Boxes.
At pentacor, we develop software modules for a wide range of customers to support their business processes and open up new opportunities for end-to-end digitalisation. The implementations range from individual services to more complex systems consisting of several services and single-page applications in combination with data management solutions. And all of this is then embedded in the existing, sometimes more, sometimes less complex environments of our customers. We have made it our mission to test each of our software components automatically (Ramon has already published an excursion into test automation in his blog entry, Test Goals – Why we Test Automatically). Our aim here is to achieve high-quality and appropriate test coverage.
Tests for our solutions can be divided into two categories: white-box tests and black-box tests. But what are the differences? Which tests belong in each category? And what challenges do we have to overcome in our projects during implementation? This is exactly what I would like to share with you in this blog. So: “Are you ready to think outside the (white and black) box?” Yes? Then, let’s get started.
Black Box Tests
We typically use black box tests to check complete applications. This can be a system consisting of several services or even individual services. It is not necessary to know the internal structure – i.e. the implementation. In our black box tests, we check whether the implemented application works according to its specification. This includes functional requirements or user stories, on the one hand, and technical aspects such as conformity with an API specification (OpenAPI 3.0) on the other. In addition, this method allows us to identify whether the developers of the various services of a system have coordinated correctly or implemented them in a way that bypasses each other. Working in small, decentralised teams certainly favours this and should therefore not be ignored during testing.
Weihnachten – Photo by Valeria Aksakova – de.freepik.com
As an example, let’s assume that we have provided a software component in Spring Boot with a REST API and a corresponding single-page application (SPA) in React. The software component also communicates via REST with other systems that already exist in the customer’s IT landscape. Here, we would first test our two components (the backend and the SPA) separately in the black box test. These would then be API smoke tests against the backend to check whether the API is running at all and responds according to the API specification. The next step is to create system integration tests that use other existing services provided for testing purposes or mocks of these services. The UI of the SPA can then be validated via end-to-end tests with a mock backend or the complete system provided. This allows us to check whether both components work according to their specifications but also whether the entire system, including the existing services, behaves in a resistant and expected manner when interacting.
Challenges
This all sounds great at first, but, unfortunately, there are also some challenges with black box tests that are often not so easy to solve. This starts with integration into existing authentication and authorisation mechanisms. Nowadays, SPAs often use OAuth flows but there are rarely any test systems for this. We are often forced to recreate these mechanisms or we can at least have a test flow created via complicated configuration. In most cases, we also need test data that is valid for all systems. If we think of our example, our UI could send a token to the backend, which, in turn, requests user data (e.g. a user UUID) from an authentication service based on the token. With this user ID, user-relevant data can then be extracted from the other existing services and delivered back to the UI in aggregated form. How do we then ensure that the user ID is known everywhere for our tests and also returns the corresponding test data? Here, too, the reality is often very complicated: if we are lucky, it can be very time-consuming to provide the relevant data, which even then usually only covers a dedicated test case, or, if we are unlucky, we can only enable these tests using our own implemented mocks.
Advantages and Disadvantages
Let’s summarise the advantages and disadvantages of black box testing::
With this method, we verify that the implemented system behaves according to its specification. The complete interaction of all implemented components of the system as well as their connected external systems are tested and compared with the specification.
It also immediately becomes clear whether the system is resilient in expected and unexpected error situations. The tests can also be created by colleagues without in-depth knowledge of the implementation of the services.
The complete test setup for automated testing is very complex. Depending on what the tests require (test data, test systems, custom mocks, etc.), the test configuration and preparation can become very complex and time-consuming.
For end-to-end tests with UI integration, it is also important to provide headless browser functionality to enable test automation. Fortunately, there are already some very good libraries that can support this, such as protractor or puppeteer. As nice as it is to test the entire process, this does not reveal any defects in the source code or incorrect technical implementations. However, these issues are covered by the white box tests.
White Box Tests
With white box tests, we check exactly what we ignore in a black box: the internal structures and functionalities of the services we provide. To do this, we need an internal view of the system. This means that the code and the test are closely related and we need to understand both. Nevertheless, we not only test technical and programmatic processes (such as the handling of exception errors) but also functional aspects. In our example, this could be a test that checks whether we correctly aggregate data received in the backend or return meaningful feedback to our UI in the event of an error.
Broschüre – Photo by lifeforstock – de.freepik.com
White box tests can be tested at different levels of granularity. At pentacor, we differentiate between tests, which test individual units, methods or classes – so-called unit tests – and tests, which test paths and functionalities between units or subsystems – so-called module tests. If we think of our example from the black box test, then a classic white box test for the back end would be. For example, the testing of an aggregation component that summarises data from several sources into one object. We can then perform this test at unit level by calling the aggregation unit with test data and evaluating the return object. And as the next stage, we would run the test at module level by faking the REST request to the backend and checking the result object with the aggregated data. In doing so, the test not only runs through the aggregation unit but also other components required to execute the REST request and create the result object.
An example of white box testing of the UI is the testing of a search mask. At unit level, for example, we would check and validate the various inputs. This includes checks such as: “Have all the required entries been made?” or “Do the search patterns correspond to their specification?”. But validations of the display of individual UI elements are also carried out. These can include tests that check whether all labels are displayed in the correct style and their text in the correct language.
Challenges
At first glance, white box tests actually seem relatively clear and simple. Unfortunately, this is not always the case, as we are regularly confronted with challenges here too. It often starts with the fact that we also need meaningful test data for these tests. We then either have to build this ourselves from the specifications of the other systems or, at best, we receive the data from dedicated test systems.
The next questions we then ask ourselves are: “How do we define the boundaries of our test?”, “Where does a meaningful, functional separation of our implementation begin and where does it end again?”. The answer to these questions is not so trivial, especially when it comes to module tests. If our application only has a simple UI or a microservice as a backend, we can define demarcations relatively quickly. However, if the services become more complex, it is more difficult to decide on the boundaries. We then often make this a team task and define together how we want to structure the test boundaries.
But testing error cases is also challenging. In addition to our own programmed exceptions and their handling, all possible error cases from neighbouring systems in our components also have to be tested. We therefore not only have to deal with our own code but also with the other systems and their behaviour.
Advantages and Disadvantages
So, what are the advantages and disadvantages of white box testing?
Smaller, separate functionalities are tested and not a complete system. The internal code structures and error handling are tested intensively. However, functional and technical sequences between components are also validated. The white box tests can all run during the build time of the code, do not require any additional mocks or databases and are therefore very easy to automate.
The delineation of tests and the preparation of test data can be very time-consuming depending on the complexity of a service. We can’t just deal with the code; we also need to have an overview of the neighbouring systems. The complete functionality of our service in its embedded landscape is disregarded with these tests. This consideration is mapped via the black box tests.
Conclusion
Both methods that we use in our systems have their advantages and disadvantages. However, the disadvantages are not an excuse for us not to implement these tests. Quite the opposite – we rise to the challenge and find a solution that makes sense for our customers in terms of time and money and also allows us to sleep soundly at night. 🙂
So, rise to the challenge: be ready to think outside the box!
Image sources:
Thumbs up: OpenClipart-Vectors on Pixabay
Thumbs down: OpenClipart-Vectors on Pixabay
