Inside the out … er … outside the in?
We have already highlighted the fact that there are different, sometimes contradictory perspectives on automated software tests. The fact that the view of white and black box tests can be at least as ambivalent is perhaps surprising. This shows that the view of white/black box tests depends heavily on the context. We have decided not to bend reality any further but to accept this and engage in the resulting vicious circle. So, we have examined white and black box tests from two angles, the latter of course with black light. Enjoy reading Are you ready to think outside the box? and Testing in Boxes.
Ramon recently took a detailed look at the “Reasons and Goals for Automated Testing”. We were able to get to know a whole host of different characters, some of which seemed to be a bit contradictory. We now know what goals they are pursuing and today we are looking at different ways of testing software.
We will initially focus on two approaches: black and white. We also do what we software engineers tend to do from time to time: we think in boxes, which we can then leave again – hardly any discussion at the whiteboard can do without them. If we apply this to software testing, we are initially faced with a black box and a white box.
Black Box
The black box, our black box, is quite dark. We can barely see our hands in front of our eyes. In fact, we can’t recognise anything at all, it’s that dark..
Sometimes that’s not so wrong. What the eye does not see, the heart does not grieve over. And the fact that this thing here is a box is perhaps enough information for the moment.
What if that’s what it’s all about and nothing more? We don’t want to know any more about our box.
Despite everything, it’s easy to recognise from the outside. Six sides, four corners each, not too big and not too small – clearly a box. We can rely on it and place or build things on it, for example.
It’s the same with software. We are not interested in the details; they would probably just confuse and distract us. As a rule, the only thing that counts is behaviour: when we place something on top of our box or build a perhaps rather fragile structure on it, we want to make sure that the box doesn’t collapse and take everything with it. It should behave the way we expect it to.
Accordingly, in the software world, we understand black box testing to mean behavioural testing. We do not make any reference to internal aspects. And even if we actually know them, we switch off the light to change our perspective.
This allows us to take an open-minded approach to the test. With black box tests, we can determine whether our assumptions about the box are possibly wrong. Why they are wrong in detail is in the end irrelevant. This is especially true when we are dealing with a box over which we have no control. This also applies to the interaction of different boxes. In times of distributed architectures, this is the rule rather than the exception.
Black box tests therefore enable us to determine whether
- functions are missing or work differently than we thought,
- interfaces are faulty,
- data structures or access to external databases do not work as expected or
- the general behaviour or performance deviates fundamentally from our expectations or requirements.
Accordingly, black box tests are primarily used when the focus is on interfaces – in API smoke tests or system integration tests. We simply pretend that we have no idea about our own box. For example, we put ourselves in the shoes of a consumer of our provided API to check whether our box, as a whole, behaves in the way we think the consumer or user expects it to. In this way, we ultimately test from a comparatively independent perspective whether agreements and contracts are being adhered to ensure harmonious interaction.
Let’s assume, for example, that our crate only has 5 sides and a total of 5 corners instead of the 6 sides with 4 corners each described above, then it will still work geometrically and statically. The structure will be very stable (see also ‘pyramid’). Viewed from a very specific perspective (for example from below), our box of this shape will also look as if everything is in order. But we shouldn’t rest on our laurels!
Remember: the aim was to place something on our box and build it up. That won’t work, it will go wrong in the truest sense of the word. The interface we want to offer our users or consumers is missing or extremely buggy. So, they certainly wouldn’t be happy about it. On the contrary, they might hate our API.
In most cases, we not only provide an interface, we also consume interfaces provided by others. So, the fate described above can also befall us if we want to place and build something on third-party boxes – and expect it to work the way we thought it would.
In this scenario, we can use black box tests to ensure that the behaviour of other boxes on which we want to build corresponds to our expectations.
If this is not the case, we have little choice but to adjust our expectations to the observed behaviour. However, we cannot possibly expect this from others who want to use our interface:
“You’re just using the box wrong. If you find the right balance, you’ll be able to adjust something, don’t be like that!”
The discussion on this could be interesting – and is not at all recommended. Instead, we should take a look at our box and the communicated contract to test it from the outside, as others would see it.
And if we then realise that we can’t keep the promises we’ve made about our behaviour, it’s time to make changes. But now we should also take advantage of the fact that this box is our box and we have the opportunity to switch on the light.
White Box
Fuse in, light on! We open the box. If we’ve already realised that a corner is missing and something is wrong with the sides, we’re probably opening Pandora’s box – we need to be aware of that.
Let’s take a look inside the brightly lit, white box! We will see everything, in detail, and can test in detail and make sure that all the corners, edges and sides are there and do what they are meant to do when viewed individually.
If we know what it looks like inside the box, it definitely helps us to understand its inner workings. But we can also get lost in the details we find there. Sometimes there is a lot to see and discover in a box like this. The fact that we actually only wanted to put something in it quickly fades into the background because – we have to admit – rummaging in boxes can be damn exciting! It can make you forget the time. There’s just so much to discover!
Perhaps it’s not always so clear how everything actually works together so that you can put something on the outside of the box at the end. To understand this, we dive into every last nook and cranny. That takes time, familiarisation and general know-how. It’s time-consuming, it’s exhausting. But it enables us to gain a deep understanding of our box and thus avoid ultimately undesirable behaviour at a very early stage as a solid basis for everything else that builds on it.
In white box tests, we get very close to the small details at a low level. We may already recognise causes that later lead to our box having a top rather than a topside when viewed from the outside. If our box is actually software, this means that we have to take a close look at the code and its structure. We know which input is used where and how, what output this generates – and where this is then used in turn. We can trace paths through the application, the box, and analyse countless exceptional situations.
However, if we concentrate too much on these small details and write automated tests for every fine detail, two things will happen:
- Test coverage will move towards 100 per cent. We can safely assume that we have tested every path in our application, no matter how probable or improbable.
- Our tests are more likely to focus on the details of the implementation than on the desired or required and agreed behaviour at a higher level.
However, it is in the nature of things that requirements usually come from outside and affect behaviour. Our users are not interested in whether our crate has integrated stiffeners, for example, so that they can place their particularly heavy crate on it. If their requirements of us change only slightly, this can mean that we sometimes have to put a lot of effort into transferring the desired change from the outside to the inside of all the many subtleties and small details. If we have previously provided detailed and fine-grained tests for all of them, we may have to update something (almost) everywhere or they may even become obsolete.
That takes up a lot of time again. That’s not really ideal. We actually only wanted to modify the behaviour of the software slightly.
So, it doesn’t necessarily make sense to cover every little thing explicitly and in detail in an (automated) test. And it’s not necessary either. In the end, the behaviour is decisive. When testing this, it is helpful to understand, comprehend and verify the basic background, but then to approach the matter with a sense of proportion.
Grey Box
This is really neither ‘black’ nor ‘white’. In reality, this categorisation doesn’t work. It’s the same with our two boxes and the black box and white box test.
So, we look for a compromise. Let’s pick up our black box again and switch on the light. However, we dim it and direct it so that we can take a look at the essential parts that are of interest. We should avoid being dazzled. Our box now appears grey.
Our user wants to place a particularly heavy crate on top of our box. We know that we have installed a stiffener for this purpose, we also know how this stiffener should behave and what it should withstand, so we can focus on this in detail to make sure that there are no nasty surprises later on.
In grey box testing, we continue to focus on the desired behaviour. However, we look at the code and the internal data structures partially and only as far as is useful to us. The decisive perspective for the test is that of the user. When designing the test cases, however, it is beneficial to know and be able to take into account the underlying details.
In the example of our crate, this means that we define the test cases in such a way that we operate at the limits of the load capacity of the stiffener, with the agreed contract in mind. We want to ensure that the expectations of the users of our crate are met and that the agreements are adhered to. And this is precisely where the built-in stiffener helps.
By the way: we have installed a metal strut with a diameter of 2 cm in the centre of the crate. It glows blue. It looks nice, but nobody on the outside is interested in it – and that’s exactly why we only consider it to a limited extent in our grey box tests. To test the load capacity of the box and the stiffening of the box, it is enough to know that it is there.
The fact that it glows blue is still nice, though, and we’re thrilled about that. Ultimately, however, this is only one aspect of our enthusiasm for technology.
