Long time no see! I have not written on this post in a while, but it was for a good reason. Currently I’m focusing in other (very fun) projects!
Today we’ll talk about Software Testing, and to be honest, I’m quite obsessed (in a good way) with testing due past experiences I had while working. Well let’s dive in with an analogy!
When we consume a product, we tend to try it out first (cars, clothes), check it passes the characteristics we request from that product (the proper color of fruits and vegetables, for example) or maybe we just trust the specifications and standards the product was put through ( like checking the caducity date of milk, revise nutrition labels of groceries, etc.). We tend to test our products, we have some expectations about the product, and we hope the product could meet them all of them (this also applies with relationships, oops!).
Testing is comparing the product we currently have against our expectations.
A more formal definition (and applied to software testing) comes from the ANSI/IEEE 1059 standard:
«A process of analyzing a software item to detect the differences between existing and required conditions (i.e., defects) and to evaluate the features of the software item.»
So basically, Software testing is the process of evaluating and verifying that a software product or application does what it is supposed to do. The benefits of testing include preventing bugs, reducing development costs, and improving performance.
Types of testing
Each type of testing checks different requirement (most of the times, non-functional requirements). As some examples we have:
Testing how the software performs under different workloads. Load testing, for example, is used to evaluate performance under real-life load conditions.
Checking whether new features break or degrade functionality. Sanity testing can be used to verify menus, functions, and commands at the surface level, when there is no time for a full regression test.
Testing how much strain the system can take before it fails. Considered to be a type of non-functional testing.
Validating how well a customer can use a system or web application to complete a task.
Levels of testing
There are different kinds of testing according to what we want to test. For example, maybe we checked on the supermarket we bought a delicious onion, but this onion was not a good ingredient in a chocolate cake. In this case, we tested the product (the onion), but not its interaction with other ingredients on the system (the cake).
The different levels of testing are:
Component or Unit Testing
Unit tests are performed to test individual software units. Since the software comprises several units/modules, error detection in a single unit is simple, cost-effective, and less time-consuming. The Unit test ensures that the functional and the non-functional behavior of the component is designed as per specification. Those tests are made by the developers.
Integration testing is carried out once Unit testing is complete. It focuses on the interaction between the components or systems. Integration testing can be component integration testing or System Integration testing. Those tests are made by the developers.
Once Integration testing is completed successfully, we move on to the next level of testing, which is called System testing. IEEE defines the system tests as “A testing performed on a complete and integrated system to check the system’s compliance with its specific requirements”. Those tests are made by the testers.
Acceptance testing is performed after System testing is successfully executed. Acceptance testing is done to provide the end-users the confidence that the system will work according to their expectations. While all the other levels (Component, Integration, and System) are executed by the software producers, acceptance testing is the only level that is carried out by the customers and end-users. The amount of acceptance testing required depends on product risk, and also how much testing has been successfully executed in previous levels.
As we can see, each level of test is done by the level of requirements are needed to be achieved. In a project, first are defined general requirements about the product, and some of those requirements are business oriented. Then, the product is defined as a system which is separated in different interfaces, objects, functions, and components. The more specific is the test, the earlier it’s run this test.
Methods of Software Testing
There is a main difference on checking the food on a restaurant among a client, a food critic, and a health assessment consultant. While a client may only consider how tasty, delicious and good looking her food is, the food critic may consider other elements as time of response of the restaurant, manners of personnel, contrast, texture, flavor, odor, composition of the food, quality of the ingredients … and the health consultant will also check the conditions of the kitchen, if the refrigerator is in the right temperature, that there are no plagues on the kitchen, and basically each detail of the process of making the food. As we can see, the three of the commensals are testing the food, but in a different deepness, with different methods of (food) testing.
In the case of software, we have some different methods for testing:
Black Box Testing
This method gets its name because a QA engineer focuses on the inputs and the expected outputs without knowing how the application works internally and how these inputs are processed. The purpose of this method is to check the functionality of the software making sure that it works correctly and meets user demands. This method can be applied to any testing level but is used mostly for system and user acceptance testing.
Black box testing QA specialist doesn’t consider the internal processes of the product while conducting a test
White Box Testing
Unlike black box testing, this method requires profound knowledge of the code as it entails testing of some structural part of the application. Therefore, generally, the developers directly involved in writing code are responsible for this type of testing. The purpose of white box testing is to enhance security, the flow of inputs/outputs through the application, and to improve design and usability. This method is mainly used at the unit and integration testing levels.
Grey Box Testing
This method is a combination of the previous two since it involves testing of both functional and structural parts of the application. Using this method, an experienced tester has partial knowledge of the internal application structure and based on this knowledge can design test cases while still testing from the black-box perspective. This method is mostly applicable to the integration testing level.
Ad Hoc Testing
This is an informal testing method as it’s performed without planning and documentation. Conducting tests informally and randomly without any formal, expected results, the tester improvises the steps and arbitrarily executes them. Though defects found with this method are more difficult to reproduce given the absence of written test cases, this approach helps find important defects quickly, something which cannot be done with formal methods.
Process of testing
There are multiple strategies for testing software, but still, it follows a common process. Tasks or steps include defining the test environment, developing test cases, writing scripts, analyzing test results, and submitting defect reports. In this section (as a Kent Beck’s devote) we’ll focus on TCR in this blog:
TCR, comes from «Test && Commit || Revert», where each time a test is executed and passes properly, the code is committed, but if the test fails, the new code is deleted permanently (git reset –hard).
TCR is way more aggressive and radical than TDD due the fact here our code is reverted, and we must try again. As a result, this leads to a set of very interesting ideas:
- We will never have a failed test while developing.
- Only we will have tests «on green».
- If you don’t want to get a bunch of code reverted . . . just write the necessary code for passing the test.
- The idea of making small increments which maintain the tests on green gains more power. This method is extremely incremental, with a few lines of code per increment.
- Even though, the tests itself are made in small increments.
My experience with Testing
We could still talk a lot about software testing, in my opinion I think is a topic is getting more and more consideration in industry. The complexity of the systems makes almost impossible understand the product as a whole. For better understanding of the system, and for ensuring it’s making the requested function is needed to test it. If you can define a test, you define the goals of the system and therefore, you can build what actually is expected.
During my experience as Software Engineer, I’ve done tests for UI components (using screenshot tests for android), unit tests ( on JUnit on Java or unittest on Python), I’ve faced the creation of integration tests ( simulating the interaction of the client reciving notifications from the server and making the right operations), and as a conclusion, I can tell if part of a system is not tested, it will be more difficult to understand the functioning of that piece of code.
Most engineers doesn’t test as much as needed. I found out in one of my past jobs, there were entire products, on production, with no tests. The integration of new components and features, as long as fixing or changing things on the existing code was a total pain in the ass. For doing anything you needed a profound knowledge of the system itself. I quit this job, was very frustrating.
In other job, I heard my team was doing «tests for testing the tests», I’d love to explain why this was needed but I barely remember the context of this idea.
Testing is needed, but maybe we could fall into the trap of test everything, in anycase, at each level. It could be said overtest is the overthinking for developers.