Various Software Testing Paradigms

Types of testing

Software testing is a process used to evaluate the functionality of a software application or system. It involves executing the application or system to find errors or defects. There are many different types of software testing, each of which serves a specific purpose and tests the application or system differently.

Some common types of software testing include:

• UI testing: This type of testing focuses on the user interface of the application or system, ensuring that it’s easy to use, intuitive, and visually appealing.

• Integration testing: This type of testing verifies that different components of the application or system work together as intended.

• End-to-end testing: This testing verifies that the application or system functions correctly from start to finish, simulating real-world use cases.

• Regression testing: This type of testing is performed after making changes to the application or system to ensure that the changes haven’t introduced new defects or caused existing functionality to break.

• Unit testing: This type of testing focuses on individual units or components of the application or system, ensuring that they function correctly in isolation.

• Mutation testing: This type of testing involves introducing artificial errors or defects into the code and verifying that the testing process can detect them.

• Property-based testing: This type of testing involves generating random inputs and verifying that the application or system produces the expected outputs.

ML-specific testing is a subcategory of software testing focused on testing machine learning models and systems. Some common types of ML-specific testing include:

• Consistency tests: These tests ensure that the behavior of functions related to each other is consistent (i.e., data transformations supposed to invert each other actually do so).

• Invariance tests: These tests verify that specific model properties are left unchanged when certain input aspects are varied.

• Negation tests: These tests verify that the model behaves as expected when the input is negated, or otherwise changed, to represent the opposite of the intended meaning.

• Directional expectation tests: These tests verify that the model output changes in the expected direction when the input is modified in a specific direction or along a specific dimension.

Test hierarchy and ontology

As you can see, many different types of software and ML-specific testing exist, and the boundaries between them can be blurry. In practice, there is no strict, formal classification of these types of testing.

However, it’s useful to consider a hierarchy of testing, where tests are organized based on their granularity. Unit tests are focused on small, individual units or components of the code, and they’re fast to run but only cover atomic use cases. On the other hand, UI tests test the entire user interface and take more time to run, but they provide a comprehensive test of the code.

Martin Fowler introduced the concept of a testing pyramid, which suggests that there’s a trade-off between the isolation of a test and its running time and long-term maintainability. Tests that are more isolated, such as unit tests, are faster to run and easier to maintain, while tests that affect multiple components and integration between them, such as integration or end-to-end tests, are more expensive in terms of running time and maintenance.