Software testing 101: Get started with software testing types

What is software testing?#

Software testing is a cyclical process used by developers to continually evaluate and correct the functionality of the features during the development process. Software testing compares the current build of the software with software requirements to confirm there are no missing requirements.

It also verifies that software can function correctly across different mediums or with existing integrated software.

Without software testing during development, you’d only know if your software worked when it reached the end-user!

How does it work?#

There are many ways to test software. In general, developers first decide a behavior or feature that needs validation, create a test that confirms the feature, then either correct the feature or move on if it passes.

In early software design philosophy, testing was undervalued and often completely ignored. Now, programs have become more complex, are implemented at a greater scale, and to a more diverse array of devices and operating systems.

Software testing has become an essential part of the modern development cycle. It acts as an ongoing form of quality assurance and verifies that the software can respond to all possible use cases or environments.

Here are some of the benefits of software testing:

• Ensures full functionality: Ensure all targeted features are included in the final product.
• Early warning: Warns of program defects during development, before they negatively affect user experience.
• Verified Device Support: Test software functionality on all targeted devices to ensure consistent user experience.
• Incremental development: Testing frameworks allow you to track measurable progress toward fulfilling all program requirements.

Black Box vs. White Box Testing#

Black Box Testing: The internal structure of the software is hidden from the tester. In other words, the tester knows what the software product is supposed to do but not how achieves that.

The tester witnesses only the results or behavior of the programming and does not need to be a programmer themselves.

This tester is often someone outside the development process to give an outsider opinion. Black box testing is primarily used to test program behavior and assess user experience.

White Box Testing: White box testing is the opposite of black-box testing; the tester does know the internal structure of the software. These testers evaluate the logic of the program in the source code through the use of specific test-case inputs.

By tracking the flow of the test inputs, the tester can verify that test cases are being handled correctly behind the scenes.

White box testers are often programmers within the development process and are used to check source code efficiency.

Automation vs. manual testing#

Manual testing:#

Manual testing involves a human tester plays the role of the end-user and checks test cases one at a time. This is the traditional form of testing and can find problems difficult to recognize by automated testing frameworks (visual appearance of a web app element, confusing layout, etc).

Automation Testing:#

Automation testing (or test automation) is the process of using software, called a testing framework to create automated test cases that compare the current program output with the expected output. The most common frameworks are Selenium and Cucumber.

The two most common testing approaches for automated testing frameworks are graphical user interface testing a that simulates user interface events like clicks or keystrokes and API testing that bypass the user interface to validate underlying behavior.

Automation testing is used to perform output-driven tests quickly or to schedule repeated tests for maintenance testing.

Functional testing methodologies#

Functional testing is a type of Black Box quality assurance test that generates test cases from software requirements and specifications. Below are some common examples of different functional testing methodologies.

Top Functional Methodologies:

• Unit testing
• Integration testing
• System testing
• Acceptance testing
• Regression Testing
• Smoke Testing

Common Functional testing process#

Most fundamental testing moves through the same 4 steps, widening in test scope at each step. The process starts with unit testing to evaluate individual components and ends with acceptance testing to evaluate how the product relates to the original plan.

Unit testing:

Unit testing is used to test individual components of a program separate from the other components. For example, in object-oriented programs, you would unit test an individual class before trying to connect it to other classes.

This type of testing is often completed by the developer to catch defects without needing to wait for a full test cycle. Unit testing is most often automated to get quick results but can be done manually.

Integration Testing:

Integration testing is used to test how multiple connected program components work together. This type of testing is often done after unit testing; first, validate each component individually, then how the components work together.

For example, you could integration test a parent class and two related child classes to ensure that test case inputs are assigned to the expected class with all expected attributes.

Integration testing is completed by the developer to verify that connected components join together seamlessly, usually through automated tests.

System Testing:

System testing is used to test a full product build with all components together. While integration testing tests modules of connected components, system testing tests how the program works with all modules integrated and catches defects in inter-module operations.

For example, we’d first integration test all our program modules like account login, search website, etc. then connect all modules and run test cases through the program like “create an account and post to the forum”. System testing is often performed by a separate testing team to avoid developer confirmation bias.

Acceptance Testing:

Acceptance testing (or user acceptance testing) is a test performed late in the development process to assess if all originally specified requirements are met by the final product build.

Both internal and external testers review the original product specifications and business requirements then check off each as they use the product. There are many ways to do acceptance testing with the most common being alpha testing (internal) and beta testing (external).

Specialized Functional Methodologies#

​ There are also functional tests fine-tuned to test specific aspects of a program beyond the process above. Below are the most common specialized functional methodologies.

Regression Testing:

Regression testing is used to test product integrity after an update or change. Regression test suites run automated tests either on the whole program or just the changed portions of the program. They then compare the output to logged output from earlier product builds.

If the outputs match, the test is successful. If they’ve changed (in unexpected ways), the test proves there was a regression or reduction of functionality.

Regression testing is the most common form of maintenance testing, that checks how a program is performing after launch. Regression tests can be regularly scheduled to provide continuous testing.

Test suite: A collection of tests automatically run in sequence by an automation test framework on scheduled intervals or by a tester.

Smoke Testing:

Smoke testing (sometimes called Sanity Testing) is used to quickly test only the most essential functions. These tests verify simple but core functionalities like “Does the program start?” or “Does the interface open/close?”.

Smoke tests are done as an indicator for later testing either to clarify if more testing is needed or to test if a current product build is stable enough for more rigorous testing.

The advantage of smoke testing is that it provides results quicker than more extensive testing suites to help determine the next step of the development process.

Non-functional testing methods#

Many non-functional methodologies overlap in focus because of the subjective nature of their definition.

Top non-functional methodologies:

• Performance testing
• Security testing
• Usability testing
• Compatibility testing
• Stress testing

Let’s take a deeper look at each of these methodologies.

Performance testing:

Performance testing is a general form of testing that assesses the speed, responsiveness, and reliability of software under a set workload. If software works but fails to meet the desired standard in any of these categories, it will be sent back to developers to improve the performance before continuing in the software development lifecycle.

Security Testing:

Security testing is used to find weaknesses in the security of information systems or software that use sensitive information like account-based systems or financial software.

There are many forms of security testing, like penetration testing or vulnerability scans, but all seek to address:

• Confidentiality: Sensitive information is restricted.
• Integrity: Data cannot be copied or modified.
• Authentication: Users must validate that they are who they claim to be.
• Authorization: Users must have permission to view sensitive information.
• Availability: Information must be available to authorized users when they need it.
• Non-repudiation: Users on both ends of communication must validate their credentials before the communication occurs.

Usability Testing:

Usability testing is used to determine where real end-users encounter difficulty or confusion. This is primarily done with a controlled cohort of end-users observed by a researcher. Testers are asked to perform certain tasks, such as “create an account” but not informed how to accomplish it.

They then use the product to accomplish the tasks and provide qualitative feedback about the experience. This methodology allows developers to get real-life feedback on how usable and intuitive their program is without advanced instruction.

This is increasingly linked to accessibility testing, that records how easily differently-abled end-users can operate the software. For example, how well text-to-speech software can communicate a web application’s visual elements.

Compatibility testing:

Compatibility testing evaluates how well software performs in different computing environments. This is often done automatically with a testing framework. The framework uses multiple virtual machines that emulate different target devices to run the same input.

The output from each VM is recorded and compared to determine if all outputs are the same and how performance differs across the platforms. This ensures end-users have a consistent product experience regardless of where they use it.

For example, if you were creating a mobile app for iOS and Android, you’d have a compatibility test verifying the app performs at a target level on both platforms.

Stress testing:

Stress testing is when developers push their software to an extreme use case to determine the breakpoints. The most common stress test is to maximize concurrent users to find how much the current build can scale.

Performance is tracked throughout the stress test so developers can find soft-breakpoints, or points when the user experience degrades below acceptable levels. Ultimately, stress testing aims to find where a system fails so you can avoid those failure conditions on the live product version.

For example, imagine you’re developing an online video game. You’d do a stress test by getting as many concurrent players on the game at one time as you could. You could then record how the server performs (speed, response, etc.) and also find when the server crashes (the breakpoint).

This overlaps with heavily with load testing. Load testing records how the software performs with expected workloads while stress testing records how the software performs at maximum workload.

Software testing life-cycle#

Let’s take a deeper look at each of these 6 steps:

Requirement Analysis#

You and your development team meet with product and marketing teams to discuss the end requirements and features of the product. For each requirement, the group brainstorms a testable specification that will indicate if that requirement has been met.

These specifications can be things like “runtime must be lower than X” or " customers must be able to easily operate the user interface". You’ll use these specifications for later steps.

Test Plan#

​ In this step, you and your development team brainstorm the specifics of how you’ll develop the test. Some common points are “what resources will we need?”, “what quantitative metric can we use to test our requirements?” and “what are initial risk factors that may affect test results?”.

The most important aspect of this step is to keep test metrics/cases specific and rooted in the product specifications.

Test Case Development#

In this step, you’ll create a test case or test suite of cases that verify that the target requirement was met. For general testing, you could use the functional testing process or for more specific requirements you could opt for a non-functional test such as the usability test.

Usually, this is done by translating the specification found in step 1 into code. It’s helpful to divide large specifications into multiple sub-conditions so you can see how far along the process the program fails.

You and the development team will also split test cases into automation and manual testing categories based on their metric and complexity.

Test Environment Setup#

In this step, you’ll create your test environment. Most products are released on multiple platforms meaning you’ll have to create at least one environment per platform. This is primarily done through testing frameworks and multiple virtual machines.

You’ll also create test inputs here that will create consistent output if run through the program. Good test inputs cover a full range of use cases and result in the same output if run repeatedly.

Test Execution#

In this step, you and your team will execute the test and record all decided metrics. Most teams will run tests multiple times to get multiple comparable data points. Note any critical or non-critical program defects to be revisited in the next development cycle.

You may also recognize that your metrics do not report all the data you’ll need. This is a good time to reassess your chosen metrics for future tests.

Test Case Closure and Analysis#

This step is about recovering solid, reportable takeaways from the tests. Most companies will have you write either a daily or weekly report that summarizes how each test went and what changes will be made as a result of the test.

From here you can either:

• Tweek the test and repeat for more information (different metrics, refined testing environments, etc.).
• Return to develop solutions for the product using the test results (optimize for runtime, increase scalability, etc.)

Using agile testing practices, you’ll complete this test cycle before you create the product code as well as after. This allows you to speed up development as you keep the test specifications in mind during product development.

Software testing best practices#

• Don’t rely fully on automated testing. Automated tests only look for defects the programmer knows to look for. Make sure to have at least one set of manual tests to catch unexpected defects.

• Write test cases in plain language or pseudo code along with your code. Your managers and newer team members will thank you for saving them the time of parsing test script.

• Use only controlled, insulated test environments to avoid outside interference. Using a personal machine or the public cloud subjects your tests to rogue variables that may affect the performance or output.

• Choose specific and quantifiable metrics. For both specifications and test cases, make sure your metrics only measure a single attribute and can be numerically tracked to aid reporting.

• Test before the final quality assurance step. This splits ups the testing workload throughout the process and will save you time often lost from overhauling a defective central component.

• Make incremental tests. Create sub-conditions within your tests to track where a program fails in the test.

• Maximize test coverage. Try to cover 100% of possible use cases to prepare the program for any input or environment.

• Have team members create tests for unit and integration testing. Avoid confirmation bias by having another developer create tests for your program. This is a good trick when external testing is not available.

• Use helpful test names. Name your tests after the condition or requirement it tests. Avoid empty names like Test1 or performanceTest in favor of StressTest_10000user.

• Use software testing tools like Selenium and Reflect. Testing can be difficult to keep track of. Use automated frameworks/tools to simplify your testing and make them more sharable across your team.

What to learn next#

Congratulations on finishing your first look into the world of software testing! If you’re interested in continuing your testing career, the next step is to learn an automation framework. Selenium is one of the most used testing frameworks in the world and can be learned fast!

To help you pick up Selenium quickly, Educative has created the course Design a Test Automation Framework with Selenium and Java. This concise course teaches you the basics of Selenium and provides dozens of interactive examples. By the end, you’ll be a master of building the types of complex automated tests interviewers are looking for.

Happy learning!

Continue reading about software testing#

• What is ethical hacking and penetration testing? Get paid to hack
• Performance Testing Tutorial: Automation, Gatling, and Jenkins
• Sass Tutorial: Unit Testing with Sass True