Performance testing 101: Finding and fixing bottlenecks

Types of performance testing#

• Stress testing places a system under an extreme workload to determine the point at which the app will crash
• Load testing measures the response time of a system under typical workloads
• Spike testing places a system under rapidly changing workloads to see how it performs
• Endurance testing exposes a system to a specific workload level for an extended period of time to see how long it can sustain performance
• Volume testing checks to see how efficiently a system can process varying database volumes
• Scalability testing gradually increases the workload to see how efficiently a system can scale up to maintain its performance

Scalability Thought Exercise

How many end-users will your application need to support at release?

Now, think about how many users it’ll need to support a year or two from then.

Key performance indicators (KPIs)#

These are some common metrics used for performance testing.

• Requests per second (RPS): The rate of requests per second that are received, processed and responded to by a system
• Throughput or Transactions per second (TPS): The rate of transactions completed by a system
• Availability: The amount of time that an application is available to an end-user Processor usage: The amount of time spent by the processor to execute a request
• Memory usage: The amount of physical memory utilized to execute a request
• Latency: Amount of time between a user request and the start of a system’s response
• Bandwidth: The number of bits per second flowing through a network connection in a given time
• Response time: The amount of time it takes for an application to respond to a user request once it arrives
• CPU interruptions per second: The average number of hardware interrupts that a process receives
• Other: Error rate, average load time, private bytes, hit ratios, committed memory, top waits, page faults, maximum active sessions

The performance testing process#

Performance testing methodologies can vary depending on your specific needs, but for the most part, they tend to follow the same basic steps.

Step 1: Identify your testing environment#

Ideally, you’ll want to be able to choose an appropriate performance testing tool and testing environment. If you don’t have control over these factors, you’ll want to know as much as you can about your testing environment before designing tests. During this stage, you’ll need to identify relevant information related to the:

• Performance testing environment
• Deployment environment
• Available testing tools
• Hardware, software, and network configurations

Knowing what environment you’ll be working in and what tools will be available will help you identify potential roadblocks before testing begins.

Step 2: Identify your KPIs and performance targets#

Next, you’ll want to identify which key performance indicators will provide the information you need. You will also need to identify realistic benchmarks, constraints, and other performance-related criteria.

At this stage, you should be thinking about what “good performance” actually means for your application. There isn’t an industry-standard definition of what good performance entails, so you’ll have to base your goals around what good performance would look like from the perspective of your end-users.

Here are some questions to consider when defining your KPI benchmarks:

• What are the conditions or constraints of this test?
• What is an acceptable result for this KPI?
• What is an unacceptable result for this KPI?
• What is the goal for this KPI?

Step 3: Plan and design your performance test#

In this phase, you’ll be thinking about how usage may vary among end users, as well as typical use cases. Consider different scenarios that reflect real-world platform traffic and traffic spike conditions, and then design your tests to examine how a system performs in these situations.

Then, create a plan for how the performance testing will take place. Once relevant stakeholders have approved the proposed method and designs, you can move on to configuring the test environment and script development.

Step 4: Configure your testing environment#

During this stage, you or your business should configure and prepare any tools or resources needed to perform the test.

Ideally, your performance test environment would be a 1:1 mirror of the deployment environment, but that’s often impractical due to the cost and complexity required to reproduce your server content and architecture.

In general, you do want to make sure that the number of application tiers in your performance testing environment is identical to the live tier deployment model. It’s also better if the size of the application database is a close approximation of the live one.

Step 5: Implement your designs#

Develop and test your performance test. You want to run a pilot test to check and see if your performance test can measure the targeted KPIs.

Step 6: Run your performance tests#

You can now execute your performance tests. During the performance tests, you will be monitoring and recording performance data.

Step 7: Analyze findings, report test results, and retest#

Once you’re done gathering performance data, you can start analyzing it and identify different ways to improve platform performance.

The results are the most critical deliverable for performance testing. Create a descriptive and concise test report. Share your findings with the developers, along with any recommended solutions. Then, retest!

Automated performance testing tools#

Whether you’re testing in-house or working for a client, you’ll need to select and use a testing tool that is compatible with the application you’re testing. This is an important step that is often overlooked and can lead to unnecessary problems down the road.

Here are some of the most popular performance testing tools used today.

LoadRunner Professional and LoadRunner Cloud#

LoadRunner is a highly efficient industry-standard performance testing tool that supports performance for Windows, Mac, and web-based applications. It specializes in finding bottlenecks before the application is implemented or deployed. LoadRunner can be a somewhat costly tool to use but comes with a slew of features.

Key features:

• Great for handling a large number of users when testing
• Supports cloud testing
• Supports mobile testing
• Interactive user transaction simulation
• Continuous rigorous testing
• Advanced forecasting features
• Supports testing on a wide range of applications

LoadRunner supports the following integrations:

• Jenkins
• DynaTrace
• UFT
• SiteScope
• AppDynamics
• Selenium
• HP Diagnostics
• Microsoft Visual Studio

WebLOAD#

WebLOAD is a flexible performance testing tool that would be a solid choice for small, medium, and large enterprises. WebLOAD is capable of simulating hundreds of thousands of concurrent users.

Key features:

• One of the best testing tools for stress and load testing
• Includes an IDE with a wide range of tools including response validation, parameterization, messaging, and more
• Scripts can be run dynamically and on different virtual clients
• Provides a detailed rundown of collected data

It also provides the following integrations:

• Jenkins
• AppDynamics
• Selenium
• DynaTrace
• Amazon Web Dynamics
• New Relic
• Perfecto Mobile

Apache JMeter#

JMeter is a popular open-source load and performance testing tool. It supports various applications, servers, and protocols, including HTTP, TCP, SOAP, Web, LDAP, etc. It is one of the most popular performance testing tools available but is less efficient at detecting threats in large-scale applications than in small applications.

Key features:

• Extreme versatility
• Dynamic and static testing resources
• Fast and intuitive GUI
• Plugin architecture makes it highly extensible
• 100% Java scripted
• Multithreading framework
• Free!

JMeter is compatible with the following integrations:

• Microsoft Visual Studio
• Maven
• BlazeMeter
• DynaTrace
• Jenkins
• CloudGen
• JSUnit
• Meliora TestLab

Bonus: Profiling#

Your program runs, software bugs have been addressed, and the internal logic of the program is solid. However, your program is just not performing as well as it needs to for some reason. Software systems can be complex, and that complexity tends to scale as a company grows. Where complexity abounds, so do performance bottlenecks.

A software system can become so complex that the source of these bottlenecks can become obscured to even the most experienced engineers. Luckily, profiling can make it much easier to locate bottlenecks.

Performance testing is most often a proactive measure to address bottlenecks and other performance issues in your application. On the other hand, profiling is a reactive measure that helps finetune the performance testing process.

If your test data shows a problem with your system, you need to find out where that performance problem is. Finding out where a system is running slow is difficult, especially for modern applications with hundreds of thousands of lines of code and numerous components.

Profilers help programmers see which sections of a program use up the most time and resources. Profilers measure the duration and frequency of function calls while considering the time complexity and available memory of a program. If performance issues occur, be prepared to diagnose the issue with a profiling tool like Prefix or Scalene before jumping into optimization.

“Rushing to optimize before the bottlenecks are known may be the only error to have ruined more designs than feature creep.”

Eric S. Raymond