Debugging Scaling Issues

Identify scaling issues

Based on the software system, its use cases, runtime environments, etc., there could be many ways to inundate it with input size and rate. So, scaling issues could affect a program’s behavior in many ways, and its symptoms could also as varied. We have already learned how a large input size affects the program’s runtime, consider the following scenarios:

• Imagine if a process is being bombarded with requests, the rate of incoming requests is more than the rate at which it can process them because of resource constraints. So, the time taken to process the request could spike enormously, leading to high latencies. Sometimes, this delay could be so significant that it could look as if the program is stuck or hung. However, this is not a hang and the process would eventually recover by itself if the requests stop at some point and the program catches up. But, if the requests keep piling up for a long time, it could very well lead to the user suspecting as much.

• Consider a process that processes an input of a certain size. Now, suppose this size exceeds a certain level. The process could eat up a lot of its resources (memory, sockets, threads, etc.) trying to process this request. Though this sounds similar to a resource leak, it is different because the resources are not leaked but are just used at a very high rate. The resources will be released back to the system as the requests are eventually processed. However, if the incoming requests are so many that the system runs out of them, then the result on the system is the same as that of a memory leak.

• There could be scenarios where multiple processes share resources. In such situations, more than one process is facing a scaling issue. So, process A and process B facing scaling issues simultaneously might cause different scaling issues than process A and process C. So, scaling issues could also be inconsistent.

Given all this, when an engineer has to deal with a scaling issue, how would they identify its cause to be the scale or load on the system? One very good indicator is the performance and operational metrics we talked about before. The symptoms, along with these metrics, will give a very compelling case that the issue on hand is because of increased scale. Any anomalies in input or resource usage patterns around the time or leading up to the time of failure are a direct indicator that the issue on hand is because of scale. Consider the graph below. It tracks a metric that measures overall program performance in terms of its latency or the average time it took for the program to respond to its inputs.