Libraries

Open source and vendor code libraries

Libraries are notorious sources of blocking threads, whether they are open-source packages or vendor code. Many libraries that work as service clients do their own resource pooling inside the library. When a problem occurs, these often make request threads block forever. Of course, these never allow you to configure their failure modes, like what to do when all connections are tied up waiting for replies that’ll never come.

If it’s an open source library, then we may have the time, skills, and resources to find and fix such problems. Better still, we might be able to search through the issue log to see if other people have already done the hard work for us.

On the other hand, if it’s vendor code, then we may need to exercise it ourselves to see how it behaves under normal conditions and under stress. For example, what does it do when all connections are exhausted? If it breaks easily, we need to protect our request-handling threads.

If we can set timeouts, do so. If not, we might have to resort to some complex structure such as wrapping the library with a call that returns a future. Inside the call, we use a pool of our own worker threads.

Then when the caller tries to execute the dangerous operation, one of the worker threads starts the real call. If the call makes it through the library in time, then the worker thread delivers its result to the future.

If the call does not complete in time, the request-handling thread abandons the call, even though the worker thread might eventually complete. Once we’re in this territory, beware. Go too far down this path and we’ll find we’ve written a reactive wrapper around the entire client library.

If we’re dealing with vendor code, it may also be worth some time beating them up for a better client library.

Blocked thread

A blocked thread is often found near an integration point. These blocked threads can quickly lead to chain reactions if the remote end of the integration fails. Blocked threads and slow responses can create a positive feedback loop, amplifying a minor problem into a total failure.