Benchmarking

Need for benchmarking

Use benchmarking tools to compare different implementations to the same problem. Proper benchmarking requires care. We need to guarantee a measurement won’t affect subsequent ones. For example, each measurement should run on a new process to make sure the data generated in the first measurement won’t affect the second one by triggering the garbage collector. We also need to make sure we warm up the code before each measurement, to make sure the VM won’t kick in dynamic code loading for some cases but not others. The details are tricky so most folks use a benchmarking tool instead of rolling their own solutions.

Imagine that our application needs to find the longest word from a file. Each word is on a separate line. Initially, everything works fine, but due to business demands, the file we have to process steadily grows until our measurements notice the difference.

Profiling trims the scope of the problem down to the following code snippet: