Using Other Advanced Tools

We have learned how trace can be used to debug live systems with minimum impact. There are many other libraries in Erlang/OTP and in the community we can use to understand production systems. We’ll look at three Erlang tools:

• The :runtime_tools library
• The :crash_dump viewer
• The :recon project

Debugging with runtime tools

We’ve already seen one of the :runtime_tools. The :dbg module enables tracing. We only scratched the surface. The useful library also includes more advanced functionality such as the following:

• It includes an Observer backend for remote debugging. If we want to remotely analyze a production node using the Observer tool, the node must have runtime_tools running. For the details of remote observing, see this article by Plataformatec.

• Integration with OS-level tracers, such as Linux Trace Toolkit, DTRACE, and SystemTap.

• It includes microstate accounting, a tool that measures how much time the runtime spends in several low-level tasks in a short time interval.

As adopters get more advanced, :runtime_tools is an important application for learning about production systems.

Exploring crash dumps

Whenever a production system terminates abruptly, the Erlang VM will write a crash report to the disk. It is a quite detailed file with a lot of useful information. In truth, for a typical dump, we’ll likely get more information than we can handle.

Let’s generate and analyze one below: