Break Our Application Like a Server (Part II)

How to kill BEAM processes with Observer

The BEAM is a resilient virtual machine. Supervisor processes monitor child processes and can be configured to handle failure differently based on the application’s needs. The most common configuration is to restart any failed child process, using the one_for_one supervisor option. The child process then initializes itself back to a healthy state. We can see this in the handle_continue callback of our Sneakers23.Inventory.Server process. If it were to crash, it would pull the current inventory from the database and continue in a healthy state. Other restart strategies are not covered in this course. The restart strategy we should use depends on the supervision structure of our application.

Designing a process tree that is guaranteed to come back online correctly can be tricky. We should test the initialization of processes with automated tests, but the QA process can also help us ensure the correctness of our processes in practice. In the following scenario, we’ll kill various processes in our system with the observer tool. Any process that we kill should re-initialize in a healthy state with minimal interruption to connected shoppers.

Define the test

A shopper is initially connected to the store, waiting for a shoe to release. Many processes in the application then crash. The system restores itself to a healthy state.

Write steps for the test

1. Start the server in a freshly seeded state.
2. Load the webpage.
3. Kill the Sneakers23.Inventory, Sneakers23.Replication.Server, Sneakers23.Repo processes.

Write expectations for the test

• The shopper sees “coming soon” after step 2.
• The shopper is not affected after step 3.

We select the custom processes built for this application as the ones we’ll kill. It’s more likely that these processes could have bugs that would prevent graceful restarts. We’ve added the database because databases are known to go down in production.

We’ll use an Observer to kill the processes listed above. We could do this on the command line, but it’s helpful to visualize the process tree during the test. After we start the Observer in the instructions below, find the “Applications” tab at the top—we will see a large sideways tree. The processes to kill are all named, so we should find them without issue. They’re all in the same column, close to the left-hand side of the tree. We will see the following view when we right-click a process and select “Kill process”: