WAL

Introduction

WAL, which stands for write-ahead log(also known as a commit log), is a temporary append-only data structure used for crash recovery and transaction recovery.

Random disk access for reads and writes is quite expensive. Hence, database implementations provide a fronting buffer manager and page cache in the main memory for faster access. The database applies all the write requests to the buffer manager in the main memory and asynchronously batches and syncs the data with the disk. Similarly, the database reads all the data from the buffer manager. The data is paged into the main memory if it is not present in the buffer manager.

The problem with using the buffer manager as an intermediary is the volatility of the main memory. System crashes and restarts erase the contents of the buffer manager and lead to data loss if it’s not synced with the disk. To mitigate this problem, database implementers provided an additional fast and disk-resident durable data structure called WAL. Most popular relational and NoSQL databases use WAL for crash recovery.

Recovery process

The typical lifecycle of a database update operation follows this progression:

• Step 1: Persist the write requests in the buffer manager.

• Step 2: Append the write requests to WAL before returning success to the client. Appending to WAL makes the data durable until it’s synced with the disk.

• Step 3: The system periodically batches data and syncs with the disk using fsync.

The process of syncing the contents to the disk is called a checkpoint. The contents up to the latest sync point are discarded in the WAL after the checkpoint process is complete. On a system restart, WAL contents are replayed in order and synced with the disk before making the data available for future requests.