System Design Interview Prep Crash Course/

...

/

Design a Blob Store

Design a Blob Store

Learn to design a blob store.

We'll cover the following...

A blob store is a storage solution for unstructured data. We can store photos, audio, videos, binary executable codes, or other multimedia items in a blob store. Every type of data is stored as a blobA blob (binary large object) consists of a collection of binary data stored as a single unit.. It follows a flat data organization pattern where there are no hierarchies, that is, directories, subdirectories, and so on.

Mostly, it’s used by applications with a particular business requirement called write once, read many (WORM), which states that data can only be written once and that no one can change it. These blobs can’t be deleted until a specified interval, and they also can’t be modified to protect critical data.

A blob store storing and streaming large unstructured files, like audio, video, images, and documents
A blob store storing and streaming large unstructured files, like audio, video, images, and documents

Note: It isn’t necessary for all applications to have this WORM requirement. However, we are assuming that the blobs that are written can’t be modified. Instead of modifying, we can upload a new version of a blob if needed.

Requirements

Let’s understand the functional and non-functional requirements below:

Functional requirements

Here are the functional requirements of the design of a blob store:

  • Create a containerA container is like a folder in a file system used to group blobs. Don’t mix up this container with a Docker container.: The users should be able to create containers in order to group blobs. For example, if an application wants to store user-specific data, it should be able to store blobs for different user accounts in different containers. Additionally, a user may want to group video blobs and separate them from a group of image blobs. A single blob store user can create many containers, and each container can have many blobs, as shown in the following illustration. For the sake of simplicity, we assume that we can’t create a container inside a container.
  • Put data: The blob store should allow users to upload blobs to the created containers.
  • Get data: The system should generate a URL for the uploaded blob so that the user can access that blob later through this URL.
  • Delete data: The users should be able to delete a blob. If the user wants to keep the data for a specified period of time (retention time), our system should support this functionality.
  • List blobs: The user should be able to get a list of blobs inside a specific container.
  • Delete a container: The users should be able to delete a container and all the blobs inside it.
  • List containers: The system should allow the users to list all the containers under a specific account.

Non-functional requirements

Here are the non-functional requirements of a blob store system:

  • Availability: Our system should be highly available.
  • Durability: The data, once uploaded, shouldn’t be lost unless users explicitly delete that data.
  • Scalability: The system should be capable of handling billions of blobs.
  • Throughput: For transferring gigabytes of data, we should ensure a high data throughput.
  • Reliability: Since failures are a norm in distributed systems, our design should detect and recover from failures promptly.
  • Consistency: The
...