Number of Partitions

Rebalancing Partitions

As a system ages and user access patterns change over time, some partitions may handle a greater portion of the overall load on the system than their fair share. In such circumstances, partitions may be rebalanced, which involves moving data from partitions under high load to partitions with low load so that the overall load is, as evenly as possible, distributed amongst all the partitions in the system. Another goal of rebalancing to move only as much data as is required to achieve the balance and no more. Finally, the system should continue to be available for reads and writes while rebalancing is in process.

Rebalancing partitions is an expensive operation as large amounts of data is moved over the network. Rebalancing can be automated or manual while some systems offer partially automated rebalancing that still requires human intervention and oversight.

Fixed number of partitions

Some databases mandate setting a fixed number of partitions from the beginning. Say you have only 5 nodes available in the system then each node may be assigned several dozen partitions. If a new node is added to the system, it takes over some of the partitions from each of the existing nodes to load balance. Similarly, if a node leaves the clusters, the partitions the leaving node is responsible for are taken over by the remaining nodes in the system. However, the maximum number of nodes in the system is limited by the number of partitions. For example, we could have a timeseries data of some sort that we partition per day for the full year resulting in 365 partitions. The system can now have at most 365 partitions and with only 5 nodes starting out at the beginning, each node will be assigned 73 partitions. The number of partitions can’t be split or merged and remain constant. Partitions are moved to other nodes in entirety and generally without any downtime.