Redis Partitioning
Partitioning is the process of dividing data across multiple Redis instances, so that each instance stores only a subset of the keys.
Advantages of Partitioning
- It allows us to construct larger databases by utilizing the combined memory of multiple machines.
- It enables us to scale computational power by leveraging multiple cores and machines, and network bandwidth by using multiple machines and network adapters.
Disadvantages of Partitioning
Some Redis features do not perform well with partitioning:
- Operations involving multiple keys are usually not supported. For example, you cannot perform an intersection operation on two sets if they are mapped to different Redis instances.
- Redis transactions involving multiple keys cannot be used.
- Data handling becomes more complex when partitioning is used, such as managing multiple RDB/AOF files and backing up persistence files from multiple instances and hosts.
- Adding or removing capacity is also more complex. While Redis clusters mostly support transparent data balancing by adding or removing nodes at runtime, other systems like client partitioning and proxies do not support this feature. However, a technique called presharding can be helpful.
Types of Partitioning
Redis supports two types of partitioning. Assuming there are four Redis instances R0, R1, R2, R3, and multiple keys representing users like user:1, user:2, there are different ways to decide which instance a key should be stored in. That is, there are different systems to map a key to a specific Redis server.
Range Partitioning
The simplest form of partitioning is range partitioning, which maps a range of objects to a specific Redis instance.
For example, users with IDs from 0 to 10000 are stored in instance R0, users with IDs from 10001 to 20000 are stored in instance R1, and so on.
This method is feasible and used in practice, but it requires a mapping table from range to instance. This table needs to be managed, and a mapping table for various objects is typically not a good approach for Redis.
Hash Partitioning
Another partitioning method is hash partitioning. This works for any key and does not require it to be in the format of object_name:
- Use a hash function to convert the key into a number, such as the crc32 hash function. Executing crc32(foobar) on the key foobar will output an integer like 93024922.
- Modulo this integer to convert it into a number between 0 and 3, which maps it to one of the four Redis instances. For example, 93024922 % 4 = 2, which means the key foobar should be stored in instance R2. Note: The modulo operation is the remainder of division, typically implemented with the % operator in various programming languages.