In Part III we finished setting up Redis Sentinel and performed a test by shutting down the Primary and watching as Sentinel failed over to one of the Secondaries. In this article we are going to throw all of that away and start fresh, only this time we will deploy Redis Cluster to our 3 nodes.

If you are following along with the Vagrant setup described in Part II then now is the time to run vagrant destroy.

This will wipe out the current setup. Once that is done you need to start it all back up with vagrant up

Once the new environment is built, go ahead and finish setting up Docker swarm, again described in Part II.

Now that is done, we should have 3 machines, redis1, redis2, and redis3. Redis1 should be a docker swarm Manager node with 2 and 3 joined as Workers. Lets get started building out our cluster.

Like I mentioned in the previous post, it’ll be best if we define our own image. We started that work with the redis-sentinel image. We’re going to update things based on that. To start create a new file called redis-cluster.conf, and populate with this:

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

Then update your Dockerfile to match:

FROM redis:5
COPY redis-sentinel.conf /usr/local/etc/redis/redis-sentinel.conf
COPY redis-cluster.conf /usr/local/etc/redis/redis-cluster.conf
#sentinel
EXPOSE 26379
#cluster primary
EXPOSE 6379
#cluster replica
EXPOSE 6380 
#CMD [ "redis-server", "/usr/local/etc/redis/redis.conf" ]

Now lets build the image and push it up to docker hub

docker build -t wjdavis5/redis_wordpress:latest .
#you might notice I changed the name from the previous tutorial
docker push wjdavis5/redis_wordpress:latest

Now lets start off by creating our 3 Primaries on the Docker Swarm:

sudo docker service create --name redis_cluster1 --network host --constraint node.hostname==redis1 --hostname redis_cluster1 --publish published=6379,target=6379,mode=host wjdavis5/redis_wordpress:latest redis-server /usr/local/etc/redis/redis-cluster.conf --port 6379
sudo docker service create --name redis_cluster2 --network host --constraint node.hostname==redis2 --hostname redis_cluster2 --publish published=6380,target=6380,mode=host wjdavis5/redis_wordpress:latest redis-server /usr/local/etc/redis/redis-cluster.conf --port 6380
sudo docker service create --name redis_cluster3 --network host --constraint node.hostname==redis3 --hostname redis_cluster3 --publish published=6381,target=6381,mode=host wjdavis5/redis_wordpress:latest redis-server /usr/local/etc/redis/redis-cluster.conf --port 6381

Once those are running we can initiate the cluster:

#update to use your ip addresses

vagrant@redis1:~$ sudo docker run -it redis:latest redis-cli --cluster create 192.168.10.123:6379 192.168.10.124:6380 192.168.10.125:6381 --cluster-replicas 0
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: aaeab6aa905df8bfac214651af1b424a4cebee60 192.168.10.123:6379
   slots:[0-5460] (5461 slots) master
M: 3bf86faf87cc78977516fb2e0da5f0986026d892 192.168.10.124:6380
   slots:[5461-10922] (5462 slots) master
M: cfa19d648dd63c3993daa66b011df2d85dc7a5ac 192.168.10.125:6381
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 192.168.10.123:6379)
M: aaeab6aa905df8bfac214651af1b424a4cebee60 192.168.10.123:6379
   slots:[0-5460] (5461 slots) master
M: cfa19d648dd63c3993daa66b011df2d85dc7a5ac 192.168.10.125:6381
   slots:[10923-16383] (5461 slots) master
M: 3bf86faf87cc78977516fb2e0da5f0986026d892 192.168.10.124:6380
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Notice in the output the lines :

M: aaeab6aa905df8bfac214651af1b424a4cebee60 192.168.10.123:6379
   slots:[0-5460] (5461 slots) master
M: cfa19d648dd63c3993daa66b011df2d85dc7a5ac 192.168.10.125:6381
   slots:[10923-16383] (5461 slots) master
M: 3bf86faf87cc78977516fb2e0da5f0986026d892 192.168.10.124:6380
   slots:[5461-10922] (5462 slots) master

These identify the master ids in the redis cluster, we’ll need them for later. We are going to configure the instances as follows:

You’ll notice that with each instanced pinned to a Docker Host, and the Replica for a Master being on a different machine, we afford ourselves some fault tolerance.

Ok, now on to configuring the replicas. First we need to create the Docker Services:

sudo docker service create --name redis_cluster4 --network host --constraint node.hostname==redis1 --hostname redis_cluster4 --publish published=6382,target=6382,mode=host wjdavis5/redis_wordpress:latest redis-server /usr/local/etc/redis/redis-cluster.conf --port 6382
sudo docker service create --name redis_cluster5 --network host --constraint node.hostname==redis2 --hostname redis_cluster5 --publish published=6383,target=6383,mode=host wjdavis5/redis_wordpress:latest redis-server /usr/local/etc/redis/redis-cluster.conf --port 6383
sudo docker service create --name redis_cluster6 --network host --constraint node.hostname==redis3 --hostname redis_cluster6 --publish published=6384,target=6384,mode=host wjdavis5/redis_wordpress:latest redis-server /usr/local/etc/redis/redis-cluster.conf --port 6384

Once those are running we need to connect to Redis1 and issue the commands to join the replicas to the masters:

This is the part where you’ll need the master ids from the previous output. Here is the redis-cli help that may be of use:

wjd@DESKTOP-J7C5B0A:/mnt/c/Users/willi/redi/redis-5.0.3/src$ ./redis-cli --cluster help
Cluster Manager Commands:

  add-node       new_host:new_port existing_host:existing_port
                 --cluster-slave
                 --cluster-master-id <arg>
  help

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

And now the commands to initiate the replicas.

sudo docker run -it redis:latest redis-cli --cluster add-node 192.168.10.123:6382 192.168.10.123:6379 --cluster-slave --cluster-master-id 3bf86faf87cc78977516fb2e0da5f0986026d892

sudo docker run -it redis:latest redis-cli --cluster add-node 192.168.10.124:6383 192.168.10.123:6379 --cluster-slave --cluster-master-id cfa19d648dd63c3993daa66b011df2d85dc7a5ac

sudo docker run -it redis:latest redis-cli --cluster add-node 192.168.10.125:6384 192.168.10.123:6379 --cluster-slave --cluster-master-id aaeab6aa905df8bfac214651af1b424a4cebee60

#example output:

vagrant@redis1:~$ sudo docker run -it redis:latest redis-cli --cluster add-node 192.168.10.125:6384 192.168.10.123:6379 --cluster-slave --cluster-master-id aaeab6aa905df8bfac214651af1b424a4cebee60
>>> Adding node 192.168.10.125:6384 to cluster 192.168.10.123:6379
>>> Performing Cluster Check (using node 192.168.10.123:6379)
M: aaeab6aa905df8bfac214651af1b424a4cebee60 192.168.10.123:6379
   slots:[0-5460] (5461 slots) master
S: 36afb4e506bef2245e192a83abd856fdea252b34 192.168.10.124:6383
   slots: (0 slots) slave
   replicates cfa19d648dd63c3993daa66b011df2d85dc7a5ac
M: cfa19d648dd63c3993daa66b011df2d85dc7a5ac 192.168.10.125:6381
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: c4f9af7131f8dc89f0cd6f493c2e8c6b15d0ddf0 192.168.10.123:6382
   slots: (0 slots) slave
   replicates 3bf86faf87cc78977516fb2e0da5f0986026d892
M: 3bf86faf87cc78977516fb2e0da5f0986026d892 192.168.10.124:6380
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.10.125:6384 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 192.168.10.123:6379.
[OK] New node added correctly.

We now have a fully functional Redis Cluster running on Docker Swarm with 3 Masters and 1 Replica / Master spread across the swarm in a fault tolerant way.

In the next post we’ll cover some basic Kernel Tuning that is essential to hosting Redis (or really any application) in Production.