In part 2 we looked at the 2 different ways Redis is setup in production, Cluster and Sentinel, and we started working towards configuring a Redis Sentinel Setup.

We’re going to continue with Configuring Redis Sentinel using our already running Redis1, Reds2, and Redis3 machines.

The next step in the process is to setup the Redis-Sentinel services in Docker. We are going to Pin our services to specific nodes to ensure we don’t ever schedule multiple Redis-Sentinel instances on the same node.

However this time we need to provide a configuration to our sentinel setup. There are several ways we could do that, however, the best solution in terms of future maintainability is to just go ahead and build your own Docker image with your configuration baked in. Once you have done that you can use Docker Hubs CI/CD pipeline to fully automate things. The contents of that thought could easily encompass a full post. So we’ll come back to that part.

For now using your favorite tool (I’m using VS Code) lets get a new project going. If you’re using VS Code create a directory where you like to store projects, open code and select Open Folder, selecting your new folder. I then like to “Save Workspace” into the same folder so I can easily launch Code back into this project.

Once you have the workspace setup create a new file called: Dockerfile

Notice the lack of file extension? You can name it something else like Redis-Sentinel.Dockerfile if you’d like

Next create another file called redis-sentinel.conf and give it these contents:

#https://redis.io/topics/sentinel
#You can read the documentation at that link
port 26379
sentinel monitor mymaster 192.168.10.117 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1

• The 1st line is the url for the sentinel docs so you can get all smart
• The 3rd line tells redis to start on port 26379
• The 4th line defines the master we want our sentinel to monitor, it has the Ip, port, and quorum configuration at the end. The Quorum is the number of sentinels that must agree to imitate a fail over.
• Next is how long a master must fail PING checks before being considered down
• Next is how long we’ll wait before Sentinel will try to fail over the same master again.
• parallel-syncs sets the number of slaves that can be reconfigured to use the new master after a failover at the same time. The lower the number, the more time it will take for the failover process to complete, however if the slaves are configured to serve old data, you may not want all the slaves to re-synchronize with the master at the same time

Now update your Dockerfile with these contents:

FROM redis:5
COPY redis-sentinel.conf /usr/local/etc/redis/redis-sentinel.conf
#CMD [ "redis-server", "/usr/local/etc/redis/redis.conf" ]

Notice a couple things here:

• I specify redis:5 – using a specific version instead of latest is a must for production
• We copy the configuration file to the container
• The CMD is for my reference later when we create the containers. We’ll eventually update the redis-server installation to use this custom image as well and I want to be able to use the same image for both.

I have also created a .dockerignore file with this in it:
.vagrant

This prevents the vagrant stuff from being sent to the docker daemon during build.

Now that we have our Dockerfile built and our redis-sentinel.conf file ready we can build the image

docker build -t wjdavis5/redis_sentinel_wordpress .

This isnt a docker tutorial, but in short this command builds an image tagged as “wjdavis5/redis_sentinel_wordpress” based on the current directory “.”

Once it is built we can push it to Docker Hub

docker push wjdavis5/redis_sentinel_wordpress:latest

Now we can move back over to our Docker host and create our services.

sudo docker service create --name redis_sentinel1 --constraint node.hostname==redis1 --hostname redis_sentinel1 --mode global --publish published=26379,target=26379,mode=host wjdavis5/redis_sentinel_wordpress:latest redis-sentinel /usr/local/etc/redis/redis-sentinel.conf

sudo docker service create --name redis_sentinel2 --constraint node.hostname==redis2 --hostname redis_sentinel2 --mode global --publish published=26379,target=26379,mode=host wjdavis5/redis_sentinel_wordpress:latest redis-sentinel /usr/local/etc/redis/redis-sentinel.conf

sudo docker service create --name redis_sentinel3 --constraint node.hostname==redis3 --hostname redis_sentinel3 --mode global --publish published=26379,target=26379,mode=host wjdavis5/redis_sentinel_wordpress:latest redis-sentinel /usr/local/etc/redis/redis-sentinel.conf

Now we should be able to connect to an instance of redis sentinel, using the docker run command from before, only this time we need to specify the port -p 26379

vagrant@redis1:~$ sudo docker run -it redis:latest redis-cli -h 192.168.10.117 -p 26379
192.168.10.117:26379> info
# Server
redis_version:5.0.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:9f27eb593282148b
redis_mode:sentinel
os:Linux 4.4.0-131-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:6.3.0
process_id:1
run_id:6f31f65979e629341da22a4b49dc820ebf2d2f2f
tcp_port:26379
uptime_in_seconds:216
uptime_in_days:0
hz:15
configured_hz:10
lru_clock:4492786
executable:/data/redis-sentinel
config_file:/usr/local/etc/redis/redis-sentinel.conf

# Clients
connected_clients:4
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# CPU
used_cpu_sys:0.276000
used_cpu_user:0.272000
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.10.117:6379,slaves=2,sentinels=4
192.168.10.117:26379>

Yeah mine says 4 sentinels b/c created an additional while testing this out, you should see 3. The next thing we want to do is test failover. To do that we first want to tail the logs for redis-sentinel. We can do that with the docker logs command:

sudo docker ps|grep sentinel|awk '{print $1}'|xargs sudo docker logs -f

Now from another window connect to the master (should be redis1) and issue the shutdown command:

vagrant@redis1:~$ sudo docker run -it redis:latest redis-cli -h 192.168.10.117
192.168.10.117:6379> shutdown
not connected>

Now if you watch the redis-sentinel log you should see the fail over initiate

20 Jan 2019 15:01:07.934 * +sentinel-address-switch master mymaster 192.168.10.117 6379 ip 172.17.0.3 port 26379 for 08aa07e38473a554911f3aebfd720692b3b7c948
1:X 20 Jan 2019 15:08:36.236 # +sdown master mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.236 # +sdown master mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.326 # +odown master mymaster 192.168.10.117 6379 #quorum 4/2
1:X 20 Jan 2019 15:08:36.326 # +new-epoch 1
1:X 20 Jan 2019 15:08:36.326 # +try-failover master mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.330 # +vote-for-leader 08aa07e38473a554911f3aebfd720692b3b7c948 1
1:X 20 Jan 2019 15:08:36.330 # 63e82269c31d372eaee4d3bde15aea8a2c2e65f4 voted for 08aa07e38473a554911f3aebfd720692b3b7c948 1
1:X 20 Jan 2019 15:08:36.330 # d7b04497a0b905f692aea802979d030d5c73d0e9 voted for 08aa07e38473a554911f3aebfd720692b3b7c948 1
1:X 20 Jan 2019 15:08:36.330 # 08aa07e38473a554911f3aebfd720692b3b7c948 voted for 08aa07e38473a554911f3aebfd720692b3b7c948 1
1:X 20 Jan 2019 15:08:36.383 # +elected-leader master mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.383 # +failover-state-select-slave master mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.445 # +selected-slave slave 192.168.10.121:6379 192.168.10.121 6379 @ mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.445 * +failover-state-send-slaveof-noone slave 192.168.10.121:6379 192.168.10.121 6379 @ mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.516 * +failover-state-wait-promotion slave 192.168.10.121:6379 192.168.10.121 6379 @ mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.670 # +config-update-from sentinel 63e82269c31d372eaee4d3bde15aea8a2c2e65f4 172.17.0.3 26379 @ mymaster 192.168.10.117 6379
1:X 20 Jan 2019 15:08:36.670 # +switch-master mymaster 192.168.10.117 6379 192.168.10.118 6379

When we issued the shutdown command it should have terminated the docker container that was running. Because we configured this as a Docker Service when the container terminates Docker is automagically going to reschedule the container to execute again. So at this point you should be able to reconnect to redis1, run an info command, and see that it is now a secondary (slave)

# Replication
role:slave
master_host:192.168.10.118
master_port:6379

Congratulations, you now have created a Redis Sentinel deployment on Docker swarm. In the next post we’ll cover configuring Redis Cluster.