O/S : CentOS 7 Minimal
0. PreInstall
필요한 패키지 설치
|
[root@localhost ~]# yum update -y [root@localhost ~]# yum install wget net-tools make gcc tcl libc*-dev -y [root@localhost ~]# yum install ruby -y [root@localhost ~]# gem update [root@localhost ~]# ruby -v ruby 2.0.0p648 (2015-12-16) [x86_64-linux] |
리눅스 설정
|
[root@localhost ~]# vi /etc/selinux/config SELINUX=disabled
[root@localhost ~]# systemctl stop firewalld [root@localhost ~]# systemctl disable firewalld
[root@localhost ~]# reboot |
1. Install redis
[root@localhost ~]# wget http://download.redis.io/redis-stable.tar.gz
[root@localhost ~]# tar xvzf redis-stable.tar.gz
[root@localhost ~]# cd redis-stable
[root@localhost redis-stable]# make install
[root@localhost redis-stable]# make test
\o/ All tests passed without errors! <--- 이 메세지가 나오면 OK
[root@localhost redis-stable]# make install
2. Copy & Backup config file
a_master.conf는 마스터 노드용 설정을 저장, c_slave.conf는 슬레이브 노드용 설정을 위한 conf 파일 임.
[root@localhost redis-stable]# mkdir /root/redis-stable/conf
[root@localhost redis-stable]# cp /root/redis-stable/redis.conf /root/redis-stable/conf/c_slave.conf
[root@localhost redis-stable]# cp /root/redis-stable/redis.conf /root/redis-stable/conf/a_master.conf
[root@localhost redis-stable]# mv /root/redis-stable/redis.conf /root/redis-stable/conf/redis.conf.bak
3. Install ruby for redis
[root@localhost ~]# gem install redis
Fetching: redis-4.0.1.gem (100%)
ERROR: Error installing redis:
The last version of redis (>= 0) to support your Ruby & RubyGems was 3.3.5. Try installing it with `gem install redis -v 3.3.5`
redis requires Ruby version >= 2.2.2. The current ruby version is .
[root@localhost ~]# gem install redis -v 3.3.5
Fetching: redis-3.3.5.gem (100%)
Successfully installed redis-3.3.5
Parsing documentation for redis-3.3.5
Installing ri documentation for redis-3.3.5
1 gem installed
4. Configuration Master,Slave conf each servers
Reis Cluster는 최소한 3개의 Master Node가 있어야 한다. 테스트는 3개의 Master와 3개의 Slave를 구성하여 테스트를 진행 함.
|
|
|
세 개의 서버에서 위의 표에 맞게 마스터 및 슬레이브 포트 및 설정파일을 변경 한다.
다음의 예는 서버1에서 마스터, 슬레이브 설정 파일을 변경 한다. 서버2, 서버3은 위의 포트를 참조해서 변경해 준다.
|
Master (/root/redis-stable/conf/a_master.conf) |
Slave ( /root/redis-stable/conf/c_slave.conf) |
|
# bind 127.0.0.1 <-- 주석처리 protected-mode no port 6379 pidfile /var/run/redis_6379.pid cluster-enabled yes cluster-config-file /root/redis-stable/conf/nodes-6379.conf cluster-node-timeout 15000 |
# bind 127.00.1 protected-mode no port 6381 pidfile /var/run/redis_6381.pid cluster-enabled yes cluster-config-file /root/redis-stable/conf/nodes-6381.conf cluster-node-timeout 15000 |
5. Setting Aliases
서버 시작과 종료를 편하게 하기 위해서 Alias 설정. 필요 없으면 건너 뛰어도 됨. (개인적인 편의를 위해서 만듦)
|
[root@localhost ~]# cat .bash_profile # Master port 6379 alias mstart='redis-server /root/redis-stable/conf/a_master.conf &' alias mstop='redis-cli -p 6379 shutdown' alias mrediscli='redis-cli -c -p 6379'
# Slave port 6381 alias sstart='redis-server /root/redis-stable/conf/c_slave.conf &' alias sstop='redis-cli -p 6381 shutdown' alias srediscli='redis-cli -c -p 6381'
[root@localhost ~]# . ./.bash_profile |
6. Start redis
3개의 서버에서 Redis 마스터와 슬레이브 시작
처음 시작시에는 Cluster모드로 뜨지만 연결되지 않은 상태 이기에 No Cluster configuration found라는 메세지가 나온다.
일단 Running in cluster mode로 뜨면 정상
Server1 : mstart, sstart
Server2 : mstart, sstart
Server3 : mstart, sstart
다음은 서버 1에서 수행 시킨 결과 임.
|
Master |
[root@localhost ~]# mstart [1] 20558 [root@localhost ~]# 20558:C 14 Mar 15:53:19.522 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 20558:C 14 Mar 15:53:19.522 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=20558, just started 20558:C 14 Mar 15:53:19.522 # Configuration loaded 20558:M 14 Mar 15:53:19.524 * Increased maximum number of open files to 10032 (it was originally set to 1024). 20558:M 14 Mar 15:53:19.526 * No cluster configuration found, I'm 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 <---- ServerID _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 4.0.8 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in cluster mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 20558 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-'
|
|
Slave |
[root@localhost ~]# sstart [2] 20562 [root@localhost ~]# 20562:C 14 Mar 15:53:24.184 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 20562:C 14 Mar 15:53:24.184 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=20562, just started 20562:C 14 Mar 15:53:24.185 # Configuration loaded 20562:M 14 Mar 15:53:24.186 * Increased maximum number of open files to 10032 (it was originally set to 1024). 20562:M 14 Mar 15:53:24.189 * No cluster configuration found, I'm 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d <----- ServerID _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 4.0.8 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in cluster mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6381 | `-._ `._ / _.-' | PID: 20562 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' |
클러스터로 묶이지 않은 상태이기 때문에 클러스터 정보를 저장하고 있는 파일들을 확인해 보면 각자 자신의 정보만이 저장되어 있다.
nodes*.conf 파일에 있는 내용이 redis-cli -c 로 접속 후 cluster nodes 명령을 치면 나오는 내용 임.
|
[root@localhost ~]# ls -al /root/redis-stable/conf/nodes* -rw-r--r-- 1 root root 114 Mar 14 15:53 /root/redis-stable/conf/nodes-6379.conf -rw-r--r-- 1 root root 114 Mar 14 15:53 /root/redis-stable/conf/nodes-6381.conf
[root@localhost ~]# cat /root/redis-stable/conf/nodes-6379.conf 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 :0@0 myself,master - 0 0 0 connected vars currentEpoch 0 lastVoteEpoch 0
[root@localhost ~]# cat /root/redis-stable/conf/nodes-6381.conf 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d :0@0 myself,master - 0 0 0 connected vars currentEpoch 0 lastVoteEpoch 0 |
7. Make Master
일단 마스터들을 클러스터로 묶어 주는 작업을 진행한다.
다른 문서에서는 redis-trib.rb create –replicas 마스터1 마스터2 마스터3 슬레이브1 슬레이브2 슬레이브3 이렇게 한번에 주면 알아서 Master와 Slave로 만들어 주는데
여기서는 명시적으로 마스터와 슬레이브를 각각 추가하는 방식을 사용함.
명시적으로 redis-trib.rb create 명령을 이용하여 Master를 클러스터로 묶음.
레디스 클러스터는 16384 개의 Hash Slot을 사용할 수 있는데, 클러스터로 묶이 마스터 개수만큼 각각 나눠서 Hash Slot을 가진다.
여기서는 3개의 마스터 서버를 구성하므로, 3개의 서버가 16384개의 Hash Slot을 나누어 관리 한다.
클러스터 생성 명령은 아무 서버에서 수행해도 된다.
클러스터가 생성되면 각각의 클러스터 노드끼리의 통신을 위한 포트들이 하나씩 더 생기게 되는데, 기본으로 설정된 port + 10000으로 생성된다.
예로 port가 6379 이면 클러스터 통신용 포트는 16379가 된다.
그래서 방화벽을 오픈할 때는 두개의 포트를 오픈해 주어야 한다.
명령어 : redis-trib.rb create 서버1:포트 서버2:포트 서버3:포트
|
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb create 192.168.100.244:6379 192.168.100.161:6380 192.168.100.204:6381 >>> Creating cluster >>> Performing hash slots allocation on 3 nodes... Using 3 masters: 192.168.100.244:6379 192.168.100.161:6380 192.168.100.204:6381 M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master <-- 첫 번째 Master가 가지게 되는 hash slot M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:5461-10922 (5462 slots) master <-- 두 번째 Master가 가지게 되는 hash slot M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:10923-16383 (5461 slots) master <-- 세 번째 Master가 가지게 되는 hash slot Can I set the above configuration? (type 'yes' to accept): yes <-- yes 입력
>>> Nodes configuration updated >>> Assign a different config epoch to each node 20558:M 14 Mar 16:06:32.964 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH >>> Sending CLUSTER MEET messages to join the cluster 20558:M 14 Mar 16:06:33.006 # IP address for this node updated to 192.168.100.244 Waiting for the cluster to join. >>> Performing Cluster Check (using node 192.168.100.244:6379) M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master 0 additional replica(s) <-- 아직은 Slave 서버가 없음 M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:10923-16383 (5461 slots) master 0 additional replica(s) <-- 아직은 Slave 서버가 없음 M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:5461-10922 (5462 slots) master 0 additional replica(s) <-- 아직은 Slave 서버가 없음 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 20558:M 14 Mar 16:06:37.952 # Cluster state changed: ok |
클러스터가 정상적으로 묶였는지 클러스터 정보 확인
|
변경 된 node정보 파일 확인
[root@localhost ~]# ls -al /root/redis-stable/conf/nodes* -rw-r--r-- 1 root root 376 Mar 14 16:06 /root/redis-stable/conf/nodes-6379.conf -rw-r--r-- 1 root root 114 Mar 14 15:53 /root/redis-stable/conf/nodes-6381.conf
[root@localhost ~]# cat /root/redis-stable/conf/nodes-6379.conf 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521011193153 3 connected 10923-16383 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521011193152 2 connected 5461-10922 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 myself,master - 0 0 1 connected 0-5460 vars currentEpoch 3 lastVoteEpoch 0 |
|
Redis에 접속해서 정보 확인
[root@localhost ~]# mrediscli 127.0.0.1:6379> cluster nodes 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521011243000 3 connected 10923-16383 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521011243765 2 connected 5461-10922 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 myself,master - 0 1521011242000 1 connected 0-5460 |
|
[root@localhost ~]# netstat -na | grep LISTEN tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN <-- Redis Port tcp 0 0 0.0.0.0:6381 0.0.0.0:* LISTEN <-- Redis Port tcp 0 0 0.0.0.0:16379 0.0.0.0:* LISTEN <-- 클러스터 사이의 통신 포트 tcp 0 0 0.0.0.0:16381 0.0.0.0:* LISTEN <-- 클러스터 사이의 통신 포트 tcp6 0 0 :::6379 :::* LISTEN tcp6 0 0 :::6381 :::* LISTEN tcp6 0 0 :::16379 :::* LISTEN tcp6 0 0 :::16381 :::* LISTEN |
8. Add Slave
슬레이브 서버를 redis-trib.rb add-node --slave 명령어를 이용하여 하나 씩 추가해 준다.
슬레이브를 추가시에 마스터 서버를 지정해 주기 위하여 Master서버의 ID를 알아야 한다.
Master를 클러스터 묶을 때와 같이 아무 서버에서나 해 줘도 된다. 여기서는 서버1에서 수행해 줌.
명령어 : redis-trib.rb add-node --slave --master-id 마스터서버의Server-id slave서버:포트 master서버:포트
|
|
Server ID |
Master |
Slave |
|
Server1 (192.168.100.244) |
05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 |
6379 |
6381 |
|
Server2 (192.168.100.161) |
3b10f1119a8cb372bd745f0347876282d50bee83 |
6380 |
6379 |
|
Server3 (192.168.100.204) |
51ec89eb2062001a11ce1bca6799aa727bc9618a |
6381 |
6380 |
|
[root@localhost ~]#/root/redis-stable/src/redis-trib.rb add-node --slave --master-id 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.244:6381 192.168.100.204:6381 [root@localhost ~]#/root/redis-stable/src/redis-trib.rb add-node --slave --master-id 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.161:6379 192.168.100.244:6379 [root@localhost ~]#/root/redis-stable/src/redis-trib.rb add-node --slave --master-id 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.204:6380 192.168.100.161:6380 |
|
[root@localhost ~]# mrediscli 127.0.0.1:6379> cluster nodes 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 myself,master - 0 1521014518000 1 connected 0-5460 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521014519422 2 connected 5461-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521014519000 3 connected 10923-16383 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 slave 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 0 1521014516000 1 connected 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521014520424 2 connected 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521014519000 3 connected |
9. Key 분산 테스트
클러스터를 생성 하였으니, 정상적으로 키가 분산 되는지 테스트 진행
600개의 키를 생성하는 벌크 파일을 생성 후 Load 시킴
|
input.txt :
set key1 value1 set key2 value2 set key3 value3 ..... set key600 value600 |
|
Server1 Master에 접속해서 수행
[root@localhost ~]# cat input.txt | redis-cli -p 6379 >> input.out
[root@localhost ~]# cat input.out MOVED 9189 192.168.100.161:6380 <-- 키가 다른 마스터 서버에 생성됨 OK OK MOVED 13120 192.168.100.204:6381 MOVED 9057 192.168.100.161:6380 OK OK MOVED 13004 192.168.100.204:6381 MOVED 8941 192.168.100.161:6380 MOVED 5850 192.168.100.161:6380 OK ... 생략 |
정상적으로 분산되어서 키가 생성되었는지 확인
|
|
Server1 |
Server2 |
Server3 |
|
Master |
127.0.0.1:6379> info keyspace # Keyspace db0:keys=201,expires=0,avg_ttl=0 |
127.0.0.1:6380> info keyspace # Keyspace |
127.0.0.1:6381> info keyspace # Keyspace |
|
Slave |
127.0.0.1:6381> info keyspace # Keyspace |
127.0.0.1:6379> info keyspace # Keyspace db0:keys=201,expires=0,avg_ttl=0 |
127.0.0.1:6380> info keyspace # Keyspace |
뭔가 이상하다. 분명 키가 분산되었는데, 1번 마스터와 1번 슬레이브서버에만 키가 생성되었고, 나머지는 키가 없다.
접속해서 조회를 해 보면
|
Server1 Master :
127.0.0.1:6379> keys *60 1) "key60" 2) "key360"
1번서버에 없는 키를 조회해 보자.
127.0.0.1:6379> get key160 -> Redirected to slot [15558] located at 192.168.100.204:6381 (nil)
해당 키에 대한 메타 정보는 있는것 같은데 키 값이 생성이 안되었음. |
클러스터는 -c 옵션을 사용해서 사용해야 하나 해서 다시 redis-cli -c 옵션을 이용해서 다시 key를 생성해 줌.
|
Server1 Master 에서 기존의 키 삭제
127.0.0.1:6379> flushall 20558:M 14 Mar 17:36:45.841 * DB saved on disk OK
새로 벌크 로드
[root@localhost ~]# cat input.txt | redis-cli -c -p 6379 >> input_c.out
Server1 Master : 127.0.0.1:6379> info keyspace # Keyspace db0:keys=201,expires=0,avg_ttl=0 127.0.0.1:6379> keys *60 1) "key60" 2) "key360" 127.0.0.1:6379> get key160 -> Redirected to slot [15558] located at 192.168.100.204:6381 "value160" |
|
|
Server1 |
Server2 |
Server3 |
|
Master |
127.0.0.1:6379> info keyspace # Keyspace db0:keys=201,expires=0,avg_ttl=0 |
127.0.0.1:6380> info keyspace # Keyspace db0:keys=203,expires=0,avg_ttl=0 |
127.0.0.1:6381> info keyspace # Keyspace db0:keys=196,expires=0,avg_ttl=0 |
|
Slave |
127.0.0.1:6381> info keyspace # Keyspace db0:keys=196,expires=0,avg_ttl=0 |
127.0.0.1:6379> info keyspace # Keyspace db0:keys=201,expires=0,avg_ttl=0 |
127.0.0.1:6380> info keyspace # Keyspace db0:keys=203,expires=0,avg_ttl=0 |
클러스터 사용시 -c 옵션을 사용해야 한다.
10. Failover 테스트
시나리오 : 서버1 마스터 서버가 Down된 것을 가정해 본다.
현재 상태 확인 :
|
[root@localhost ~]# echo "cluster nodes" | redis-cli -c -p 6379
05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 myself,master - 0 1521088849000 1 connected 0-5460 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521088850636 2 connected 5461-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521088850000 3 connected 10923-16383
3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 slave 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 0 1521088851638 1 connected 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521088851000 2 connected 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521088851000 3 connected |
|
Server1 |
Server2 |
|
[root@localhost ~]# ps -ef | grep redis root 20558 1 0 Mar14 ? 00:01:59 redis-server *:6379 [cluster] root 20562 1 0 Mar14 ? 00:01:57 redis-server *:6381 [cluster]
# 1번 마스터 중지 (또는 DEBUG SEGFAULT 명령수행) [root@localhost ~]# mstop
[root@localhost ~]# ps -ef | grep redis root 20562 1 0 Mar14 ? 00:01:57 redis-server *:6381 [cluster] |
클러스터 정보 조회 [root@localhost ~]# echo "cluster nodes" | redis-cli -c -p 6380
05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 master,fail - 1521089039112 1521089037000 1 disconnected 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 myself,master - 0 1521089122000 2 connected 5461-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521089122853 3 connected 10923-16383
3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 master - 0 1521089121000 4 connected 0-5460 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521089123856 2 connected 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521089122000 3 connected |
Server1의 슬레이스 서버가 마스터로 승격 되었음을 확인 할 수 있다.
새로운 키를 Insert 해 본다.
|
Server2 :
[root@localhost ~]# redis-cli -c -p 6379 127.0.0.1:6379> set key601 value601 OK 127.0.0.1:6379> keys *601 1) "key601" 127.0.0.1:6379> get key601 "value601" |
Server1의 기존 마스터를 Start 해보자
|
Server1 |
Server2 |
|
[root@localhost ~]# ps -ef | grep redis root 20562 1 0 Mar14 ? 00:01:58 redis-server *:6381 [cluster]
#기존 1번 마스터 재 기동 [root@localhost ~]# mstart .... 18267:M 15 Mar 13:52:59.649 * Ready to accept connections 18267:M 15 Mar 13:52:59.651 # Configuration change detected. Reconfiguring myself as a replica of 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 18267:S 15 Mar 13:52:59.651 * Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 18267:S 15 Mar 13:52:59.651 # Cluster state changed: ok 18267:S 15 Mar 13:53:00.663 * Connecting to MASTER 192.168.100.161:6379 18267:S 15 Mar 13:53:00.663 * MASTER <-> SLAVE sync started 18267:S 15 Mar 13:53:00.663 * Non blocking connect for SYNC fired the event. 18267:S 15 Mar 13:53:00.664 * Master replied to PING, replication can continue... 18267:S 15 Mar 13:53:00.665 * Trying a partial resynchronization (request 700bba7ba2b36a730e88d8a638e3edf93f02113a:1). 18267:S 15 Mar 13:53:00.668 * Full resync from master: cf343df57691b7018efcd58918995192f96cc1f5:119951 18267:S 15 Mar 13:53:00.668 * Discarding previously cached master state. 18267:S 15 Mar 13:53:00.730 * MASTER <-> SLAVE sync: receiving 3546 bytes from master 18267:S 15 Mar 13:53:00.730 * MASTER <-> SLAVE sync: Flushing old data 18267:S 15 Mar 13:53:00.731 * MASTER <-> SLAVE sync: Loading DB in memory 18267:S 15 Mar 13:53:00.737 * MASTER <-> SLAVE sync: Finished with success
[root@localhost ~]# ps -ef | grep redis root 18267 16390 0 13:52 pts/1 00:00:00 redis-server *:6379 [cluster] root 20562 1 0 Mar14 ? 00:01:58 redis-server *:6381 [cluster]
기동 시점에 Full resync가 발생 함.
생성된 키 조회를 위하여 기존 마스터 서버에 접속해 본다.
[root@localhost ~]# redis-cli -c -p 6379 127.0.0.1:6379> info replication # Replication role:slave ... 생략 127.0.0.1:6379> get key601 -> Redirected to slot [977] located at 192.168.100.161:6379 "value601" 192.168.100.161:6379> info replication # Replication role:master
|
클러스터 정보 조회 [root@localhost ~]# echo "cluster nodes" | redis-cli -c -p 6380
05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 slave 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 0 1521089881450 4 connected 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 myself,master - 0 1521089880000 2 connected 5461-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521089880000 3 connected 10923-16383
3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 master - 0 1521089880448 4 connected 0-5460 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521089879445 2 connected 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521089879000 3 connected
|
위의 테스트를 자세히 보면,
슬레이브에서 키를 조회 하게 되면, 마스터 서버로 redirect 되는 것을 확인 할 수 있다. 그리고 이후 명령어들은 마스터에서 수행되어 진다.
그래서 다시 info replication을 해보면 role이 master로 나오는 것을 확인 할 수 있다.
아마도 슬레이브에서 추가, 변경 작업이 발생되지 않게 하기 위한 것 같은데, 이것은 어디까지나 추측일 뿐이다.
다시 월래 상태로 되돌리기 위하여 Server2의 승급되었던 6379를 내렸다가 role이 바뀐 후 다시 기동시킨다.
11. Add Master
3마스터,3슬레이브에 새로운 마스터 하나 추가를 해보고, 데이터가 재 분배 되는지 확인해 본다.
|
|
Master |
Slave |
|
Server1 (192.168.100.244) |
6379 |
6381 |
|
Server2 (192.168.100.161) |
6380 |
6379 |
|
Server3 (192.168.100.204) |
6381 |
6380 |
|
Server4 (192.168.100.147) |
6382 |
|
서버4에서 redis-trib.rb add-node 명령을 이용하여 Master를 추가 시킨다.
명령어 : redis-trib.rb add-node 추가할서버:포트 기존클러스터서버:포트
|
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb add-node 192.168.100.147:6382 192.168.100.244:6379 >>> Adding node 192.168.100.147:6382 to cluster 192.168.100.244:6379 >>> Performing Cluster Check (using node 192.168.100.244:6379) M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381 slots: (0 slots) slave replicates 51ec89eb2062001a11ce1bca6799aa727bc9618a S: 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380 slots: (0 slots) slave replicates 3b10f1119a8cb372bd745f0347876282d50bee83 M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379 slots: (0 slots) slave replicates 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 192.168.100.147:6382 to make it join the cluster. [OK] New node added correctly. [root@localhost ~]# 19816:M 15 Mar 14:49:27.669 # IP address for this node updated to 192.168.100.147 19816:M 15 Mar 14:49:32.752 # Cluster state changed: ok
[root@localhost ~]# mrediscli 127.0.0.1:6382> cluster nodes 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 master - 0 1521092983000 5 connected 0-5460 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521092986495 2 connected 5461-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521092985492 3 connected 10923-16383 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382@16382 myself,master - 0 1521092984000 0 connected 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521092982000 3 connected 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 slave 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 0 1521092984000 5 connected 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521092984489 2 connected 127.0.0.1:6382> info keyspace # Keyspace |
위에서 보면 master가 추가 되었는데 Hash Slot은 할당 되지 않았다.
12. Reshard hash slot
새로운 Master를 추가 하거나, 기존의 마스터의 데이터를 새로운 destinatio으로 옮기고자 할 때 사용.
Source는 sharding 작업을 할 노드를 적어 주면 된다. 그러나 destination은 하나의 서버만 지정될 수 있다.
명령어 : redis-trib.rb reshard 작업을할서버:포트
|
작업은 마스터 서버 아무곳에서나 해도 된다. 여기서 새로 추가한 서버4에서 작업을 하게 함.
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb reshard 192.168.100.147:6382 >>> Performing Cluster Check (using node 192.168.100.147:6382) M: 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382 slots: (0 slots) master 0 additional replica(s) S: 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380 slots: (0 slots) slave replicates 3b10f1119a8cb372bd745f0347876282d50bee83 M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381 slots: (0 slots) slave replicates 51ec89eb2062001a11ce1bca6799aa727bc9618a S: 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379 slots: (0 slots) slave replicates 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 4096 <-- 전체 Hash Slot이 16384이므로 4개로 분산하면 4096개씩 가지면 됨. What is the receiving node ID? 02c94678bdefa05b1e35d0ba36b1e524079d9a42 <--- 새로 추가한 서버4에 Hash Slot을 옮겨 올 것이므로 서버4의 서버 아이디 입력 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:all <-- 모든 서버로 부터 받아올 것이므로 all 입력. 특정 서버에서 전체 hash slot을 옮겨 오고 싶으면 해당 서버의 아이디를 입력
Ready to move 4096 slots. Source nodes: M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:5461-10922 (5462 slots) master 1 additional replica(s) Destination node: M: 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382 slots: (0 slots) master 0 additional replica(s) Resharding plan: Moving slot 5461 from 3b10f1119a8cb372bd745f0347876282d50bee83 생략 ... Do you want to proceed with the proposed reshard plan (yes/no)? yes Moving slot 5461 from 192.168.100.161:6380 to 192.168.100.147:6382: 19816:M 15 Mar 15:45:46.839 # New configEpoch set to 6 19816:M 15 Mar 15:45:46.839 # configEpoch updated after importing slot 5461 Moving slot 5461 from 192.168.100.161:6380 to 192.168.100.147:6382: 생략... Moving slot 6826 from 192.168.100.161:6380 to 192.168.100.147:6382: <-- 161 서버로 부터 5462~6826까지의 hash slot을 가져옴 Moving slot 10923 from 192.168.100.204:6381 to 192.168.100.147:6382: 생략... Moving slot 12287 from 192.168.100.204:6381 to 192.168.100.147:6382: <-- 204 서버로 부터 10923~12287까지의 hash slot을 가져옴 Moving slot 0 from 192.168.100.244:6379 to 192.168.100.147:6382: 생략... Moving slot 1364 from 192.168.100.244:6379 to 192.168.100.147:6382: <-- 244 서버로 부터 0~1364까지의 hash slot을 가져옴
[root@localhost ~]# echo "cluster nodes" | redis-cli -c -p 6382 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 master - 0 1521096412000 5 connected 1365-5460 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521096412545 2 connected 6827-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521096415553 3 connected 12288-16383 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382@16382 myself,master - 0 1521096414000 6 connected 0-1364 5461-6826 10923-12287 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521096416555 3 connected 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 slave 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 0 1521096414000 5 connected 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521096414550 2 connected
|
생성 되어 있던 key 분산 확인
|
|
Server1 |
Server2 |
Server3 |
Server4 |
|
Master |
# Keyspace db0:keys=149,expires=0,avg_ttl=0 |
# Keyspace db0:keys=153,expires=0,avg_ttl=0 |
# Keyspace db0:keys=149,expires=0,avg_ttl=0 |
# Keyspace db0:keys=150,expires=0,avg_ttl=0 |
13. Delete Node
추가 되었던 Server4를 삭제해 본다.
Hash slot을 가지고 있으면 삭제가 되지 않으므로, Hash slot을 다른 노드로 옮기는 작업을 한 후에 삭제 해야 한다.
삭제할 노드가 slave 이면 reshard해줄 필요가 없음.
명령어 : redis-trib.rb del-node 작업을할서버:포트 삭제할서버id
|
|
Server ID |
Master |
Slave |
|
Server1 (192.168.100.244) |
05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 |
6379 |
6381 |
|
Server2 (192.168.100.161) |
3b10f1119a8cb372bd745f0347876282d50bee83 |
6380 |
6379 |
|
Server3 (192.168.100.204) |
51ec89eb2062001a11ce1bca6799aa727bc9618a |
6381 |
6380 |
|
Server4 (192.168.100.147) |
02c94678bdefa05b1e35d0ba36b1e524079d9a42 |
6382 |
|
|
서버1에서 작업 수행 : 삭제할 서버에 Hash slot이 있는 상태에서 삭제 시도. [root@localhost ~]# /root/redis-stable/src/redis-trib.rb del-node 192.168.100.244:6379 02c94678bdefa05b1e35d0ba36b1e524079d9a42 >>> Removing node 02c94678bdefa05b1e35d0ba36b1e524079d9a42 from cluster 192.168.100.244:6379 [ERR] Node 192.168.100.147:6382 is not empty! Reshard data away and try again. |
1365개씩 다른 마스터로 hash slot을 다시 돌려 준다.
|
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb reshard 192.168.100.147:6382 >>> Performing Cluster Check (using node 192.168.100.147:6382) M: 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382 slots:0-1364,5461-6826,10923-12287 (4096 slots) master 0 additional replica(s) S: 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380 slots: (0 slots) slave replicates 3b10f1119a8cb372bd745f0347876282d50bee83 M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:1365-5460 (4096 slots) master 1 additional replica(s) M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381 slots: (0 slots) slave replicates 51ec89eb2062001a11ce1bca6799aa727bc9618a S: 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379 slots: (0 slots) slave replicates 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1365 What is the receiving node ID? 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:02c94678bdefa05b1e35d0ba36b1e524079d9a42 Source node #2:done
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb reshard 192.168.100.147:6382 >>> Performing Cluster Check (using node 192.168.100.147:6382) M: 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382 slots:5461-6826,10923-12287 (2731 slots) master 0 additional replica(s) S: 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380 slots: (0 slots) slave replicates 3b10f1119a8cb372bd745f0347876282d50bee83 M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381 slots: (0 slots) slave replicates 51ec89eb2062001a11ce1bca6799aa727bc9618a S: 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379 slots: (0 slots) slave replicates 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1366 What is the receiving node ID? 3b10f1119a8cb372bd745f0347876282d50bee83 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:02c94678bdefa05b1e35d0ba36b1e524079d9a42 Source node #2:done
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb reshard 192.168.100.147:6382 >>> Performing Cluster Check (using node 192.168.100.147:6382) M: 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382 slots:10923-12287 (1365 slots) master 0 additional replica(s) S: 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380 slots: (0 slots) slave replicates 3b10f1119a8cb372bd745f0347876282d50bee83 M: 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381 slots: (0 slots) slave replicates 51ec89eb2062001a11ce1bca6799aa727bc9618a S: 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379 slots: (0 slots) slave replicates 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 M: 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1365 What is the receiving node ID? 51ec89eb2062001a11ce1bca6799aa727bc9618a Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:02c94678bdefa05b1e35d0ba36b1e524079d9a42 Source node #2:done
[root@localhost ~]# echo "cluster nodes" | redis-cli -c -p 6382 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521099486597 8 connected 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 master - 0 1521099485597 7 connected 0-5460 02c94678bdefa05b1e35d0ba36b1e524079d9a42 192.168.100.147:6382@16382 myself,master - 0 1521099483000 6 connected <-- hash slot이 없음. 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521099485000 9 connected 10923-16383 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521099483594 9 connected 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 slave 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 0 1521099483000 7 connected 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521099484596 8 connected 5461-10922
|
삭제 하고자 하는 서버에 Hash slot을 비웠으므로 다시 서버4 삭제
|
[root@localhost ~]# /root/redis-stable/src/redis-trib.rb del-node 192.168.100.244:6379 02c94678bdefa05b1e35d0ba36b1e524079d9a42 >>> Removing node 02c94678bdefa05b1e35d0ba36b1e524079d9a42 from cluster 192.168.100.244:6379 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node.
서버4의 레디스는 자동으로 종료 됨.
[root@localhost ~]# echo "cluster nodes" | redis-cli -c -p 6379 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 192.168.100.244:6379@16379 myself,master - 0 1521099619000 7 connected 0-5460 3b10f1119a8cb372bd745f0347876282d50bee83 192.168.100.161:6380@16380 master - 0 1521099617000 8 connected 5461-10922 51ec89eb2062001a11ce1bca6799aa727bc9618a 192.168.100.204:6381@16381 master - 0 1521099619305 9 connected 10923-16383 7cee3a6801e31d98bdbfc1f7f5c93d072ee1047d 192.168.100.244:6381@16381 slave 51ec89eb2062001a11ce1bca6799aa727bc9618a 0 1521099618302 9 connected 3c3f1adf36d9ed7a4b533776454cbe6afc2a29a4 192.168.100.161:6379@16379 slave 05d5d6f2f55e70b27cae1be2f59eca54dbfbdd81 0 1521099616000 7 connected 02d4c50f18105a30c4304adad0784024f74e6643 192.168.100.204:6380@16380 slave 3b10f1119a8cb372bd745f0347876282d50bee83 0 1521099617298 8 connected |