Redis acc

REDIS Management & Cell Architecture

charsyam@naver.com

Redis.io

Redis/Twemproxy Contributor

카카오 홈 개발자

Agenda REDIS Management REDIS HA CELL Architecture

Agenda REDIS Management REDIS HA CELL Architecture

Redis Single Threaded In Memory

Persistent Replication

Collections

Redis is Single Threaded!

Single Thread Means!

Don’t execute long task.

Example: Keys command Flushall/flushdb command

O(n)

Keys *

di = dictGetSafeIterator(c->db->dict); allkeys = (pattern[0] == '*' && pattern[1] == '\0'); while((de = dictNext(di)) != NULL) { …… stringmatchlen(pattern,plen,key,sdslen(key),0) }

Memcache’s flush is faster than Redis’s flush.

Memcache’s flush is faster than

Redis’s flush?

FlushAll-Redis for (i = 0; i < ht->size && ht->used > 0; i++) { dictEntry *he, *nextHe; if ((he = ht->table[i]) == NULL) continue; while(he) { nextHe = he->next; dictFreeKey(d, he); dictFreeVal(d, he); zfree(he); ht->used--; he = nextHe; } }

FlushAll-Memcache

if (exptime > 0) settings.oldest_live = realtime(exptime) - 1; else /* exptime == 0 */ settings.oldest_live = current_time - 1;

FlushAll-Memcache if (settings.oldest_live != 0 && settings.oldest_live <= current_time && it->time <= settings.oldest_live) { do_item_unlink(it, hv); do_item_remove(it); it = NULL; }

FlushAll

Cache Item Count Time

Memcache 1,000,000 1~2ms

Redis 1,000,000 1000ms(1 second)

Redis Single Threaded In Memory

Persistent Replication

Collections

In Memory Fast but can’t store big data. 32bit: Maximum 3G Data in Memory 64bit: No Max memory. using swap memory in OS.

Redis Single Threaded In Memory

Persistent Replication

Collections

Collections Memcached supports just K-V List Set Sorted Set Hash

Don’t insert too many items into one collections.

Delete collections

Item Count Time

list 1,000,000 1000ms(1 second)

set

Sorted set

hash

Redis Single Threaded In Memory

Persistent Replication

Collections

Redis can make its memory snapshot.

RDB is not Relation DBMS.

RDB is redis snapshot name.

Fork()

RDB - BAD • Bad Performance in Large memory.

• Twice memory problem.

• Denying write problem when storing RDB fails.

• If you just want Cache. Turn off RDB

RDB – Large Memory • Performance is relevant to Memory

Size.

17.1 GB 34.2 GB

68.4 GB

117 GB

RDB – Twice Memory • fork() and COW(copy on write) Issue

–In Write Heavy System:

RDB – Twice Memory

RDB – Twice Memory

RDB – Twice Memory

Real Case Study • Background

–Can’t write to Redis Server

–Sentinel doesn’t find Server’s failure.

Real Case Study • Reason –If redis fails to save RDB, Redis basically

denies write operations from client.

–“MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”

Real Case Study • Reason if (server.stop_writes_on_bgsave_err &&

server.saveparamslen > 0 && server.lastbgsave_status == REDIS_ERR && c->cmd->flags & REDIS_CMD_WRITE) { flagTransaction(c); addReply(c, shared.bgsaveerr); return REDIS_OK; }

Real Case Study • Solution #1

• Solution #2

config set stop-writes-on-bgsave-error no

Turn off RDB Setting

AOF

AOF is append only file.

AOF memorizes all requests.

AOF *3\r\n$3\r\nset\r\n$1\r\nA\r\n$3\r\n123\r\n*3\r\n$3\r\nset\r\n$1\r\nB\r\n$3\r\n123\r\n*3\r\n$3\r\nset\r\n$1\r\nC\r\n$3\r\n123\r\n

AOF *3 $3 Set $1 A ……

AOF Rewrite When size of AOF grows than redis rewrite memory state to AOF.

Redis Single Threaded In Memory

Persistent Replication

Collections

Redis support master/slave replication

Replication •Support Chained Replication

Master 1st Slave 2nd Slave

1st slave is master of 2nd slave

Replication

Master Slave replicationCron

Health check

Replication

Master Slave replicationCron

Health check

Replication

Master Slave replicationCron

When master reruns, Resync with Master

Mistake: Replication

Master Slave replicationCron

Slave will has no data after resyncing

If master has no data.

Replication Don’t forget “slave of no one”

When Redis forks… 1. BGSAVE(RDB) 2. AOF Rewrite(AOF) 3. Replication(RDB) even though, you turned it off

Agenda REDIS Management REDIS HA CELL Architecture

Redis Sentinel

What Redis Sentinel

do?

1. Check Redis Liveness

1. Check Redis Liveness -Subjective Down -Objective Down

2. Choose good slave.

Good Slave NOT SDOWN, ODOWN, DISCONNECTED

NOT DEMOTE

Ping reply > info_validity_time

Slave_priority != 0

Info reply > info_validity_time

Choose new master

Sort Slave Priority

Runid

3. Promote it

Promoting Send Slaveof no one to new master

Send Slaveof [new master ip] [addr] to Other redis

Notify new master to clietns +switch-master

Set DEMOTE mark to old-Master

Is Sentinel Really Good?

Not Mature #1

Sentinel Conf port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 2000 sentinel can-failover mymaster yes sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 900000

Quorum Count sentinel monitor mymaster 127.0.0.1 6379 2

We should check this count.

Sentinel Conf sentinel down-after-milliseconds mymaster 2000

If this value is Too small, It can cause some trouble.

-sdown/+sdown loop

Compare with Zookeeper Conf tickTime=2000 dataDir=/var/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

Not Mature #2

If redis is down, when redis is loading data(RDB,

AOF), sentinel can’t register redis.

Even though, Sentinel is not mature. It is useful.

Redis Failover

Twemproxy

Twemproxy Redis

Redis

Redis

Client

Client

Client

Client

Client

Client

hashing

Twemproxy Pros

One connection per server Zero Copy Multi Hashing Algorithm Support Nickname

Twemproxy Cons

Not Support Full Command of Redis(Pub/Sub or MULTI)

Not Support HA

Twemproxy

Redis

Redis

Redis

Twemproxy

Twemproxy

L4

Redis-Sentinel TwemProxy Agent

http://www.codeproject.com/Articles/656965/Redis-Sentinel-TwemProxy-Agent

HA Proxy

HA Proxy

VRRP Twemproxy

Twemproxy

Twemproxy

Redis Master

Redis Slave

Redis Slave

Shard

Redis Master

Redis Slave

Redis Slave

Shard Sentinel

Agent Pub/Sub

Restart

Sharding

Sharding

WEB/AS

Master Master Master User A Data User C Data User D Data

User B Data User X Data User Z Data

User Y Data User E Data User F Data

Cell Architecture

Cell 0 Cell 1 Cell 2

User Cell Info Server

ID: CharSyam

CellID: 1, Status: Normal

Get/set

Cell Architecture

A Cell is Full-Set Can serve Users

Cell Examples #1

WEB/AS

Master Slave

WEB/AS WEB/AS

Cell Examples #2

WEB/AS WEB/AS WEB/AS

Master

Slave Slave Slave

READ only WRITE only

Failure of Cell Architecture

Cell 0 Cell 1 Cell 2

User Cell Info Server

ID: CharSyam

CellID: 1, Status: Normal

Can’t Service

Failure of Cell Architecture

Cell 0 Cell 1 Cell 2

User Cell Info Server

ID: CharSyam

Can’t response

Get/set

• Benefits

–Easy to extend

–Failure is limited to some Users in same cell.

–Can deploy specific feature to some cell users.

• Liabilities

–To need more servers.

• To build full-set

Cell Architecture

Q&A

Thank you