Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering

Skalbarhet och tillgänglighet med MySQL-replikering

Sven Sandberg ([email protected])Principal Software Engineer, MySQL Replication Core Team(För tillfället föräldraledig)

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. | `2

Safe Harbour Statement

The following is intended to outline our general product direction. It is

intended for information purposes only, and may not be incorporated into any

contract.

It is not a commitment to deliver any material, code, or functionality, and

should not be relied upon in making purchasing decisions. The development,

release, and timing of any features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.


OverviewMySQL Replication Design

Setting Up



Setting Up


Industry Leaders Rely On MySQLReplication Design Setting Up

CloudOEM & ISVsWeb & Enterprise


Replication: Copy Changes Master → Slave

MySQL Master Server

– Changes data

– Sends changes to slave

MySQL Slave Server– Receives changes from master

– Applies received changes to database

M S

Replication Design Setting Up



M M/S S

S

S

S

M

SM

M

Server can be master, slave or both

Master can have multiple slaves

Slave can only have one master




M/S

SM

M

Circular replication is also possible

You can simulate multiplemasters by time-sharing

M/S

M/S

M/S

M/S

M/S

SM

MSwitch

every 5 sec



Why Replication? – Performance

Read scale-out

M S

write clients read clients




Read scale-out

M S

write clients read clients

Morereads?More

slaves!




Read scale-out

M SS

S

S

M

write clients read clientsread clients

write clients

Morereads?More

slaves!



Why Replication? – Redundancy

If master crashes, promote slave to master

C

B

A





C

B

A


Crash




C

B

A

B is thenew master



Why Replication? – Long-distance Data Distribution

CB

BAAC

Image fromwww.ginkgomaps.com




Cool!So, give me a 7 minintro to replication.



Setting Up


All Changes Written to Binary Log

binary log

Client




binary log

Clientcreate table t (a int);




create...

binary log

Client

Table t

create table t (a int);




create...

binary log

Client

Table t

create table t (a int);insert into t values (1);




create...insert...

binary log

Client

Table t1

create table t (a int);insert into t values (1);



Slave Initiates Replication

B

binary log

A

binary log

Client


1. Slave sendsrequest to start replication

to master

2. Master sendsstream of replication data

to slave


create...

Binary Log Sent to Slave, Re-executedReplication Design Setting Up

B

binary log

A

binary log

Client


create...

Binary Log Sent to Slave, Re-executed

Table t


B

binary log

create...

A

binary log

Client


create...


Table t Table t


create...

B

binary log

create...

A

binary log

Client


create...insert...


Table t Table t


create...

B

binary log

create...

A

binary log

Client


create...insert...


Table t1

Table t


create...

B

binary log

create...insert...A

binary log

Client


create...insert...


Table t1

Table t1


create...insert...B

binary log

create...insert...A

binary log

Client


Actually, Changes Land in Slave's Relay Log

B

binary logrelay log

A

binary log

ClientDumpthread

IOthread

SQLthread


Network


create...


B

binary logrelay log

A

binary log

ClientDumpthread

IOthread

SQLthread


Network


create...


B

binary logrelay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t


Network


create...


B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t


Network


create...


create...

B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t Table t


Network


create...insert...


create...

B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t Table t


Network


create...insert...


create...

B

binary log

create...

relay log

create...insert...A

binary log

ClientDumpthread

IOthread

SQLthread

Table t1

Table t


Network


create...insert...


create...

B

binary log

create...insert...

relay log

create...insert...A

binary log

ClientDumpthread

IOthread

SQLthread

Table t1

Table t


Network


create...insert...


create...insert...B

binary log

create...insert...

relay log

create...insert...A

binary log

ClientDumpthread

IOthread

SQLthread

Table t1

Table t1


Network


create...

Replication is Asynchronous

create...

B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t Table tAtomic → db and logconsistent after crashAtomic → db and logconsistent after crash

Asynchronous Asynchronous Asynchronous

(Atomictoo)

(Atomictoo)


Network



Replicationseems fantastic!

I want to try!



Setting Up


Setting Up Replication

Master's my.cnf:

serverid=1logbin=masterbin

Slave's my.cnf:

serverid=2logbin=slavebinrelaylog=slaverelaybinlogslaveupdates



Setting Up Replication

On master: create replication user

> CREATE USER `rpl_user`@`slave_host`;> GRANT REPLICATION SLAVE ON *.* TO `rpl_user`@`slave_host` IDENTIFIED BY 'secret';> FLUSH PRIVILEGES;

On slave: initiate replication

> CHANGE MASTER TO MASTER_HOST = 'master_host', MASTER_PORT = <port>, MASTER_USER = 'rpl_user', MASTER_PASSWORD = 'secret';> START SLAVE;



Replicate!

On master: execute some SQL

> USE test_db;> CREATE TABLE t1 (a INT);> INSERT INTO t1 VALUES (1);

On slave: check that it replicated:

> SELECT * FROM test_db.t1;

a

1



Check Replication Status

On master:

> SHOW BINARY LOGS;

> SHOW BINLOG EVENTS;

Log_name File_size

master-bin.000001 426

Log_name Pos Event_type sid end_pos Info

master-bin.000001 4 Format_desc 1 120 Server ver: 5.6.6-m9-log, Binlog ver: 4

master-bin.000001 120 Query 1 217 use `test`; CREATE TABLE t1 (a INT)

master-bin.000001 217 Query 1 296 BEGIN

master-bin.000001 296 Query 1 395 use `test`; INSERT INTO t1 VALUES (1)

master-bin.000001 395 Xid 1 426 COMMIT /* xid=21 */

One binary logso far

Five events:One CREATE and

one INSERT



Check Replication Status

On slave:

> SHOW SLAVE STATUS\G…

Slave_IO_Running Yes

Slave_SQL_Running Yes

…

Last_IO_Errno 0

Last_IO_Error

Last_SQL_Errno 0

Last_SQL_Error

…

Both slave threadsare running

If any thread stopswith an error,look for error

message here




Next? Vote!1. Fail-over + Crash tolerance

2. Fail-over + Q&A3. Q&A


Overview

MySQL Replication

Crash-Tolerance Failover

Reduce Risk for Data Loss

– Semi-sync– Transactional meta-data

Performance– Transactional meta-data– Multi-threaded slave


Crash-Tolerance

Can happen to anyone:– Hardware failure

– Human mistake

– Bug

– Natural disaster

Features in MySQL 5.6:– Reduce admin overhead

– Prevent unplanned downtime

– Prevent planned downtime

Crash-Tolerance Fail-over Reduce Risk for Data Loss Performance


Crash-Tolerance

Fail-over: “back on track” faster & more automatically – Global Transaction Identifiers

Reduce risk for data loss: less administration → less downtime– Semi-synchronous replication → slave has all changes from master

– Update replication metadata transactionally → safe on disk

Performance: help slave keep up with master– Replication metadata in table – leverage disk sync from storage engine

– Multi-threaded slave

5.5

5.6




OK.Tell me about

fail-over!


Overview

MySQL Replication






Fail-over

Handle server crash with minimal disruption

Example 1: tree

C

B

A

Crash-tolerance Fail-over Reduce Risk for Data Loss Performance


Fail-over


Example 1: tree

C

B

A


Crash


Fail-over

C

B

A


Example 1: tree



Fail-over

C

B

A

Make B the new master


Example 1: tree



Fail-over

CA B


Example 1: tree

Example 2: line



Fail-over

CA B


Example 1: tree

Example 2: line


Crash


Fail-over

CA B


Example 1: tree

Example 2: line



Fail-over

CA B

Make A direct master of C


Example 1: tree

Example 2: line



Fail-over

C

A

B

D


Example 1: tree

Example 2: line

Example 3: circle



Fail-over

C

A

B

D


Example 1: tree

Example 2: line

Example 3: circle


Crash


Fail-over

C

A

B

D


Example 1: tree

Example 2: line

Example 3: circle



Fail-over

C

A

Make a shortcutin the circle

B

D


Example 1: tree

Example 2: line

Example 3: circle



Fail-over

C

A

B

D


Example 1: tree

Example 2: line

Example 3: circle

Example 4: scheduled maintenance



Fail-over

C

A

B

D

Takeout D!


Example 1: tree

Example 2: line

Example 3: circle




Fail-over

C

A

B

D

MaintainD offline,without

disruptingservice


Example 1: tree

Example 2: line

Example 3: circle




Fail-over

C

A

B

D


Example 1: tree

Example 2: line

Example 3: circle




Fail-over


Example 1: tree

Example 2: line

Example 3: circle


Example 5: arbitrary topology


A

B

C

D

E

F


Fail-over: Design

Server assigns Global Transaction Identifier (GTID) to every transaction:

– server_uuid:numbera61678ba488942799e5845ba840af334:1

– server_uuid identifies the server; globally unique

– number is incremented by 1 for each transaction on this server

Writes GTID to binary log Slave preserves GTID when re-executing transaction



Fail-over: Design: New Events In Binary Log

> SHOW BINLOG EVENTS;

Log_name Pos Event_type sid end_pos Info

master-bin.000001 4 Format_desc 1 120 Server ver: 5.6.12-m9-log, Binlog ver: 4

master-bin.000001 120 Prev_gtids 1 151

master-bin.000001 151 Gtid 1 199 SET @@GTID_NEXT= '<uuid>:1'

master-bin.000001 199 Query 1 296 use `test`; CREATE TABLE t1 (a INT)

master-bin.000001 296 Gtid 1 344 SET @@GTID_NEXT= '<uuid>:2'

master-bin.000001 344 Query 1 423 BEGIN

master-bin.000001 423 Query 1 522 use `test`; INSERT INTO t1 VALUES (1)

master-bin.000001 522 Xid 1 602 COMMIT /* xid=21 */

3 new events:1 Previous_gtids +

2 Gtid


fileheader

trans-action

trans-action


Fail-over: Design: Protocol

Master-slave protocol:– Slave sends to master:

range of identifiers of executed transactions to master

– Master sends all other transactions to slave

(slave)

id1,trx1,id2,trx2

(master)

id1,trx1,id2,trx2,id3,trx3

binlog

A 2. id3, trx3, …

1. id1…id2 binlog

B




Example 1: tree

A

(crashed)(master)

Aid1,trx1,id2,trx2,id3,trx3

binlog

(slave)

Cid1,trx1

binlog(slave)

id1,trx1,id2,trx2 B

binlog




Example 1: tree

A

(crashed)(master)


binlog

(slave)

Cid1,trx1

binlog(slave)

id1,trx1,id2,trx2 B

binlog


Crash



Example 1: tree

(slave)

Cid1,trx1

binlog(slave)

id1,trx1,id2,trx2 B

binlog

A

(crashed)(crashed)


binlog id1

id2, trx2,...




Example 1: tree Example 2: circle

B

binlog

A

binlog

C

binlog

client client





id1,trx1 B

binlog

A

binlog

C

binlog

client client trx1





id1,trx1,id2,trx2 B

binlog

id2,trx2

A

binlog

C

binlog

client clienttrx2 trx1






B

binlog

id2,trx2

A

binlog

C

binlog

client clienttrx2 trx1, trx3






B

binlog

id2,trx2

A

binlog

C

binlog



Crash





B

binlog

id2,trx2

A

binlog

C

binlog


id2

id1,trx1,id3,trx3,...




Wow!Show me how to

configure forfail-over!


Fail-over: Set Up

Prerequisites:– Configure replication as usual

– Use transactional storage engine for all tables (InnoDB)

– Don't use CREATE TABLE … SELECT



Fail-over: Set Up

Prepare the server for fail-over

1. Sync and stop all servers

2. Add to every my.cnf:gtidmode=onenforcegtidconsistency=onlogbinlogslaveupdates

3. Start all servers

4. Create replication user on every server that may become master

5. Execute:

> CHANGE MASTER TO MASTER_AUTO_POSITION = 1



Fail-over: Perform

Perform fail-over

On each slave:> CHANGE MASTER TO MASTER_HOST = '<host>', MASTER_PORT = <port number>, MASTER_USER = '<user name>' MASTER_PASSWORD = 'secret';


Credentials onnew master


Fail-over: Perform

Perform fail-over

C

B

ACHANGE MASTER TOMASTER_HOST = 'B',MASTER_PORT = <B's port>,MASTER_USER = '<user name>'MASTER_PASSWORD = 'secret';



Fail-over: Perform

Perform fail-over

CHANGE MASTER TOMASTER_HOST = 'C',MASTER_PORT = <C's port>,MASTER_USER = '<user name>'MASTER_PASSWORD = 'secret';

C

A

B

D




I love GTIDs!Can fail-over be

automated?


Fail-over: Utilities

Mysqlrpladmin– Promote one slave to master– mysqlrpladmin switchover master='user[:pass]@host[:port][:sock]' discoverslaveslogin='user[:pass]' newmaster='user[:pass]@host[:port][:sock]'

– Many other admin tasks



Fail-over: Utilities

Mysqlfailover– Monitor master– If master goes away, do automatic fail-over– mysqlfailover master='user[:password]@host[:port[:socket]]' discoverslaveslogin='user:password'



Fail-over: Utilities Video tutorial

– dev.mysql.com/techresources/articles/mysqlreplicationutilities.html

Blog– drcharlesbell.blogspot.co.uk/2012/04/mysqlutilitiesandglobaltransaction.html

Doc– dev.mysql.com/doc/workbench/en/mysqlfailover.html– dev.mysql.com/doc/workbench/en/mysqlrpladmin.htm

Download– www.mysql.com/downloads/workbench/




Now I cando fail-over!

But how do I avoidlosing data after

a crash?


Overview

MySQL Replication






Overview - less manual work- reduce downtime

MySQL Replication






Overview - less manual work- reduce downtime

MySQL Replication





a.k.a. “slave has allchanges from master”


Asynchronous vs Semi-sync Replication

By default, replication is asynchronous

– Master does not wait for slave

– Fast

– ACK to app before slave receives change → changes lost if master dies

New in MySQL 5.5: semi-synchronous replication plugin– Master acks to app

only after changes are in slave's relay log

– Master has to wait

– Less risk for lost updates



create...


create...

B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t Table t


Network


create...


create...

B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t Table tAtomic → db and logconsistent after crashAtomic → db and logconsistent after crash

Asynchronous Asynchronous Asynchronous

(Atomictoo)

(Atomictoo)


Network


create...

Semi-synchronous Replication

create...

B

binary log

create...

relay log

create...

A

binary log

ClientDumpthread

IOthread

SQLthread

Table t Table t

Asynchronous

Network

Semi-synchronous replication:statement is in slave's relay log

before client ack



Semi-synchronous Replication: Set Up

Consists of two plugins: master plugin, slave plugin

Installation: read the manual :-)– http://dev.mysql.com/doc/refman/5.6/en/replicationsemisyncinstallation.html

After installation:– On master:SET GLOBAL rpl_semi_sync_master_enabled = {0|1};

– On slave:SET GLOBAL rpl_semi_sync_slave_enabled = {0|1};




So semi-syncensures slave has all

changes before my appgets notified.

Anything else that helpsconsistency after

a crash?


Overview

Meta-datasafe on disk

MySQL Replication






Replication Metadata in Tables

Slave threads maintain their current positions– IO thread: where to read from master

– SQL thread: where to read from relay log

Positions stored on disk– Before: in file

– 5.6: in table


insert...

B

binary log

insert...

relay log

IOthread

SQLthread


Replication Metadata in Tables

File: updated aftertransaction commit


Trans-action

Updateposition

Trans-action

Updateposition

Crashpossible

Table: updated atomically attransaction commit

Consistent after server crash


Replication Metadata in Tables: Set Up

Only meaningful if using transactional storage engine

Add to slave's my.cnf:

masterinforepository=TABLEslaveinforepository=TABLEdefaultstorageengine=InnoDB




Nice!Also, won't slave be more

in sync if we improveslave performance?


Overview

MySQL Replication





Slave less behind→ less lost updates,short fail-over time


Overview

MySQL Replication





Slave less behind→ less lost updates,short fail-over time

Fewer disk syncs


Performance: Replication Metadata In TablesCrash-tolerance Fail-over Reduce Risk for Data Loss Performance

File: separate disk syncfor the file

Trans-action

Updateposition

Trans-action

Updateposition

Sync

Table: leverage sync from storage engine

(even better: group commit)

Again, use transactional engine

Sync Sync



Ahh, sotransactional meta-data

is double-good!How else can I improve

slave performance?


Overview

MySQL Replication






Performance: Parallel Slave Threads

B

binary logrelay log

A

binary log

Client

Client

Parallel writerthreads on master

Writes are serializedin binary log

Single applieron slave →

Can becomebottleneck

Client




B

binary logrelay log

A

binary log

Client

ClientClient

Coordinatorthread (reading)

Multiple workerthreads! (executing)




Parallelize by schema (database)

– Different worker threads never update same schema

– Effective if you partition data by schema

Consistency– Consistent within schema

– Eventually consistent between schemas

– Commit order may differ between master and slave, among slaves





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1

relay log

3 workerthreadsCoordinator

thread

w1w2w3





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1


thread

DB1w1w2w3

relay log





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1


thread

DB1

DB2

w1w2w3

relay log





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1


thread

DB1

DB2

w1w2w3

DB3

relay log





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1


thread

DB1

DB2

w1w2w3

DB3

Coordinator waits for w1 to finishbefore feeding DB1 update to w3

relay log





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1


thread

DB2

w1w2w3

DB3


relay log





B

Trx1: DB1Trx2: DB2

Trx3: DB3+DB1


thread

DB2

w1w2w3

DB3


DB1

relay log



Performance: Parallel Slave Threads: Set Up

On slave:

> STOP SLAVE;> SET SLAVE_PARALLEL_WORKERS = 4;> START SLAVE;

Depends on number of CPUsand number of databases.

No exact formula –measure what's best for you!




Thank you,that's a lot of info!

Can yousummarize?


SummaryMySQL Replication






Semi-sync

GlobalTransactionIdentifiers

ReplicationBase

TransactionalMetadata

Parallel slave

My.cnf:serverid=<different number for each server>logbin=<host>binrelaylog=<host>relaybinlogslaveupdatesmasterinforepository=TABLEslaveinforepository=TABLEdefaultstorageengine=InnoDBgtidmode=onenforcegtidconsistency=on

Client commands:> CHANGE MASTER TO MASTER_AUTO_POSITION = 1;> SET SLAVE_PARALLEL_WORKERS = 4;> SET GLOBAL rpl_semi_sync_master_enabled = {0|1};> SET GLOBAL rpl_semi_sync_slave_enabled = {0|1};

Summary



Thank you!Questions?

Email: [email protected] Blog: svenmysql.blogspot.co.uk

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. | 127

Technology

Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering