My sql fabric webinar tw2

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1

MySQL Fabric:Easy Management of MySQL Servers

杜修文 [email protected] Principal Sales ConsultantOracle LLC

July 17, 2014

2

Insert Picture Here

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |

An extensible and easy-to-use framework for managing a farm of MySQL server supporting high-availability and sharding

MySQL Fabric

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |3

MySQL Fabric: Goals & Features

Connector API Extensions Support Transactions Support full SQL

Decision logic in connector Reducing network load

Load Balancing Read-Write Split Distribute transactions

Global Updates Global tables Schema updates

Shard Multiple Tables Using same key

Sharding Functions Range (Consistent) Hash

Shard Operations Using built-in executor Shard move Shard split


Transaction Handling


Routing Transactions Pre-declare properties of transactions Detecting transaction boundaries Push as much as possible to server

Managing Session State Move session state between servers

− Easy to use− Expensive and error prone

Reset state after each transaction− Transactions start with default session state

Transaction Handling


Routing Transactions

Shard #2

Shard #1

Shard #3

Switch

StateStore Executor

QUERYKEY

KEY

QUERY

Contain decision logicfor routing queries

Contain informationabout location of shards

SHARD#


Birds-eye View

High Availability Groups (Shards)

MySQL FabricNode

Application

XML-RPC

SQL

Connector

Connector

Connector

Operator


MySQL Fabric: Prerequisites

MySQL Servers (version 5.6.10 or later) Server for meta-data backing store Servers being managed

Python 2.6 or 2.7 No support for 3.x yet

MySQL Utilities 1.4.0 Available at http://labs.mysql.com/


MySQL Fabric: Configuration

Backing Store MySQL server Persistent storage for state Storage engine-agnostic

Protocol Address where node will be Currently only XML-RPC

Logging Chatty: INFO (default) Moderate: WARNING URL for rotating log

[storage]address = localhost:3306user = fabricpassword =database = fabricconnection_timeout = 6

[protocol.xmlrpc]address = localhost:8080threads = 5

[logging]level = INFOurl = file:///var/log/fabric.log

10

Insert Picture Here


Connecting to aMySQL Fabric Farm


Fabric-aware Connector API



Load Balancing Read-Write Split Distribute transactions

Fabric-aware Connectors Connector/J Connector/Python Connector/PHP

Fabric-aware Frameworks Doctrine Hibernate

Focus on Connector/Python

12

Insert Picture Here


Architecture forSharding


Benefits of Sharding

Write scalability Can handle more writes

Large data set Database too large Does not fit on single server

Improved performance Smaller index size Smaller working set Improve performance

UID 10000-20000 UID 20001-40000


MySQL Fabric: Sharding Goals & Features



Shard Multiple Tables Using same key

Global Updates Global tables Schema updates

Sharding Functions Range (Consistent) Hash

Shard Operations Using built-in executor Shard move Shard split


Mapping the Sharding Key

What is a sharding key? Single column Multi column

− Same table?− Different tables?

How is the key transformed? Hash Range User-defined

ComputeShard#

Key

Shard#

(X)(X,Y,...)

RANGEHASHSomething else


Sharded Tables

In desperate needof sharding!

Foreign keys


Multi-table Query with Sharded Tables

SELECT first_name, last_name, salaryFROM salaries JOIN employees USING (emp_no)WHERE emp_no = 21012 AND CURRENT_DATE BETWEEN from_date AND to_date;

Referential Integrity Constraint Example query joining salaries and employees Same key, same shard: co-locate rows for same user

JOIN normally based on equality Using non-equality defeats purpose of foreign key


Global Tables

Does not needto be sharded


Multi-table Query with Global Tables

SELECT first_name, last_name, GROUP_CONCAT(dept_name) FROM employees JOIN dept_emp USING (emp_no) JOIN departments USING (dept_no)WHERE emp_no = 21012 GROUP BY emp_no;

JOIN with departments table Has no employee number, hence no sharding key Table need to be present on all shards

How do we update global tables?


Sharding Architecture

Shards

MySQL Fabric Node

Application

GlobalGroup

Global Updates

ShardUpdates

Replication

Support global updatefor off-line shards


Demo


Demo Server Side – Startup Instances

3306

3606

FabricRepository

3506

Fabric Node

32274

3308 3314 3408 3406


Demo Server Side – Configuration of Fabric

/etc/mysql/fabric.cfg[DEFAULT]prefix = sysconfdir = /etclogdir = /var/log[storage]address = localhost:3506user = fabricpassword = welcome1database = fabricauth_plugin = mysql_native_passwordconnection_timeout = 6connection_attempts = 6connection_delay = 1

[servers]user = fabricpassword = welcome1[protocol.xmlrpc]address = localhost:32274threads = 5user = adminpassword =welcome1 disable_authentication = norealm = MySQL Fabricssl_ca = ssl_cert = ssl_key = [executor]executors = 5[logging]level = INFOurl = file:///var/log/fabric.log


Startup Fabric

•Fabric help command mysqlfabric help commands•Instantiate Fabricmysqlfabric manage setup•Bring up Fabricmysqlfabric manage start --deamonize


Demo – Configure by Fabric

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314 3408 3406


Fabric Group Operation

Create 1 global group, 2 shard groupmysqlfabric group create gp-globalmysqlfabric group create gp1mysqlfabric group create gp2

Assign Servers to groupsmysqlfabric group add gp-global 127.0.0.1:3606mysqlfabric group add gp1 127.0.0.1:3306mysqlfabric group add gp1 127.0.0.1:3308mysqlfabric group add gp1 127.0.0.1:3314mysqlfabric group add gp2 127.0.0.1:3406mysqlfabric group add gp2 127.0.0.1:3408

Pick the master server by Fabricmysqlfabric group promote gp1mysqlfabric group promote gp2


Fabric Group Operations

• Let Fabric manage auto-failover for youmysqlfabric group activate gp-globalmysqlfabric group activate gp1mysqlfabric group activate gp2

• Define Range shard on to global groupmysqlfabric sharding create_definition RANGE gp-global

• Add shard table – employees to the shard created (shard map id = 1) shard by id columnmysqlfabric sharding add_table 1 employees.employees id

• Assign rows with id value 1~999 to group gp1, 1000 above to group gp2mysqlfabric sharding add_shard 1 gp1/1,gp2/1000 --state=ENABLED


Demo – Databases is Ready

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406


Demo Application Side– Need Connector/J 5.1.30 or later

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406

JavaConnector/J


Demo Application Side– Cache Fabric Configuration on creating DB Connection

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406

JavaConnector/JConnection

Object


Java Application- Global tables and DDLs are accessed by property fabricServerGroup

import com.mysql.fabric.jdbc.FabricMySQLConnection;…rawConnection = DriverManager.getConnection( baseUrl + database + "?fabricServerGroup=gp-global“,user, password);statement = rawConnection.createStatement();…statement.executeUpdate("create table employees.employees (emp_no int not null,first_name varchar(50),last_name varchar(50), department_id int,primary key (emp_no))");…statement.executeUpdate("insert into employees.departments values (1,'Sales'),(2,'Financial')");


Demo Environment – Dispatch Request Based On Cached Configuration with Connection

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406


Object – Shard Table

Write Sharded Table

Write Global Table



GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406



Write Sharded Table

Write Global Table



GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406



Write Sharded Table

Write Global Table


Shared tables are accessed by property fabricShardTable and fabericShardKey

rawConnection = DriverManager.getConnection( "jdbc:mysql:fabric://127.0.0.1:32274/mysql?fabricShardTable=employees.employees“, user, password);

…for (int i = 0; i < 4; ++i) { connection.setShardKey(ids[i].toString()); ps.setInt(1, ids[i]); ps.setString(2, firstNames[i]); ps.setString(3, lastNames[i]); ps.setString(4, departmentIds[i]); ps.executeUpdate();}


Demo Environment – Read/Write Split within Shard

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406



Read Only Operations

Read Global Table


Read Only Operations

FabricMySQLConnection newConn = (FabricMySQLConnection) rawConnection;

newConn.setReadOnly(true);ps = newConn.prepareStatement( "select emp_no, first_name, last_name, d.name, i.variable_value from employees.employees e, information_schema.global_variables i, employees.departments d where e.department_id = d.department_id and i.variable_name='port' and emp_no = ?");


Demo Environment – Failure Detected

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308 3314-M 3408-M 3406


Object

Write Global Table

Change Master to 3606

Change Master to 3308

X


Demo Environment – Failover within Shard

GP1

3306

GP-Global

3606

GP2

FabricRepository

3506

Fabric Node

32274

3308-M 3314-M 3408-M 3406


Object

Write Shard Table

Write Global Table

X

40

Insert Picture Here


Closing Remarks


Thoughts for the Future

Connector multi-cast Scatter-gather UNION of result sets More complex operations?

Internal interfaces Improve extension support Improve procedures

support

Command-line interface Improving usability Focus on ease-of-use

More protocols MySQL-RPC Protocol? AMQP?

More frameworks?

More HA group types DRBD MySQL Cluster

Fabric-unaware connectors?

Documents

My sql fabric webinar tw2