美团点评技术沙龙08 - 分布式监控系统实践

CAT (Central Application

Tracking)

1

•

• 2010

• slb

2

• CAT

• CAT

•

3

CAT

• CAT(Central Application Tracking) Java

• CAT

4

• 1

• 2

•

5

CAT Logview•

• ID IP ID •

6

Logview

7

Logview

8

APMTransac'on

EventProblem Hearbeat JVM Memory Thread Matrix Cross SOA RPC Cache

Dependency … …

9

• CAT

• CAT

•

10

CAT

•

•

•

11

A

B

C

TCP

Logview Report

Consumers Consoles

12

HttpThread

Call

SQL

Cach

......

ThreadContext

Call

SQL

Cache

.....

Sender

threads

Message Queue

SenderSender

ServerServer

Server

13

•

•

• CPU

• 2%

•

• netty

14

-IO• java message tree id java

MappedByteBuffer

•

• cpu io

15

-Memory

• MessageTreemessageTree

•

16

Receiver

threads

File A

B

C

Receiver

Receiver

threads

Analyzer

Analyzer

Analyzer

17

Analyzer

Receiver

Receiver

ReceiverAnalyzer2

Analyzer3

Analyzer1

Analyzer1domain hash

18

•

•

• CPU

•

•

•

19

•

•

20

url/sql

Exception

Cpu,IO

Transaction

Event

Heartbeat

Metric

21

KeyValue

•

•

•

22

• Transaction

• Event

• Problem

• Heartbeat

• ……

23

•

•

•

• XML, JSON, Binary...

•

24

25

26

cpu

27

•

•

28

• ID ShopWeb-0a010680-375030-2

•

• /2012/10/13/14/ShopSerivce-ShopWeb-10.1.6.1

• /2012/10/13/14/ShopService-ShopWeb-10.1.6.2

• 375030 => 2012-10-13 14:00:00

• ShopService => domain

• 10.1.6.1/2 => IP

• 0a010680 => 10.1.6.128 ID

29

• 1 transaction report event report name

• 2 swap off

• 3 numactl --interleave=all

• 4 jvm

30

• CAT

• CAT

•

31

CAT• 2011-11

• 2012-3 MVP

• 2012-6

• 2012-12 150+ 500+

• 2013-12 400+ 1500+

• 2014-12 800+ 3000+

• 2015-9 1500+ 7000+

• 2016-6 2600+ 12000+

32

MVP

• Demo 1

• MVP 3

•

33

•

• vip

34

•

•

• ***

35

•

•

• mat

36

code

•

•

•

37

•

• sql rpc web

•

38

• jetty server

• hdfs

• mysql

39

•

• 2-3

•

•

40

• scope

•

•

41

Open Source• http://github.com/dianping/cat

42

• https://github.com/dianping/cat/issues/753

43

QA

• Thanks

44