ç¾›¢€œ¯²™é¾™03 - ®‍—¶•°®»““解†³–¹ˆ

  • View
    1.168

  • Download
    7

Embed Size (px)

Text of ç¾›¢€œ¯²™é¾™03 -...

  • - judasheng@meituan.com

    2015-11-03

  • RtDW

  • RtDW

  • ROI

  • RtDW

  • ApplicationDB

    Flume Agent

    Flume Collector

    Canal

    Kafka

    Hadoop

    Nginx

    Flume Agent

    App

  • Flume Canal Binlog

    Kafka

    org binloglog app

    Storm

    LogParserTopology

    AppTopology

    Batch HBase

    State Redis

    Dim RedisStorm

  • Input/Dim/State/Output

  • RtDW

  • , ,

  • 1

    ,,

    Bolt_Join_Dimcity_id, money

    Bolt_Aggrcity_id: SUM(money)

    Statecity_id_20151125: xxx

  • ODS

    Fact Dim

    ETL

    ETL

    Summary

    ODS

    Dim

    Fact+Dim

    State

    DB:SummaryDB:Fact+Dim

  • Join Dim

    Fact+DimDB

  • RedisHBase

    Storm

    Kafka .

    Kafka.Fact

    binloglog

    rtdw.order

    StormRtdw State

    Dim

    RtdwSync RtdwSync

    MySQL

    Dim

    rtdw.deal rtdw.poi rtdw.user

    /MySQL/Redis/HBase

    App

    MySQL.RtDW

    RtDW

    Batch

    Hive

    Batch

  • 2

    Deal,From,City

    , ,

    Bolt_Join_Dimuser_id, deal_id, order_id, city_id, money

    Bolt_Join_Dimuser_id, deal_id,

    order_id, from

    user_id, deal_id, order_id, city_id, from, money

  • -

    (Fact)

    DimRedisJoin

    FactJoin

    Fact

    Dim

    Fact

    Key Part A Part B

  • -

    Deal/POI

    DimRedisJoin

    StateRedis

    Fact

    Dim

    Key Part A

    State

  • -

    Fact

    Dim

    Fact

  • -

    State

    State Reids Cluster

    State Redis Cluster

    State Redis Cluster

  • 3

    Bolt_Cal_Newbuyeruser_id, order_id

    Hive ,user_id, order_id

    HBasenewbuyer_batch

    HBasenewbuyer_rt

  • -

    Redis StateSet DealUV

    HBaseKey

    Redis StateHyperLogLog

  • RtDW

  • RtDW

  • RtDW

  • RtDW

  • RtDW

  • RtDW

  • DimState DB

  • RtDW