Microsoft Big Data 解決方案與案例分享

DBI 312

Rich HoTechnical Architect微軟技術中心

Agenda

• What is Big Data?• Microsoft Big Data Strategy• Key Benefits of Microsoft Big Data• Demo• Case Study

What is Big Data ?

By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent.

– Gartner, Mark Beyer, “Information Management in the 21st Century”

10x increase every five years

85% from new

data types

Dataexplosion

VolumeVelocityVariety

Easy Accessibility of External Data

Hadoop

Cloud

Cheap, Distributed Storage &

Processing

The world of Data is changing

Three V in Big Data

Relational Data

10x increase every 5 years

VOLUME

4.3 connected devices per adult

VELOCITY

VARIETY

85% from new data types

Big Data

We are moving to Big Data

Any Data, Any Size, Anywhere

Traditional Data

Big Data

Data Volume Petabytes+Terabytes

Data Variety UnstructuredStructured

Data Velocity StreamingBatch

Garbage? +++++-----

With Big Data, says:

以前我們將尿布和啤酒放在一起而創造新業績

現在我們將比老爸更早一步知道他女兒懷孕了 !

Traditional data flow

New exploratory data flow

What is Big Data?

Increasing Data Volumes

Increasing Data & Analysis Complexing

Emerging Technologies

Microsoft Big Data Strategy

Microsoft Big Data Stacks

NON-RELATIONAL

100111

DATA MANAGEMENT

SHAREAND GOVERN

DISCOVERAND RECOMMEND

TRANSFORMAND CLEAN

INSIGHTS

DATA ENRICHMENT

OPERATIONAL

SELF-SERVICE MOBILE

PREDICTIVE

REAL-TIMECOLLABORATIVE

MA

RK

ETPLA

CE

Exte

rnal D

ata

and S

erv

ices

RELATIONAL MULTIDIMENSIONAL STREAMING

Fast

L

oad

Source Systems

Historical Data(Beyond Active Window)

Summarize & Load

Big Data Sources (Raw, Unstructured) Data & Compute

Intensive Application

ERPCRM LOB

APPS

Integrate/Enrich

Enterprise Data Warehouse

Enterprise ETL with SSIS, DQS, MDS

Distributed environments for Non-structured data

Data Marts

Reporting

Multi-dimensional Analysis

Business Insights

Interactive Reports

Performance Scorecards

Crawlers

Bots

Devices

Sensors

Online Transaction

Enterprise Big Data Blueprint

Hadoop on Windows

Hadoop on Azure

Fast

L

oad

Source Systems

Historical Data(Beyond Active Window)

Summarize & Load

Big Data Sources (Raw, Unstructured) Data & Compute

Intensive Application

ERPCRM LOB

APPS

Integrate/Enrich

Hadoop on Windows Azure

Hadoop on Windows Server

Business Insights

Interactive Reports

Performance Scorecards

Crawlers

Bots

Devices

Sensors

Enterprise Data Warehouse

Data Marts

Reporting

Multi-dimensional Analysis

Enterprise ETL with SSIS, DQS, MDS

Online Transaction

Microsoft Big Data Platform

Fast

L

oad

Source Systems

Historical Data(Beyond Active Window)

Summarize & Load

Big Data Sources (Raw, Unstructured) Data & Compute

Intensive Application

ERPCRM LOB

APPS

Integrate/Enrich

Hadoop on Windows Azure

Hadoop on Windows Server

Business Insights

Interactive Reports

Performance Scorecards

Crawlers

Bots

Devices

Sensors

Enterprise Data Warehouse

Data Marts

Reporting

Multi-dimensional Analysis

Enterprise ETL with SSIS, DQS, MDS

Online TransactionSQL Server

EE

SQL Server PDW

SQL Server FT

Microsoft Big Data Platform

Fast

L

oad

Source Systems

Historical Data(Beyond Active Window)

Summarize & Load

Big Data Sources (Raw, Unstructured) Data & Compute

Intensive Application

ERPCRM LOB

APPS

Integrate/Enrich

Hadoop on Windows Azure

Hadoop on Windows Server

Business Insights

Interactive Reports

Performance Scorecards

Crawlers

Bots

Devices

Sensors

Enterprise Data Warehouse

Data Marts

Reporting

Multi-dimensional Analysis

Enterprise ETL with SSIS, DQS, MDS

Online Transaction

SQL Azure

Microsoft Big Data on Public Cloud

Hadoop on Azure (CTP)

Request a New Cluster (4 nodes, 2 TB)

Allocating NN & DN

Starting NN & DN

NN & DN ready! (after 9 mins)

Hadoop on Azure ready! (after 13 mins)

Access Hadoop on Azure through RDP

Key Benefits of Microsoft Big Data

Microsoft Big Data Benefits

• New Hadoop-based distribution on Windows / Azure

• Hive ODBC driver

• Hive add-in for Excel

Broader Access of Hadoop

• Analyze Big Data with familiar tools (Excel, PowerPivot, Power View)

• JavaScript based simple programming

Breakthrough Insights

• System Center Integration

• Active Directory Integration

Enterprise Ready

Breakthrough Insights

Hive ODBC Driver integrates Hadoop to SQL Server Analysis Services, PowerPivot, and Power View

Breakthrough Insights

Benefits

Key

Featu

res

Broader Access of Hadoop

Hive add-in for Excel

Hive ODBC driver

Broader Access of Hadoop

Benefits

Key

Featu

res

Enterprise Ready (SCOM Integration)

System Center to monitor performances of Head nodes and Data nodes

Enterprise Ready

Benefits

Key

Featu

res

1. Hadoop on Azure - JavaScript Console - Job Execution2. Excel Hive Connectivity3. Windows Marketplace & Hadoop on Azure Integration

Case Study: Mining Big Mobility Data

Microsoft Big Data Case Study: T-Drive

Microsoft Big Data Case Study: T-Drive

Video

Microsoft Research AsiaBig Data Case Study

Thank You

Connect. Share. Discussshttp://www.microsoft.com/taiwan/techdays2012/

Microsoft Certification & Training Resources

http://www.microsoft.com/learning/zh/tw/

Resources for IT Professionals

http://social.technet.microsoft.com/Forums/zh-tw/categories

/

Resources for Developershttp://social.msdn.microsoft.com/Forums/zh-tw/categories

/

Resources

請協助完成「本課程問卷」，並在離開教室時交給工作人員！

填妥大會背包中的大會問卷，可於活動第三天兌換問卷禮哦！

感謝您的合作。