Big Data Analyticssiva/talks/nls/myBigData.pdf · Good Ugly Bad.. Big Data Analytics िशवकुमारG. Sivakumar சிவகுமார் Computer Science and Engineering

Good Ugly Bad

.

...... Big Data Analytics

िशवकुमार G. Sivakumar சிவகுமார்

Computer Science and Engineeringभारतीय ौोिगकी संान म ुबंई (IIT Bombay)

[email protected]

October 17, 2013

• The Good (Data Deluge, Web 1.0, 2.0, 3.0, Analytics,Opportunities, Benefits)

• The Ugly? (Underlying Technologies, Hadoop, MapReduce, NoSQL)• The Bad (Hype, Challenges, Cost)

िशवकुमार G. Sivakumarசிவகுமார்Computer Science and Engineering भारतीय ौोिगकी संान म ुबंई (IIT Bombay) [email protected] Data Analytics

Good Ugly Bad

.. Blind men and the Elephant - अ-गज ायः


Good Ugly Bad

.. Dengue Trends


Good Ugly Bad

.. Twitter Trends


Good Ugly Bad

.. Google Trends

Popular search terms on Oct 16th.

What’s Wonobo?


Good Ugly Bad

.. Dengue Trends (Validation)

Thought Experiments: How did they do this? Is this really BigData Analytics?Compare with: Phailin, Real-time Traffic control forMumbai/Chennai, Amazon/Facebook recommendations.


Good Ugly Bad

.. Internet (Web 1.0)

Milestones

12

3

4 5 6

02

http://www.isc.org/

9796959493888270s

10 80k 1M 4.5M 16M1k

100M30M2k

5

25

90 150

20k 50k 800k

500 200k 1.2M

Academic WWW(steroids)

Java

LAN−boom!

(TCP/IP)

(DoD funds)

Hosts

INTERNET GROWTH

99

Users

Countries

Domains

WWW sites

Commercial UsersE−commerce

147M

All

Motto: Information AnyTime, AnyWhere, AnyForm, AnyDevice, ...WebTone like DialTone

Basic Hardware (sine qua non!)Democratized access to information! (Digital Divide)


Good Ugly Bad

.. Social Networking (Web 2.0)The OS/system software that empowers users to become producersof knowledge and ensures their right to collaboration/assembly.Examples: Wikipedia, Flickr, Orkut, Twitter, ....Mantras: Architecture of participation, Wisdom of crowds, Betteras more use - Long tail, Tagging, commenting, blogs, Open access(source/content) for Remix/Mashup


Good Ugly Bad

.. पवू प (Purva Paksha)

.

......

Web 1.0 may have democratized access to information, but it is likedrinking water from a fire hose!Search engines provide partial solutions, but cannot combine, categorizeand infer!

.

......

Web 2.0 may have allowed right to assembly/collaboartion, but• Proliferated unreliable, contradictory information.• Facilitated malicious uses including loss of privacy, security.

.

......

What do you want from Web 3.0?What you want to see/hear when you wakeup?I have a dream ...


Good Ugly Bad

.. Semantic WebThe application layer tapping the hardware (Web 1.0) and OS(Web 2.0)?

..Ramana Maharishi. author-of..

NaanYaar?

.Aksharamanamalai

.Vichara

Mani Mala

.

Realityin FortyVerses

.

contemporaries

.

.

KanchiChan-

drasekaraSaraswathi

.

Jiddu Kr-ishnamurti

.

Place: Tiru-vannamali,Tamil Nadu

.

.Lived30/12/1879

to14/4/1950

.

Combined, categorized information inferred from various sites,languages. www.dbpedia.org comes close today!


Good Ugly Bad

.. Approach 1: Artificial IntelligenceCan AI of computers match NS of humans? Consider chess, oncethe holy grail of AI.

Does not play the human way at all!Mostly parallelized search in hardware (200 millionpositions/second!) and almost no learning/improvement. But,enough to beat humans. Has changed chess irrevocably! (seeRybka analysis on left).Very tough challenge: Natural Language! (out of sight, out ofmind)


Good Ugly Bad

.. Approach 2: Smart Data

Embed all the knowledge in the data.The Semantic Web will enable machines to understandsemantic documents and data, NOT human speech andwriting.

Natural language is a real problem (invisible idiot! ;-)Two broad approaches here also.

• Mostly manual process. Exhaustive annotation with metadata.• Computers do most of the grunt work. Humans only supervise

and edit.Big Data Analtyics is only tip of the iceberg with main successes inmarketing and advertisement (Amazon, Google, Facebook,Twitter).


Good Ugly Bad

.. Tomorrow’s Information, Yesterday’s Technologies

How much useful data is accessible today? What types?(structured, semi-structured, unstructured) What characteristics?(4Vs)What actionable inferences can we make? (wireless moksha!)


Good Ugly Bad

.. 4-Vs of Big Data (IBM’s Version)


Good Ugly Bad

.. So, What’s Big Data?


Good Ugly Bad

.. Size of Web (Indexed)


Good Ugly Bad

.. data.gov.in


Good Ugly Bad

.. data.gov.in DataSets


Good Ugly Bad

.. Why Analytics?

िचनीया िह िवपदां आदाववे ितियान कूपखननं यंु दी े विना गहृेThe effect of disasters should be thought of beforehand. It is notappropriate to start digging a well when the house is ablaze withfire.There is a tide in the affairs of men, Which taken at the flood,leads on to fortune. Omitted, all the voyage of their life is boundin shallows and in miseries. Shakespeare


Good Ugly Bad

.. What is Analytics?

Includes the following.Databases, Queries, Text Mining, Data Mining, DataWarehouse, OLAP, Statistics, Pattern Recognition,Machine Learning, Artificial Intelligence

Looking for value!• Need to build models for data analysis. (who?)• Must ultimately result in some meaningful action and return on

investment• 3 step process: Identify objectives, Identify controls, Collect data

needed and Analyze


Good Ugly Bad

.. Bhuta Bhavya Bhavatprabhu ( भतूभ भवभःु )

॥ हिरः ॐ ॥िवं िववु षारो भतूभभवभःु।

• Bhuta (What happened? Reactive)Reports- Standard, ad-hoc, Query Drill down

• Bhavat (What is happening?)Statistical Analysis, Anamolies, Alerts

• Bhavya (What will happen? Pro-active)Predictive Forecast, Optimize

Analytics can convert data to knowledge to wisdom.


Good Ugly Bad

.. Typical Examples of analysis

• BIWhich products give higher revenue, how it varies overmonths (seasonality), which contribute to profitability, whichregions, which salesmen are doing well, …

• Predictive/online analyticsLikely response to new products, repeat customers, high riskcustomers, who visits our online resources, which videos getgood rating, is there negative sentiment, …

• Marketing analytics:Which advt. media to select, how sales will be affected ...


Good Ugly Bad

.. What’s special in Big Data Analytics?What more/different do we need from

• Databases (RDBMS)Used for Transaction processing, consolidation, reports,business performance

• Data WarehouseUsed to Co-relate various data, strategies for pricing, supplychains, cross-sale, ….

Can these not handle Web log for profiling and recommendations,Sentiment analysis for product evaluation, positioning, marketing?No. Biggest problem is the ACID properties requirement whichmakes scale-out and fault-tolerance near impossible.Some Weaknesses:

• Disk oriented storage and indexing structures• Multithreading to hide latency• Locking-based concurrency control mechanisms• Log-based recovery


Good Ugly Bad

.. Big Data Analytics Requirements

Think: Facebook likes, Shopping cart recommendations, ...• Ingest data at very high speeds and rates• Scale easily to meet growth and demand peaks• Support integrated fault tolerance• Support a wide range of real-time (or “near-real-time”)

analytics• Integrate easily with high volume analytic datastores


Good Ugly Bad

.. High Speed Data Ingestion

• Support millions of write operations per second at scale• Read and write latencies below 50 milliseconds• Do not need ACID-level consistency guarantees (Eventual is

fine!)• Support one or more well-known application interfaces

• SQL• Key/Value• Documents (NLP)


Good Ugly Bad

.. Scaling Requirements

• Scale-out on commodity hardware• Built-in database partitioning

Manual sharding and/or add-on solutions not practicalDatabase must automatically implement defined partitioningstrategy

• Application should see a single database instance• Database should encourage scalability best practices

For example, replication of reference data minimizes need formulti-partition operations


Good Ugly Bad

.. Fault-tolerance Requirements

• Database should transparently support built-in“Tandem-style” HA

• Users should be able to easily increase/decrease faulttolerance levels

• Database should be easily and quickly recoverable in the eventof severe hardware failures

• Database should be able to automatically detect and managea variety of partition fault conditions

• Downed nodes should be rejoinable without the need forservice windows


Good Ugly Bad

.. hadoop.apache.org


Good Ugly Bad

.. Open Source Big Data Stack


Good Ugly Bad

.. Goals of HDFS/HBase

• Need to process Multi Petabyte Datasets• Very Large Distributed File System

10K nodes, 100 million files, 10 PB Provides very highaggregate bandwidth

• Assumes Commodity HardwareProvide very high reliability : replication; detect failures andrecover Nodes fail every day

• Optimized for Batch Processing• Data locations exposed so that computations can move to

where data resides• Files are insufficient data abstractions : HBase

Need tables, schemas, partitions, indices• Need for an open data format

flexible schema


Good Ugly Bad

.. HDFS Architecture

Files are broken into 128MB blocks, each block replicated onmultiple data nodes.


Good Ugly Bad

.. Map-Reduce

Borrowed from functional programming (1980s!)


Good Ugly Bad

.. Hive: (No)SQL over Hadoop

This is the Big Analytics...• Supports normal SQL operators

Projections, equi-joins, group by, order by• Can run on selected samples of data• Provides custom scripts using map-reduce• Map-reduce based implementation of joins, group by, sorting

(order by)• Optimization

Merging sequential map-reduce jobs, sharing common reads,early selections and projections


Good Ugly Bad

.. Hype or Real?

In the Big Data Analytics context consider the following..Wikipedia Definition..

......

Technology is the usage and knowledge of tools, techniques, crafts,systems or methods of organization in order to solve a problem orserve some purpose.

Technology is not created in a vaccuum. It has a context and issocially relevant.Mother:Necessity Father:Profit!As can be expected, mother is always good, father can be verybad! IPR/Profit: Boon or Bane?There are many who believe that Google/Facebook has damagedthe human brain! ;-)


Good Ugly Bad

.. Hype or Real?

In the Big Data Analytics context consider the following..Wikipedia Definition..

......

Technology is the usage and knowledge of tools, techniques, crafts,systems or methods of organization in order to solve a problem orserve some purpose.

Technology is not created in a vaccuum. It has a context and issocially relevant.Mother:Necessity Father:Profit!As can be expected, mother is always good, father can be verybad! IPR/Profit: Boon or Bane?There are many who believe that Google/Facebook has damagedthe human brain! ;-)


Good Ugly Bad

.. Big Data Challenges (Gartner Survey)


Good Ugly Bad

.. Hype Cycle (Gartner 2013)


Good Ugly Bad

.. MIT’s DataHub (VLDB 2013 Keynote talk)

The middle-path?
