Embed Size (px)
DESCRIPTION
Building a Real Time Analytics Platform on Big Data in the Cloud
Citation preview
1
Building a Real Time Analytics Platform on
Big Data in the Cloud Sanjay Sharma
Principal Architect
June 12 2013
Impetus
© 2013 Impetus Technologies
• Software Solutions and Services Company • Leader in Innovation led Technology services • 17 years of customer success, 1500 people • Big Data, Enterprise Mobility, Test and
Performance Engineering, Carrier Grade Large Systems
© 2013 Impetus Technologies
?
Why s9ll Batch?
$$$
Credits * hAp://thenextweb.com/insider/2013/02/07/mailboxs-‐user-‐access-‐queue-‐is-‐a-‐work-‐of-‐frustra9ng-‐genius/ ** hAp://en.wikipedia.org/wiki/Moore's_law
Batch to Real Time Now is the Time
© 2013 Impetus Technologies
OLTP Online
Transac'ons Processing
NOW
OLAP/DW Online
Analy'cal Processing/DW
ZZZZZZZZZ…
Transac9ons & Analy9cal Processing Now is the Time
© 2013 Impetus Technologies
Age of Horizontal Scalability -‐Divide and Conquer
Online Transac'ons Processing
Online Analy'cal
Processing/ DW
OLTP OLAP
© 2013 Impetus Technologies
Online Transac9ons Processing
Online Analy9cal
Processing/ DW
On Demand Scalability -‐CLOUD
OLTP OLAP
© 2013 Impetus Technologies
State of Play: Analy9cs/ BI
Database RESULTS
Time Time
Ingest/ETL SQL Machine Data
© 2013 Impetus Technologies
Faster Analy9cs: Streaming Analy9cs/ Complex Event Processing
Database RESULTS
Time Time
RESULTS
Ingest/ETL SQL
Approach 1
© 2013 Impetus Technologies
Faster Analy9cs: In-‐memory Analy9cs
Database RESULTS
Time Time
RESULTS
Ingest/ETL SQL
Approach 2
• Streaming/CEP Analytics – Storm by Twitter
– Spark Streaming by Berkley Amplabs – IBM Streams – SQLstream – MS StreamInsight – StreamBase – Oracle CEP/Streams – Esper
© 2013 Impetus Technologies
• In-Memory Analytics – SAP Hana – Oracle Exalytics – SAS in-memory analytics – MS Powerpivot
Some Real Time So_ware
© 2013 Impetus Technologies
• Others – NoSQL databases
• Cassandra, MongoDB, HBase
– Grid solutions – MPP and NewSQL – Druid, Kognitio – Solr/Elastic Search/Marklogic – Splunk – BI products with OLAP
• Hadoop based – Summarization/ cube
preparation engine – Impala/Stinger/Hawq/
Drill – Hstreaming
Some Real Time So_ware
• Specialized – Hardware accelerators / GPU
cards • Sisense
© 2013 Impetus Technologies
Real Time Analy9cs Strategy -‐Key Components
RESULTS Ingest SQL
Machine Data
OLTP OLAP/DW
ETL
Discovery & Search
Deeper Analytics
Streaming Analytics
Real time Analytics
© 2013 Impetus Technologies
NoSQL + Search
Machine Data
Real Time Analy9cs Strategy -‐Big Data Reference Architecture Example
© 2013 Impetus Technologies
Real Time Analy9cs Strategy -‐Data Inges9on/ Streaming – Customer Example
Machine Data
Inges9on Engine (Real 9me +
Batch components)
Real Time Processing Engine
(CEP/Analy9cs/ Rule Engine)
Real Time Data
Storage Engine (Store + Indexing/Search)
Business Process Engine (Business Process/ Rule management)
Kafa Storm + Esper
HBase+ Solr
JBoss Drools/jBPM
© 2013 Impetus Technologies
NoSQL + Search
Machine Data
Real Time Analy9cs Strategy -‐Big Data Reference Architecture Example
Impetus provides vendor-neutral Consulting & Services for Big Data Analytics
We take enterprises through the journey of
successful Big Data adoption
We deliver: Low-risk acceleration to Business value at moderate cost
© 2013 Impetus Technologies
Impetus
