View
390
Download
2
Category
Preview:
DESCRIPTION
Citation preview
Data Warehousing & Business Intelligence in the Cloud
Seoul, Korea COEX Convention Centre 24th October 2013
Data Analytics in the
Cloud
Blair Layton
Business Development Manager
(Databases) – Amazon Web Services
(APAC)
The Explosion of Data
Existing Challenges with Analytics
The Cloud
The Explosion of Data
Existing Challenges with Analytics
The Cloud
We are constantly producing more data
• Insert big data infographic here
From all types of industries
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Take a look a data processing “pipeline”
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Data is available everywhere, contains customer insight and costs little to generate, but..,
What has changed in this pipeline
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Highly constrained
Everything else has constraints
Big Gap in turning data into actionable
information
The Explosion of Data
Existing Challenges with Analytics
The Cloud
Provision all your infrastructure and tools before you get results
Challenge 1: Capex Intensive
Source: Oracle technology global price list 11/1/2012
Cost of your infrastructure dictates what analytics you can perform
Most data never makes it to a data warehouse
1990 2000 2010 2020
The Data Analysis Gap
Enterprise Data
Data in Warehouse
Enterprise Data is growing at over 50% yearly
Data Warehousing growing at less than 10% yearly
Most data is left on the floor
Sources: Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
Setup takes months of planning and work
Challenge 2: Hard to setup, manage and scale
Enterprises average between 3 and 4 DBAs per data warehouse
Gartner: Critical factors in calculating the data warehouse TCO, July 2009
Extending your data-warehouse can be heavy on time and cost
Managing a data analytics platform requires expensive staff
Complex tuning and management skills required
Very hard to move up the stack
These make it extremely hard to move up the Business Intelligence Maturity Stack
The Explosion of Data
Existing Challenges with Analytics
The Cloud
AWS Services
AWS Global Infrastructure
Application Services
Networking
Deployment & Administration
Database Storage Compute
AWS Global Infrastructure
9 Regions
25 Availability Zones
Continuous Expansion
• $5.2B retail business
• 7,800 employees
• A whole lot of servers
Every day, AWS adds enough
server capacity to power that
whole $5B enterprise
Powering the Most Popular Internet Businesses
Broad ecosystem of consulting partners..
We have partners and technologies ready to help
Solving Problems for Organizations Around the World
No Upfront Investment
Replace capital expenditure with variable expense
Low ongoing cost
Customers leverage our economies of scale
Flexible capacity
No need to guess capacity requirements and over-
provision
Speed and agility
Infrastructure in minutes not weeks
Focus on business
Not undifferentiated heavy lifting
Global Reach
Go global in minutes and reach a global audience
37 PRICE REDUCTIONS
Value proposition of the AWS cloud
Architected for Enterprise Security Requirements
“The Amazon Virtual Private Cloud
[Amazon VPC] was a unique option that
offered an additional level of security and
an ability to integrate with other aspects of
our infrastructure.”
Dr. Michael Miller, Head of HPC for R&D
(August 19, 2013)
Gartner “Magic Quadrant for Cloud Infrastructure as a Service,” Lydia Leong, Douglas Toombs, Bob Gill, Gregor Petri, Tiny Haynes, August 19, 2013. This Magic Quadrant graphic was published by Gartner, Inc. as part of a
larger research note and should be evaluated in the context of the entire report.. The Gartner report is available upon request from Steven Armstrong (asteven@amazon.com). Gartner does not endorse any vendor, product or
service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner's research organization
and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Gartner Magic Quadrant for Cloud Infrastructure as a Service
The Explosion of Data
Existing challenges with analytics
The Cloud
Data is a competitive edge
Hard and expensive to setup, manage and scale
Lowers cost and improves agility
Summarizing the problem and the opportunity
Data Analytics in the Cloud
Easy and inexpensive to get started
Easy to setup, scale and manage
Low cost to enable analytics on all your data
Open and flexible
The Solution
Technology Process View
Data source 1
Data source n
Unstructured data sources
Extract Transform, Load and Cleanse Data
warehouse
Data source 1
Analytics
Analytics
The diagram above shows functional architecture components of any data warehousing project.
Source systems
Data source 1
Data source n
Unstructured data sources
Extract Transform, Load and Cleanse Data
warehouse
Data source 1
Analytics
Analytics
The diagram above shows functional architecture components of any data warehousing project.
Data Integration
Data source 1
Data source n
Unstructured data sources
Extract Transform, Load and Cleanse Data
warehouse
Data source 1
Analytics
Analytics
The diagram above shows functional architecture components of any data warehousing project.
The Data Warehouse
Data source 1
Data source n
Unstructured data sources
Extract Transform, Load and Cleanse Data
warehouse
Data source 1
Analytics
Analytics
The diagram above shows functional architecture components of any data warehousing project.
Business Intelligence and Analytics
Data source 1
Data source n
Unstructured data sources
Extract Transform, Load and Cleanse Data
warehouse
Data source 1
Analytics
Analytics
The diagram above shows functional architecture components of any data warehousing project.
Data Analytics -Technology Stack
Data Integration
Data Warehouse
Business Intelligence
AWS Cloud
Amazon Redshift
Amazon Redshift
Data warehousing done the AWS way
• Pay as you go, no up front costs
• Fast, cheap, easy to use
• SQL
• Easy to provision Deploy
Customer quotes
“[Amazon Redshift] took an industry famous for its opaque pricing,
high TCO and unreliable results and completely turned it on its head.”
“Redshift is twenty times faster than Hive…The cost saving is even
more impressive…Our analysts like [it] so much they don’t want to go
back.”
“Team played with Redshift today and concluded it is awesome. Un-
indexed complex queries returning in < 10s.”
“Queries that used to take hours came back in seconds. Our analysts
are orders of magnitude more productive.”
Amazon Redshift lets you start small and grow big
Extra Large Node (HS1.XL) 3 spindles, 2 TB, 16 GB RAM, 2 cores
Single Node (2 TB)
Cluster 2-32 Nodes (4 TB – 64 TB)
Eight Extra Large Node (HS1.8XL) 24 spindles, 16 TB, 128 GB RAM, 16 cores, 10 GigE
Cluster 2-100 Nodes (32 TB – 1.6 PB)
Note: Nodes not to scale
Amazon Redshift Pricing – Singapore & Sydney
Price Per Hour for XL Node ($US)
On-Demand $ 1.25
1 Year Reservation $ 0.75
3 Year Reservation $ 0.45
Simple Pricing
Number of Nodes x Cost per Hour
No charge for Leader Node
Pay as you go
So for example…….
• 1 XL node reserved for 3 years:
= 0.45c x number of hours in a month
= $340 per month
• 1 XL node cluster gives you: • 2 Cores
• 16 GB RAM
• 2 TB Disk
• Plus 2 TB storage in S3 for backups & snapshots
Amazon Redshift is easy to use
• Provision in minutes
• Monitor query performance
• Point and click resize
• Built in security
• Automatic backups
Use cases
• Reporting Data-warehouse behind an OLTP system
• Data Mart to take load off the existing data warehouse
• Log file analysis for clickstream or gaming data (e.g. Advertising, Retail, Gaming)
• Query-able archive for data compliance (e.g. Telco - Call detail Records)
• Machine generated sensor data analysis (e.g. Utility - smart meters, Resources - equipment failure prediction)
• As a data analytics system for live data (Gaming, Advertising)
Amazon Partner Network
(Technology Partners)
Flexibility & choice are key in the Cloud
Application Services
Compute Storage Database
Networking
AWS Global Infrastructure
Deployment & Administration
Thank you
Extending data
integration into the Cloud
Colm Daniel
World Wide Cloud Alliances
Ron Lunasin
Sr. Director – Cloud Product
Management
Today’s Agenda
• Informatica Cloud Overview
• Informatica Cloud Amazon Redshift Connector
• Demonstration
• Next Steps
• Q&A
The Industry Leader in Cloud Integration Informatica:
#1 by Customer Count
2000+ companies
#1 by Customers/Analysts
Gartner AppExchange
#1 by Data Processed
+40B transactions/month
#1 by Connectivity
Informatica Cloud Marketplace
Top Right @ the Core: Gartner Magic Quadrants
Employees in 26 Countries…. and growing!
Global Presence & Global Perspective
New Cloud Connectors
http://www.informaticacloud.com/connectivity
New!
Cloud Integration Customer Success Stories
Synchronizing Salesforce CRM with Netsuite and other business apps
1.5M rows of data synchronized daily
App Integration Data Replication
Decreased operational issues from 70% to 30% of IT workload
Enabled faster, more accurate decision-making based on timely, trusted data
Data Migration
Consolidated Smith Barney and Morgan Stanley data on Day 1 of merger
Managers didn’t lose momentum in ongoing recruiting efforts
Extend PowerCenter
Hybrid deployment gives integration flexibility and scalability to meet various use cases
Lowered time and resources needed for integrations by 80%
iPaaS *(Build)
Reduce time to build and distribute connectivity to 3rd party data sources
Customize cloud integration templates to execute sophisticated integration workflows
Informatica Cloud
The Industry’s Most Comprehensive Cloud Integration
and Data Management Solution
Cloud Integration Connecting your cloud apps
Cloud Data Quality and MDM Delivering the “Single Customer View”
Cloud Process Automation Guiding users to work efficiently with the data
Our Mission:
Unleash the Potential Of the Cloud
Cloud Amazon Redshift
Connector Ron Lunasin, Cloud Platform Adoption
Recognition of “The Next Wave” back in 2004
Cloud based Integration
Client / Server based Integration
Challenges with Traditional Approaches to Cloud Integration
Prism ETI
Mainframe based Integration
Move to the Cloud…
IT transitions from skeptic to partner to driver
Increasing IT involvement
in Cloud decision making
Pre-2010
LOB Owned (Outside of IT)
LOB Led (IT Approved)
Business-IT Collaboration
Cloud First (IT Led)
2010-2012
2012-2013
2013
Cloud is the Reality in the Enterprise
90% Cloud decisions and operations
involve IT (IDC)
Driven by IT
Large, Accelerating Market
66% SaaS POs
signed by IT (IDC)
76% enterprises
have a formal cloud strategy
(Forrester)
74% using cloud
will increase cloud spend > 20%
(IDC)
Led by Large
Enterprises 4-6x growth rate of
on-premise IT
20-27% CAGR
$20-40B market (Forrester, IDC, Gartner, 451Group)
84% of net new software is now SaaS
(IDC)
60% of all companies
using SaaS w/in 12 months
(Forrester)
SaaS largest category
PaaS fastest growing
(Forrester)
Informatica Cloud and Amazon Redshift:
Enabling cost-effective data warehousing
• Redshift Connector pre-release announced in February
• General availability in August 2013
InformaticaCloud.com/Amazon-Redshift
What did it use to take…
• Budget large capital expenditure
• Schedule a sales meeting with Oracle, IBM, Teradata, etc…
• Formal POC (Proof of Concept)
• Procure software and hardware
• Install and setup
• Start project
What it takes now…
• Go to the web and sign-up
• Start project!
2
1
Informatica Cloud Architecture Overview
4 Secure Agent
Your Company 3
Marketplace
Amazon Redshift
Informatica Cloud Amazon Redshift demonstration
Firewall
Informatica Cloud Secure Agent
Metadata Mappings
Build mapping and execute job
1
1
Retrieve Account Data 2
2
3 Put Account Data into Flat File
4 Transfer compressed Flat File to S3
5 Initiate copy from S3
6 Load data into Amazon Redshift
6
3
5 4
Best practices to remember…
• The Amazon S3 bucket that holds the data files must be created in the same
region as your cluster
– Files are deleted from Amazon S3 bucket when upload is complete
• Choose a batch size where the number of batches matches the number of
slices in your cluster
– Each XL node has 2 slices, each 8XL node has 16
– If you have a 2 node XL cluster and 40,000 rows of data, choose a batch size of
10,000
– The Informatica Cloud Redshift connector can maximize Amazon’s parallel
processing capabilities this way
Next Steps
• Get started with Amazon Redshift
• Get started with Informatica Cloud
– InformaticaCloud.com
• Learn more about our Redshift Connector
– InformaticaCloud.com/Amazon-Redshift
Q&A Colm Daniel, cdaniel@informatica.com
Ron Lunasin, rlunasin@informatica.com
Thank you
AWS Reporting &
Analysis
Ben Connors
Worldwide Head of Alliances - Jaspersoft
• Analysis of Cloud market motivations
• Overview of Cloud trends
• Cloud User category expectations
• How BI/Jaspersoft fits into Cloud strategies
• Demos
• Summary
© 2013 Jaspersoft Corporation 71
Session Overview
Industry Movement to the Cloud
• Cloud Growth –
– Cloud IT spend will grow from 3% - 17% of total (Morgan Stanley)
• Motivations:
– Agility
– Lower cost
– Faster time to value
– Less risk
• Use cases:
– CRM, ERP, HR, Online Gaming, Manufacturing, Expense Reporting, Big Data, Consumer Applications, Etc.
• Workloads:
– Dev/Test
– ‘Spiky’
– High Growth
– Reliable production
• BI usage matches these Cloud trends
© 2013 Jaspersoft Corporation. 72
Cloud Computing Growth
© 2013 Jaspersoft Corporation. 73 http://www.forbes.com/sites/louiscolumbus/2013/02/19/gartner-predicts-infrastructure-services-will-accelerate-cloud-computing-growth/
Asia/Pacific Cloud Growth
© 2013 Jaspersoft Corporation. 74 http://techaisle.com/blog/2012/11/lots-of-clouds-in-the-forecast-and-a-holiday-story/
Top Cloud Applications
0
10
20
30
40
50
Deployed
In 12 months
• INTERNAL BUSINESS APPLICATIONS TOP THE LIST; MOBILE SITES NEXT
What kinds of applications have you delivered using a cloud environment? Which do you plan to deliver during the next 12 months?
Source: Forrester Cloud Developer Survey, Q3 2012
© 2013 Jaspersoft Corporation. 75
2013: Current/future BI
Cloud adoption trends
© 2013 Jaspersoft Corporation. 76
TechTarget 2013 Analytics & Data Warehousing Reader Challenges & Priorities Survey
Does your organization run or plan to run any part of its BI, analytics and data warehousing
systems in the cloud?
15%
13%
32%
41%
Yes, active cloud user
Plan to start using the cloud in the next 12 months
Considering, but no set plans
No
60% planning, considering, or actively using
N = 559
• The cloud continues to play a critical role in supporting BI, analytics, and DW initiatives with 3 out of 5 respondents reporting that they are planning, considering or actively using the cloud.
• Business User
– Efficient access to IT resources w/o red tape and delays
• Application Developer
– Platform with dev tools, middleware, capacity, configuration mgt.
• IT Operations
– Elastic capacity, secure, standard, keep users happy
• Management
– Control expenses & risk, delight customers/partners, move fast
© 2013 Jaspersoft Corporation 77
Constituents - Cloud Expectations
Example Industry Use Cases
for Business Intelligence
Industry Data Analyzed
Online Gaming # players vs. time, spend/player, popularity of weapons, scene usage
Education Student attendance, test scores, teacher performance, spend/student
Telecom Customer churn, data traffic patterns, billing per service
Government Crime data, demographics, health trends, economic
Advertising Click-through rates, conversion rates, regional variation
Retail Product sales, Profits, Customer traffic, Product correlations
Manufacturing Inventory, quality, vendor performance, logistics
78 © 2013 Jaspersoft Corporation
Current State of Business Intelligence
• Standalone
• Expensive
• Desktop-based
• High Latency
© 2013 Jaspersoft Corporation. 79
Competing on Time and Information
80
“The New Factors of Production: Time and Information” Brian Gentile, Jaspersoft
But business users don’t have access to timely,
actionable data
Why?
Most don’t spend their day inside a BI tool …nor
do they want to!
© 2013 Jaspersoft Corporation.
Embedded BI - Why?
• For Best Decisions, Information Should Be:
– Relevant
– Timely
– Actionable
81 © 2013 Jaspersoft Corporation.
Embedded BI
• Maintains
– Context/Relevance
– Motivation/Timeliness
– Train of thought/Timeliness
– Actionable/Within application or beyond
– Security
• Broadens User Community
– Executives
– More knowledge workers
– Self-serve, Interactive
82 © 2013 Jaspersoft Corporation.
4xC Barriers to
Embedded BI Adoption
© 2013 Jaspersoft Corporation. 83
Complex to Deploy
Cost Complex to Embed
Complex to Use
NEED: Develop for free. Pay only for what you use when
deploy
NEED: Deploy
with push-
button ease or use as a service
NEED: Embed
self-service BI through standard
APIs
NEED: Easy to
build and use BI assets
Simple, Low-Cost Embedded BI
3rd Gen Embedded BI
Breaks Barriers
© 2013 Jaspersoft Corporation. 84
Complex to Deploy
Cost Complex to Embed
Complex to Use
Free + usage-based pricing
HTML5/CSS+ RESTful
web services
Push-button on-premises deployment and Cloud BI service
Easy to build for BI Builders on any data and self-serve for BI Consumers on any device
3rd Generation Embedded BI
We Need “Intelligence Inside”
85
We want information to FIND US, not the other way round
“We need Intelligence Inside the applications and business processes we use every day.”
– Pipeline dashboard inside SaaS CRM app
– Performance report inside partner portal
– Salary data visualizations inside HR intranet
– Portfolio analytics inside client website
– Tickets crosstab inside custom helpdesk app
– Interactive charts inside native mobile app
© 2013 Jaspersoft Corporation.
Embeddable Architecture Open web standard
architecture makes
integration with any
app easy to perform
Cloud Ready Multi-tenant architecture,
100’s of SaaS
customers, top selling BI
solution on Amazon
Affordable Up to 80% less than
traditional BI platforms
while delivering significant
power & capabilities
Proven Platform Millions of users,
380,000 community
members, deployed in
130,000+ applications
Full Self-Service BI Suite Address all user requirements with
interactive reports, dashboards,
analysis, and data integration
Jaspersoft: The Intelligence Inside
Product Overview
Jaspersoft Products
88
Reporting Engine
Visual Report
Design Environment
Ad Hoc Reports, Dashboards,
In-Memory Analysis Server
Powerful OLAP
Data Analysis
Studio
© 2013 Jaspersoft Corporation.
Design Any Report . . .
© 2013 Jaspersoft Corporation. 89
… Dashboard
90 © 2013 Jaspersoft Corporation.
… or Analytic View
91 © 2013 Jaspersoft Corporation.
... Using Any Data Type
POJO files
Relational Files Relational Big Data Files
© 2013 Jaspersoft Corporation. 92
Redshift
© 2013 Jaspersoft Corporation. 93
… bringing Intelligence to Any App
… with a World-Class BI Platform
94
Reporting, Dashboards, Visualization, OLAP Analysis
Columnar-Based In-Memory Engine
Data Connectivity to Any Data
10
0%
Web
Sta
nd
ard
s: C
SS, .
JS, .
JSP,
Jav
a
Exte
nsi
ve A
PIs
: HTT
P, S
OA
P, R
EST
HTML5 Browser, Native Mobile Apps
Business Metadata Layer
Data Integration
Data Virtualization Direct
Redshift EMR On-Premises RDS SaaS
Jaspersoft Customers
Software & Technology
Financial Services
Public Sector
Telecommunications
Travel & Transportation
Manufacturing
Healthcare/Pharmaceutical
© 2013 Jaspersoft Corporation.
Jaspersoft AWS Hourly: 500+ Customers in 6 Months!
95
Jaspersoft/AWS Customer:
BizFlow/Samsung Korea
• Business Process Management (BPM)
• Challenge
– Monitor/Analyze Business Activities
• Solution
– Jaspersoft on Cloud
• Results
– Customers avoid infrastructure
– Increased BizFlow revenue
– Self-service BI
– Higher value analytics
http://www.bizflow.com/business-process-management/samsung-heavy-industries
© 2013 Jaspersoft Corporation. 96
Jaspersoft/AWS Customer:
Sage Human Capital
• Recruiting Firm for High Tech companies
• Challenge
– Visibility for recruiting process status
• Internal
• External
• Solution
– Jaspersoft on AWS
• Results
– Dashboards set up in two hours
– Disrupting the industry “Jaspersoft for AWS allows me to have big company analytics for a small business price. With this information, we can be proactive instead of reactive.” - Paul Grewal, CEO Sage Human Capital
© 2013 Jaspersoft Corporation. 97
Jaspersoft/AWS Customer:
Blue Consulting
• Administration Systems for Schools
• Challenge
– Data from many systems
– Difficult for everyone, including teachers, to access
• Solution
– Jaspersoft on AWS, Amazon Redshift
• Results
– Over 200 schools provide reporting to teachers, even at home
– More informed decisions, educational approaches, resource optimization
“Our users LOVE Jaspersoft ad hoc reporting, and the performance of the system with Redshift.” -Russ Davis, Founder & CEO
98 © 2013 Jaspersoft Corporation.
© 2013 Jaspersoft Corporation. 99
Jaspersoft BI for AWS Overview
Jaspersoft 5 Demo
100 © 2013 Jaspersoft Corporation.
Jaspersoft Integrated with Amazon Redshift
• Jaspersoft is the first BI service that you can buy per hour
– No user limitations, no monthly fee,
– less than $1 per hour
• First BI service to automatically
connect to your AWS data
– 10 minutes from launch to visualizing your data in RDS or Redshift
– AWS Security Integration
• Released February, 2013
– Over 500 customers
101
Jaspersoft Pro on AWS
Thank you
Recommended