Kazunori SatoDeveloper Advocate, Cloud Platform

The Googliness ofGoogle Cloud Platform

+Kazunori Sato@kazunori_279

Developer Advocate, Cloud Platform- Developer Community Support- gcp ja night, Docker Meetup, GCPUG- Product Launch Support

Cloud Technology Innovations

2012 2013

MapReduce

Spanner/F1

2003 2006 2007 2010 2011

GFS

Omega

Colossus

Cloud Storage

Dremel

BigQuery

Bigtable

Cloud Datastore

Paxos impl.

2004

Management MobileDeveloper Tools

Compute

Networking

Big Data

Storage

Google Compute Engine

Google App Engine

Google BigQuery Data Analytics with Google’s massively parallel query engine

IaaS with Google’s network and computing infra

PaaS with Google’s container technology

Google BigQuery

72 hours / min100 petabytes500M active users

How Big is Big Data in Google?

In Google, we useGoogle BigQuery

for processing Big Data

SELECT top(appId, 20) AS app, count(*) AS countFROM installlog.2013ORDER BY count DESC

Scans 100B rowsin 10 sec, without index.

Google BigQuery: Massively Parallel Query Service

Cost of BigQuery

Storing 1TB = $26 / month

Query on 10GB = $0.05 / query

select top(title), count(*) from publicdata:samples.wikipedia

To scan 1 TB in 1 sec,you need 5,000 disks in parallel

Massively Parallel Processing

Mixer 0

Mixer 1 Mixer 1

Shard Shard Shard Shard

ColumnIO on Colossus SELECT state, year

COUNT(*)GROUP BY stateWHERE year >= 1980 and year < 1990

ORDER BY count_babies DESCLIMIT 10

COUNT(*)GROUP BY state

Tree structure for fast aggregation

Streaming API + Fluentd: real-time import at 1M+ rows/s

Google App Engine

Google App Engine: PaaS from Google

Google confidential | Do not distribute

Google’s Container Technology

Every Google service runs in containers

Launches 2B containers every week

Google confidential | Do not distribute

Scaling Out by Google’s Container Technology

Instantiationin 40 ms

GAE Architecture And Optmization

The Traditional Web Development

Hardware Failures

Traffic Spike

Growing Big Data

Complex Design

Complex Dev.

Complex Admin

Cost

GAE Architecture And Optmization

Google App Engine Encapsulates Them All

Hardware Failures

Traffic Spike

Growing Big Data

Simpler Design

Simpler Dev

No Admin

No Initial Funding

Source: Google Internal Data

4.75 Millionactive applications

作用中應用程式 　　　active applications

700 million

“App Engine enabled us to focus on developing the application. We wouldn’t have gotten here without the ease of development that App Engine gave us.”Bobby Murphy, CTO

Snapchat sends

photos and videos each day Google App Enginescaled seamlesslyduring growth to

millions of users

Small team is ableto innovate quickly

and expandglobally

Google Compute Engine

The State of the Art Network Technology

Andromeda: The latest Software Defined Network (SDN)

- Intra/Inter zone throughput = 9G bps- Inter regional private network by default

　Google-scale Load Balancing

region 2 region 3region 1

VMs VMs VMs

:::::::::::::::::::::::::::::::::::::::::::::::HTTP LB

DNS :)

One global IP: 11.22.33.44

● Load Balancing infra shared with Google.com

● Easily handles 1M req/s

● TCP anycast with single global IP

● Fast fail-over in seconds

Live Migration

source: Google

Run and manage an optimized cluster

KubernetesOpen Source

Workload

API & Tools Google Container Engine (GKE)

Kubernetes ● Growing Community ● Linux containers support● Open source orchestration

create and manage

Container Engine ● クラスタをすぐに起動可能

● クラスタ環境の最適化

● Kubernetesを実行管理する

GKEMaster GKE Nodes

Compute Engine

VMs

Google Container Engine

Getting Started

http://gcpug.jp/

Thanks !