Taste Java in the Clouds

朱仲傑Jacky Chu

Agenda

• Evolution of computing with the network

• What is Cloud Computing

• Cloud Computing in the world

• How to develop a cloud application

• Hadoop Overview

• Your First Cloud Program

• Who needs Cloud Computing

Evolution of Computing with the Network

Internet Services around the World

What is Cloud Computing

松下問童子，

言師採藥去。

只在此山中，

雲深不知處。

尋隱者不遇

唐賈島

Cloud Computing is …

• Services and data in the cloud

• Accessible with any device

• Connected to the cloud with a browser

The Key Technical Issue for Developers is …

Cloud vs. Grid

Cloud Computing Grid Computing

Leading Service Provider Academic Circles

Standard No Yes

Open Source Partial Fully

Hardware PC Heterogeneous

Characteristic Small data for each processA large number of repeat process

Huge data for once process

Cloud Computing in the World

How to Develop a Cloud Application

MapReduce

Storage

Database

http://www.gridgainsystems.com/wiki/display/GG15UG/MapReduce+Overview

MapReduce

DataData

(Key1, Value)(Key2, Value)

…(Key n, Value)

(Key1, Value)(Key2, Value)

…(Key n, Value)

(Key1, Value)(Key3, Value)

…(Key m, Value)

(Key1, Value)(Key3, Value)

…(Key m, Value)

(Key2, Value)(Key3, Value)

…(Key n, Value)

(Key2, Value)(Key3, Value)

…(Key n, Value)

(Key1, Value)(Key2, Value)(Key3, Value)

…(Key m, Value)(Key n, Value)

(Key1, Value)(Key2, Value)(Key3, Value)

…(Key m, Value)(Key n, Value)

Map Reduce

Hadoop History

Doug Cutting

Hadoop Overview

HDFS

Map Class

• Extends MapReduceBase

• Implements Mapper– <K1, V1, K2, V2>– <WritableComparable, Writable,

WritableComparable, Writable>

• map– K1, V1– OutputCollector<K2, V2>– Reporter

Key Class

• org.apache.hadoop.io.WritableComparable

• BooleanWritable, BytesWritable

• IntWritable, LongWritable– VIntWritable, VLongWritable

• FloatWritable

• Text, UTF8

• NullWritable

Value Class

• org.apache.hadoop.io.Writable

• Super Interface of WritableComparatable

• ArrayWritable, MapWritable

• ObjectWritable

Reduce Class

• Extends MapReduceBase

• Implements Reducer– <K2, V2, K3, V3>– <WritableComparable, Writable,

WritableComparable, Writable>

• reduce– K2, Iterator<V2>– OutputCollector<K3, V3>– Reporter

Job Configuration

• org.apache.hadoop.mapred.JobConf• new JobConf(Class)• setJobName(String)• setMapperClass(Class <? extends Mapper>)• setCombinerClass(Class <? extends Reducer>)• setReducerClass(Class <? extends Reducer>)• setInputFormat(Class <? extends InputFormat>)• setOutputFormat(Class <? extends OutputFormat>)

Input/Output Format

• org.apache.hadoop.mapred.*

• FileInputFormat– setInputPath(JobConf, Path...)– The array of Paths– org.apache.hadoop.fs.Path

• FileOutputFormat– setOutputPath(JobConf, Path)

Run Job

• JobClient– runJob(JobConf)

DEMO

Who needs Cloud Computing

• ASP (Application Service Provider)

• SaaS (Software as a Service)

• PaaS (Platform as a Service)

The Key is …

References

• Google AppEngine– http://code.google.com/appengine

• Hadoop– http://hadoop.apache.org

• Amazon Web Services– http://aws.amazon.com

• Windows Azure– http://www.microsoft.com/azure/default.mspx