[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫

Copyright © GREE, Inc. All Rights Reserved.

The Trial and Error in Releasing GREE Chat

Shun Ozaki, Takayuki Hasegawa

Scala Matsuri2014 B-6

GREE's First Scala Product


• Shun Ozaki • @wozaki

• Joined in April, 2013• Android Application

• Takayuki Hasegawa• @hase1031• Joined in April, 2013• NLP, Machine Learning

Self Introduction

2/56


• Released at June 2

GREE Chat

3/56


How to build our system for hundreds of thousands daily users

Agenda

4/56


Share our knowledge we got through the development of GREE Chat

• Why Scala?• Team development• Decision of frameworks• Obstacles & workarounds

Goal

5/56


• Reason of Selecting Scala

• Learning Scala in a Team

• Architecture of GREE Chat

• Obstacles

• Summary

Outline


• Reason of Selecting Scala• Learning Scala in a Team


• Obstacles

• Summary

Outline


Decide the Language based on the Requirement


• Hundreds of thousands daily users

• Real-time response Connect with streaming

• Run on a small number of servers Utilize server resource effectively

• Maintain for over 5 years

Requirements

9/56


◎ GREE uses PHP so heavily• Many libraries and know-how in GREE

△ Streaming• #connections = #processes

△ Single thread, multi process• Overhead of spawning an OS process

△ Maintainability • Dynamic typing

10/56


◎Compatible with concurrent programming

◎One process, many connections

◎High maintainability• Static typing• Functional programming (no side effects)

◎Stimulate new technology learning in GREE• Open to new languages & technologies!

Scala!

11/56



• Learning Scala in a Team• Architecture of GREE Chat

• Obstacles

• Summary

Outline


After adopting Scala …


Only two Scala programmers in a team of seven

Shortage of Scala experts

2／7

14/56


Only two Scala programmers in a team of seven

We must build up our skill!

Shortage of Scala experts

2／7

15/56


1. Self study

2. Study club

3. Pair programming

Learning of Scala

16/56


1. Self Study

Understand the basic syntax


1. Self Study

Books

18/56


1. Self Study

Documents by Twitter

https://twitter.github.io/scala_school/ http://twitter.github.io/effectivescala/ 19/56

https://twitter.github.io/scala_school/





http://twitter.github.io/effectivescala/








1. Self Study

Source code from OSS

20/56


2. Study Club

Share what we learned by ourselves

Introduction to functional programming


• Each engineer solves the problems provided by Scala experts• e.g., Binary Tree, Fibonacci Sequence

2. Study Club

22/56


• Example of factorial generation algorithm

2. Study Club

Discussion with Members

Use var Use recursion

23/56


Use var Use recursion

• Example of factorial generation algorithm

2. Study Club

Discussion with Members

No side effect

24/56


3. Pair Programming

Practice by using the knowledge obtained from study club


• Effective learning• Learn about symbols (e.g., +, @), which cannot

be searched easily on the Internet

• Supplement the knowledge not taught at study club• Complex syntaxes (e.g., implicit, currying)

• Development tools (e.g., sbt, IntelliJ)

3. Pair Programming

26/56


• Reasons of selecting Scala• Compatible with concurrent programming

• Maintainability because of static typing

• Learning of Scala• Self Study, Study Club, Pair Programming

• Big burdens on Scala experts• Needed for teaching team members

• Maintain the quality of codes

Summary: Adopting Scala

27/56




• Architecture of GREE Chat• Obstacles

• Summary

Outline


• Servers separated by Queues• API Server: Process users’ requests• EventBus Server: Process events in Queue• Stream Server: Supply event to connected users

Architecture of Backend (simplified)

29/56


Role of Each Server and Framework


• Enqueue events• e.g., Send message

Join/leave conversation

• Techniques to process a lot of requests• Delegate heavy tasks (e.g., Disk I/O) to others• Logic is written to run asynchronously

• scala.concurrent.Future

API ServerProcess users’ requests

31/56


• RPC system for JVM based on Netty• Support us to write asynchronous logic

• Used by large scale web services• e.g., Twitter, Tumblr, Foursquare, Pinterest

• Equipped with various clients• Redis, Memcached• With retry policy, connection pools

32/56


• Examples of tasks• Logging• Inquire GREE internal system• Store events in DB• etc.

• Problems of concurrent processing• Deadlock, race condition, …

EventBus ServerProcess events in Queue

33/56


Framework to write concurrent, distributed logic more easily

• No need to handle shared resources• Race condition, dead-lock

• Logic separation• Parent-Child

• Fault tolerance

34/56


Example of Akka

35/56


Example of Akka

Match by Event type

36/56


Example of Akka

Send copy to child

37/56


Example of Akka

log

38/56


• Connect by streaming

• Increase connections as much as possible• Keep users data in the memory to reduce I/O• Asynchronous I/O by Finagle, Akka

• Task division by Akka• Supply events to users• Update users data• Send KeepAlive to users

Stream ServerSupply events to connected users

39/56


• Each server is separated by Queue• API, EventBus, Stream Server

• Framework is selected by considering asynchronous processing• Finagle, Akka

• Task is divided by Akka• Easy to write concurrent logic

Summary: Architecture

40/56





• Obstacles• Summary

Outline


1. JVM

• Full GC

2. Self-created Scala library

• Sharding

• ID Architecture

Obstacles

42/56


Full GC Problems


• Scala runs on JVM• We want to prevent “stop the world”

• GC in young generation is faster than full GC• Try not to let it store object in old generation

Full GC

Old generationYoung generation

Eden TenuredSurvivor 0

Survivor 1

44/56


• Use method that has side effects• Reuse object everywhere• Forget to release used resources

Causes of Uncollected Reference Problem

45/56


• Use method that has side effects• Reuse object everywhere• Forget to release used resources

Difficult to recognize by developers

Causes of Uncollected Reference Problem

46/56


• Use val variable (final variable)• Don’t use mutable variable• Don’t leave references to objects for too long

Short-Lived Object

Create new objects instead of updating old ones

47/56


Sharding: Scalable Storage

48/56


Scale up?

49/56


Scale out!

50/56


• Sharding is necessary!• Considering capacity and #accesses

• We have Cascade, a sharding library• For PHP, not Scala• So, we created Aurora and make it OSS

• Aurora and application were developed simultaneously• So difficult to integrate them during

development• Code modification must be done in many parts

Horizontal Partitioning

51/56


Ref ： MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performancehttp://kccoder.com/mysql/uuid-vs-int-insert-performance/

ID Architecture: UUID and InnoDB

http://kccoder.com/mysql/uuid-vs-int-insert-performance/











Ref ： MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performancehttp://kccoder.com/mysql/uuid-vs-int-insert-performance/

UUID

AUTO_INCREMENT #record

#time












• At first, we used UUID as primary key• But this caused MySQL to insert randomly

• Created a library to generate ID instead• Referred to Snowflake by Twitter• Sequential insertion based on sorted created

time

Drop UUID, Take Time-Based

54/56


• JVM• Use short-lived objects to avoid Full GC

• Self-created Scala library• Sharding

• Made Scala library OSS• Integration of library to application was difficult

• ID Architecture• Adopt time-based ID for sequential insertion

Summary: Obstacles & Workaround

55/56





• Obstacles

• Summary

Outline


• Reasons for selecting Scala• Compatible with concurrent programming

• Plan Scala learning within the team• Learning cost is expensive

• Architecture & Frameworks• Each server is separated by Queue

• Finagle, Akka

• Shortage of libraries • We must have made libraries by ourselves• Needed to pull request to OSS repository

Summary

57/56


Thanks!

@tomoyoshi_ogura, @kyo_ago, @takc923

@yoshie_777, @le_chang, @beketa, @j5ik2o

and

ScalaMatsuri staffs

Technology

[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫