© Fraunhofer IESE

1

CU! - Continuous Usability mit offline-fähigen Apps

Susanne BraunMatthias Naab

08.03.2016JavaLand 2016

© Fraunhofer IESE

2

Fraunhofer IESE The research institution for software and systems engineering methods

Founded in 1996, headquartered in Kaiserslautern

approx. 240 employees

Our solutions can be scaled flexibly and are suitable for companies of any size

Our most important business areas:

Automotive and Transportation Systems

Automation and Plant Engineering

Health Care

Information Systems

Energy Management

E-Government

© Fraunhofer IESE

3

© Fraunhofer IESE

3

About ACES

The Fraunhofer Approach for Modeling Software and System Architectures

Compiled Best Practices from literature, scaled and tailored for effective architecting in practice

More than 20 years of architecting experiences across domains:Embedded Systems, Information Systems, Smart Ecosystems

ACES –Architecture-Centric Engineering Solutions

© Fraunhofer IESE

4

© Fraunhofer IESE

4

Always Connected

Quelle: http://theoatmeal.com/comics/smartphone

© Fraunhofer IESE

5

© Fraunhofer IESE

6

© Fraunhofer IESE

6

User Experience Expectations of Mobile Users

Quelle: http://theoatmeal.com/comics/smartphone

© Fraunhofer IESE

7

What Mobile Users expect from high quality Apps

Continuous Availability of Mobile Services and Data

Responsive and Polished UIs

Economic Consumption of Battery & Data Plans

Apps need to be offline usable and ship with smart & economic

data synchronization

© Fraunhofer IESE

8

© Fraunhofer IESE

8

Mobile Data Synchronization Classes

One User One Device

One User Multiple Devices

Multiple Users Multiple Devices

Complexity of the Mobile Sync Solution

© Fraunhofer IESE

9

© Fraunhofer IESE

9

Write Conflicts & Lost-Updates

Sync

Sync Read(c) = 100

Write(c) = c+20= 120

Read(c) = 100

Write(c) = c+50= 150

Sync

Sync

Update c += 50 lost

Conflict resolved with Last-Writer-

Wins c = 120

t

Distributed Counter c = 100

c = 120

c = 170

Sync c = 120

Sync c = 150

© Fraunhofer IESE

10

© Fraunhofer IESE

10

ACID & Replication

„Update anywhere-anytime-anyway transactionalreplication has unstable behavior as the workload scales up: a

ten-fold increase in nodes and traffic gives a thousand foldincrease in deadlocks or reconciliations.”

Jim Gray, 1996*

* http://research.microsoft.com/apps/pubs/default.aspx?id=68247

© Fraunhofer IESE

11

© Fraunhofer IESE

11

CAP Theorem*

C onsistency

A vailability PNetworkartition Tolerance

App offering full offline operations

* Eric Brewer

© Fraunhofer IESE

12

© Fraunhofer IESE

12

How to achieve Convergence?

Anti-Entropy

Exchange data versions or update operations between replicas

Reconciliation

Conflict Detection

Conflict Resolution

Highly Application Dependent

© Fraunhofer IESE

13

© Fraunhofer IESE

13

Approaching Mobile Data Synchronization

Mobile Data Synchronization

Data Shipping

Write Set Shipping

…

Operation Shipping

CommutativeOperations Design

Operational Transformation

CQRS Pattern

…

© Fraunhofer IESE

14

Couchbase Mobile

© Fraunhofer IESE

15

© Fraunhofer IESE

15

Couchbase Mobile

Mobile Devicesrunning

Couchbase Lite

Channels

Data Center

Sync Gateway

Couchbase Server / Cluster

Bucket

Bucket

Node Node Node

BucketvBucket

© Fraunhofer IESE

16

© Fraunhofer IESE

16

Couchbase Mobile Channels

User Documents

D1

D2

D3

D4

Channels

© Fraunhofer IESE

17

© Fraunhofer IESE

17

Couchbase Mobile Scalability

Sync Gateway

Couchbase Server / Cluster

Get Changes

Get ChangesChannel 2

Get ChangesChannel 1

[*, 42] D1

[Channel 1, 42] D1

[Channel 2, 42] D1

… …

Channel Changes View

Number ofRequests!

View Growth!

© Fraunhofer IESE

18

© Fraunhofer IESE

18

Couchbase Mobile Conflict Resolution

Automatic and deterministicdetermination of winner version

Without communication overhead

Winner:

Undeleted version with greatest depth in version graph

If more than one: lexigrophic comparison on revision

„Busiest Writer“ wins or arbitrary in worst case

Support for manual conflict resolution

Guarantee that conflicting versions areavailable

No Guarantee that all historic versions areavailabe

AutomaticWinner

ConflictingVersions

© Fraunhofer IESE

19

Zumero

© Fraunhofer IESE

20

© Fraunhofer IESE

20

Zumero

Mobile Devicesrunning SQLite

SQL Conditions

Microsoft Windows / IIS-Plattform

ZumeroServer

MSSQL-Server

App Tables Zumero MetadataTables, Trigger

EntityEntity

© Fraunhofer IESE

21

© Fraunhofer IESE

21

Zumero Conflict Resolution

Support for automatic merge of updateson different entity column values

Database Constraint Violations cannot beautomatically merged

Strategies for conflicting updates on same entity column:

Last-Writer-Wins

First-Writer-Wins

Support for manual conflict resolution

Guarantee that conflicting entities areavailable

Historic entity versions are maintained in metadata tables (unofficial & undocumented)

AutomaticMerge

ConflictResolution Strategy

© Fraunhofer IESE

22

Products & Frameworks

DB Replication

Couchbase Mobile

Zumero

Frameworks

Amazon Cognito

Android Sync Adapter Framework

© Fraunhofer IESE

23

My John Deere Mobile Case Study

© Fraunhofer IESE

24

Important Design Decisions

Consistency Guarantees (What are the desired Consistency Guarantees? How are theyachieved?)*

Update Format (Exchange of data items vs. exchange of update operations?)*

Change Tracking (How to track updates that need to be propagated?)*

Metadata (What metadata is stored and communicated about replicated items?)*

Sync State (What state is maintained at a device for each synchronization partnership?)*

Change Enumeration (How do devices determine which updates still need to bepropagated to which other devices?)*

Communication (What transport protocols are used?)*

Ordering (How to decide on the order in which received upates should be applied?)*

Filtering (How are the contents of partial replicas specified and managed?)*

Conflicts (How are conflicting updates detected and resolved?)*

Synchronization Time (How to decide on the optimal point in time forsynchronization? Impact on battery consumption, data traffic and probability for conflicts.)

* Source: Replicated Data Management for Mobile Computing, Douglas Terry, p. 27

© Fraunhofer IESE

25

© Fraunhofer IESE

25

MJDM Sync Solution Architecture

iOS / Android Devicesrunning

SQLite with ORM

Apache Tomcat

Sync REST

Service

MySQL: Current Entity Data

Flat Objects

Flat Objects

Flat Objects

Spring Context

JPA / Hibernate

Cassandra Java

Driver

Cassandra: Time-Series Data (Historic Entities, Events)

EntityID:YearTimestamp Timestamp Timestamp…

EntityID:YearTimestamp Timestamp Timestamp…

HistoryService

© Fraunhofer IESE

26

© Fraunhofer IESE

26

Change Tracking & Change Enumeration

JPA Entity Base Class• id: UUID• syncState: updated | inserted | deleted | insync• revNr: numeric version

Globally uniqueidentifier

• Track and enumeratechanges on the client

• Mark entities as deletedon client and server

• Version number of theentity

• Managed by Sync Service• Enumerate changes on

the server• Conflict Detection

© Fraunhofer IESE

27

© Fraunhofer IESE

27

Conflict Detection

Client must keep and ship revNr of each entity in the Write Set

Server preprocess Write Set of Client

Server iterates over all entities in Write Set and

Checks if revNr of Client < revNr of Server

Conflict

Server MySQL DB

ID REV …

11

8

3f49f674-….

6d47d45-….

Entity Id: 3f49f674-….revNr: 10

Entity Id: 6d47d456-….revNr: 8

Client Write Set

© Fraunhofer IESE

28

© Fraunhofer IESE

28

Conflict Resolution

Entity Id: 3f49f674-….revNr: 10c

Dirty Client Entity

Entity Id: 3f49f674-….revNr: 10

Entity in version 10

Client Write Set

Cassandra

MySQLEntity

Id: 3f49f674-….revNr: 12c

Merged Entity

Field Changes• …• ….

Field-by-field

comparisonvia

Reflection

Apply to currententity

Entity Id: 3f49f674-….revNr: 11c

Current Entity

• Three-Way-Merge• Last-Writer-Wins (conflicts on the same entity field)

© Fraunhofer IESE

29

© Fraunhofer IESE

29

Two-Phase Protocol

Sync REST

Service

PUT JSON Write Set• Conflict detection• Conflict resolution• Increase sync counter• Update revision number of entities• Update syncState (insync | deleted)• Persist entities• Insert historic entity versions

OK

Sync REST

Service

GET /42• Calculate Write Set • Entities with revision number > 42• Entities either deleted or insync

JSON Write Set

• All entities „insync“• syncState deleted entites are now deleted on client

Atomic

Atomic

© Fraunhofer IESE

30

Caution: Avoid Disadvantageous Data Modelling

Field

Ag Plan

Ag Plans ofField

1

*

Field

Ag Plan

Field of AgPlan

1

*

• Insertion of Ag Plan• = Collection update in Field

• New version of Field• Can conflict with other

Field updates.

• Insertion of Ag Plan• = Insertion of Ag Plan!• -> Cannot conflict with any

other update!

© Fraunhofer IESE

31

Lessons Learned

© Fraunhofer IESE

32

Impact on Data Modelling

© Fraunhofer IESE

33

Impact on Code Complexity

© Fraunhofer IESE

34

The Return of the Fat Clients

in conjunction with Data-CentricArchitectures…

© Fraunhofer IESE

35

Data Validation can become a Show Stopper

© Fraunhofer IESE

36

Write Set Shipping Cries for Generative Approaches

© Fraunhofer IESE

37

Authorization Changes Usually Require a Full Sync

© Fraunhofer IESE

38

Full Sync Usually Requires Special Optimization

© Fraunhofer IESE

39

Historic Data is Valuable

© Fraunhofer IESE

40

Design for Economic ResourceConsumption

© Fraunhofer IESE

41

Design for Great User Experience

is not trivial…

© Fraunhofer IESE

42

“By the end of 2014, the number of mobile-connected devices will exceed the number of

people on earth”1

“Last year’s mobile data traffic was nearly 30 times the s ize of the entire global Internet in 2000”1

Questions?“The Future of Enterprise Applications Is Mobility”2