Lucas Jellema (AMIS)NLJUG JFall 2013

6th November 2013, Nijkerk, The Netherlands

On the integrity of data in Java Applications

Agenda

• What is integrity?• Enforcing data constraints

– throughout the application architecture• Transactions• Exclusive Access to …• The Distributed World

3

Definition of Integrity

• Truth– Nothing but the truth

• The Only Truth• [Degree of] success or

completeness ofactions is known

4

Sufficient Integrity

IntegrityIntegrity

Corruptible

π48,23

7,0

“five”

4233,0000002

Uncorrupted

CompleteConsistentReliable

Correct

5

Conference Application

6

Conference Application

Client(HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierJPA

RDBMS

EJB

POJO Domain Model

7

Validation at entry time

8

Validation at entry timeClient and View

9

Validation at entry timeClient and View

10

More validation at entry time – bean Validation

11

Validation at entry timeBean Validation in View

12

Engage Bean Validation in Web Tier

13

Record (Type) level rules

• Program should be Kidswhen age < 18; either Developer or Management when age > 18

• Using JavaScript – when either field changes

(handle nulls)– on submit of the entire

record

• Using Bean Validation: custom type validator– in either web-tier or JPA

14

Type Level Constraints with Bean Validation

15

Type Level Bean Validation: Custom Validator

16

Validation Implementation options & considerations

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierJPA

RDBMS

EJB

Mobile ClientClient

(pure HTML 5 & Java Script)

RESTful Services

POJO Domain Model

Native HTML 5; JavaScript

Native

Custom;JSF Validator;

Bean Validation

Custom;Bean Validation

Custom;Bean Validation

Native HTML 5; JavaScript

17

But wait – there is more!

• More User Interfaces• More Attendee

Instances• More Entities

& More types of Constraints

• More Users, Sessions,and Transactions

• More Nodes in the Middle Tier Cluster

• More Data Stores

18

Domain model

• Attendee• Speaker• Session• Room• Slot• Attendance

– Booked– Realized

19

Multiple-Instances-of-Single-Entity constraints

• Constraints that cover multiple same type objects/instances– Attendee’s Registration Id is unique– No more than 5 conference attendees from the same company– Not more than two sessions by the same speaker– At most one session scheduled per room per slot– Only one keynote session in a slot– Sessions from up to a maximum of three tracks can be scheduled in the same room

20

Inter entity constraints

• Attendees can only attend one hands-on session during the conference• A person cannot attend another session in a slot in which the session

(s)he is speaker of is scheduled• No more planned session attendances are allowed than the capacity of

the room in which the session is scheduled to take place• If the room capacity is smaller than 100, then no more than 2 people from

the same company may sign up for it• Attendees from Amsterdam cannot attend sessions in room 010

• Common challenge:– Many data change events

can lead to constraint violation

21

Event Analysis for Inter Entity Constraint

• No more planned session attendances are allowed than the capacity of the room in which the session is scheduled to take place

Create, Update (session reference)

Update (capacity [decrease])

Update (room reference)

22

Constraint classification

• Based on event-analysis (when can the constraint get violated) we discern these categories of contraints– Attribute– Tuple– Entity– Inter Entity

• Each category has its own implementation methods,options and considerations– Multi record instance rules cannot

meaningfully be enforced in client/web-tier

23

Nous ne sommes pas ‘Sans Famille’

24

Nous ne sommes pas ‘Sans Famille’

RDBMS

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierJPA

EJB

Mobile ClientClient

(pure HTML 5 & Java Script)

RESTful Services

POJO Domain Model

25

Multiple clients forData Source

RDBMS

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierJPA

EJB

Mobile ClientClient

(pure HTML 5 & Java Script)

RESTful Services

POJO Domain Model

ESB.NET

BatchDBA/

Application Admin

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierJPA

EJB

Mobile ClientClient

(pure HTML 5 & Java Script)

RESTful Services

POJO Domain Model

26

Integrity Enforcement in the Persistent Store

• All data is available• Persistent store is the final stop: the buck stops here

– Any alternative data manipulation (channel) has to go to the persistent store– Mobile, Batch, DBA, ESB

• Built-in (native) mechanisms for constraint enforcement– Productive development, proven robustness, scalable performance– For example:

Column Type, PK/UK, FK, Check; trigger

• Transactions• Enforcing integrity is integral part of persisting data

– Without final validation, persistent store cannot take responsibility for integrity

27

Multiple-Instances-of-Single-Entity constraints

• No more than 5 conference attendees from the same company

28

Implementation Consideration for Multiple-Entity-Instance rule

• Implementation – how and where?– Is the entire set of data available– Is all associated info available– Is the data set stable?– Can the constraint elegantly be implemented (natively? good framework support?)– Are all data access paths covered?

29

Implementing Multi-Instance constraint ‘5 max per company’

Business TierJPA

Register New Attendee – method A- Ensure L2 Cache is up to date in terms of Attendees (fetch all attendees into cache)- Inspect the collection of attendees for same company- Persist Attendee if collection does not hold 5 (or more)

POJO Domain Model

Attendees

Register New Attendee – method B- Select count of attendees in same company from the Data Store- Inspect the long value- Persist Attendee if long is < 5

L2 CacheAttendees

30

Max 5 per companyJPA Facade enforcement

31

Max 5 per Company – Flaws in JPA Enforcement

• Persist does not [always] ‘post to database’– When more than one attendee is added in a transaction, prior ones are not counted

when the latter are validated

Business TierJPA

Attendees

Facade

POJO Domain Model

Thread 1select countpersistselect countpersist

32

One thread persisting two attendees in a row – no flush

33

Max 5 per Company – Flaws in JPA Enforcement

• Persist does not [always] ‘post to database’– When more than one attendee is added in a transaction, prior ones are not counted

when the latter are validated

Business TierJPA

Attendees

Facade

POJO Domain Model

Thread 1select countpersistselect countpersistcommit

34

Flush after persist for complete picture

35

Web Tier

ClientHTML 5 & Java Script

Session A

JPA Facade enforcement in a multi-threaded world

Business TierJPA

Attendees

Facade

POJO Domain Model

Thread 1 Thread 2select countpersist

select countpersist

ClientHTML 5 & Java Script

Session B

36

Web Tier

ClientHTML 5 & Java Script

Session A

JPA Facade enforcement in a multi-threaded world

Business TierJPA

Attendees

Facade

POJO Domain Model

Thread 1 Thread 2select countpersistcommit

select countpersistcommit

ClientHTML 5 & Java Script

Session B

37

Two threads inter-leaving

38

Database Solution?

39

Data Trick – Materialized View with Check Constraint

40

Transactions

• Logically consistent set of data manipulations– Atomic units of work– Succeed or fail together– Any changes inside a transaction are invisible to other sessions/transactions until the

transaction completes (commits)– Note: during a transaction, constraints may be violated; the only thing that matters:

commit [time]– Transaction ends with succesful commit or rollback –

In both cases, transaction-owned locks are released

• ACID (in RDBMS)– vs BASE (in NoSQL)

• Note: post vs. commit with RDBMS– Post means do [all] data manipulation (insert, update, delete) but do not commit [yet]– Only upon commit are changes persisted and published

41

Perfect Integrity

42

Fine grained locking

Attendees

Unique Key UK1 on (FirstName, LastName)

Transaction 1 Transaction 2

insert … ('John','Doe',…)

43

Fine grained locking

Attendees

Unique Key UK1 on (FirstName, LastName)

Transaction 1 Transaction 2

insert … ('John','Doe',…)

update <JANE> set firstname ='John'

insert … ('Jane','Doe',…)

44

Fine grained locking

Attendees

Unique Key UK1 on (FirstName, LastName)

Transaction 1 Transaction 2

insert … ('John','Doe',…)

update <JANE> set firstname ='John'

commit

insert … ('Jane','Doe',…)

Lock onUK1_JOHN_

DOE

45

Web Tier

ClientHTML 5 & Java Script

Session A

JPA Facade enforcementExclusive Constraint Checking

Business TierJPA

Attendees

Facade

POJO Domain Model

Thread 1 Thread 2take lockselect countpersistcommit

take lock…select countrollback

ClientHTML 5 & Java Script

Session B

LockMgrATT_MAX

46

Two threads and Lock on Constraint

47

Two threads and Lock on Constraint

48

Distributed or Global Transaction

• One logical unit of work - involving data manipulations in multiple resources (global transaction composed of local transactions)

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business Tier

RDBMS

EJB

Mobile ClientClient

(pure HTML 5 & Java Script)

RESTful Services

POJO Domain Model

RDBMSJMS

ERPJCA

49

Implementation for Distributed Transaction

• Typical approach: two-phase commit– Each resource locks and validates – then reports OK or NOK back to the transaction

overseeer– When all resources have indicated OK

then phase two:all resources commit and release locks

– When one or more resources signal NOK, then phase two: all resources roll back/undo changes and release locks

• With regards to integrity:– With a distributed transaction,

the integrity for each participant is handled as before; this will result in ‘constraint-locks’ in multiple separate resources

50

Distributed (aka global) transaction inside container

• Java EE containers (and various non-EE JTA implementations) support global (distributed) transactions within a JVM– JTA (JSR-907) – based on X/Open XA architecture

• Key element is Transaction Monitor (the container) and Resource Managers (JDBC, EJB, JMS, JCA)

• One non-XA resource can participate (file system, email, …) in a global transaction:– All XA-resources perform Phase One – The non-XA resource does its thing– Upon success of the non-XA resource: others perform Phase two by comitting– Upon failure of the non-XA resource: others roll back

51

Distributed transactions across/outside containers

Step 2:Payment

RDBMS

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierJPA

EJB

Mobile ClientClient

(pure HTML 5 & Java Script)

RESTful Services

POJO Domain Model

52

Container

Distributed transactions across/outside containers

• Transaction involving remote containers, Web Services, File System or any stateless transaction participant

• There is no actual common, shared vehicle (like a global XA transaction)– There is not really a coordinated two-phase commit

• Transaction consists of – Any resource does its thing – lock, validate, commit (or rollback), report back– If all resources report succes: great, done– If one resource reports failure the all other resources should perform ‘compensation’

– i.e. rollback/undo effects of a committed transaction

Remote/Stateless Enterprise Resource

Remote/Stateless Enterprise Resource

LocalEnterprise Resource

Transaction

commit

compensate

commit

53

Compensation

• How to implement a compensation mechanism?• How long after the commit can compensation be requested?• What is the state of the enterprise resource between commit and the

compensation expiry time?• Should the invoker notify the resource that compensation is no longer

required (so the ‘logical locks’/’temporary state’ can be updated)– i.e. the global distributed transaction has succussfully completed

Enterprise Resource

commit

compensate

54

RESTful transaction is a distributed transaction

Resource A Resource B Resource C

Client

PUT

PO

ST

DELETE

Domain Model/JPA Cache

55

RESTful transaction is a distributed transaction

Resource A Resource B Resource C

Client

PUT

PO

ST

DELETE

Domain Model/JPA

56

Distributed Constraints

• Constraints that involve data collections in multiple enterprise resources

RDBMSRDBMSJMS

Client(JSF based HTML 5 & Java Script)

Web TierJavaServer Faces

Business TierEJB

Mobile ClientClient

(pure HTML 5 & JS)

RESTful Services

POJO Domain Model

ERPJCA

Table XTable Y

57

Distributed Constraints

• Not more than three attendees (resource A) from the same company may attend a session (resource B)– Insert/Update Attendance requires validation – as does update of Attendee.company

Client

Web Tier

Java EE Business Tier

Client Client

ATTENDANCESATTENDEES

Distributed Lock Manager

Web Tier

Java EE Business Tier

MAX_3_COMP_ATT

58

Distributed Constraints

• Not more than three attendees (resource A) from the same company may attend a session (resource B)– Insert/Update Attendance requires validation – as does update of Attendee.company

Client

Web Tier

Java EE Business Tier

Client Client

ATTENDANCESATTENDEES

Distributed Lock Manager

Web Tier

Java EE Business Tier

MAX_3_COMP_ATT

59

Distributed Constraints

• Not more than three attendees (resource A) from the same company may attend a session (resource B)– Insert/Update Attendance requires validation – as does update of Attendee.company

Client

Web Tier

Java EE Business Tier

Client Client

ATTENDANCESATTENDEES

Distributed Lock Manager

Web Tier

Java EE Business Tier

MAX_3_COMP_ATT

ESB

61

Java global (distributed) lock managers

• Within JVM: SynchronousQueue• Across JVMs: Apache ZooKeeper, HazelCast, Oracle Coherence, …

JVM

JVM

JVM

62

Summary

• Which level of integrity is required?• Change undermines integrity

– Data change is trigger for constraint validation

• Exclusive lock on multi-record validation– released when transaction commits

• Ensure that all data access paths are covered– Not all data manipulations may come through the Java middle tier

• Transactions may include multiple enterprise resources– That may not be able to participate in a distributed transaction and have to support a

compensation mechanism

• True integrity and real robustness are very hard to achieve– Much harder than is commonly assumed

64

Handling Integrity Really Well...

Lucas Jellema (AMIS)

Email: lucas.jellema@amis.nlTwitter: @lucasjellema

Blog: http://technology.amis.nlWebsite: http://www.amis.nl