A Amadeus Journey to Oracle Grid Infrastructure 11gR2 A · PDF fileDifferent migration windows ... VMax 2 0 1 1 A m a d e u s I T G r ... HPUXv3 or RHEL 5 1A lib Oracle Clusterware

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, Better

© 2

011

Amad

eus

IT G

roup

SA


1

Amadeus Journey toOracle Grid Infrastructure 11gR2

© 2

011

Amad

eus

IT G

roup

SA


2

Edgar Canaan System Programmer

Franz Fischer Project Manager

Vitor Pacheco Manager

Nicolae Hachi DBA Manager

© 2

011

Amad

eus

IT G

roup

SA


3 2010 movie following the well known internet series “ Did you know? …”

© 2

011

Amad

eus

IT G

roup

SA


4

The largest transactionprocessor in travel

Common / overlapping platforms & applications

Common data centre

Common customers

Common sales & marketing infrastructure

DISTRIBUTION BUSINESS

Provision of indirect distribution

services

713 airlines (over 436 bookable)

116 Insurance companies

50+ cruise and ferry lines

204+ tour operators

87,000+ hotel properties

29 car rental companies

103 railways

Travel providers

Consumers/General public

Corporate traveldepartments

Travel buyers

Travel agencies

Travel Management companiesBusiness travel agenciesLeisure travel agenciesOnline travel agencies

Consolidators Single-site agency

Travel search companiesAirline sale offices and airline websites

connected to Amadeus direct sell technology

IT SOLUTIONSIncluding direct distribution

technology

Transaction-based business model Operate globally in the growing travel and technology market Two highly synergistic and profitable businesses: Distribution and IT Solutions

© 2

011

Amad

eus

IT G

roup

SA


© 2

011

Amad

eus

IT G

roup

SA


5

© 2

011

Amad

eus

IT G

roup

SA


6

© 2

011

Amad

eus

IT G

roup

SA


Amadeus technology is present across the entire journey

7

Traveler checks-in and receives boarding pass

Traveler receives notification via mobile that return flight schedule has changed

Action Travel agency / traveler searches for best air fare and hotel content

Traveler needs to change their travel date after ticket has been issued

Travel agency/ Traveler makes air and hotel bookings and issues air e-ticket/ hotel voucher paying with credit card

Altéa Inventory

Altéa Departure Control System

Solutions for mobile applications

Amadeus technology

Availability & shopping technology

TA front office Altéa Reservation Revenue maximization

(dynamic availability, journey data)

Self-booking tool for corporations

Ticket Changer GDS booking

Altéa e-commerce

E-ticketing & voucher

Credit card authorization

Sales management solution

Revenue integrity

JourneyTrip planning & pricing

Booking & ticketing On the moveCheck-in & boardingChange ticketing Post-trip

Traveler completes travel expense report

Travel agency issues invoice and reconciles travel to invoiced data

Travel provider / travel agency conducts revenue management and accounting activities

© 2

011

Amad

eus

IT G

roup

SA


© 2

011

Amad

eus

IT G

roup

SA


8

UXs3 ProjectUniX stack 3 Qualification

© 2

011

Amad

eus

IT G

roup

SA


Objectives

Validate and introduce a new software stack (s3/s4) for the Oracle DB tier Operating System:

HP-UX V3 – s3 RedHat 5.7 – s4

Oracle GI (ASM & Oracle Clusterware 11gR2) Database Engine: Oracle 11gR2

RAC (services Active/Passive) – s3 RAC One Node – s4

Migration of all existing Oracle Databases to the new stack in both RAC and RAC One Node configurations

Prepare foundation for future DB Tier platform on Linux x-86

9

© 2

011

Amad

eus

IT G

roup

SA


Project Motivation

Reduce the number of vendors & complexity of two cluster layers (Oracle and Symantec)

Avoid resulting problems with validation, integration, patching, upgrading, change / problem & incident management

Cost avoidance by reducing Symantec licensing costs Oracle product lifecycle compliance (10.2.0.3 off prime support) Oracle announcement for discontinuation of support of Itanium Green IT (reduce growth of Data Center power & cooling

requirements)

10

© 2

011

Amad

eus

IT G

roup

SA


11

Scope

Validation of the new s3/s4 stack Migration strategy customized for different use cases

RAC, RAC One Node, HP-UX, Linux etc. Different migration windows - 15 minutes or 30 minutes

Integration of the new s3/s4 stack with spread & thin provisioning technologies Address increasing storage costs

Train operational & systems teams on Oracle 11gR2 GI Ensure operational readiness of support processes for Oracle

11gR2 GI

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, BetterTimeline overview

12

Migration Phase HP-UX -> HP-UX

Migration Phase HP-UX -> Linux RAC One Node

Migration Phase HP-UX -> Linux RAC

System Qualification

Funct.Qual. Part1

Funct.Qual. Part2

Funct.Qual. Part2

Migration Phase Linux -> Linux RAC One Node

IP NW Resiliency

Server Disk Access & SAN

DB RAC Server HA

MW Porting to 11g

GG PoC

OAQ’s dependency removal /Extension of the migration process

RAT

Patch Strategy

Backup

RDBMS Feat. Eval.

Test Refresh

APF

SecurityConcept

DBMonitoring

Migr. Strategy

ZBO

ADPStats

OEM GC

PlanStability

DR Replication

EHA

May11

Jan11 UXs3 Project

May10

Sep10

Sep11

Jan12

May12

Sep12

Jan13

May13

1st D

eploym

ent

1st M

igration

© 2

011

Amad

eus

IT G

roup

SA


Main challenges

Short time frame 32 months for validation and

migration ~ 70 production Databases ~ 8 test databases for each Prod DB

Expectations Every team has

3 different wishes, 3 different constraints

End customer base distributed across the globe

Development vs. Operations targets SW functionality vs. high availability

and stability Marketing milestone

Alignment with ongoing Projects Technical Customer driven

Capacity Requirements Double capacity during migration Budget constraints HW Delivery and provisioning

processes

13

© 2

011

Amad

eus

IT G

roup

SA


© 2

011

Amad

eus

IT G

roup

SA


14

System Infrastructure

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, BetterUNIX OBE –May 2011 global figures

Application: 83 applications (with 15 being DB

management related) 569 Oracle DBs 105 MySQL DBs

Cluster: 63 clusters 764 Service Groups 6333 Resources

Servers: 25 SuperDomes 82 HP-UX partitions 71 Linux servers (rack mounted)

Storage (estimated growth of 4TB/day): 53 disk arrays 650 TB of Filesystem and ASM

volumes 1.1 PB of DG 2.1 PB of raw storage (estimation

considering we are mainly R1 and adding BCV/clone space)

10K LUNs

15

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, BetterSystems Infrastructure – building blocs

16

Oracle11gR2 GI

HPUXv3or

RHEL 5…

1A lib

Oracle Clusterware

S-Checkmonitoring

EMC DMX 4 or VMax

XiV

1A libASM ASM

MPIOOS

MPIOOS

node 1 node n

VOMSRM

VCSOne‘orchestration framework’

10 GbE

FC

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, BetterSystems Infrastructure – RAC One Node scale-out approach

17

© 2

011

Amad

eus

IT G

roup

SA


Future DB platform – IB with RDS and iSer for low latency transport

18

© 2

011

Amad

eus

IT G

roup

SA


1Alib highlights

Making sure Storage, Systems and DBA teams refer to the same storage component.

Quickly troubleshooting hot spots is cumbersome without meaningful name common to all three groups (DBA, UNIX, SAN) when issues arise.

Provisioning Storage safely with an enterprise unique schema based on World Wide Identifier (WWUID)

Repurposing LUNs can be a risky task. Interaction with DBAs and SAN Administrator a must.

19

© 2

011

Amad

eus

IT G

roup

SA


1Alib:: discover

20

© 2

011

Amad

eus

IT G

roup

SA


1Alib:: listall

21

© 2

011

Amad

eus

IT G

roup

SA


1Alib:: iostat

22More 1Alib command examples in the Backup Slides section

© 2

011

Amad

eus

IT G

roup

SA


1Alib:: free

23

© 2

011

Amad

eus

IT G

roup

SA


1Alib:: showzone

24

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, BetterSystems Feedback

Still no RHEL 6 support. No thin provisioning awareness. Lack of official white papers with architecture and concept

documentation. No possibility for 3rd party off host mirroring integration like EMC TF

BCV/CLONE. No specific performance report 'a la AWR' for GI components (also

AWR should report in microseconds and not milliseconds for I/O related statistics)

Patching is complex and lengthy. No POSIX conformant interface.

25

© 2

011

Amad

eus

IT G

roup

SA


2626

Database

© 2

011

Amad

eus

IT G

roup

SA


Application flows

Application access to the DB performed through Amadeus own middleware Simple validation of Middleware

Client upgrade decoupled from the Server upgrade We maintain 10gR2 client for 2 months past the migration

Plan stability enforced through baselines Depends on plan completeness in the source environment Attention to deprecated hints in 11g – i.e. “ordered”

Application specific qualification done as part of the pre-migration validation Qualification to start up to 3 months prior to migration date

27

© 2

011

Amad

eus

IT G

roup

SA


Backup and Recovery10g storage layout

28

© 2

011

Amad

eus

IT G

roup

SA


Backup and RecoveryCurrent backup solution

RMAN with Flashback Recovery Area Integration with Backup server

Tivoly Data Protection for Oracle Complemented by

Data Guard in Maximum Performance Mode Second DB in primary site

Data Guard in Archiver transport mode Disaster Recovery

EMC Clone Off host backup solution Not available in the new 11gR2 environment

Alternative being studied29

© 2

011

Amad

eus

IT G

roup

SA


Amadeus Test Systems

30

OBO

MIG

OBO clone

PDT

UAT SKL

OBO clone

RMAN restore

Data guardPPT

DES

DEV

FVT

RMAN restore PHY STBY

BTPRD QRT

QAT

AP1W

W

3*D 3*D

4W4W4W

8W 8W8W

W

OnReqOnReq

Split mirror from OBO


Full disk image

Clone (copy on write)

OnReqInSyncWithPRD

© 2

011

Amad

eus

IT G

roup

SA



31

OBO

MIG

OBO clone

PDT

UAT SKL

OBO clone

RMAN restore

Data guardPPT

DES

DEV

FVT


BTPRD QRT

QAT

AP1



Full disk image


Up to 10 Test Systems per Application Virtualization as a mean to minimize Storage investments Short outage windows Large DB volume – up to 12 TB

Sum of all Test: 1.2 PB Storage snaps on write not yet available in ASM Symantec Volume Manager will be maintained in most of the Test

environments

© 2

011

Amad

eus

IT G

roup

SA


ASM

Symantec Volume Manager


32

OBO

MIG

OBO clone

PDT

UAT SKL

OBO clone

RMAN restore

Data guardPPT

DES

DEV

FVT


BTPRD QRT

QAT

AP1



Full disk image


© 2

011

Amad

eus

IT G

roup

SA


© 2

011

Amad

eus

IT G

roup

SA


33

Migration Scenarios

© 2

011

Amad

eus

IT G

roup

SA


10g-11g Migration alternatives

Data Guard Positives:

Robust technology Used in the 9i-10g migration Extensive experience Process industrialized Consistency confidence

Negatives Oracle 11gR2 does not install in HP-

UX V2 No fallback possible after migration

Upgrade duration in the range of 15+ min

Not possible cross endian Data dictionary persistency No reorganization possible (soft)

GoldenGate Positives

Cross platform No need to upgrade during outage

window = short outage window Physical reorganization is possible

Negatives Complex and effort driven Technology bugs Difficult to industrialize Consistency between the two DB’s

needs to be evaluated regularly Licensing costs

34

1 2

© 2

011

Amad

eus

IT G

roup

SA


10g-11g Migration alternatives

Functional migration Positives

No down time: products migrated one by one

No extra license costs Application exposed to 11g

progressively No big-bang approach

Negatives Development effort (very hard) Long coexistence of 10g and

11g (soft)

Data Pump Positives

Rehearse effort is small Simple process

Negatives Long duration for DB’s larger

than 10GB Constraint can be addressed

using low latency technology

35

3 4

© 2

011

Amad

eus

IT G

roup

SA


36

Amadeus Planned Outages

Amadeus classifies its services in 3 different categories Mission Critical Application

Target: continuous availability Planned outages: aligned to Quarterly Outage Windows (QOW)

Mission Critical for Reservations Target: 99.99% availability Planned outages: QOW – quarterly 15 min

Other Target: 99.95% availability Planed outages: Monthly Maintenance Window – monthly 30 min

© 2

011

Amad

eus

IT G

roup

SA


37

Migration strategy

Linux-to-Linux Data Guard Migration

HP-UX to Linux Option 1: Data pump, if respective outage window is acceptable Option 2: GoldenGate

HP-UX to HP-UX Option 1: GoldenGate – in particular to MCA’s Option 2: Data Guard – if no fallback is acceptable

1

24

21

© 2

011

Amad

eus

IT G

roup

SA


Golden GateMain Challenges

Activation of Supplemental Logging on source DB Dealing with ‘resource busy’ situations for highly accessed tables Additional amount of generated redo

Handling of unsupported data types Ex. dedicated handling of Oracle Advanced Queues

Limitation of effort / risk related to DDL replication Avoiding of DB Loads during active GG processes Disabling of batch jobs during active GG processes

Awareness and/or handling of all data model ‘specialties’, like Unused columns, Triggers, Cascading deletes, Missing Primary / Unique

Key Dedicated GG configuration needed per application

Complex qualification: 4 test runs for each Application Long preparation and test phase - 8 to 12 weeks duration

A series of new builds were required to fix an IOT related bug

38

© 2

011

Amad

eus

IT G

roup

SA


39

Golden GateTuning Challenge

Special performance based GG configuration required for DBs with high data modification rate Parallel GG processes for

Identification of independent sets of tables

Splitting table replication ondata range level

Various GG tuning parameters DB based tuning

(e.g. creation of unique keys)

Veridata Configuration Comparison of large amount of data in a reasonable time

and with a limited amount of resources Comparison of databases with on-going data changes

© 2

011

Amad

eus

IT G

roup

SA


Conclusions 1/2

40

11gR2 RDBMS part

Not a major transition Good toolkit

Grid Infrastructure Major evolution from previous Oracle versions Toolkit not yet there

IO Performance metrics not fulfilling our expectation

Migration is the biggest challenge High effort demand in the DBA teams

UXs3 Project Project extended to the end of 2012 New stacks included

S5 – RedHat 6.x, GI 11gR2, RAC (services Active/Passive) S6 – RedHat 6.x, GI 11gR2, Low latency, RAC (services

Active/Active)

© 2

011

Amad

eus

IT G

roup

SA


Conclusions 2/2

41

What we have achieved until now New stack s3 and s4 qualified

S3 completed by May 2011 S4 completed by August 2011

4 Production DB’s migrated to the s3 stack 2 migrations using Golden Gate 3 migrations achieved in less than 10 minutes

5 new DB’s deployed natively in the new s3 stack S4 platform ready for Production deployment

Ongoing Application migration qualification First migrations to the s4 stack planned for the 13th of

November

© 2

011

Amad

eus

IT G

roup

SA


Special thanks

Oracle RAC Pack Team Martina Keippel Sanjay Singh Erich Kreisler

Oracle ACS Florian Zahneisen and his

team Oracle Germany

Klaus Thielen Wolfgang Thiem Rogee Biondi

42

© 2

011

Amad

eus

IT G

roup

SA


43

© 2

011

Amad

eus

IT G

roup

SA

Brighter, Bolder, BetterQuestions?

Documents

A Amadeus Journey to Oracle Grid Infrastructure 11gR2 A · PDF fileDifferent migration windows ... VMax 2 0 1 1 A m a d e u s I T G r ... HPUXv3 or RHEL 5 1A lib Oracle Clusterware