Anil nair rac_internals_sangam_2016

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Oracle Real Application Clusters (Internals)

Anil NairSr. Principal Product Manager, Oracle Real Application Clusters (RAC) Nov 12th, 2016

@Amnair, @OracleRACpm http://www.linkedin.com/in/anil-nair-01960b6http://www.slideshare.net/AnilNair27/

15 Years of Performance Innovations

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. 3

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


• Out-of-box support for major applications• e.g. Oracle Apps, Siebel, SAP, Peoplesoft, etc.

• Unprecedented Scalability • Add nodes as needed for linear scalability

• Seamless integration with other options:• RAC + Data Guard provide site availability • RAC + Multitenant provide availability and

scalability for consolidated environments• RAC + IMDB provide availability

and scalability for DSS environments• **RAC + IMDB + Reader Nodes provide even more

flexibility when it comes to scaling your workload

4

Without Application Code Changes!#1 Choice for Scalability & Availability

✔


• Clusterware– Cluster Domain Architecture

• Automatic Storage Management– Flex Disk Group – Create Database clones

• Autonomous Health Framework

Announcing Oracle Database 12c Release 2 on Oracle Cloud

Oracle is presenting features for Oracle Database 12c Release 2 on Oracle Cloud. We will announce availability of the On-Prem release soon.

Oracle Real Application Cluster Family of Solutions


Program Agenda15 Years of Innovations

RAC Scalability & Availability Optimizations

Automated for You

Optionally tunable for your environment

New Features Not to Miss

Appendixes

Step by Step Upgrade GI from 12.1 to 12.2

12.2 Installer New Features in Action

1

2

3

A

B




Automated for You



Appendixes



1

2

3

A

B


Oracle RAC Evolution

8

Oracle 6 to Oracle8i

OPS(199x)

Oracle Database 10g

RAC (2004)

Grid Computing,Oracle Clusterware

Oracle Database 11g

Rel. 1 with RAC (2007)

Engineered Systems

Oracle9i RAC (2001)

RAC debuts

Oracle Database 11g


Oracle RAC

One Node A

DB A

DB A

RAC One Node

Oracle Database 12c


Multitenant

Oracle Confidential – Internal


15 Years of Innovations

• Oracle Clusterware – Cluster Domain Architecture• Automatic Storage Management• Autonomous Health Framework• Oracle Real Application Cluster

Leading to Oracle 12c Release 2



• Oracle Clusterware - Cluster Domain architecture• Automatic Storage Management• Autonomous Health Framework• Oracle Real Application Cluster



Domain Services Cluster

Cluster Domain

ASMIO Service

ACFS Services

ASM Service

DatabaseMember Cluster

Uses ASM Service


Uses ASM IO Service of DSC

Trace File Analyzer

(TFA) Service

Mgmt Repository

(GIMR) Service

Application Member Cluster

GI only


Uses local ASM

Shared ASM

Additional Optional Services

Rapid Home Provisioning

(RHP)Service

Private Network

SAN Storage

ASMNetworkStorage

1 2 3 4






Database Oriented Storage ManagementOracle 12c Release 2Current Organization

• Flex Diskgroups enable File Groups

• A File Group is the set of files belonging to database or PDB

• A File Group’s name defaults to the database or PDB name


Flex Diskgroup

1. Quota Groups provide means of enforcing quota management

2. Modifiable redundancy at the File Group level

3. Shadow copies of File Groups can be created and split off






Oracle 12c Release 2AutonomousHealthFramework

Cluster Verification

Utility

ORAchk

Cluster Health

Monitor

Cluster Health Advisor

Trace File Analyzer

Hang Manager

Memory Guard

Quality of Service

Management

Confidential – Oracle Restricted




Automated for You



Appendixes



1

2

3

A

B


Cache FusionA brief Refresher

• Maximum 3-way communication

• Dynamic Resource Management (DRM) attempts to optimize down to 2-way communication by moving the master to the instance where the resource is frequently accessed


Cache FusionAutomatically chooses Optimal path

• Cache fusion collects & maintains statistics on Private Network access time and Disk access time

• Cache fusion will use this information to find the optimal path Network or Disk to serve blocks

• E.g. Flash Storage may provide better access times to data than the Private Network under high load.


Performance OutliersHard to find cause


Introducing LMS CR SlavesWill help mitigate Performance outliers

• In previous releases, LMS work on incoming consistent read requests in sequential fashion

• Sessions requesting consistent blocks that require applying lot of undo may cause LMS to be busy

• Starting with Oracle RAC 12c Release 2, LMS offloads work to ‘CR slaves’ if the amount of UNDO to be applied exceeds a certain, dynamic threshold

• Default is 1 slave and additional slaves are spawned as needed


• OLTP sessions require remote undo header lookups to find– If a transaction has committed– Block cleanouts

• To reduce remote look ups, each instance maintains a hash table of recent transactions (active & committed)

• Undo Header Hash table improves scalability by eliminating remote look ups

Introducing Undo Header Hash Table

22


Provides consistent performance after planned service failoverOptimized Singleton Workload Scaling

Service-oriented Buffer Cache Access determines the data (on database object level) accessed by the service and masters this data on the node on which the (singleton) service is offered, which improves data access performance.


• Cache Fusion maintains a Service to Buffer Cache relationship– Tracks which service causes row(s)

to be read into the buffer cache

• This statistic is used to– Master the resource only on those

nodes where the service is active• Optimized “Resource Master” Dispersion

– Pre-Warm the cache during service failover amid planned downtime

Service-Oriented Buffer Cache Access

24

NodeAOracle GI

Oracle RAC

NodeBOracle GI

Oracle RAC

cons_1 cons_2


Chatty PDB(s) does not affect other PDB performancePluggable Database and Service Isolation

Pluggable Database and Service Isolation improves performance by reducing DLM operations for PDBs / Services not offered in all instances and optimizing block operations based on in-memory block separation.


• Single Domain (domain 0) at the CDB level for all PDBs

• Operations like PDB start, stop, reconfigurations etc rely on the single Domain

• Accessing a resource from hash table using single Domain affected as the number of PDBs increases (> 250) Pdb-1

Pdb-2Pdb-3Pdb-4

Domain 0

Instance SGA

Resource Hash Table

Oracle Multitenant 12c Rel. 1 Scalability Implementation


• With Oracle Database 12c Rel. 2, Each PDB gets its own PDB specific domain ID

• GES & GCS resources are balanced across PDB-specific domain ID

• Resulting in improved, consistent performance– One chatty PDB will not affect the

performance of any another PDB – Reduced Hash table size as hash table is only

created on the instances where PDB is running

Oracle Multitenant 12c Rel. 2 Scalability Optimization

Domain1Resource Hash Table

Resource Hash Table

Resource Hash Table

Resource Hash Table

Domain 2

Domain 3

Domain 4


Program Agenda15 Years of Performance Innovations

Scalability & Availability Optimizations

Automated for You



Appendixes



1

2

3

A

B


Statistics show that shortage of memory and subsequent swapping

are a major cause of downtime (Node evictions or Instance evictions)

Configure Swap on SSD device

Ensure device is formatted as ext4

Add “discard, noatime” to fstab

Configure OS parametersvm.swappiness=100 vm.panic_on_oom=0 vm.oom_kill_allocating_task=0 vm.overcommit_memory=0

Configure Swap on SSD Storage

29

#cat /sys/block/sdm/queue/rotational1

#blkid /dev/sdm/dev/sdm:UUID=“xxxx” TYPE=ext4

#cat /etc/fstab |grep sdmUUID="xxx" swap ext4 discard 1 2

#sysctl –w vm.swappiness=100#sysctl –w vm.panic_on_oom=0#sysctl –w vm.oom_kill_allocating task=0# sysctl –w vm.overcommit_memory=0#sysctl –w

#sysctl –p /etc/sysctl.conf


• Detect Node/Instance Hang/Death• Evict the dead/hung Instance/Node

• Elect a Recovery Master (RM)– One of the surviving instance process (SMON)

will get lock and be elected RM

• RM will then– read redo of evicted instance– apply recovery– signal completion

High Level Reconfiguration Stages

30

Detect

Evict

Elect Recovery

Read Redo

Apply Recovery


• Recovery Buddy feature optimizes reconfiguration

– Buddy Instances eliminate the “Elect Recovery Master” phase

– Redo-read is optimized via memory-reads

– Apply recovery is optimized as switching between read and writes is no longer required

Reduced Reconfiguration time with “Recovery Buddy”

31

Detect

Evict

Elect Recovery

Read Redo

Apply Recovery


1. Buddy Instance mapping is simple ( 1 2, 2 3, 3 4, 4 1)

2. Recovery buddies are set during instance startup

3. RMS0 on each recovery buddy instance maintains an in-memory area of redo log change

4. The in-memory area is used during recovery therefore eliminating the need to physically read the redo

Buddy Instances – Under the Hood

32

1. Inst1 is recovery buddy for Inst22. Inst2 is recovery buddy for 3 and so-on3. Recovery buddy mapping will change as new

instances join or leaveFor e.g If inst3 crashes, a new recovery buddy will be assigned to Inst 4

Inst 1 Inst 2 Inst 3 Inst 4

Recovery Buddy 2

Recovery Buddy 3

Recovery Buddy 4

MyCluster

Recovery Buddy 1


• Overall recovery times depends on reconfiguration time, which depends on the number of dirty blocks in the instance that needs recovery

• Reduce recovery time by calibrating and setting Fast_Start_MTTR_Target:1. Measure:

“Select Target_mttr,Estimated_mttrfrom gv$instance_recovery”

2. Set Fast_Start_MTTR_Target= <value> under

Better Availability Means Reducing Reconfiguration Times

33

– The estimated_mttr value is the mean time to recover should a crash occur

• Vendor Clusterware / NFS– Instance reconfiguration depends

on Oracle Clusterware. Vendor Clusterware only adds an additional layer

– Oracle Homes on will be affected by NFS hangs and can cause increased brownouts


Program Agenda15 Years of Performance Innovations

Scalability & Availability Optimizations

Automated for You



Appendixes



1

2

3

A

B



Leaf Nodes

Hang Manager

Data Guard

1

2

3



Leaf Nodes

Hang Manager

Data Guard

1

2

3


• Flex Cluster and Leaf nodes were introduced with Oracle RAC 12c Rel. 1.

• Leaf Nodes are loosely coupled nodes that are attached to Hub Nodes

• Starting with Oracle RAC 12c Rel. 2, it is now possible to run read-only workload on instances running on Leaf Nodes Reader Nodes

• A Reader Node failure does not impact the overall database activity, making it easy to scale to hundreds of nodes.

Flex Cluster – A Brief Review

37

Hub Node 1

Leaf Node 1

Leaf Node 3

LeafNode 2

Hub Node 3

Hub Node 2

StorageNetwork


How to run an Instance on Flex Cluster

38


• Configure Reader Nodes with additional memory for queries– Goal: prevent spilling to

Temp Tablespace during sorts

• Create local Temp Tablespace to improve performance for spills– Avoid cross-instance space

management– Avoid CF enqueue overhead

Reader Nodes Instance – Configuration

39

R/W Inst 1

R/W Inst 3

R/W Inst 2

Reader Node Instance 1

Reader Instance 3

Reader Instance 2


• Create and Configure– CREATE LOCAL TEMPORARY TABLESPACE FOR RIM

temp_rim TEMPFILE ‘/loc/temp_file_rim’ EXTENT Management local UNIFORM SIZE 1M AUTOEXTEND ON

– One Bigfile per Tablespace– Alter user scott local temporary tablspace blah;

• Users can be configured– local temporary to be used when user is connected

to Reader Node instance– Shared Temporary to be used when user is

connected to Read Write instance

Configure a Temporary Tablespace

40

User Shared temp

Read Write Instance

Read OnlyInstance

N

Continue SQL

Processing

User Local temp

N

DB Shared temp

N

DB Local temp

User Local temp

N

DB Local temp

N

User Shared temp

N

DB Shared temp

Session(s)



Leaf Nodes

Hang Manager

Data Guard

1

2

3


Overlooked and Underestimated – Hang Manager

• Customers experience database hangs for a variety of reasons– High system load, workload contention, network congestion or errors

• Before Hang Manager was introduced with Oracle RAC 11.2.0.2 – Oracle required information to troubleshoot a hang - e.g.:

• System state dumps• For RAC: global system state dumps

– Customer usually had to reproduce with additional events

Why is a Hang Manager required?


• Hang Manager only considers DB sessions holding resources on which sessions are waiting

• Hang Manager detects hangs across layers

• Deadlocks and User Locks are not managed by Hang Manager

How does it work?

Consider Cross layer holders like ASM

instance, Leaf nodes etc

Consider QoS policies, User Defined settings

Holder Identified

Verify

Analyze

Evaluate

Detect

Session


• Hang Manager auto-tunes itself by periodically collecting instance-and cluster-wide hang statistics

• Metrics like Cluster Health/Instance health is tracked over a moving average

• This moving Average considered during resolution

• Holders waiting on SQL*Net break/reset are fast tracked

Hang Manager Optimizations

44


• Early Warning exposed via (V$ view)

• Sensitivity can be set higher, if the user feels the default level is too conservative.

• Hang Manager behavior can be further fine-tuned by setting appropriate QoS policies

DBMS_HANG_MANAGER.Sensitivity

45

Hang Sensitivity Level

Description Note

NORMAL Hang Manager uses its default internal operating parameters to try to meet typical requirements for any environments.

Default

HIGH Hang Manager is more alert to sessions waiting in a chain than when sensitivity is in NORMAL level.



Leaf Nodes

Hang Manager

Data Guard

1

2

3


Data Guard Standby Redo Apply

• In a typical RAC Primary and RAC standby, Only one node of the standby can apply redo

• Other RAC nodes of the standby instance are typically in waiting mode even if the apply is CPU bound.

• Other instance only takes over redo apply only if the instance applying redo crashes


Multi-Instance Redo Apply

• Parallel, multi-instance recovery means “the standby DB will keep up”– Standby recovery - utilizes CPU and I/O across all nodes of RAC standby

– Up to 3500MB+/sec apply rate on an 8 node RAC

• Multi-Instance Apply runs on all MOUNTED instances or all OPEN Instances

• Exposed in the Broker with the ‘ApplyInstances’ property on standby

Utilize all RAC nodes on standby to apply Redo

recover managed standby database disconnect using instances 4;


Single-Instance Redo Apply

MRPProcesses

MRPProcesses

Standby Instance 1

RFSProcess

CoordinatorProcess

MRPProcesses

RFSProcess

CoordinatorProcess

MRPProcesses

RFSProcess

Primary Instance 1

ASYNC/SYNCProcess

Primary Instance 2

ASYNC/SYNCProcess

Primary Instance 3

ASYNC/SYNCProcess

Thread 1 Redo

Thread 2 Redo

Thread 3 Redo

SRL

SRL

SRL


Multi-Instance Redo Apply

CoordinatorProcess

MRPProcesses

Standby Instance 1

RFSProcess

CoordinatorProcess

MRPProcesses

Standby Instance 2

RFSProcess

CoordinatorProcess

MRPProcesses

Standby Instance 3

RFSProcess

Primary Instance 1

ASYNC/SYNCProcess

Primary Instance 2

ASYNC/SYNCProcess

Primary Instance 3

ASYNC/SYNCProcess

Thread 1 Redo

Thread 2 Redo

Thread 3 Redo

SRL

SRL

SRL


Multi-Instance Redo Apply – 2 Node Standby

CoordinatorProcess

MRPProcesses

Standby Instance 1

RFSProcess

CoordinatorProcess

MRPProcesses

Standby Instance 2

RFSProcess

CoordinatorProcess

RFSProcess

Primary Instance 1

ASYNC/SYNCProcess

Primary Instance 2

ASYNC/SYNCProcess

Primary Instance 3

ASYNC/SYNCProcess

Thread 1 Redo

Thread 2 Redo

Thread 3 Redo

SRL

SRL

SRL





Automated for You



Appendixes



1

2

3

A

B


Click icon to add picture

Create a new directory to install the grid home and copy the grid_home.zip to that directory


Unzip the grid_home.zip in the newly created directory (/u01/app/12.2.0/crs) in the example


Execute ./gridSetup.sh (Hint: Not runInstaller)

Step by Step Upgrade GI from 12.1 to 12.2 – Pre Steps

$mkdir /u01/app/12.2.0/crs

$scp grid_home.zip /u01/app/12.2.0/crs

1

All these steps (Step1, Step 2, Step 3 is to be executed on

First Node only

2 3


Step by Step Upgrade GI from 12.1 to 12.2 – Steps 1 to 5


Step by Step Upgrade GI from 12.1 to 12.2 Steps 5 to 7Optionally choose to enable “Automatically run configuration scripts” and provide root credentials in the next screen. In this example, we will not be enabling this option

Ensure cluvfy checks are taken care of. In the example, it seems CVU is complaining about missing mandatory patch 21255373

Click on the “more details” hyperlink to get additional details about mandatory patch 21255373


Download Patch 21255373

Login to MOS and download mandatory patch 21255373



Ensure you are using latest version of Opatch


Download the latest version of the Opatch from MOS using Patch 6880880.

The above version was latest at the time the slides were created

Ensure the latest version of opatch is being used


Apply the latest opatch to Grid home on ALL nodes

The latest Opatch is installed by unzipping the p6880880* file into the Grid Home on ALL the nodes


Now Proceed applying patch 21255373 on all the nodes


Now the patch is installed, Continue the Upgrade


Execute the root scripts on First node

1.) The log file location

Ensure you Note

2.) The last node to be downgraded cannot be a leaf node


Ensure root.sh completes successfully on All the nodes

2.) The last node to be downgraded cannot be a leaf node


Continue to the gridSetup.sh GUI -- Steps 9 & 10

Go back to the window where the installer is running to continue with the Upgrade




Automated for You



Appendixes


12.2 Installer New Features in Action (Grid Infrastructure Only)

1

2

3

A

B


New Image-based InstallationStep Image-based (12.1) Step Image-based (12.2)

1 Download shiphome zip files 1 Download image zip files

2 Unzip grid1/2.zip to some stage location (stage_loc)

3 Execute <stage_loc>/runInstaller.sh

4 Bootstraps files to some temp area (1GB)

5 Copies files from <stage_loc> to OH

6 Zip up the OH and store it as image files

7 Unzip image files to OH on all nodes of the cluster

2 Unzip image files to OH on one of the nodes of cluster

8 Execute clone.pl on all nodes of cluster

9 Run config.sh from one of the nodes 3 Run gridSetup.sh from one of the nodes


New ‘Lenient’ mode• Oracle 12c Rel. 2 Grid Infrastructure

installer supports ‘Lenient’ mode installation

• Installer allows user to by-pass nodes that are possibly mis-configured and proceed with configuration on the remaining nodes

• It is default behavior for all interactive installations

• Supported for silent(non-interactive) installations when “-lenientInstallMode” is specified on command line

67


Specify nodes using patterns/expressions

68


new –executeConfigtools option• –executeConfigTools is the new option to (re-)execute post-install

configuration tools• Works for gridSetup(GI) and runInstaller(DB)• Works with the installer’s response file• Has interactive UI• Better logging of configuration tools output for easier diagnostics• configToolAllCommands is deprecated

Technology

Anil nair rac_internals_sangam_2016