Oracle Performance Tuning

Performance Tuning BasicBejjanky Naresh Kumar

Explain Plan for this sessionThe ProblemGood performance habitsThe approach Define the problemInstance Tuning: Examine OSInstance Tuning: Examine Oracle StatsInstance Tuning: Implement and measure changesUsing SQL Trace and TKPROFSQL Trace File ExampleFormatting Trace Files with TKPROFTKPROF File Example

The ProblemPerception Oracle Performance tuning is wizardyWhy? Too many things to check.. Too many ratios to figure out..Ratios should be > 90%, some even > 99% else performance is badAny drop in ratios = performance lossLarger the Oracle caches = Better system performanceUntil 9i DBAs had a challenge to manually size SGADBAs don't need to size SGA manually, we have AMM in 10g/11g

Good Performance HabitsSeek out the bottlenecks on your systemCure the disease not just the symptomsTuning should be part of the lifecycle of an application. Most of the times tuning phase is left until the system is in production. At this time, tuning becomes a reactive fire-fighting exerciseTuning the resource hog on your system will provide you maximum benefitsSet tuning goals, stop tuning when you achieve them

The ApproachDefine the problemSeek out the OS BottlenecksSeek out the Oracle BottlenecksIdentify the real bottlenecksMake minimal changes as requiredMeasure the effect of each change

Define the problemGet the business expectations (SLAs) for the program/jobIdentify Scope of problem. Whole instance? One program or specific operation? Identify time frame when problem occursQuantify the slow downIdentify any changes. Any OS change? Has more data been loaded to system or the data volume has grown?If symptoms can be identified as local to a program, focus on that program. Application tuning.

Instance Tuning: Examine OS: Detecting CPU BottlenecksCPU Bottlenecks (sar, top, prstat) sar u 5 5essappu41:/apps/alice/aliceappl/fnd/11.5.0/resource> sar -o 5 5SunOS essappu41 5.9 Generic_118558-20 sun4u 02/09/200706:44:47 %usr %sys %wio %idle06:44:52 31 9 0 60

%usrThis is the percentage of time that the processor is in user mode %sysThis is the percentage of time that the processor is in system mode, servicing system calls. Users can cause this percentage to increase above normal levels by using system calls inefficiently%wioThis is the percentage of time that the processor is waiting on completion of I/O, from disk, NFS. If the percentage is regularly high, check the I/O systems for inefficiencies%idleThis is the percentage of time the processor is idle. If the percentage is high and the system is heavily loaded, there is probably a memory or an I/O problem.

Instance Tuning: Examine OS: Detecting CPU BottlenecksCPU Bottlenecks (sar, top, prstat) Example 1essdbpu42 $sar -u 5 5SunOS essdbpu42 5.9 Generic_112233-12 sun4u 11/19/200405:18:23 %usr %sys %wio %idle05:18:28 4 3 0 9305:18:33 9 2 0 8905:18:38 11 5 0 8405:18:44 2 3 0 9405:18:49 5 8 0 87Average 6 4 0 89

essdbpu42 $Above output system load look likes Very good because 89 % CPU is idle

Instance Tuning: Examine OS: Detecting CPU Bottlenecks

If the CPU is used by a small number of high usage programs then examine those programsIf high usage programs are non-oracle, then determine If they require that amount of cpu. If so, then can their executions be delayed to off-peak hoursIf high CPU Usage programs are oracle processes identify high CPU usage oracle sessions, Drill down at session level to examine which sql is running and why is it taking high cpu. Trace the session if need be

Instance Tuning: Examine OS: Detecting I/O BottlenecksI/O bottlenecks (sar, iostat)sar d 5 1essappu41:/apps/alice/aliceappl/fnd/11.5.0/resource> sar -d 5 1

SunOS essappu41 5.9 Generic_118558-20 sun4u 02/09/2007

06:48:23 device %busy avque r+w/s blks/s avwait avserv

06:48:28 md0 8 0.4 15 49 3.6 24.0 md1 0 0.0 0 0 0.0 0.0 md3 0 0.0 0 6 0.0 16.0

i/o requests should be evenly balanced high disk queue numbers + high service times I/O contention Check the wait events in v$system_event to see if the top wait events are i/o related

Instance Tuning: Examine OS: Detecting I/O BottlenecksI/O bottlenecks (sar, iostat)iostat reports terminal and disk I/O activity Look for service time, disk utilization, queue length,iostat will show service times, busy rates and queue lengths by diskImportant columns are svc_t, wait, %w and %bsvc_t - service time in millisec.

Long service times >50 ms indicate too much head movement. Slow service times significantly impare performance of a disk.Wait - wait is the number of entries in the queue for that device waiting to be serviced%w - is the percent if time the queue is occupied. 100% would mean that the queue never got emptied during that time period%b - is the percent of time the device is busy: 60% is severerly bad and very little useful workis performed. Disk > 65% busy on a steady basis are severely overworked

Instance Tuning: Examine OS: Detecting I/O Bottlenecksessdbdu31 $iostat xnp 5 5 extended device statisticsessdbdu31 $iostat xnp 5 5 extended device statisticsr/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device0.0 0.0 0.0 0.1 0.0 0.0 1.2 10.6 0 0 d10.0 0.0 0.0 0.1 0.0 0.0 0.3 2.8 0 0 d40.0 0.0 0.0 0.1 0.0 0.0 1.2 8.3 0 0 d51.2 0.9 24.4 5.7 0.0 0.0 0.0 12.9 0 1 d100.0 0.0 0.0 0.1 0.0 0.0 0.0 8.7 0 0 d110.3 0.2 13.3 0.9 0.0 0.0 0.0 8.5 0 0 d130.8 1.6 29.4 6.8 0.0 0.0 0.0 6.8 0 1 d160.0 0.0 0.0 0.1 0.0 0.0 0.0 3.5 0 0 d20 5.7 8.1 158.1 67.7 0.0 0.1 0.1 3.8 0 3 essnasu31s0:/vol/dev1/flora/db20.7 57.1 399.6 476.2 0.0 0.2 0.1 55 0 100 essnasu32s0:/vol/dev2/alicedev8/db

In the above example essnasu32s0:/vol/dev2/alicedev8/db file system asvc_t is 55 ms and %b (busy) is 100 % that means system is running with poor IO performance.If you see like this first we need to find out what are the instances are resides in that file system then what are the processes on that instances performing more IO activity

Instance Tuning: Examine OS: Detecting I/O Bottlenecksi/o wait events are db file sequential read, db file scattered read, db file single write and db file parallel write. These are all events corresponding to i/os performed against the data file headers, control files or data files

If any of these wait events correspond to high average time in AWR then investigate the problem

Check v$sqlarea and sort sqls by disk reads to find out oracle sessions doing physical reads

OEM will help in identifying such sql statements

Instance Tuning: Examine OS: Detecting Memory Bottlenecks vmstat S 5 1

essappu41:/apps/alice/aliceappl/fnd/11.5.0/resource> vmstat -S 5 1 kthr memory page disk faults cpu r b w swap free si so pi po fr de sr m0 m1 m3 m4 in sy cs us sy id 0 0 0 53270408 12000224 0 0 148 65 62 0 0 0 0 0 0 3353 7206 4838 27 5 67

Do not go overboard on SGA sizing just to get high cache hit ratios

An example One interface program run was performing poor. To attain high library cache hit ratios, sga (shared pool) was increased to 2gb. On examination, severe library latch contention was found. Cause was lack of bind variable usage. On fixing the application, system performance came right back.

Instance Tuning: Examine OS: Detecting Network BottlenecksLook at the network round trip ping times. (Ping, Tracert on Windows, TraceRoute on Unix)C:\Documents and Settings\kmra>tracert essdbpu51.emrsn.comTracing route to essdbpu51.emrsn.com [10.237.129.81] over a maximum of 30 hops: 1

Instance Tuning: Examine Oracle: Detecting Oracle Bottlenecks Oracle Statistics are examined and cross-referenced with OS Stats Wait events means server process/thread has to wait for an event to complete before further processingA server process can wait forA resource to become available (such as buffer or latch)An action to complete (such as an I/O)220 Wait events in Oracle8i, about 400 in Oracle9i, more than 800 in Oracle10g and above 1100 in 11g.

Instance Tuning: Oracle Wait InterfaceOWI Philosophy:The OWI is a performance tracking methodology that focuses on process bottlenecks, better known as wait events.OWI includes waits for I/O, locks, latches, background processes, network latencies OWI records and presents all the bottlenecks that a process encounters from start to finish.Keeps tracks of the number of times and amount of time a process spent on each bottleneck.Removing or even reducing the impact of major bottleneck improves performance.

Instance Tuning: Oracle Wait InterfaceOWI plays a significant role in determining the database response time.

Response Time = Service Time + Wait Time

Service time is amount of time a process spends on a CPU. Wait time is the amount of time a process wait for specific resource to be available before continuing with processing.

You can improve database response time by shortening the service time, the wait time, or both.

Instance Tuning: Oracle Wait InterfaceOWI Components:A collection of a few dynamic performance views and extended SQL trace file. OWI provides statistics for wait events to tell how many times and how long the session waited for an event to complete.V$EVENT_NAME not a dynamic view. Contains all the wait events defined for your database.V$SYSTEM_EVENT displays aggregated stats for all wait events encountered by all oracle sessions since the instance startup.V$SESSION_EVENT contains aggregated wait event stats by session for all sessions currently connected to the instance.V$SESSION_WAIT provides details info about the event or resource that each session is waiting for

Instance Tuning: Oracle Wait InterfaceI/O Related Wait Events:Db file sequential read -> is initiated by SQL that perform single-block read operations against indexes, undo segments. This is a single block read operation.db file scattered read -> is initiated by SQL that perform full scans operation against tables and indexes. This is a multi-block read.Direct path read -> waits are driven by SQL that perform direct read operation from the temporary or regular tablespaces. Direct path write -> waits are driven by SQL that perform direct write operation in the temporary or regular tablespaces.Locks related Wait Events:Latch free -> it means the process failed to obtain the latch in the willing-to-wait mode after spinning _SPIN_COUNT times and went to sleep. (shared pool, Library Cache, Cache Buffer Chains Latches)Enqueue -> Due to variety of enqueue types, this event can occur for many different reasons. The common causes and appropriate action to take depends on enqueue type and the mode that the sessions are competing for. Buffer busy waits -> occurs when a session wants to access a data block in the buffer cache that is currently in use by other session.

Instance Tuning: Examine OracleStart with v$system_event to determine the top waits on your system Column names: Event, Total_waits, Total_timeouts, Time_waited, Avg_waitDrill down to v$session_event Same column names + SID. Event information for the session. Stats are lost as session log off. Remember to get the SQL executed by SIDGet all the SQL for the session in question by joining v$session_wait, v$session, v$sqlGet the bottlenecking segment for this SQL by joining v$session_wait, dba_data_files, dba_extents

Instance Tuning: Examine OracleInitialization Parameters that effect performanceSGA_MAX_SIZEDB_CACHE_SIZEPGA_AGGREGATE_TARGETSHARED_POOL_SIZEMetalink Note 216205.1 Database Initialization Parameters and Configurations for Oracle Apps 11i.A poorly sized buffer cache results in excessive buffer gets and physical I/OA poorly sized shared pool results in library cache and shared pool latch contention due to lack of space

Instance Tuning: Implement and Measure ChangesIdentify two or three changes that could potentially improve the performanceImplement one change at a timeMeasure affect of change against the baseline data collected in performance definition phaseIts an iterative process. Know when to stop tuning.

Using SQL Trace and TKPROFalter session set events '10046 trace name context forever, level n';If the level is 4 or 12 then the trace file contains information about the bind variables. If the level is 8 or 12 then the trace file contains information about the events that the session waited for.Without any level it is equivalent to alter session set sql_trace=trueSome relevant init.ora parametersuser_dump_dest - Trace file locationtimed_statistics=TRUE - Enables Timingmax_dump_file_size - Max size trace files (unlimited by default)_trace_files_public=TRUE -changes permissions on trace files from 640 to 644sql_trace=TRUE- enables trace globally

Using SQL Trace and TKPROFHow to find your trace file select spid from v$process a, v$session b where a.addr=b.paddrand b.sid= (select distinct sid from v$mystat)

ALTER SESSION SET TRACEFILE_IDENTIFIER= balaselect username, traceid from v$process

How to Trace Someone Elses SessionExec sys.dbms_system.set_ev (sid, serial#, 10046, 12, ) - to turn onExec sys.dbms_system.set_ev (sid, serial#, 10046, 0, ) - to turn off

Oracle Form level trace Help -> Diagnostics -> Trace -> Trace with binds and waits

Setting the Trace for MWA/telnet session or a particular userLog in as a System Administrator & move to Profile/SystemCheck off the USER box - and enter username that you are tracingSearch on the profile option - 'Initialization SQL Statement - Custom.Set following at the user level - with the following string - BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||bala ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

SQL Trace File ExampleDump file bala_ora_461_DM_USER2.trc

*** TRACE DUMP CONTINUED FROM FILE ***

System name:SunOSNode name:essdbsu41Release:5.9Version:Generic_112233-12Machine:sun4uInstance name: betsys2Redo thread mounted by this instance: 1Oracle process number: 74Unix process pid: 461, image: oracle@essdbsu41 (TNS V1-V3)

*** SESSION ID:(41.34778) 2012-05-25 04:53:31.081=====================PARSING IN CURSOR #30 len=104 dep=2 uid=173 oct=42 lid=173 tim=30531412 hv=3741596375 ad='5de1c6c0'ALTER SESSION SET TRACEFILE_IDENTIFIER='DM_USER2' EVENTS =' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 'END OF STMTEXEC #30:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=4,tim=30531412=====================PARSING IN CURSOR #21 len=194 dep=1 uid=173 oct=47 lid=173 tim=30531412 hv=1642868942 ad='5de1ce8c'BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'DM_USER2' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;END OF STMTEXEC #21:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=1,dep=1,og=4,tim=30531412WAIT #1: nam='SQL*Net message to client' ela= 0 p1=1952673792 p2=1 p3=0=====================

SQL Trace File Example

SQL Trace File Example=====================PARSING IN CURSOR #1 len=125 dep=0 uid=173 oct=47 lid=173 tim=30531419 hv=651791082 ad='707b19dc'declare dummy boolean;begin fnd_profile.get_specific(:name, :userid, :respid, :applid, :val, dummy, :org_id,:server_id);end;END OF STMTPARSE #1:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=30531419WAIT #1: nam='SQL*Net message to client' ela= 0 p1=1952673792 p2=1 p3=0WAIT #1: nam='SQL*Net message from client' ela= 1 p1=1952673792 p2=1 p3=0BINDS #1: bind 0: dty=1 mxl=2000(1998) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=2000 offset=0 bfp=01b2b9e0 bln=2000 avl=09 flg=05 value="MRP_TRACE" bind 1: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=72 offset=0 bfp=01a932d0 bln=22 avl=04 flg=05 value=10159 bind 2: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=0 offset=24 bfp=01a932e8 bln=22 avl=04 flg=01 value=52348 bind 3: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=0 offset=48 bfp=01a93300 bln=22 avl=03 flg=01 value=401 bind 4: dty=1 mxl=2000(1998) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=2000 offset=0 bfp=01b2b1f0 bln=2000 avl=00 flg=05 bind 5: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=48 offset=0 bfp=01a93294 bln=22 avl=01 flg=05 value=0 bind 6: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=0 offset=24 bfp=01a932ac bln=22 avl=01 flg=01 value=0

SQL Trace File Example p1,p2,p3 values1.SQL> SELECT p1 "file#", p2 "block#", p3 "class#"FROM v$session_waitWHERE event = 'buffer busy waits';

file# block# class#---------- ---------- ----------1045 99752 1301045 99752 1301045 99752 1301045 99752 1301045 99752 1301045 99752 130

6 rows selected.

2.SQL> SELECT relative_fno, owner, segment_name, segment_typeFROM dba_extentsWHERE file_id = &fileAND &block BETWEEN block_id AND block_id + blocks - 1; 2 3 4Enter value for file: 1045old 3: WHERE file_id = &filenew 3: WHERE file_id = 1045Enter value for block: 99752old 4: AND &block BETWEEN block_id AND block_id + blocks - 1new 4: AND 99752 BETWEEN block_id AND block_id + blocks - 1

RELATIVE_FNO OWNER------------ ------------------------------SEGMENT_NAME--------------------------------------------------------------------------------SEGMENT_TYPE------------------22 SCAADMSCA_TMS_LOV_MAPTABLE

Formatting Trace Files with TKPROF tkprof produces a formatted output from the trace files generated by SQL trace or event 10046.The output generated by tkprof can be sorted by a variety of parameters to separate the more expensive statements from the rest. The first part of the parameter specifies the stageprs - parseexe - executionfch - fetchThe second part specifies the actual statisticmis - misses in library cachedsk - physical readsqry - consistent readscu - current readcpu - CPU timeela - elapsed time

tkprof SR135283.trc SR135283.trc.out sys=no explain=userid/passwd sort='(prsela,exeela,fchela)' waits=yes

TKPROF File Example 1

TKPROF: Release 11.2.0.1.0 - Production on Sun Oct 21 18:04:27 2012

Sort options: default

********************************************************************************count = number of times OCI procedure was executedcpu = cpu time in seconds executing elapsed = elapsed time in seconds executingdisk = number of physical reads of buffers from diskquery = number of buffers gotten for consistent readcurrent = number of buffers gotten in current mode (usually for update)rows = number of rows processed by the fetch or execute call********************************************************************************

alter session set events='10046 trace name context forever, level 12'

call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 0 0.00 0.00 0 0 0 0Execute 1 0.00 0.00 0 0 0 0Fetch 0 0.00 0.00 0 0 0 0------- ------ -------- ---------- ---------- ---------- ---------- ----------total 1 0.00 0.00 0 0 0 0

Misses in library cache during parse: 0Optimizer goal: CHOOSEParsing user id: 173

Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 1 0.00 0.00 SQL*Net message from client 1 0.00 0.00********************************************************************************

TKPROF File Example 2Sort options: exeela fchela ********************************************************************************count = number of times OCI procedure was executedcpu = cpu time in seconds executing elapsed = elapsed time in seconds executingdisk = number of physical reads of buffers from diskquery = number of buffers gotten for consistent readcurrent = number of buffers gotten in current mode (usually for update)rows = number of rows processed by the fetch or execute call********************************************************************************

SELECT CPT.SERIAL_NUMBER PT_SL_NUMBER , DRA.SERIAL_NUMBER DR_SL_NUMBER , CPT.INVENTORY_ITEM_ID PT_ITEM_ID , DRA.INVENTORY_ITEM_ID DR_ITEM_ID FROM CSD_PRODUCT_TXNS_V CPT , CSD_REPAIRS DRA WHERE ACTION_TYPE IN('RMA', 'WALK_IN_RECEIPT') AND DRA.REPAIR_LINE_ID = :B1 AND CPT.REPAIR_LINE_ID = DRA.REPAIR_LINE_ID AND NVL(CPT.SERIAL_NUMBER_CONTROL_CODE,1) > 1

call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 1 0.06 0.04 0 0 0 0Execute 1 0.42 0.41 0 0 0 0Fetch 1 32.35 309.40 56563 205651 0 1------- ------ -------- ---------- ---------- ---------- ---------- ----------total 3 32.83 309.86 56563 205651 0 1

Misses in library cache during parse: 1Optimizer goal: CHOOSEParsing user id: 173 (APPS) (recursive depth: 1)

TKPROF File Example 2Rows Row Source Operation------- --------------------------------------------------- 1 NESTED LOOPS OUTER 1 NESTED LOOPS OUTER 1 HASH JOIN OUTER 1 NESTED LOOPS OUTER 1 HASH JOIN OUTER 1 HASH JOIN OUTER 1 NESTED LOOPS OUTER 1 HASH JOIN OUTER 1 FILTER 6303 HASH JOIN OUTER 6303 NESTED LOOPS OUTER 6303 NESTED LOOPS 6303 NESTED LOOPS 6303 NESTED LOOPS 6303 NESTED LOOPS 6303 HASH JOIN 43 TABLE ACCESS FULL CS_TXN_BILLING_TYPES 6303 HASH JOIN 37063 NESTED LOOPS 37063 NESTED LOOPS 1 TABLE ACCESS BY INDEX ROWID CSD_REPAIRS 1 INDEX UNIQUE SCAN CSD_REPAIRS_U1 (object id 82662) 37063 TABLE ACCESS BY INDEX ROWID MTL_SYSTEM_ITEMS_B 55932 INDEX RANGE SCAN MTL_SYSTEM_ITEMS_B_N9 (object id 200613) 37063 INDEX UNIQUE SCAN MTL_SYSTEM_ITEMS_TL_U1 (object id 38056) 10900 TABLE ACCESS FULL CS_ESTIMATE_DETAILS 6303 INDEX UNIQUE SCAN CS_TRANSACTION_TYPES_B_U1 (object id 32048) 6303 INDEX UNIQUE SCAN CS_TRANSACTION_TYPES_TL_U1 (object id 32341) 6303 INDEX UNIQUE SCAN MTL_UNITS_OF_MEASURE_TL_U2 (object id 38094) 6303 INDEX UNIQUE SCAN CSD_REPAIRS_U1 (object id 82662) 0 TABLE ACCESS BY INDEX ROWID OKC_K_HEADERS_B 0 INDEX UNIQUE SCAN OKC_K_HEADERS_B_U1 (object id 85477) 6102 TABLE ACCESS FULL CSD_PRODUCT_TRANSACTIONS 976289 TABLE ACCESS FULL CSI_ITEM_INSTANCES 1 INDEX UNIQUE SCAN OE_ORDER_HEADERS_U1 (object id 41952) 459001 TABLE ACCESS FULL HZ_PARTY_SITES 459001 TABLE ACCESS FULL HZ_PARTY_SITES 1 INDEX UNIQUE SCAN HZ_LOCATIONS_U1 (object id 172918) 353157 INDEX FULL SCAN OE_ORDER_LINES_U1 (object id 42102) 1 INDEX UNIQUE SCAN QP_LIST_HEADERS_TL_PK (object id 45951) 1 INDEX UNIQUE SCAN HZ_LOCATIONS_U1 (object id 172918)

How to Understand TKPROF Example 3update ...where ...

-----------------------------------------------------------------------| call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|-----|---------|------|---------|---------|--------|| Parse | 1 | 7 | 122 | 0 | 0 | 0 | 0 || Execute | 1 | 75 | 461 | 5 | [H] 297 | [I] 3 | [J] 1 || Fetch | 0 | 0 | 0 | 0 | 0 | 0 | 0 |-----------------------------------------------------------------------

This statement is a single execute of an update.

[H] shows that this query is visiting 297 buffers to find the rows to update[I] shows that only 3 buffer are visited performing the update[J] shows that only 1 row is updated.

Reading 297 buffers to update 1 rows is a lot of work and would tend to indicate that the access path being used is not particularly efficient. Perhaps there is an index missing that would improve the access performance?

Spotting Over Parsing

select ...-------------------------------------------------------------------------| call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|---------|---------|------|--------|---------|-------|| Parse | [M] 2 | [N] 221 | 329 | 0 | 45 | 0 | 0 || Execute | 3 | [O] 9 | [P] 17 | 0 | 0 | 0 | 0 || Fetch | 3 | 6 | 8 | 0 | [L] 4 | 0 | [K] 1 |-------------------------------------------------------------------------

Misses in library cache during parse: 2 [Q]

Here we have a select that we suspect may be a candidate for over parsing.

[K] is shows that the query has returned 1 row.[L] shows that 4 buffers were read to get this row back.

Appendix: More Information

Oracle Metalink Note 39817.1 Interpreting Raw SQL_TRACE Oracle Metalink Note 216205.1 Database Initialization Parameters for Oracle Applications 11i Oracle Metalink Note 169935.1 Troubleshooting Oracle Apps Performance issues