137
Using SDA Extension s to Troub leshoot System Hang s and Perform anc e Issu es Presented by Jim Mehlhop [email protected] www.parsec.com | 888-4-PARSEC To Download http://www.parsec.com/public/SDA_Extensions.pdf 

SDA Extensions

Embed Size (px)

DESCRIPTION

openvms sda

Citation preview

  • Using SDA Extensions to Troubleshoot System Hangs and Performance Issues

    Presented by

    Jim [email protected]

    www.parsec.com | 888-4-PARSEC

    To Download

    http://www.parsec.com/public/SDA_Extensions.pdf

  • Activating SDA

    Symbols

    Read Global Symbols from System Loadable Images

    Read Global Symbols from Any Object Module

    Topics

    Format Instructions and Blocks of Data

  • Copy the System Dump File

    Send Output to a File or Device

    SDA Extensions What are they

    How they work

    They are undocumented, getting help can be a challenge

    Topics (contd)

  • Topics (contd)

    Clue Example

    Example of using SDA PCS to troubleshoot a hung system

    Examples using SDA SPL to troubleshoot high MP SYNCHRONIZATION time

    Example using FLT to isolate source of alignment faults, a major source of performance issues on Integrity Servers.

    Additional extensions

  • Activating SDA

    There are two ways of getting into SDA. One is to examine the live system and the other is to analyze a crash dump file. In both cases it reads a basic symbol table: REQSYSDEF.STB

    To analyze a live system, issue the following: $ analyze/system

    Need the CMKRNL privilege Since it is a live system, things change REMEMBER that

    To analyze a crash dump, issue the following: $ analyze/crash dump-filespec

    Need read access to the dump file Everything is static

  • Activating SDA (contd)

    Either way you start SDA, it always looks for an initialization file.

    First it looks for a logical SDA$INIT

    Second it looks in the current users default login directory for a file named sda.init

    The purpose for this file is to execute SDA commands that you ALWAYS want to execute.

    An example of this would be commands to read in symbols. We will be talking about symbols shortly.

  • Reading Symbols

    Symbols are read both automatically and under user control. Automatically

    When SDA gets activated, it reads reqsysdef.stb from the appropriate directory for the current architecture.

    User control SDA> read/exec

    Reads symbols from system loadable image list SDA> read

    Default file type is .stb, however it reads .obj files as well VMS supplies .stb files for most loadable_images and subsystems

    $DIR SYS$LOADABLE_IMAGES:*.stb (Alpha) Subsystem STBs DCLDEF, F11BXQPDEF, IMGDEF,

    IODEF, RMSDEF, SCSDEF, and a system-wide SYSDEF

  • Reading Symbols (contd)

    Users can supplement the VMS symbols

    Create a Macro source file with needed definitions

    $ edit symbols_example.mar

    Compile that source

    $ mac symbols_example.mar

  • Reading Symbols Examples #3Reading User-Created Object File

    There is an offset in a drivers Function Decision Table that indicates which functions support 64 bit addressing

    named FDT$Q_OK64BIT by default VMS does not make that symbol available to you. So we create a macro

    source program that pulls the FDT symbols in from the Macro library.

    CLASS3$ type symbols_example.mar

    .LIBRARY /SYS$LIBRARY:LIB.MLB/

    .LIBRARY /SYS$LIBRARY:STARLET.MLB/

    $FDTDEF GLOBAL

    $PFNDEF GLOBAL ;PFN DATA RECORD(Hopefully)

    $PFREEDEF GLOBAL ;FREE POOL PACKET DEFINITIONS

    $PRDEF GLOBAL ;PROCESSOR REGISTER DEFINITIONS

    $PSIGDEF GLOBAL ;CALL SIGNATURE DEFINITIONS

    $PTEDEF GLOBAL ; Page Table Entry DEFINITIONS

    $PTRDEF GLOBAL ; V7.3 PAGEFILE POINTER STRUCURE

    .END

    CLASS3$ mac symbols_example.mar

    SDA> sho sym FDT$Q_OK64BIT

    %SDA-E-BADSYM, unknown symbol "FDT$Q_OK64BIT"

    SDA> read SYMBOLS_EXAMPLE.obj

    %SDA-I-READSYM, 1394 symbols read from STAFF_DISK:[MEHLHOP]SYMBOLS_EXAMPLE.OBJ;1

    SDA> sho sym FDT$Q_OK64BIT

    FDT$Q_OK64BIT = 00000000.00000108

  • SDA Commands Format

    The format command allows you to display a block of memory that is defined as a data structure.

    It relies on the convention that the byte at offset 10(decimal) 0A(hex) defines which data structure the block represents.

    If there are symbols that represent that data structure, then SDA will display those names and the data in a formatted output as displayed in the next slide.

    If the block does not follow the convention or does not have a value that SDA understands, then SDA will either not format it or format it as the wrong type of structure.

  • SDA Commands Format (contd)

    SDA> sho proc ! To define PCB symbol

    .

    .

    .

    SDA> ex pcb+0A

    %SDA-W-UNALIGNED, unaligned address FFFFFFFF.81E8200A; converting to aligned address

    PCB+00008: 000C0580 "...."

    SDA> !0c=PCB

    SDA> format pcb

    %SDA-E-NOSYMBOLS, no "PCB" symbols found to format this block

    SDA> read sysdef

    SDA> format pcb

    FFFFFFFF.81E82000 PCB$L_SQFL 818099E8 SCH$AQ_COMT+001A4

    FFFFFFFF.81E82004 PCB$L_SQBL 818099E8 SCH$AQ_COMT+001A4

    FFFFFFFF.81E82008 PCB$W_SIZE 0580

    FFFFFFFF.81E8200A PCB$B_TYPE 0C

    FFFFFFFF.81E8200B

    00

    .

    .

    .

  • SDA Commands Format (contd)

    SDA> format phd !This structure does NOT follow convention note type=0

    %SDA-E-INVBLKTYP, invalid block type in specified block

    SDA> form phd/type=phd

    FFFFFFFF.82866000 PHD$Q_PRIVMSK 00000040.D05DC205

    FFFFFFFF.82866008 PHD$W_SIZE 0000

    FFFFFFFF.8286600A PHD$B_TYPE 00

    FFFFFFFF.8286600B 00

    FFFFFFFF.8286600C PHD$L_WSLIST 00000001

    FFFFFFFF.82866010 PHD$L_WSLOCK 0000000A

    FFFFFFFF.82866014 PHD$L_WSDYN 00000012

    FFFFFFFF.82866018 PHD$L_WSNEXT 00000263

    FFFFFFFF.8286601C PHD$L_WSLAST 000002E2

    FFFFFFFF.82866020 PHD$L_WSEXTENT 00000400

    FFFFFFFF.82866024 PHD$L_WSQUOTA 00000120

    .

    .

    .

  • SDA Commands Misc.

    SDA> show executive

    SDA> map 810B8050

    SDA> show cluster

    SDA> show lan

    SDA> copy dev::[dir]filename

    SDA> set output dev::[dir]filename

    SDA> set log dev::[dir]filename

  • Extensions What are they

    Special purpose code that calls SDA callable routines to examine memory under the protection of SDA.

    They are searched for whenever you issue an SDA command that is not a native SDA command.

    Searches for a logical name in the format of whatever the command was plus $SDA. For example SDA> PCS show trace will look for the logical PCS$SDA. If that logical does not exist it will look in the SYS$LIBRARY directory for the image PCS$SDA.

  • Extensions What are they (cont)

    Anyone can create an SDA extension. The first extension was CLUE. The second one was written by an ISV to support troubleshooting their application.

    The extensions supplied by OpenVMS can be identified by issuing the following command $DIRECTORY SYS$LIBRARY:*SDA.EXE

  • Extensions What are they (cont)

    Extensions are version/patch/Architecture specific

    Alpha

    SYS6$ show system/noprocess

    OpenVMS V8.3 on node SYS6 12-SEP-2011 16:10:23.94 Uptime 0 00:31:38

    SYS6$ dir sys$share:*$sda.exe

    Directory SYS$COMMON:[SYSLIB]

    CLUE$SDA.EXE;1 CNX$SDA.EXE;1 DECDTM$SDA.EXE;1 DKLOG$SDA.EXE;1

    EXC$SDA.EXE;1 FC$SDA.EXE;1 FLT$SDA.EXE;1 IO$SDA.EXE;1

    IPC$SDA.EXE;1 ISCSI$SDA.EXE;1 LAN$SDA.EXE;1 LCK$SDA.EXE;1

    LNM$SDA.EXE;1 MTX$SDA.EXE;1 OCLA$SDA.EXE;1 PCS$SDA.EXE;1

    PE$SDA.EXE;1 PKM$SDA.EXE;1 PRF$SDA.EXE;1 PSH$SDA.EXE;1

    PTHREAD$SDA.EXE;1 PWIP$SDA.EXE;1 RMS$SDA.EXE;1 SHAD$SDA.EXE;1

    SPL$SDA.EXE;1 TCPIP$SDA.EXE;1 TQE$SDA.EXE;1 TR$SDA.EXE;1

    USB$SDA.EXE;1 XFC$SDA.EXE;1 XX$SDA.EXE;1

    Total of 31 files.

    SYS6$

  • Integrity

    SYS226$ show system/noprocess

    OpenVMS V8.3-1H1 on node SYS226 12-SEP-2011 16:15:05.95 Uptime 49 04:31:46

    SYS226$ dir sys$share:*$sda.exe

    Directory SYS$COMMON:[SYSLIB]

    CLUE$SDA.EXE;1 CNX$SDA.EXE;1 DECDTM$SDA.EXE;1 DKLOG$SDA.EXE;1

    EXC$SDA.EXE;1 FC$SDA.EXE;1 FLT$SDA.EXE;1 IO$SDA.EXE;1

    IPC$SDA.EXE;1 ISCSI$SDA.EXE;1 LAN$SDA.EXE;1 LCK$SDA.EXE;1

    LNM$SDA.EXE;1 MTX$SDA.EXE;1 PCS$SDA.EXE;1 PE$SDA.EXE;1

    PKM$SDA.EXE;1 PKR$SDA.EXE;1 PRF$SDA.EXE;1 PSH$SDA.EXE;1

    PTHREAD$SDA.EXE;1 PWIP$SDA.EXE;1 RMS$SDA.EXE;1 SHAD$SDA.EXE;1

    SPL$SDA.EXE;1 TCPIP$SDA.EXE;1 TQE$SDA.EXE;1 TR$SDA.EXE;1

    USB$SDA.EXE;1 XFC$SDA.EXE;1 XX$SDA.EXE;1

    Total of 31 files.

    SYS226$

    Extensions What are they (cont)

  • IntegrityCLASS3$ show system/noprocess

    OpenVMS V8.4 on node CLASS3 12-SEP-2011 15:58:48.80 Uptime 5 23:26:10

    CLASS3$ dir sys$share:*$sda.exe

    Directory SYS$COMMON:[SYSLIB]

    CLUE$SDA.EXE;1 CNX$SDA.EXE;1 DECDTM$SDA.EXE;1 DKLOG$SDA.EXE;1

    EXC$SDA.EXE;1 FC$SDA.EXE;1 FLT$SDA.EXE;1 GSP$SDA.EXE;1

    ICC$SDA.EXE;1 IO$SDA.EXE;1 IPC$SDA.EXE;1 ISCSI$SDA.EXE;1

    LAN$SDA.EXE;1 LCK$SDA.EXE;1 LES$SDA.EXE;1 LNM$SDA.EXE;1

    MTX$SDA.EXE;1 NET$SDA.EXE;1 PCS$SDA.EXE;1 PE$SDA.EXE;1

    PKD$SDA.EXE;1 PKM$SDA.EXE;1 PKR$SDA.EXE;1 PRF$SDA.EXE;1

    PSH$SDA.EXE;1 PTHREAD$SDA.EXE;1 PWIP$SDA.EXE;1 RMS$SDA.EXE;1

    SHAD$SDA.EXE;1 SPL$SDA.EXE;1 SWIS$SDA.EXE;1 TCPIP$SDA.EXE;1

    TQE$SDA.EXE;1 TR$SDA.EXE;1 USB$SDA.EXE;1 XFC$SDA.EXE;1

    XX$SDA.EXE;1

    Total of 37 files.

    Extensions What are they (cont)

  • How they work

    When you use an extension command SDA loads the extension into YOUR process address space.

    They may then collect for display information that is contained in the OS

    Many of them also include an image in the SYS$LOADABLE_IMAGES directory that gets loaded into system space (GH) with a load subcommand to the extension. This image will stay in memory until either unloaded or the system reboots

  • How they work(cont)

    For those extensions that collect information that is not normally collected by the OS the extension will also allocate a ring buffer in nonpaged pool.

    This allows for you to start collection, log off, and later examine the collection after an event (including a system crash).

  • How they work(cont)

    Process address space System address space

    SDA.EXE

    Shareable images

    xxx$SDA

    xxx$DEBUG

    (system loadable image)

    Ring Buffer

  • How they work(cont)CLASS8$ an/sy

    OpenVMS system analyzer

    SDA> pcs load

    PCS$DEBUG load status = 00000001

    SDA> sho exec pcs$debug

    VMS Executive layout

    --------------------

    Image Base End Length Image Offset SymVec

    --------------------------------------- ----------------- ----------------- ----------------- ----------------- --------

    PCS$DEBUG

    Code FFFFFFFF.818E6000 FFFFFFFF.818E76EF 00000000.000016F0 00000000.00010000

    Data (read only) FFFFFFFF.886FD200 FFFFFFFF.886FD367 00000000.00000168 00000000.00020000

    Data (read/write) FFFFFFFF.886FD400 FFFFFFFF.886FD40F 00000000.00000010 00000000.00030000

    Short data (read/write) FFFFFFFF.886FD600 FFFFFFFF.886FD84F 00000000.00000250 00000000.00060000

    Linked 29-JUN-2006 23:20 LDRIMG = 88FA7A40 SeqNum = 000000B6 GP = FFFFFFFF.888FD600

    SDA> pcs unload

    PCS$DEBUG unload status = 00000001

    SDA> sho exec pcs$debug

    No loadable image matching "PCS$DEBUG" found

    SDA>

  • Getting help

    Some have limited help that you get by specifying just the command and no qualifiers (subcommand)

    Some have built in help

    Some have a help library

  • Getting help (cont)

    SDA> flt

    Alignment Fault Tracing Utility FLT commands:

    FLT LOAD

    FLT UNLOAD

    FLT START TRACE [/BUFFER=pages]

    [/BEGIN=pc_range_low]

    [/END=pc_range_high]

    [/MODE=(K,E,S,U)] (default is ALL modes)

    [/INDEX=pid] (default is ALL processes)

    FLT STOP TRACE

    FLT SHOW TRACE [/SUMMARY]

  • SDA> pe help

    PE$SDA Extension on CLASS2 (AlphaServer 4100 5/400 4MB) crash at 24-APR-2007 15:15:58.06

    ----------------------------------------------------------------------------------------

    PE

    This command executes the PEDRIVER specific SDA extension image

    which is a collection of PEDRIVER specific commands which mirror

    the display functions of SYS$SYSTEM:SCACP.EXE in SDA.

    By default this image is loaded from SYS$SHARE. However, it can

    be redirected with the following logical:

    $ define pe$sda [dir-spec]pe$sda.exe

    Format:

    SDA> PE command

    Additional information available:

    Press RETURN for more.

    SDA>

    PE$SDA Extension on CLASS2 (AlphaServer 4100 5/400 4MB) crash at 24-APR-2007 15:15:58.06

    ----------------------------------------------------------------------------------------

    BUS CHANNEL LAN_DEVICE TRACE HELP VC

    PE Subtopic? vc

    Getting help(cont)

  • PE Subtopic? vc

    PE

    VC

    Displays PEDRIVER virtual circuit data. Each VC is NISCACP

    communications path between the local system and a remote

    system comprised of a set of channels.

    Format

    PE VC

    Additional information available:

    Additional information available:

    Press RETURN for more.

    SDA>

    PE$SDA Extension on CLASS2 (AlphaServer 4100 5/400 4MB) crash at 24-APR-2007 15:15:58.06

    ----------------------------------------------------------------------------------------

    Parameters Qualifiers

    /ALL /COUNTERS /SUMMARY /VC /n

    Example

    PE VC Subtopic? exam

    PE

    VC

    Example

    SDA> PE VC/ERRORS/VC=FOO

    Displays VC error counters for all VCs whose name (remote node name)

    starts with FOO.

    PE VC Subtopic?

    PE Subtopic?

    Topic?

    Getting help(cont)

  • SDA> pwip

    SDA:PWIP> help

    HELP

    DEBUG [ON]

    DEBUG OFF

    DEBUG BREAKPOINT

    PWIP HEADER

    PWIP DATAGRAMS [ BRIEF | FULL | RAW | NUM ]

    PWIP LISTENERS [ BRIEF | FULL | RAW | NUM ]

    PWIP SESSIONS [ BRIEF | FULL | RAW | NUM ]

    PWIP [ALL] ( BRIEF of Datagrams, Listeners & Sessions)

    PWIP TRACE

    PWIP DEFAULT [ BRIEF | FULL | RAW ] (When NUM specified)

    SDA:PWIP>

    Getting help(cont)

  • SDA> spawn /nowait dir sys$help:*sda.hlb

    SDA>

    Directory SYS$COMMON:[SYSHLP]

    SDA.HLB;1 TCPIP$SDA.HLB;1 XFC$SDA.HLB;1

    Total of 3 files.

    SDA> tcpip help

    TCPIP

    Sorry, no documentation on TCPIP

    Additional information available:

    About_TCPIP$SDA Development_Qualifiers READ SEARCH

    SHOW SYSCONFIG TAG FORMAT

    Topic? devel

    Sorry, no documentation on DEVEL

    Additional information available:

    TCPIP

    Topic? tcpip

    Getting help(cont)

  • Topic? tcpip

    TCPIP

    The TCPIP verb prefixes all of the TCPIP (TCP/IP Services for

    The TCPIP verb prefixes all of the TCPIP (TCP/IP Services for

    Press RETURN for more.

    SDA>

    OpenVMS) SDA extension's commands.

    Format:

    TCPIP [qualifiers]

    TCPIP FORMAT ...

    TCPIP READ ...

    TCPIP SEARCH ...

    TCPIP SHOW ...

    TCPIP SYSCONFIG ...

    TCPIP TAG ...

    Additional information about the TCPIP extension can be found

    under the `About_TCPIP$SDA' topic.

    Additional information available:

    About_TCPIP$SDA Development_Qualifiers READ SEARCH

    SHOW SYSCONFIG TAG FORMAT

    Getting help(cont)

  • SDA> tcpip help format

    TCPIP

    FORMAT

    Formats the block of memory specified by the "address" parameter.

    To do this the "MALLOC_HDR", located at negative 40 hex from the

    address is interpreted to determine the type of the block, and

    in some cases the subtype of the block.

    Format:

    TCPIP FORMAT address

    /VERIFY

    Additional information available:

    Parameters

    Press RETURN for more.

    SDA>

    Getting help(cont)

  • Clue Crash

    SDA> clue crash

    Crashdump Summary Information:

    ------------------------------

    Crash Time: 16-MAY-2006 08:36:28.28

    Bugcheck Type: SSRVEXCEPT, Unexpected system service exception

    Node: SYS6 (Cluster)

    CPU Type: Digital Personal WorkStation

    VMS Version: V8.2

    Current Process: MEHLHOP

    Current Image: $6$DKC200:[MEHLHOP.BOOTCAMP.SOURCE]CRASH.EXE;1

    Failing PC: 00000000.00030194 LAN$C_MCA_L_LAT+00061

    Failing PS: 10000000.00000003

    Module: CRASH

    Offset: 00030194

    Boot Time: 10-MAY-2006 21:25:06.00

    System Uptime: 5 11:11:22.28

    Crash/Primary CPU: 00/00

    System/CPU Type: 1E05

    Saved Processes: 0

    Pagesize: 8 KByte (8192 bytes)

    Physical Memory: 640 MByte (81920 PFNs, contiguous memory)

    Press RETURN for more.

  • Clue Crash (cont)

    SDA>

    Crashdump Summary Information:

    ------------------------------

    Dumpfile Pagelets: 1310720 blocks

    Dump Flags: olddump,writecomp,errlogcomp

    Dump Type: raw,full,shared_mem

    EXE$GL_FLAGS: poolpging,init,bugdump

    Paging Files: 1 Pagefile and 1 Swapfile installed

    Stack Pointers:

    KSP = 00000000.7FF87BE8 ESP = 00000000.7FF8C000 SSP = 00000000.7FF9CC80

    USP = 00000000.7AE35AB0

    General Registers:

    R0 = 00000000.0000000C R1 = 00000000.7FF87E38 R2 = FFFFFFFF.81C38CC0

    R3 = FFFFFFFF.818CA460 R4 = 00000000.7FF87CC0 R5 = 00000000.7FF87E38

    R6 = 00000000.7FF87E80 R7 = 00000000.7FF87FC0 R8 = 00000000.7FF9CDE8

    R9 = 00000000.7FF9DDF0 R10 = 00000000.7FFA4F28 R11 = 00000000.7FFCDBE8

    R12 = 00000000.7FFCDA68 R13 = 00000000.00010080 R14 = 00000000.00000000

    R15 = 00000000.7AF1C5E0 R16 = 00000000.000003C4 R17 = 00000000.7FF87CC0

    R18 = 00000000.00000020 R19 = 00000000.00000002 R20 = 00000000.00000000

    R21 = 00000000.00000000 R22 = 00000000.7FFF03C8 R23 = 00000000.7FFF0040

    Press RETURN for more.

  • Clue Crash (cont)

    SDA>

    Crashdump Summary Information:

    ------------------------------

    R24 = 00000000.00000000 AI = 00000000.00000003 RA = FFFFFFFF.81810200

    PV = FFFFFFFF.818C9260 R28 = 00000000.000005AC FP = 00000000.7FF87BF0

    PC = FFFFFFFF.8009B43C PS = 28000000.00000000

    Exception Frame:

    R2 = 00000000.00000003 R3 = FFFFFFFF.818CA460 R4 = FFFFFFFF.82027880

    R5 = 00000000.00000088 R6 = 00000000.00030050 R7 = 00000000.7FF87FC0

    PC = 00000000.00030194 PS = 10000000.00000003

    Signal Array: 64-bit Signal Array:

    Arg Count = 00000005 Arg Count = 00000005

    Condition = 0000000C Condition = 00000000.0000000C

    Argument #2 = 00000000 Argument #2 = 00000000.00000000

    Argument #3 = 00000000 Argument #3 = 00000000.00000000

    Argument #4 = 00030194 Argument #4 = 00000000.00030194

    Argument #5 = 00000003 Argument #5 = 10000000.00000003

    Mechanism Array:

    Arguments = 0000002C Establisher FP = 00000000.7AE35AB0

    Press RETURN for more.

  • Clue Crash (cont)

    SDA>

    Crashdump Summary Information:

    ------------------------------

    Flags = 00000001 Exception FP = 00000000.7FF87E80

    Depth = FFFFFFFD Signal Array = 00000000.7FF87E38

    Handler Data = 00000000.00000000 Signal64 Array = 00000000.7FF87E50

    R0 = 00000000.00000000 R1 = 00000000.00000000 R16 = 00000000.00000000

    R17 = 00000000.00010040 R18 = 00000000.00000005 R19 = 00000000.00000000

    R20 = 00000000.00000000 R21 = FFFFFFFF.FFFFFFFF R22 = 00000000.00010040

    R23 = FFFFFFFF.8016663C R24 = 00000000.00000001 R25 = 00000000.00000001

    R26 = 00000000.00030178 R27 = 00000000.00010090 R28 = 00000000.00000006

    System Registers:

    Page Table Base Register (PTBR) 00000000.0000BA35

    Processor Base Register (PRBR) FFFFFFFF.81C38000

    Privileged Context Block Base (PCBB) 00000000.22B16080

    System Control Block Base (SCBB) 00000000.00000A61

    Software Interrupt Summary Register (SISR) 00000000.00000000

    Address Space Number (ASN) 00000000.00000034

    AST Summary / AST Enable (ASTSR_ASTEN) 00000000.0000000F

    Floating-Point Enable (FEN) 00000000.00000001

    Interrupt Priority Level (IPL) 00000000.00000000

    Machine Check Error Summary (MCES) 00000000.00000000

    Virtual Page Table Base Register (VPTB) FFFFFEFC.00000000

    Press RETURN for more.

  • Clue Crash (cont)

    SDA>

    Crashdump Summary Information:

    ------------------------------

    Failing Instruction:

    LAN$C_MCA_L_LAT+00061: LDL R0,(R0)

    Instruction Stream (last 20 instructions):

    LAN$C_MCA_L_LAT+00011: LDQ R7,#X0028(SP)

    LAN$C_MCA_L_LAT+00015: LDQ R13,#X0030(SP)

    LAN$C_MCA_L_LAT+00019: LDQ FP,#X0038(SP)

    LAN$C_MCA_L_LAT+0001D: ADDQ SP,#X40,SP

    LAN$C_MCA_L_LAT+00021: RET R31,(R28)

    LAN$C_MCA_L_LAT+00025: BIS R31,R31,R31

    LAN$C_MCA_L_LAT+00029: BIS R31,R31,R31

    LAN$C_MCA_L_LAT+0002D: SUBQ SP,#X10,SP

    LAN$C_MCA_L_LAT+00031: STQ R26,(SP)

    LAN$C_MCA_L_LAT+00035: STQ R13,#X0008(SP)

    LAN$C_MCA_L_LAT+00039: BIS R31,R27,R13

    LAN$C_MCA_L_LAT+0003D: ADDQ R13,#X10,R27

    LAN$C_MCA_L_LAT+00041: BSR R26,#X000006

    LAN$C_MCA_L_LAT+00045: LDQ R28,(SP)

    LAN$C_MCA_L_LAT+00049: LDQ R13,#X0008(SP)

    Press RETURN for more.

  • Clue Crash (cont)

    SDA>

    Crashdump Summary Information:

    ------------------------------

    LAN$C_MCA_L_LAT+0004D: ADDQ SP,#X10,SP

    LAN$C_MCA_L_LAT+00051: RET R31,(R28)

    LAN$C_MCA_L_LAT+00055: BIS R31,R31,R31

    LAN$C_MCA_L_LAT+00059: BIS R31,R31,R31

    LAN$C_MCA_L_LAT+0005D: BIS R31,R31,R0

    LAN$C_MCA_L_LAT+00061: LDL R0,(R0)

    LAN$C_MCA_L_LAT+00065: RET R31,(R26)

    LAN$C_MCA_L_LAT+00069: HALT

    LAN$C_MCA_L_LAT+0006D: HALT

    LAN$C_MCA_L_LAT+00071: HALT

  • Clue Stack

    Clue Stack attempts to decode and annotate the stack. It sometimes has problems

    with null stack frames and in fact in this version got confused with the CHF structure

    as well.

    SDA> clue stack

    Stack Decoder:

    --------------

    Normal Process Kernel Stack:

    Stack Pointer 00000000.7FF87BE8

    Stack Limits (low) 00000000.7FF84000

    (high) 00000000.7FF88000

    Press RETURN for more.

  • Clue Stack (cont)

    SDA>

    SSRVEXCEPT Stack:

    -----------------

    Stack Pointer SP => 00000000.7FF87BE8

    Information saved by Bugcheck:

    a(Signal Array) 00000000.7FF87BE8 00000000.7FF87E38

    EXE$EXCPTN[E] Stack Frame:

    PV 00000000.7FF87BF0 FFFFFFFF.818C9260 EXE$EXCPTN

    Entry Point FFFFFFFF.8009B380 EXE$EXCPTN_C

    00000000.7FF87BF8 00000000.00000000

    return PC 00000000.7FF87C00 FFFFFFFF.8009A93C SYS$CALL_HANDL_C+0002C

    saved R2 00000000.7FF87C08 00000000.00000000

    saved R4 00000000.7FF87C10 FFFFFFFF.82027880 KTB

    saved R5 00000000.7FF87C18 00000000.00000088

    saved R6 00000000.7FF87C20 00000000.00030050 SYS$K_VERSION_03+00050

    saved FP 00000000.7FF87C28 00000000.7FF87C40

    SYS$CALL_HANDL Temporary Storage:

    00000000.7FF87C30 FFFFFFFF.818C9028 SYS$CALL_HANDL

    00000000.7FF87C38 00000000.00000000

    Press RETURN for more.

  • Clue Stack (cont)

    SDA>

    SSRVEXCEPT Stack:

    -----------------

    SYS$CALL_HANDL Stack Frame:

    PV 00000000.7FF87C40 FFFFFFFF.818C9028 SYS$CALL_HANDL

    Entry Point FFFFFFFF.8009A910 SYS$CALL_HANDL_C

    00000000.7FF87C48 00000000.00000000

    return PC 00000000.7FF87C50 FFFFFFFF.80098370 CHF_BADSTACK_C+00060

    saved FP 00000000.7FF87C58 00000000.7FF87EE0

    Fixed Exception Context Area:

    Linkage Pointer 00000000.7FF87C60 FFFFFFFF.818C8240 CHF$GR_LINKAGE_SECTION

    a(Signal Array) 00000000.7FF87C68 00000000.7FF87E38

    a(Mechanism Array) 00000000.7FF87C70 00000000.7FF87CC0

    a(Exception Frame) 00000000.7FF87C78 00000000.7FF87E80

    Exception FP 00000000.7FF87C80 00000000.7FF87EE0

    Unwind SP 00000000.7FF87C88 00000000.00000000

    Reinvokable FP 00000000.7FF87C90 00000000.00000000

    Unwind Target 00000000.7FF87C98 00000000.00000008

    #Sig Args/Byte Cnt 00000000.7FF87CA0 82027880.81EEA8C0

    a(Msg)/Final Status 00000000.7FF87CA8 8200E640.81DE2400

    Flags/Arguments 00000000.7FF87CB0 00000005.00000260 BUG$_NETTRANCNT

    Press RETURN for more.

  • Clue Stack (cont)

    SDA>

    SSRVEXCEPT Stack:

    -----------------

    a(Establisher FP) 00000000.7FF87CB8 818BCF64.000008F8 BUG$_SEQ_NUM_OVF

    Mechanism Array:

    Flags/Arguments 00000000.7FF87CC0 00000001.0000002C

    a(Establisher FP) 00000000.7FF87CC8 00000000.7AE35AB0

    reserved/Depth 00000000.7FF87CD0 FFFFFFFF.FFFFFFFD

    a(Handler Data) 00000000.7FF87CD8 00000000.00000000

    a(Exception Frame) 00000000.7FF87CE0 00000000.7FF87E80

    a(Signal Array) 00000000.7FF87CE8 00000000.7FF87E38

    saved R0 00000000.7FF87CF0 00000000.00000000

    saved R1 00000000.7FF87CF8 00000000.00000000

    saved R16 00000000.7FF87D00 00000000.00000000

    saved R17 00000000.7FF87D08 00000000.00010040 SYS$K_VERSION_16

    saved R18 00000000.7FF87D10 00000000.00000005

    saved R19 00000000.7FF87D18 00000000.00000000

    saved R20 00000000.7FF87D20 00000000.00000000

    saved R21 00000000.7FF87D28 FFFFFFFF.FFFFFFFF

    saved R22 00000000.7FF87D30 00000000.00010040 SYS$K_VERSION_16

    saved R23 00000000.7FF87D38 FFFFFFFF.8016663C AMAC$EMUL_CALL_NATIVE_C+0005C

    Press RETURN for more.

  • Clue Stack (cont)

    SDA>

    SSRVEXCEPT Stack:

    -----------------

    saved R24 00000000.7FF87D40 00000000.00000001

    saved R25 00000000.7FF87D48 00000000.00000001

    saved R26 00000000.7FF87D50 00000000.00030178 LAN$C_MCA_L_LAT+00045

    saved R27 00000000.7FF87D58 00000000.00010090 LAN$C_CCA_W_BROADCAST+00009

    saved R28 00000000.7FF87D60 00000000.00000006

    saved F0 00000000.7FF87D68 08000000.00000000

    saved F1 00000000.7FF87D70 00000000.00000000

    saved F10 00000000.7FF87D78 00000000.00000000

    saved F11 00000000.7FF87D80 00000000.00000000

    saved F12 00000000.7FF87D88 00000000.00000000

    saved F13 00000000.7FF87D90 00000000.00000000

    saved F14 00000000.7FF87D98 00000000.00000000

    saved F15 00000000.7FF87DA0 00000000.00000000

    saved F16 00000000.7FF87DA8 00000000.00000000

    saved F17 00000000.7FF87DB0 00000000.00000000

    saved F18 00000000.7FF87DB8 00000000.00000000

    saved F19 00000000.7FF87DC0 00000000.00000000

    saved F20 00000000.7FF87DC8 00000000.00000000

    saved F21 00000000.7FF87DD0 00000000.00000000

    Press RETURN for more.

  • Clue Stack (cont)

    SDA>

    SSRVEXCEPT Stack:

    -----------------

    saved F22 00000000.7FF87DD8 00000000.00000000

    saved F23 00000000.7FF87DE0 00000000.00000000

    saved F24 00000000.7FF87DE8 00000000.00000000

    saved F25 00000000.7FF87DF0 00000000.00000000

    saved F26 00000000.7FF87DF8 00000000.00000000

    saved F27 00000000.7FF87E00 00000000.00000000

    saved F28 00000000.7FF87E08 00000000.00000000

    saved F29 00000000.7FF87E10 00000000.00000000

    saved F30 00000000.7FF87E18 00000000.00000000

    a(Signal64 Array) 00000000.7FF87E20 00000000.7FF87E50

    SP Align = 10(hex) [...............]

    Signal Array:

    Arguments 00000000.7FF87E38 00000005

    Condition 00000000.7FF87E3C 0000000C

    Argument #2 00000000.7FF87E40 00000000

    Argument #3 00000000.7FF87E44 00000000

    Argument #4 00000000.7FF87E48 00030194 LAN$C_MCA_L_LAT+00061

    Argument #5 00000000.7FF87E4C 00000003

    Press RETURN for more.

  • Clue Stack (cont)

    SDA>

    SSRVEXCEPT Stack:

    -----------------

    64-bit Signal Array:

    Arguments 00000000.7FF87E50 00002604.00000005

    Condition 00000000.7FF87E58 00000000.0000000C

    Argument #2 00000000.7FF87E60 00000000.00000000

    Argument #3 00000000.7FF87E68 00000000.00000000

    Argument #4 00000000.7FF87E70 00000000.00030194 LAN$C_MCA_L_LAT+00061

    Argument #5 00000000.7FF87E78 10000000.00000003

    Interrupt/Exception Frame:

    saved R2 00000000.7FF87E80 00000000.00000003

    saved R3 00000000.7FF87E88 FFFFFFFF.818CA460 EXE$CMODEXECX

    saved R4 00000000.7FF87E90 FFFFFFFF.82027880 KTB

    saved R5 00000000.7FF87E98 00000000.00000088

    saved R6 00000000.7FF87EA0 00000000.00030050 SYS$K_VERSION_03+00050

    saved R7 00000000.7FF87EA8 00000000.7FF87FC0

    saved PC 00000000.7FF87EB0 00000000.00030194 LAN$C_MCA_L_LAT+00061

    saved PS 00000000.7FF87EB8 10000000.00000003 IPL INT CURR PREV

    SP Align = 10(hex) [...............] 00 0 Kern User

    Press RETURN for more.

    .

    .

    .

  • Troubleshooting a Hung System

    This process works in the case where you have a system that is periodically hanging to the point where you cannot get to a $ prompt.

    In this case it was an artificially created case where the system was hanging at IPL 8

    Note: When this occurs on an Integrity Server you do NOT see the ^P menu on the console.

    Prior to the hang you MUST use the PCS extension to load and start a trace

    The ring buffer will continue to be written until the system is crashed

  • Troubleshooting a Hung System Integrity

    CLASS8$ run new

    (system hangs ^P followed by ^P)

    Crash (y/n)> y

    **** OpenVMS I64 Operating System V8.3 - BUGCHECK ****

    ** Bugcheck code = 0000064D: OPERCRASH, Operator forced system crash

    ** Crash CPU: 00000000 Primary CPU: 00000000 Node Name: CLASS8

    ** Supported CPU count: 00000002

    ** Active CPUs: 00000000.00000003

    ** Current Process: MEHLHOP

    ** Current PSB ID: 00000001

    ** Image Name:

    **** Starting compressed selective memory dump at 14-MAY-2007 17:22...

    ................................................................................

    .............................................

    ** System space, key processes, and key global pages have been dumped.

    ** Now dumping remaining processes and global pages...

    ................................

    ...Complete ****

    **** Primary HALTED with code HWRPB_HALT$K_WARM_REBOOT

    ***********************************************************

  • Troubleshooting a Hung System Integrity (cont)

    CLASS8$ an/cr SYSDUMP_HANG_IPL8

    OpenVMS system dump analyzer

    ...analyzing an I64 compressed selective memory dump...

    Dump taken on 14-MAY-2007 17:27:14.60 using version V8.3

    OPERCRASH, Operator forced system crash

    SDA> sho exception

    Exception Frame Summary

    -----------------------

    Exception Frame Type Stack IIP / Ret_Addr Trap_Type / Service_Number

    ----------------- ---- ----- ----------------- --------------------------

    00000000.7FF42D80 ORIGINAL_INTSTK Kernel FFFFFFFF.810B5B70 00000041 Bugcheck Breakpoint Trap

    00000000.7FF430A0 INTSTK Kernel FFFFFFFF.80145950 00000080 External Interrupt

    00000000.7FF430F0 INTSTK Kernel FFFFFFFF.80442A20 00000028 Report alignment fault

    00000000.7FF43150 INTSTK Kernel FFFFFFFF.7E34A000 00000028 Report alignment fault

    00000000.7FF433B0 INTSTK Kernel 000002D0.02000000 00000060 Interval Clock Interrupt

    00000000.7FF433C0 INTSTK Kernel FFFFFFFF.805100E0 00000060 Interval Clock Interrupt

    00000000.7FF436E0 STALE_SSENTRY Kernel FFFFFFFF.883AA600 0100019C SYS$PERSONA_EXPORT_ARB

    00000000.7FF43700 SSENTRY Kernel FFFFFFFF.80B3D690 01000022 SYS$DALLOC

    00000000.7FF438F0 SSENTRY Kernel FFFFFFFF.80B396B0 010000F1 SYS$KRNDWN

    00000000.7FF43990 SSENTRY Kernel FFFFFFFF.805A4890 0100008A SYS$RMSRUNDWN

    00000000.7FF43B40 SSENTRY Kernel FFFFFFFF.80B5C3D0 01000028 SYS$DELPRC

    00000000.7FF43D30 SSENTRY Kernel FFFFFFFF.80B5AC40 0100018E SYS$EXIT_INT

    00000000.7FF43F40 SSENTRY Kernel 00000000.00020080 01000019 SYS$CMKRNL

  • Troubleshooting a Hung System Integrity (cont)

    SDA> sho exception 7FF430A0

    Exception Frame at 00000000.7FF430A0

    ------------------------------------

    IPL = 08

    TRAP_TYPE = 00000080 External Interrupt

    IVT_OFFSET = 00003000 External Interrupt

    IIP = FFFFFFFF.80145950 EXE$TIMEOUT_C+00320

    IIPA = FFFFFFFF.80145950 EXE$TIMEOUT_C+00320

    IFA = FFFFFFFF.7E34A000

    IPSR = 00001410.08026010 RT TB LP DB SI DI PP SP DFH DFL DT PK I IC MFH MFL AC BE UP

    1 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 0 0 0

    IA BN ED RI SS DD DA ID IT MC IS CPL

    0 1 0 2 0 0 0 0 1 0 0 0

    PREVSTACK = 00

    BSP = 00000000.7FF2F208

    BSPSTORE = 00000000.7FF2F208

    BSPBASE = 00000000.7FF2F208

    RNAT = 00000000.00000000

  • Troubleshooting a Hung System Integrity (cont)

    SDA> pcs sho trace

    PC sampler information:

    -----------------------

    Timestamp CPU PC IPL M Pid Routine Module

    ---------------------- --- ----------------- --- - -------- -------------------------------- ------

    14-MAY 17:27:14.605993 00 FFFFFFFF.8003C200 21 I 34400430 BOO$READB_C SYS$PLATFORM_SUPPORT

    14-MAY 17:27:14.594993 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    14-MAY 17:27:14.583994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    14-MAY 17:27:14.572994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    .

    .

    .

    14-MAY 17:27:13.483994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    14-MAY 17:27:13.472994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    14-MAY 17:27:13.461993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.450993 00 FFFFFFFF.80145980 8 I 34400430 EXE$TIMEOUT_C+00350 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.439993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.428993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.417993 00 FFFFFFFF.80145950 8 I 34400430 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.406993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.395993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.384993 00 FFFFFFFF.80145930 8 I 34400430 EXE$TIMEOUT_C+00300 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.373993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.362993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.351993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.340993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.329993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.318993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.307993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN

    Press RETURN for more.

  • Troubleshooting a Hung System Integrity (cont)

    SDA>

    PC sampler information:

    -----------------------

    Timestamp CPU PC IPL M Pid Routine Module

    ---------------------- --- ----------------- --- - -------- -------------------------------- ------

    14-MAY 17:27:13.296993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.285993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.274993 00 FFFFFFFF.80145950 8 I 34400430 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.263993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.252993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.241993 00 FFFFFFFF.80145930 8 I 34400430 EXE$TIMEOUT_C+00300 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.230993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.219993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.208993 00 FFFFFFFF.801459E0 8 I 34400430 EXE$TIMEOUT_C+003B0 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.197993 00 FFFFFFFF.801459E0 8 I 34400430 EXE$TIMEOUT_C+003B0 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.186993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.175993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.164993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.153993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.142993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.131993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    14-MAY 17:27:13.120993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN

  • Troubleshooting a Hung System Integrity (cont)

    SDA> pcs sho trace/statistics

    PC sampler information:

    -----------------------

    PC IPL Pid Count Routine Module

    ----------------- --- -------- ------------ -------------------------------- ------

    FFFFFFFF.80145970 8 34400430 575 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.801459A0 8 34400430 537 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.801459D0 8 34400430 489 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.801459E0 8 34400430 352 EXE$TIMEOUT_C+003B0 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.80145950 8 34400430 343 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.801459B0 8 34400430 337 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.80145940 8 34400430 274 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.80145930 8 34400430 249 EXE$TIMEOUT_C+00300 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.80145980 8 34400430 182 EXE$TIMEOUT_C+00350 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.80145990 8 34400430 144 EXE$TIMEOUT_C+00360 SYSTEM_PRIMITIVES_MIN

    FFFFFFFF.810B4D90 21 34400430 102 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    FFFFFFFF.805100E0 2 34400430 6 IO_ROUTINES+F10E0 IO_ROUTINES

    FFFFFFFF.804682D0 2 34400430 6 IOC$SCAN_IODB_C+00030 IO_ROUTINES

    FFFFFFFF.805100A0 2 34400430 4 IO_ROUTINES+F10A0 IO_ROUTINES

    FFFFFFFF.80510080 2 34400430 3 IO_ROUTINES+F1080 IO_ROUTINES

    FFFFFFFF.80510070 2 34400430 3 IO_ROUTINES+F1070 IO_ROUTINES

    FFFFFFFF.80468340 2 34400430 3 IOC$SCAN_IODB_C+000A0 IO_ROUTINES

    Press RETURN for more.

    SDA> set out delme.out

    SDA> pcs sho trace/full

    SDA> set out tt

    SDA> spawn

    CLASS8$ edit/read delme.out

  • Troubleshooting a Hung System Integrity (cont)

    OpenVMS Operating System, Version V8.3 -- System Dump Analysis 14-MAY-2007 17:27:14.60 Page 2

    PC sampler information:

    Time = 14-MAY 17:27:14.605993

    CPU = 00

    IPL = 21

    Mode = Interrupt

    PID = 34400430 MEHLHOP

    PC = FFFFFFFF.8003C200 BOO$READB_C SYS$PLATFORM_SUPPORT

    B0 = FFFFFFFF.810B4D90 SYS$SRDRIVER+11B90

    R2 = 00000000.7FF43040

    R3 = 00000000.F4050005

    R4 = 00000000.00000000

    R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)

    R6 = FFFFFFFF.89008600 DDB

    R7 = 00000000.00000038

    Time = 14-MAY 17:27:14.594993

    CPU = 00

    IPL = 21

    Mode = Interrupt

    PID = 34400430 MEHLHOP

    PC = FFFFFFFF.810B4D90 SYS$SRDRIVER+11B90 SYS$SRDRIVER

    B0 = FFFFFFFF.810B4D90 SYS$SRDRIVER+11B90

    R2 = 00000000.7FF43040

    R3 = 00000000.00000060

    R4 = 00000000.00000000

    R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)

    R6 = FFFFFFFF.89008600 DDB

    R7 = 00000000.00000038

    .

  • Troubleshooting a Hung System Integrity (cont)

    OpenVMS Operating System, Version V8.3 -- System Dump Analysis 14-MAY-2007 17:27:14.60 Page 32

    PC sampler information:

    Time = 14-MAY 17:27:13.450993

    CPU = 00

    IPL = 8

    Mode = Interrupt

    PID = 34400430 MEHLHOP

    PC = FFFFFFFF.80145980 EXE$TIMEOUT_C+00350 SYSTEM_PRIMITIVES_MIN

    B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250

    R2 = FFFFFFFF.88033228 ECC$REENABLE+00028

    R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST

    R4 = 00000000.00000000

    R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)

    R6 = FFFFFFFF.89008600 DDB

    R7 = 00000000.00000038

    Time = 14-MAY 17:27:13.439993

    CPU = 00

    IPL = 8

    Mode = Interrupt

    PID = 34400430 MEHLHOP

    PC = FFFFFFFF.801459B0 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN

    B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250

    R2 = FFFFFFFF.88033228 ECC$REENABLE+00028

    R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST

    R4 = 00000000.00000000

    R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)

    R6 = FFFFFFFF.89008600 DDB

    R7 = 00000000.00000038

  • Troubleshooting a Hung System Integrity (cont)

    Time = 14-MAY 17:27:13.428993

    CPU = 00

    IPL = 8

    Mode = Interrupt

    PID = 34400430 MEHLHOP

    PC = FFFFFFFF.80145970 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN

    B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250

    R2 = FFFFFFFF.88033228 ECC$REENABLE+00028

    R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST

    R4 = 00000000.00000000

    R5 = FFFFFFFF.88F7C760

    R6 = FFFFFFFF.89008600 DDB

    R7 = 00000000.00000038

    Time = 14-MAY 17:27:13.417993

    CPU = 00

    IPL = 8

    Mode = Interrupt

    PID = 34400430 MEHLHOP

    PC = FFFFFFFF.80145950 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN

    B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250

    R2 = FFFFFFFF.88033228 ECC$REENABLE+00028

    R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST

    R4 = 00000000.00000000

    R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)

    R6 = FFFFFFFF.89008600 DDB

    R7 = 00000000.00000038

  • Troubleshooting a Hung System AXP

    SYS6$ run hang_ddb_corrupt

    (Hit the HALT switch instead of ^P)

    halted CPU 0

    halt code = 1

    operator initiated halt

    PC = ffffffff80051bac

    >>>crash

    CPU 0 restarting

    **** OpenVMS Alpha Operating System V8.2 - BUGCHECK ****

    ** Bugcheck code = 0000064C: OPERCRASH, Operator forced system crash

    ** Crash CPU: 00 Primary CPU: 00 Active CPUs: 00000001

    ** Current Process = MEHLHOP

    ** Current PSB ID = 00000001

    ** Image Name =

    **** Starting full memory dump at 30-APR-2007 16:51...

    ................................................................................

    ................................................................................

    .

    .

    .

    ................................................................................

    ................................................................................

    ....................

    ...Complete ****

    halted CPU 0

    halt code = 5

    HALT instruction executed

    PC = ffffffff80081b34

    CPU 0 booting

  • Troubleshooting High MP Synch time

    CLASS8$ run hispin

    CLASS8$ an/sy

    OpenVMS system analyzer

    SDA> spl sho trace

    Spinlock Trace Information:

    ---------------------------

    Timestamp CPU Spin/Forklock/IPL Caller's/Fork PC EPID Operation Trace Buffer

    ---------------------- --- --------------------- -------------------------------------- -------- ----------------- -----------------

    16-MAY 11:54:57.378645 01 881C0B00 MMG 8067DB60 SYS$VM+35660 00000000 Release FFFFFFFF.7DC06E20

    16-MAY 11:54:57.378640 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06DF8

    16-MAY 11:54:57.378618 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06DD0

    16-MAY 11:54:57.378617 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06DA8

    16-MAY 11:54:57.378608 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06D80

    16-MAY 11:54:57.378606 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06D58

    16-MAY 11:54:57.378595 01 881C0B00 MMG 8067DB60 SYS$VM+35660 00000000 Release FFFFFFFF.7DC06D30

    16-MAY 11:54:57.378591 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06D08

    16-MAY 11:54:57.378589 01 881C0B00 MMG 8067BA70 SYS$VM+33570 00000000 Release FFFFFFFF.7DC06CE0

    16-MAY 11:54:57.378586 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06CB8

    16-MAY 11:54:57.378528 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06C90

    16-MAY 11:54:57.378527 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06C68

    16-MAY 11:54:57.378523 01 881C0B00 MMG 80673A70 MMG_STD$IN_REGION_64_C+00830 00000000 Release FFFFFFFF.7DC06C40

    16-MAY 11:54:57.378523 01 881C0B00 MMG 80673900 MMG_STD$IN_REGION_64_C+006C0 00000000 Acquire FFFFFFFF.7DC06C18

    16-MAY 11:54:57.378520 01 881C0B00 MMG 806E8D70 MMG_STD$CLONE_REGIONS_64_C+00 00000000 Release FFFFFFFF.7DC06BF0

    16-MAY 11:54:57.378520 01 881C0B00 MMG 806E8D00 MMG_STD$CLONE_REGIONS_64_C+00 00000000 Acquire FFFFFFFF.7DC06BC8

    16-MAY 11:54:57.378506 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06BA0

    Press RETURN for more.

  • Troubleshooting High MP Synch time (cont)

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Timestamp CPU Spin/Forklock/IPL Caller's/Fork PC EPID Operation Trace Buffer

    ---------------------- --- --------------------- -------------------------------------- -------- ----------------- -----------------

    16-MAY 11:54:57.378504 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06B78

    16-MAY 11:54:57.378500 01 881C0B00 MMG 8067DB60 SYS$VM+35660 00000000 Release FFFFFFFF.7DC06B50

    16-MAY 11:54:57.378491 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06B28

    16-MAY 11:54:57.378487 01 881C0A00 SCHED 8052C250 PROCESS_MANAGEMENT+1A550 00000000 Release FFFFFFFF.7DC06B00

    16-MAY 11:54:57.378484 01 881C0A00 SCHED 8052BE80 PROCESS_MANAGEMENT+1A180 00000000 Acquire FFFFFFFF.7DC06AD8

    16-MAY 11:54:57.378483 01 881C0A00 SCHED 80586AD0 SCH$QEND_C+00AF0 00000000 Release FFFFFFFF.7DC06AB0

    16-MAY 11:54:57.378482 01 881C0A00 SCHED 80586500 SCH$QEND_C+00520 00000000 Acquire FFFFFFFF.7DC06A88

    16-MAY 11:54:57.378482 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire FFFFFFFF.7DC06A60

    16-MAY 11:54:57.378481 01 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 34600442 Release FFFFFFFF.7DC06A38

    16-MAY 11:54:57.377999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06A10

    16-MAY 11:54:57.377999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC069E8

    16-MAY 11:54:57.376999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC069C0

    16-MAY 11:54:57.376999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06998

    16-MAY 11:54:57.376233 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire (spin) FFFFFFFF.7DC06970

    16-MAY 11:54:57.376232 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire FFFFFFFF.7DC06948

    16-MAY 11:54:57.376232 00 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 00000000 Release FFFFFFFF.7DC06920

    16-MAY 11:54:57.375999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC068F8

    16-MAY 11:54:57.375999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC068D0

    16-MAY 11:54:57.374999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC068A8

    16-MAY 11:54:57.374999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06880

    16-MAY 11:54:57.373999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06858

    16-MAY 11:54:57.373999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06830

    16-MAY 11:54:57.373980 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire (spin) FFFFFFFF.7DC06808

    16-MAY 11:54:57.373981 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire FFFFFFFF.7DC067E0

    16-MAY 11:54:57.373980 01 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 34600442 Release FFFFFFFF.7DC067B8

    16-MAY 11:54:57.372999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06790

    16-MAY 11:54:57.372999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06768

    16-MAY 11:54:57.371999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06740

    16-MAY 11:54:57.371999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06718

    16-MAY 11:54:57.371731 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire (spin) FFFFFFFF.7DC066F0

    16-MAY 11:54:57.371730 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire FFFFFFFF.7DC066C8

    16-MAY 11:54:57.371730 00 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 00000000 Release FFFFFFFF.7DC066A0

    16-MAY 11:54:57.370999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06678

    16-MAY 11:54:57.370999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06650

  • Troubleshooting High MP Synch time (cont)

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Timestamp CPU Spin/Forklock/IPL Caller's/Fork PC EPID Operation Trace Buffer

    ---------------------- --- --------------------- -------------------------------------- -------- ----------------- -----------------

    16-MAY 11:54:57.362728 00 TIMER 80143560 SYSTEM_PRIMITIVES_MIN+0012356 Frk Dsp Start FFFFFFFF.7DC060D8

    16-MAY 11:54:57.362728 00 881C0D00 TIMER 80143140 EXE_STD$IOFORK_CPU_C+00EC0 00000000 Acquire FFFFFFFF.7DC060B0

    16-MAY 11:54:57.362727 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire FFFFFFFF.7DC06088

    16-MAY 11:54:57.362728 00 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 00000000 Release FFFFFFFF.7DC06060

    16-MAY 11:54:57.361999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06038

    16-MAY 11:54:57.361999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06010

    16-MAY 11:54:57.361000 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC05FE8

    16-MAY 11:54:57.360999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC05FC0

    16-MAY 11:54:57.360512 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire (spin) FFFFFFFF.7DC05F98

    16-MAY 11:54:57.360511 01 IPL 08 Frk Dsp End FFFFFFFF.7DC05F70

    16-MAY 11:54:57.360511 01 88E09C80 EWA$PORT 80295960 SYS$LAN_CSMACD+32360 00000000 Restorel FFFFFFFF.7DC05F48

    16-MAY 11:54:57.360511 01 88E09C80 EWA$PORT 802957B0 SYS$LAN_CSMACD+321B0 00000000 Acqnoipl FFFFFFFF.7DC05F20

    16-MAY 11:54:57.360511 01 88E4D540 PEA$PORT 81128000 SYS$PEDRIVER+53A00 00000000 Restorel FFFFFFFF.7DC05EF8

    16-MAY 11:54:57.360511 01 88E4D540 PEA$PORT 81127EE0 SYS$PEDRIVER+538E0 00000000 Acqnoipl FFFFFFFF.7DC05ED0

    16-MAY 11:54:57.360510 01 IPL 08 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E Frk Dsp Start FFFFFFFF.7DC05EA8

    16-MAY 11:54:57.360510 01 IPL 08 Frk Dsp End FFFFFFFF.7DC05E80

    16-MAY 11:54:57.360510 01 88E09C80 EWA$PORT 80295960 SYS$LAN_CSMACD+32360 00000000 Restorel FFFFFFFF.7DC05E58

    16-MAY 11:54:57.360510 01 88E09C80 EWA$PORT 802957B0 SYS$LAN_CSMACD+321B0 00000000 Acqnoipl FFFFFFFF.7DC05E30

    16-MAY 11:54:57.360510 01 88E4D540 PEA$PORT 81128000 SYS$PEDRIVER+53A00 00000000 Restorel FFFFFFFF.7DC05E08

    16-MAY 11:54:57.360509 01 88E4D540 PEA$PORT 81127EE0 SYS$PEDRIVER+538E0 00000000 Acqnoipl FFFFFFFF.7DC05DE0

    16-MAY 11:54:57.360509 01 IPL 08 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E Frk Dsp Start FFFFFFFF.7DC05DB8

    16-MAY 11:54:57.360508 01 IPL 08 Frk Dsp End FFFFFFFF.7DC05D90

    16-MAY 11:54:57.360508 01 88E09C80 EWA$PORT 810C5220 SYS$EW5700+1E320 00000000 Restorel FFFFFFFF.7DC05D68

    16-MAY 11:54:57.360505 01 88E4D540 PEA$PORT 811284C0 SYS$PEDRIVER+53EC0 00000000 Restorel FFFFFFFF.7DC05D40

    16-MAY 11:54:57.360505 01 88E09C80 EWA$PORT 8026B760 LAN$RETURN_RCV_VCRP_C+00240 00000000 Restorel FFFFFFFF.7DC05D18

    16-MAY 11:54:57.360505 01 88E09C80 EWA$PORT 8026B5E0 LAN$RETURN_RCV_VCRP_C+000C0 00000000 Acqnoipl (own) FFFFFFFF.7DC05CF0

    16-MAY 11:54:57.360503 01 88E4D540 PEA$PORT 811283D0 SYS$PEDRIVER+53DD0 00000000 Acquire (nospin) FFFFFFFF.7DC05CC8

    16-MAY 11:54:57.360501 01 88E4D540 PEA$PORT 811284C0 SYS$PEDRIVER+53EC0 00000000 Restorel FFFFFFFF.7DC05CA0

    16-MAY 11:54:57.360501 01 88E09C80 EWA$PORT 8026B760 LAN$RETURN_RCV_VCRP_C+00240 00000000 Restorel FFFFFFFF.7DC05C78

    16-MAY 11:54:57.360500 01 88E09C80 EWA$PORT 8026B5E0 LAN$RETURN_RCV_VCRP_C+000C0 00000000 Acqnoipl (own) FFFFFFFF.7DC05C50

    16-MAY 11:54:57.360491 01 88E4D540 PEA$PORT 811283D0 SYS$PEDRIVER+53DD0 00000000 Acquire (nospin) FFFFFFFF.7DC05C28

    16-MAY 11:54:57.360486 01 88E09C80 EWA$PORT 810C4960 SYS$EW5700+1DA60 00000000 Acqnoipl FFFFFFFF.7DC05C00

    16-MAY 11:54:57.360486 01 IPL 08 810C48C0 SYS$EW5700+1D9C0 Frk Dsp Start FFFFFFFF.7DC05BD8

  • Troubleshooting High MP Synch time (cont)

    SDA> spl sho trace/sum

    Spinlock Trace Information: (at 16-MAY-2007 13:13:42.55, trace time 00:00:42.377022)

    ------------------------------------------------------------------------------------

    Events Acquires Releases Acq Own Acq NoSpin Spinwaits %

    Spinlock /sec /sec /sec /sec /sec /sec Spinwait

    ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

    MEGA 2.0 1.0 1.0 0.0 0.0 0.0 0.0

    HWCLK 2000.0 1000.0 1000.0 0.0 0.0 0.0 0.0

    INVALIDATE 4.9 2.4 2.4 0.0 0.0 0.0 0.0

    PERFMON 10.7 5.3 5.3 0.0 0.0 0.0 0.0

    SCHED 1872.0 935.6 935.6 0.0 0.0 0.7 0.1

    MMG 169.5 83.3 84.6 0.1 1.3 0.2 0.2

    TIMER 617.5 306.5 306.5 0.0 0.0 4.4 1.4

    TX_SYNCH 0.9 0.5 0.5 0.0 0.0 0.0 0.0

    IOLOCK8 147.7 73.6 73.8 0.3 0.0 0.0 0.1

    LCKMGR 372.7 130.8 131.5 0.0 0.7 109.8 83.5

    FILSYS 4.6 2.3 2.3 0.0 0.0 0.0 0.0

    Dynamic 631.5 279.0 315.1 22.3 14.8 0.2 0.1

    ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

    5833.9 2820.3 2858.8 22.8 16.7 115.3 4.1

    Press RETURN for more.

  • Troubleshooting High MP Synch time (cont)

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Events Acquires or Spins % Own

    Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset

    ---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------

    MEGA 1.0 1.0 Rel/s 0.0 0.0 0.0 80147620 SYSTEM_PRIMITIVES_MIN+001276 SYSTEM_PRIMITIVES_MIN 00127620

    MEGA 1.0 1.0 Acq/s 0.0 0.0 0.0 80146BB0 SYSTEM_PRIMITIVES_MIN+00126B SYSTEM_PRIMITIVES_MIN 00126BB0

    HWCLK 1000.0 1000.0 Acq/s 0.0 0.0 0.0 801496B0 SYSTEM_PRIMITIVES_MIN+001296 SYSTEM_PRIMITIVES_MIN 001296B0

    HWCLK 1000.0 1000.0 Rel/s 0.0 0.0 0.0 80149960 SYSTEM_PRIMITIVES_MIN+001299 SYSTEM_PRIMITIVES_MIN 00129960

    INVALIDATE 1.7 1.7 Rel/s 0.0 0.0 0.0 80725F30 SYS$VM+DDA30 SYS$VM 000DDA30

    INVALIDATE 1.7 1.7 Acq/s 0.0 0.0 0.0 80724E50 MMG$TBI_DATA_64_THREADS_C+00 SYS$VM 000DC950

    INVALIDATE 0.7 0.7 Acq/s 0.0 0.0 0.0 80728030 SYS$VM+DFB30 SYS$VM 000DFB30

    INVALIDATE 0.7 0.7 Rel/s 0.0 0.0 0.0 80729110 SYS$VM+E0C10 SYS$VM 000E0C10

    PERFMON 2.1 2.1 Acq/s 0.0 0.0 0.0 80440E40 PMS$END_RQ_C+001B0 IO_ROUTINES 00021E40

    PERFMON 2.1 2.1 Acq/s 0.0 0.0 0.0 80441F10 PMS$START_RQ_C+00230 IO_ROUTINES 00022F10

    PERFMON 2.1 2.1 Rel/s 0.0 0.0 0.0 80442230 PMS$START_RQ_C+00550 IO_ROUTINES 00023230

    PERFMON 1.4 1.4 Rel/s 0.0 0.0 0.0 804414E0 PMS$END_RQ_C+00850 IO_ROUTINES 000224E0

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Events Acquires or Spins % Own

    Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset

    ---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------

    PERFMON 0.7 0.7 Rel/s 0.0 0.0 0.0 80440F70 PMS$END_RQ_C+002E0 IO_ROUTINES 00021F70

    PERFMON 0.7 0.7 Acq/s 0.0 0.0 0.0 80440820 PMS$END_IO_C+00190 IO_ROUTINES 00021820

    PERFMON 0.7 0.7 Rel/s 0.0 0.0 0.0 80440930 PMS$END_IO_C+002A0 IO_ROUTINES 00021930

    PERFMON 0.3 0.3 Acq/s 0.0 0.0 0.0 80441840 PMS$START_IO_C+00190 IO_ROUTINES 00022840

    PERFMON 0.3 0.3 Rel/s 0.0 0.0 0.0 80441980 PMS$START_IO_C+002D0 IO_ROUTINES 00022980

    PERFMON 0.0 0.0 Rel/s 0.0 0.0 0.0 011B8720

    SCHED 686.8 686.8 Acq/s 0.0 0.0 0.0 011A8CE0

    SCHED 686.8 686.8 Rel/s 0.0 0.0 0.0 011A8E80

    SCHED 136.0 136.0 Rel/s 0.0 0.0 0.0 011AD290

    SCHED 136.0 136.0 Acq/s 0.0 0.0 0.0 011ACFB0

    SCHED 32.7 32.7 Acq/s 0.0 0.0 0.0 815B89E0 PSDC$LOADABLE_IMAGE+147E0 PSDC$LOADABLE_IMAGE 000147E0

    SCHED 32.7 32.7 Rel/s 0.0 0.0 0.0 815B8770 PSDC$LOADABLE_IMAGE+14570 PSDC$LOADABLE_IMAGE 00014570

    SCHED 23.1 23.1 Rel/s 0.0 0.0 0.0 8052C250 PROCESS_MANAGEMENT+1A550 PROCESS_MANAGEMENT 0001A550

    SCHED 17.0 16.8 Acq/s 0.2 1.1 0.0 805CF370 PROCESS_MANAGEMENT+BD670 PROCESS_MANAGEMENT 000BD670

    SCHED 14.1 13.8 Acq/s 0.3 2.2 0.0 805654E0 SCH$CALC_CPU_LOAD_C+00E20 PROCESS_MANAGEMENT 000537E0

    SCHED 13.8 13.8 Rel/s 0.0 0.0 0.0 80564FB0 SCH$CALC_CPU_LOAD_C+008F0 PROCESS_MANAGEMENT 000532B0

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Events Acquires or Spins % Own

    Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset

    ---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------

    MMG 53.2 53.1 Acq/s 0.1 0.2 0.0 800C0E40 MMG$READ_DATA_PHYS_C+00280 SYSTEM_PRIMITIVES_MIN 000A0E40

    MMG 53.1 53.1 Rel/s 0.0 0.0 0.0 800C1AE0 MMG$READ_DATA_PHYS_C+00F20 SYSTEM_PRIMITIVES_MIN 000A1AE0

    MMG 19.8 19.7 Acq/s 0.1 0.5 0.0 800C1D50 MMG$WRITE_DATA_PHYS_C+001E0 SYSTEM_PRIMITIVES_MIN 000A1D50

    MMG 19.7 19.7 Rel/s 0.0 0.0 0.0 800C2AC0 MMG$WRITE_DATA_PHYS_C+00F50 SYSTEM_PRIMITIVES_MIN 000A2AC0

    MMG 2.0 2.0 Acq/s 0.0 0.0 0.0 80678C10 MMG$PAGEFAULT_C+00190 SYS$VM 00030710

    MMG 1.6 1.6 Acq/s 0.0 0.0 0.0 8017D660 MMG$PTEREF_64_C+002C0 SYSTEM_PRIMITIVES_MIN 0015D660

    MMG 1.1 1.1 Rel/s 0.0 0.0 0.0 80673A70 MMG_STD$IN_REGION_64_C+00830 SYS$VM 0002B570

    MMG 1.1 1.1 Rel/s 0.0 0.0 0.0 80679610 MMG$PAGEFAULT_C+00B90 SYS$VM 00031110

    MMG 1.1 1.1 Rel/s 0.0 0.0 0.0 806A6950 SYS$VM+5E450 SYS$VM 0005E450

    MMG 0.9 0.9 Acq/s 0.0 0.0 0.0 80673900 MMG_STD$IN_REGION_64_C+006C0 SYS$VM 0002B400

    TIMER 118.6 116.2 Acq/s 2.4 2.1 0.0 80143D90 SYSTEM_PRIMITIVES_MIN+00123D SYSTEM_PRIMITIVES_MIN 00123D90

    TIMER 103.6 101.6 Acq/s 2.0 2.0 0.0 80157B80 EXE$INSTIMQ_C+000C0 SYSTEM_PRIMITIVES_MIN 00137B80

    TIMER 103.6 103.6 Rel/s 0.0 0.0 0.0 801440E0 SYSTEM_PRIMITIVES_MIN+001240 SYSTEM_PRIMITIVES_MIN 001240E0

    TIMER 101.6 101.6 Rel/s 0.0 0.0 0.0 80157D00 EXE$INSTIMQ_C+00240 SYSTEM_PRIMITIVES_MIN 00137D00

    TIMER 72.2 72.2 Acq/s 0.0 0.0 0.0 80143140 EXE_STD$IOFORK_CPU_C+00EC0 SYSTEM_PRIMITIVES_MIN 00123140

    TIMER 72.2 72.2 Rel/s 0.0 0.0 0.0 80143450 SYSTEM_PRIMITIVES_MIN+001234 SYSTEM_PRIMITIVES_MIN 00123450

    TIMER 12.6 12.6 Rel/s 0.0 0.0 0.0 80144590 SYSTEM_PRIMITIVES_MIN+001245 SYSTEM_PRIMITIVES_MIN 00124590

    TIMER 12.6 12.6 Acq/s 0.0 0.0 0.0 805E00A0 PROCESS_MANAGEMENT+CE3A0 PROCESS_MANAGEMENT 000CE3A0

    TIMER 12.6 12.6 Rel/s 0.0 0.0 0.0 805E02F0 PROCESS_MANAGEMENT+CE5F0 PROCESS_MANAGEMENT 000CE5F0

    TIMER 3.0 3.0 Acq/s 0.0 0.0 0.0 805DF180 PROCESS_MANAGEMENT+CD480 PROCESS_MANAGEMENT 000CD480

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Events Acquires or Spins % Own

    Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset

    ---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------

    TX_SYNCH 0.5 0.5 Rel/s 0.0 0.0 0.0 80C45790 EXE$GENERATE_UID_C+004C0 SYS$TRANSACTION_SERVICES 00074A90

    TX_SYNCH 0.5 0.5 Acq/s 0.0 0.0 0.0 80C45390 EXE$GENERATE_UID_C+000C0 SYS$TRANSACTION_SERVICES 00074690

    IOLOCK8 48.6 48.6 Rel/s 0.0 0.0 0.0 810EF680 SYS$PEDRIVER+1B080 SYS$PEDRIVER 0001B080

    IOLOCK8 48.6 48.6 Acq/s 0.0 0.0 0.0 810EF200 SYS$PEDRIVER+1AC00 SYS$PEDRIVER 0001AC00

    IOLOCK8 9.9 9.9 Acq/s 0.0 0.0 0.0 812B5D10 SYS$LTDRIVER+17810 SYS$LTDRIVER 00017810

    IOLOCK8 8.9 8.9 Rel/s 0.0 0.0 0.0 812B60C0 SYS$LTDRIVER+17BC0 SYS$LTDRIVER 00017BC0

    IOLOCK8 6.9 6.9 Acq/s 0.0 0.0 0.0 80145B00 EXE$TIMEOUT_C+004D0 SYSTEM_PRIMITIVES_MIN 00125B00

    IOLOCK8 6.9 6.9 Rel/s 0.0 0.0 0.0 80145D60 EXE$TIMEOUT_C+00730 SYSTEM_PRIMITIVES_MIN 00125D60

    IOLOCK8 2.0 1.9 Acq/s 0.0 1.2 0.0 810F5B20 SYS$PEDRIVER+21520 SYS$PEDRIVER 00021520

    IOLOCK8 1.9 1.9 Rel/s 0.0 0.0 0.0 810F5C90 SYS$PEDRIVER+21690 SYS$PEDRIVER 00021690

    IOLOCK8 1.0 1.0 Rel/s 0.0 0.0 0.0 8125E610 NETDRIVER+32D10 NETDRIVER 00032D10

    IOLOCK8 1.0 1.0 Rel/s 0.0 0.0 0.0 812B7570 SYS$LTDRIVER+19070 SYS$LTDRIVER 00019070

    LCKMGR 232.4 124.7 Acq/s 107.7 86.4 0.0 00020200 SYS$K_VERSION_08+001E0

    LCKMGR 124.7 124.7 Rel/s 0.0 0.0 0.0 000202E0 SYS$K_VERSION_08+002C0

    LCKMGR 2.2 2.0 Acq/s 0.2 11.9 0.0 807A87A0 LCK$CALCULATE_LOCKIDTBL_MAX_ LOCKING 0004FBA0

    LCKMGR 2.0 2.0 Rel/s 0.0 0.0 0.0 807A8820 LCK$CALCULATE_LOCKIDTBL_MAX_ LOCKING 0004FC20

    LCKMGR 1.2 1.0 Acq/s 0.2 23.8 0.0 80146A90 SYSTEM_PRIMITIVES_MIN+00126A SYSTEM_PRIMITIVES_MIN 00126A90

    LCKMGR 1.2 1.0 Acq/s 0.2 19.0 0.0 80DA3580 SYS$CLUSTER+98B80 SYS$CLUSTER 00098B80

    LCKMGR 1.0 0.6 Acq/s 0.4 59.3 0.0 8077DA50 LCK$ENQUE_C+00B90 LOCKING 00024E50

    LCKMGR 1.0 1.0 Rel/s 0.0 0.0 0.0 80147820 SYSTEM_PRIMITIVES_MIN+001278 SYSTEM_PRIMITIVES_MIN 00127820

    LCKMGR 1.0 1.0 Rel/s 0.0 0.0 0.0 80DA37A0 SYS$CLUSTER+98DA0 SYS$CLUSTER 00098DA0

    LCKMGR 0.8 0.8 Rel/s 0.0 0.0 0.0 8076FD70 LOCKING+17170 LOCKING 00017170

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Spinlock Trace Information:

    ---------------------------

    Events Acquires or Spins % Own

    Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset

    ---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------

    FILSYS 0.4 0.4 Rel/s 0.0 0.0 0.0 806BCB50 SYS$VM+74650 SYS$VM 00074650

    FILSYS 0.4 0.4 Acq/s 0.0 0.0 0.0 806BCA10 SYS$VM+74510 SYS$VM 00074510

    FILSYS 0.4 0.4 Acq/s 0.0 0.0 0.0 8082CD10 F11BXQP+2EF10 F11BXQP 0002EF10

    FILSYS 0.4 0.4 Rel/s 0.0 0.0 0.0 8082CD80 F11BXQP+2EF80 F11BXQP 0002EF80

    FILSYS 0.3 0.3 Acq/s 0.0 0.0 0.0 80841CF0 F11BXQP+43EF0 F11BXQP 00043EF0

    FILSYS 0.3 0.3 Rel/s 0.0 0.0 0.0 808426D0 F11BXQP+448D0 F11BXQP 000448D0

    FILSYS 0.2 0.2 Acq/s 0.0 0.0 0.0 80819D20 F11BXQP+1BF20 F11BXQP 0001BF20

    FILSYS 0.2 0.2 Rel/s 0.0 0.0 0.0 808196D0 F11BXQP+1B8D0 F11BXQP 0001B8D0

    FILSYS 0.2 0.2 Acq/s 0.0 0.0 0.0 80819570 F11BXQP+1B770 F11BXQP 0001B770

    FILSYS 0.2 0.2 Rel/s 0.0 0.0 0.0 80819E50 F11BXQP+1C050 F11BXQP 0001C050

    Dynamic 50.8 50.8 Acq/s 0.0 0.0 0.0 810EC540 SYS$PEDRIVER+17F40 SYS$PEDRIVER 00017F40

    Dynamic 50.8 50.8 Rel/s 0.0 0.0 0.0 810EC730 SYS$PEDRIVER+18130 SYS$PEDRIVER 00018130

    Dynamic 48.7 48.6 Acq/s 0.0 0.0 0.0 810EF390 SYS$PEDRIVER+1AD90 SYS$PEDRIVER 0001AD90

    Dynamic 48.6 48.6 Rel/s 0.0 0.0 0.0 810EF550 SYS$PEDRIVER+1AF50 SYS$PEDRIVER 0001AF50

    Dynamic 23.0 23.0 Acq/s 0.0 0.0 0.0 805CEE30 PROCESS_MANAGEMENT+BD130 PROCESS_MANAGEMENT 000BD130

    Dynamic 20.8 20.8 Rel/s 0.0 0.0 0.0 81348B00 TCPIP$INTERNET_SERVICES+1C80 TCPIP$INTERNET_SERVICES 0001C800

    Dynamic 20.8 20.8 Acq/s 0.0 0.0 0.0 81348670 TCPIP$INTERNET_SERVICES+1C37 TCPIP$INTERNET_SERVICES 0001C370

    Dynamic 16.8 16.8 Rel/s 0.0 0.0 0.0 805CF3F0 PROCESS_MANAGEMENT+BD6F0 PROCESS_MANAGEMENT 000BD6F0

    Dynamic 16.6 16.6 Rel/s 0.0 0.0 0.0 80625DE0 PROCESS_MANAGEMENT+001140E0 PROCESS_MANAGEMENT 001140E0

    Dynamic 16.6 3.6 Acq/s 0.0 0.0 13.0 80624980 SCH_STD$QAST_C+00300 PROCESS_MANAGEMENT 00112C80

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Forklock Trace Information: (at 16-MAY-2007 13:13:42.55, trace time 00:00:42.377022)

    ------------------------------------------------------------------------------------

    Total CPU ID

    Forklock Events/sec 0 1

    ---------- ---------- ------------

    IPL 06 0.1 0.1 0.0

    IPL 08 103.3 54.5 48.8

    TIMER 72.2 72.2 0.0

    IOLOCK8 0.2 0.0 0.2

    LCKMGR 0.3 0.3 0.0

    ---------- ------------

    Totals 176.0 127.0 49.0

    Press RETURN for more.

    SDA>

    Forklock Trace Information:

    ---------------------------

    Average Minimum Maximum

    Forklock Event/sec % Time Held (nsec) (nsec) (nsec) Fork PC

    ---------- --------- --------------- --------- --------- --------- ----------------------------------------

    IPL 06 0.1 0.0 114915 43565 153208 80185C10 EXE$KP_DEALLOCATE_KPB_C+00680

    ---------- --------- ---------------

    Totals 0.1 0.0

    IPL 08 50.8 0.0 2496 1523 41758 810EC410 SYS$PEDRIVER+17E10

    IPL 08 20.8 0.0 6701 2216 63843 81348580 TCPIP$INTERNET_SERVICES+1C280

    IPL 08 16.3 0.0 6056 1819 22713 810C48C0 SYS$EW5700+1D9C0

    IPL 08 6.5 0.0 1613 886 4989 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E80

    IPL 08 3.9 0.0 695 554 1149 813C2200 TCPIP$INTERNET_SERVICES+95F00

    IPL 08 2.0 0.0 6275 4036 10754 8024D320 SYS$EIDRIVER+1AD20

    IPL 08 1.2 0.0 5305 3159 10131 81120360 SYS$PEDRIVER+4BD60

    IPL 08 1.1 0.0 11958 2144 46351 8151D030 TCPIP$INTERNET_SERVICES+001F0D3

    IPL 08 0.4 0.0 9222 5728 15571 80FC1240 SYS$TTDRIVER+2AF40

    IPL 08 0.3 0.0 6576 4236 11479 80FC02C0 SYS$TTDRIVER+29FC0

    ---------- --------- ---------------

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Forklock Trace Information:

    ---------------------------

    Average Minimum Maximum

    Forklock Event/sec % Time Held (nsec) (nsec) (nsec) Fork PC

    ---------- --------- --------------- --------- --------- --------- ----------------------------------------

    Totals 103.3 0.0

    TIMER 72.2 0.1 17443 172 4664164 80143560 SYSTEM_PRIMITIVES_MIN+00123560

    ---------- --------- ---------------

    Totals 72.2 0.1

    IOLOCK8 0.2 0.0 2812 1237 5833 80297130 SYS$LAN_CSMACD+33B30

    IOLOCK8 0.0 0.0 2555 2538 2573 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E80

    ---------- --------- ---------------

    Totals 0.2 0.0

    LCKMGR 0.3 0.0 13346 2586 30121 80D45030 SYS$CLUSTER+3A630

    ---------- --------- ---------------

    Totals 0.3 0.0

    ========== ========= ===============

    Totals 176.0 0.2

  • Troubleshooting High MP Synch time (cont)

    After looking at the data I wanted to know WHAT was accessing the SCHED spinlock.

    Ran problem program again

    Forced a crash

    Analyzed that crash

    SDA> set output hispin.out

    SDA> spl show trace

    $ search hispin.out pc

    SDA> show process/id=pid from search

  • Troubleshooting High MP Synch time (cont)

    SDA> set out delme_hispin.out

    SDA> spl sho trace

    CLASS8::MEHLHOP 21:36:33 SDA CPU=00:01:02.53 PF=1554 IO=3376 MEM=1257

    CLASS8::MEHLHOP 21:36:41 SDA CPU=00:01:09.29 PF=1554 IO=3648 MEM=1257

    CLASS8::MEHLHOP 21:38:11 SDA CPU=00:02:23.72 PF=1554 IO=6720 MEM=1257

    CLASS8::MEHLHOP 21:39:17 SDA CPU=00:03:16.56 PF=1554 IO=9079 MEM=1257

    Interrupt

    CLASS8$ exit

    CLASS8$ search delme_hispin.out 011A8CE0

    20-MAY 21:29:35.000984 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CF40

    20-MAY 21:29:35.000983 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CEF0

    20-MAY 21:29:35.000983 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CEA0

    20-MAY 21:29:35.000982 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CE50

    20-MAY 21:29:35.000982 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CE00

    20-MAY 21:29:35.000981 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CDB0

    20-MAY 21:29:35.000981 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CD60

    20-MAY 21:29:35.000980 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CD10

    20-MAY 21:29:35.000979 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CCC0

    20-MAY 21:29:35.000979 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CC70

    20-MAY 21:29:35.000978 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CC20

    20-MAY 21:29:35.000978 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CBD0

    20-MAY 21:29:35.000977 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CB80

    20-MAY 21:29:35.000977 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CB30

    CLASS8$ an/cr sys$system

  • Troubleshooting High MP Synch time (cont)

    SDA> sho proc/id=34E00430

    Process index: 0030 Name: PSDC$DC_SERVER Extended PID: 34E00430

    --------------------------------------------------------------------

    Process status: 00140011 RES,PSWAPM,PHDRES,LOGIN

    status2: 00000000

    PCB address 89126980 JIB address 89122A40

    PHD address 8DA92000 Swapfile disk address 00000000

    KTB vector address 89126CA8 HWPCB address FFFFFFFF.8DA92080

    Callback vector address 00000000 Termination mailbox 0034

    Master internal PID 00010030 Subprocess count 0

    Creator extended PID 00000000 Creator internal PID 00000000

    Previous CPU Id 00000000 Current CPU Id 00000000

    Previous ASNSEQ 0000000000000001 Previous ASN 0000000000FD6DEB

    Initial process priority 15 # open files remaining 125/128

    Delete pending count 0 Direct I/O count/limit 100/100

    UIC [00001,000004] Buffered I/O count/limit 100/100

    Abs time of last event 0134E8B6 BUFIO byte count/limit 261440/261440

    # of threads 1 ASTs remaining 98/100

    Swapped copy of LEFC0 00000000 Timer entries remaining 14/16

    Swapped copy of LEFC1 00000000 Active page table count 0

    Global cluster 2 pointer 00000000 Process WS page count 2385

    Global cluster 3 pointer 00000000 Global WS page count 21

    PCB Specific Spinlock 89127280 Subprocesses in job 0

  • Troubleshooting High MP Synch time (cont)

    Press RETURN for more.

    SDA>

    Process index: 0030 Name: PSDC$DC_SERVER Extended PID: 34E00430

    --------------------------------------------------------------------

    Thread index: 0000

    ------------------

    Current capabilities: System: 000C QUORUM,RUN

    User: 0000.00000000

    Permanent capabilities: System: 000C QUORUM,RUN

    User: 0000.00000000

    Current affinities: 00000000.00000000

    Permanent affinities: 00000000.00000000

    Thread status: 00140011

    status2: 00000000

    KTB address 89126980 HWPCB address FFFFFFFF.8DA92080

    PKTA address 7FFEFF98 Callback vector address 00000000

    Internal PID 00010030 Callback error 00000000

    Extended PID 34E00430 Current CPU id 00000000

    State HIB Flags 00000000

    Base priority 15 Current priority 15

    Waiting EF cluster 4 Event flag wait mask 0000000D

    CPU since last quantum FFFFFFFE Mutex count 0

    ASTs active NONE

    SDA>

  • Troubleshooting High Alignment Faults

    CLA