BasicsofKernelCrashDumpAnalysis

Agenda: Basicsofkernelcrashdumpanalysis

Initialanalysisofkernelcrashdump

Initialanalysisofmemorysubsystem

Whatarethetools/datarequiredtoperformakernelcrashdumpanalysis?

a)crashutility(/usr/bin/crash)

Providedby"crash"package.Eg:crash6.0.42.el6.x86_64.rpm

Nocrossplatformsupport.oUse32bitversionofcrashforvmcorecapturefrom32bitkernel.oUse64bitversionofcrashforvmcorecapturedfrom64bitkernel.oSimilarlyfors390,s390xandPPC.

Howtoinstall?

Installitusingyumorrpmcommand.

#yuminstallcrash

OR

#rpmivhcrash6.0.42.el6.x86_64.rpm

Compilefromupstreamsourcecode(notsupportedbyRedHatSupport):

#gitclonehttps://github.com/crashutility/crash.git#cdcrash/#make#makeinstall

b)Kernelsymbolfile(vmlinux)ofthecrashedkernel

Providedby"kerneldebuginfo"package.Eg:kerneldebuginfo2.6.32431.el6.x86_64.rpm

Theversionandarchofkerneldebuginfopackagemustmatchwiththeversionofkernelfromwhichvmcorewascaptured.

Howtoinstall?

Subscribeto"rhel6serverdebugrpms"channel.

#yumconfigmanagerenablerhel6serverdebugrpms#subscriptionmanagerreposenablerhel6serverdebugrpms

Installitusingyumordebuginfoinstallcommand:

#yuminstallkerneldebuginfo2.6.32431.el6#debuginfoinstallkerneldebuginfo2.6.32431.el6

OR

DownloadkerneldebuginfocommonandkerneldebuginforpmpackagesfromRedHatCustomerPortalandinstallitusingrpmcommand.

#rpmivhkerneldebuginfocommon2.6.32431.el6.x86_64.rpm#rpmivhkerneldebuginfo2.6.32431.el6.x86_64.rpm

Howtouse(vmlinux)withoutinstallingkerneldebuginfopackageonthesystem?

Extractthekerneldebuginforpmpackageusingrpm2cpiocommand.

#rpm2cpiokerneldebuginfo2.6.32431.el6.x86_64.rpm|cpioidv

c)Amachineofsamearchitectureasofkernelfromwhichvmcorewascaptured.

Akernelcrashdump(vmcore)capturedfromax86_64machinecanonlybeviewedonax86_64machine.

Similarly,akernelcrashdump(vmcore)capturedfromas390machinecanonlybeviewedonas390machine.

d)kernelcrashdumpfile(vmcore)

Capturedusingkdump/diskdump/netdump/xendump/LKCD/vmss2coremechanisms.

e)Sourcecodeofthecrashedkernel(optional)

Providedby"src"rpmpackageofkernel.Eg:kernel2.6.32431.el6.src.rpm

Theversionof"src"rpmpackageofkernelmustmatchwiththeversionofkernelfromwhichvmcorewascaptured.

Howtoinstall?

Installitusingyumorrpmcommand.

#yuminstallkernelsrc#rpmivhkernel2.6.32431.el6.src.rpm

OR

Extractthekernelsrcrpmpackageusingrpm2cpiocommand.

#rpm2cpiokernel2.6.32431.el6.src.rpm|cpioidv

Howtoopenakernelcrashdump(vmcore)foranalysis?

a)Typicalpostmortemdebugging:[Offlinemode]

Syntax:#crash/path/to/vmlinux/path/to/vmcore

Kernelobjectfileandmemoryimagearesupplied,respectively.

#crashosreleasevmcore2.6.32431.el6.x86_64

#crashd1vmcore|grepRELEASEOSRELEASE=2.6.32431.el6.x86_64

#crash/var/crash/vmcore/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux

b)Livememorydebugging:[Onlinemode]

Syntax:#crash/path/to/vmlinux

/dev/crashusedbydefaultforlivememoryimage.

#crash/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux

c)Livememorydebugging(withvmlinuxsearch):[Onlinemode]

Syntax:#crash

Predefineddirectoriesaresearchedforpropervmlinux

Versionstringmatchedtotherunningkernel(/proc/version)

Whatarethebasiccommandsofcrashutility?

osys:Displayessentialsysteminformation.

olog:Displaythekernelringbufferlog.

obt:Displayakernelstackbacktrace.

obtf:Displayallstackdatacontainedinaframe.

odis:Disassembleafunctionorsymbol.

okmemi:Displaysinformationabouttheuseofkernelmemory.

okmems:Displaysbasickmalloc()slabdata.

oipcs:SystemVIPCfacilities.

ops:Displaysprocessesinthesystem.

orunq:Displaysthetasksontherunqueuesofeachcpu.

omount:Displaysinformationaboutthemountedfilesystems.

ofiles:Displaytheopenfilesofthecurrentcontext.

odev:Displaycharacterandblockdevicedata.

odevd:DisplaydiskI/Ostatistics.

onet:Displaythesystem'snetworkdevicelist.

oirq:DisplayIRQdata.

omod:Displaythecurrentlyinstalledmodules.

omodt:Displaymodulesthatare"tainted".

oexit:Exitthecrashsession.

oextend:Dynamicallyloadsorunloadscrashextensionsharedobject.

ohelp:Gethelp.

InitialAnalysisofKernelCrashDump

Howtocheckbasicsysteminformation?

"sys"commanddisplaysessentialsysteminformation.

crash>sysKERNEL:/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinuxDUMPFILE:/var/crash/127.0.0.12016012117:45:34/vmcoreCPUS:4DATE:ThuJan2117:45:302016UPTIME:00:02:01LOADAVERAGE:0.49,0.30,0.11TASKS:264NODENAME:hvyas.example.comRELEASE:2.6.32431.el6.x86_64VERSION:#1SMPSunNov1022:19:54EST2013MACHINE:x86_64(2790Mhz)MEMORY:5.8GBPANIC:"SysRq:Triggeracrash"

Howtocheckhardwareinformation?

Theoption(i)of"sys"commanddumpstheDesktopManagementInterface(DMI)stringdataifavailableinthekernel.

crash>sysiDMI_BIOS_VENDOR:SeabiosDMI_BIOS_VERSION:0.5.1DMI_BIOS_DATE:01/01/2007DMI_SYS_VENDOR:RedHatDMI_PRODUCT_NAME:KVMDMI_PRODUCT_VERSION:RHEL6.5.0PCDMI_PRODUCT_SERIAL:DMI_PRODUCT_UUID:3D51C070C307BD3A281DCA1A689C22C6DMI_CHASSIS_VENDOR:RedHatDMI_CHASSIS_TYPE:1DMI_CHASSIS_VERSION:DMI_CHASSIS_SERIAL:DMI_CHASSIS_ASSET_TAG:DMI_SMBIOS_VERSION:2.4

OR

crash>log|grepDMI:DMI:RedHatKVM,BIOS0.5.101/01/2007

Howtocheckkernelringbuffer(dmesg)?

"log"or"dmesg"commanddumpsthekernellog_bufcontents.

crash>log|tailn38SysRq:TriggeracrashBUG:unabletohandlekernelNULLpointerdereferenceat(null)IP:[]sysrq_handle_crash+0x16/0x20PGD140aff067PUD140a9f067PMD0Oops:0002[#1]SMPlastsysfsfile:/sys/devices/pci0000:00/0000:00:01.2/usb1/11/speedCPU2Moduleslinkedin:nfsdlockdnfs_aclauth_rpcgssexportfsautofs4sunrpcbnx2fccnicuiofcoe8021qlibfcoegarpstplibfcllcscsi_transport_fcscsi_tgtxt_NFQUEUEiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesipv6uinputmicrocodevirtio_balloonvirtio_netsnd_hda_intelsnd_hda_codecsnd_hwdepsnd_seqsnd_seq_devicesnd_pcmsnd_timersndsoundcoresnd_page_alloci2c_piix4i2c_coreext4jbd2mbcachevirtio_blkpata_acpiata_genericata_piixvirtio_pcivirtio_ringvirtiodm_mirrordm_region_hashdm_logdm_mod[lastunloaded:speedstep_lib]

Pid:2187,comm:bashNottainted2.6.32431.el6.x86_64#1RedHatKVMRIP:0010:[][]sysrq_handle_crash+0x16/0x20RSP:0018:ffff8801422bde18EFLAGS:00010096RAX:0000000000000010RBX:0000000000000063RCX:0000000000000000RDX:0000000000000000RSI:0000000000000000RDI:0000000000000063RBP:ffff8801422bde18R08:0000000000000000R09:203a207152737953R10:0000000000000000R11:0000000000000000R12:0000000000000000R13:ffffffff81b01a40R14:0000000000000286R15:0000000000000007FS:00007ffdf778f700(0000)GS:ffff880028280000(0000)knlGS:0000000000000000CS:0010DS:0000ES:0000CR0:0000000080050033CR2:0000000000000000CR3:0000000140815000CR4:00000000000006e0DR0:0000000000000000DR1:0000000000000000DR2:0000000000000000DR3:0000000000000000DR6:00000000ffff0ff0DR7:0000000000000400Processbash(pid:2187,threadinfoffff8801422bc000,taskffff88018d4e8ae0)Stack:ffff8801422bde68ffffffff8134b982ffff88018d4e8ae0ffff88010000000000000000000003000000000000000002ffff880140ace3c000007ffdf77990000000000000000002fffffffffffffffbffff8801422bde98ffffffff8134ba3eCallTrace:[]__handle_sysrq+0x132/0x1a0[]write_sysrq_trigger+0x4e/0x50[]proc_reg_write+0x7e/0xc0[]vfs_write+0xb8/0x1a0[]sys_write+0x51/0x90[]system_call_fastpath+0x16/0x1bCode:d08881a31cfe81c9c36666662e0f1f840000000000554889e50f1f440000c7050d077500010000000faef804250000000001c9c3554889e50f1f4400008d47RIP[]sysrq_handle_crash+0x16/0x20RSPCR2:0000000000000000

Howtodeterminethepanictask?

Theoption(p)of"set"commandsetsthecontexttothepanictask,orbacktothecrashtaskonalivesystem.

crash>setpPID:2187COMMAND:"bash"TASK:ffff88018d4e8ae0[THREAD_INFO:ffff8801422bc000]CPU:2STATE:TASK_RUNNING(SYSRQ)

Howtocheckthebacktraceofpanictask?

"bt"commanddisplaysakernelstackbacktrace.Ifnoargumentsaregiven,thestacktraceofthecurrentcontextwillbedisplayed.

crash>btPID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"#0[ffff8801422bd9e0]machine_kexecatffffffff81038f3b#1[ffff8801422bda40]crash_kexecatffffffff810c5d92#2[ffff8801422bdb10]oops_endatffffffff8152b510#3[ffff8801422bdb40]no_contextatffffffff8104a00b#4[ffff8801422bdb90]__bad_area_nosemaphoreatffffffff8104a295#5[ffff8801422bdbe0]bad_areaatffffffff8104a3be#6[ffff8801422bdc10]__do_page_faultatffffffff8104ab6f#7[ffff8801422bdd30]do_page_faultatffffffff8152d45e#8[ffff8801422bdd60]page_faultatffffffff8152a815[exceptionRIP:sysrq_handle_crash+22]RIP:ffffffff8134b6c6RSP:ffff8801422bde18RFLAGS:00010096RAX:0000000000000010RBX:0000000000000063RCX:0000000000000000RDX:0000000000000000RSI:0000000000000000RDI:0000000000000063RBP:ffff8801422bde18R8:0000000000000000R9:203a207152737953R10:0000000000000000R11:0000000000000000R12:0000000000000000R13:ffffffff81b01a40R14:0000000000000286R15:0000000000000007ORIG_RAX:ffffffffffffffffCS:0010SS:0018#9[ffff8801422bde20]__handle_sysrqatffffffff8134b982#10[ffff8801422bde70]write_sysrq_triggeratffffffff8134ba3e#11[ffff8801422bdea0]proc_reg_writeatffffffff811f328e#12[ffff8801422bdef0]vfs_writeatffffffff81188f78#13[ffff8801422bdf30]sys_writeatffffffff81189871#14[ffff8801422bdf80]system_call_fastpathatffffffff8100b072RIP:0000003e2c2db560RSP:00007fff425f6548RFLAGS:00010202RAX:0000000000000001RBX:ffffffff8100b072RCX:00000000004b5e34RDX:0000000000000002RSI:00007ffdf7799000RDI:0000000000000001RBP:00007ffdf7799000R8:000000000000000aR9:00007ffdf778f700R10:0000000000000000R11:0000000000000246R12:0000000000000002R13:0000003e2c58e7a0R14:0000000000000002R15:0000003e2c58e7a0

ORIG_RAX:0000000000000001CS:0033SS:002b

Howtodisplayparentalhierarchyofaprocess?

The(p)optionof"ps"commanddisplaystheparentalhierarchyofselected,orall,tasks.

crash>psp2187PID:0TASK:ffffffff81a8d020CPU:0COMMAND:"swapper"PID:1TASK:ffff880192d39500CPU:2COMMAND:"init"PID:1865TASK:ffff88014cf00040CPU:1COMMAND:"sshd"PID:1877TASK:ffff88018d606040CPU:2COMMAND:"sshd"PID:2161TASK:ffff8801420beaa0CPU:1COMMAND:"bash"PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"

Howtodisplaythechildtaskofaprocess?

The(c)optionof"ps"commanddisplaysthechildrenofselected,orall,tasks.

crash>psc2187PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"(nochildren)

crash>psc2161PID:2161TASK:ffff8801420beaa0CPU:1COMMAND:"bash"PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"

Howtochecklistofopenfilesbypanictask?

"files"commanddisplaystheopenfilesofthecurrentcontext.

crash>files2187PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"ROOT:/CWD:/rootFDFILEDENTRYINODETYPEPATH0ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/01ffff880140ace3c0ffff8801410738c0ffff880141043078REG/proc/sysrqtrigger2ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/010ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/0254ffff88018cc71500ffff8801434ef240ffff88014104c4b8REG/root/system_crash.sh255ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/0

Howtochecktaskpriorityandpolicy?

"task"commandcanusedtodeterminethetaskpriorityandpolicy.

"task"commanddisplaysthecontentsofatask'stask_structandthread_infostructures.

crash>setpPID:2187COMMAND:"bash"TASK:ffff88018d4e8ae0[THREAD_INFO:ffff8801422bc000]CPU:2STATE:TASK_RUNNING(SYSRQ)

crash>task2187Rpolicy,prio,rt_priorityPID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"policy=0,prio=120,rt_priority=0,

Schedulingpolicies:#defineSCHED_NORMAL0#defineSCHED_FIFO1#defineSCHED_RR2#defineSCHED_BATCH3#defineSCHED_IDLE5

OR

crash>setpPID:2187COMMAND:"bash"TASK:ffff88018d4e8ae0[THREAD_INFO:ffff8801422bc000]CPU:2STATE:TASK_RUNNING(SYSRQ)

crash>task_struct.policy,prio,rt_priority0xffff88018d4e8ae0policy=0prio=120rt_priority=0

Howtocheckthecommandlineargumentsandenvironmentstringsoftask?

Theoption(a)of"ps"commanddisplaystheargumentandenvironmentdataforthetask.

crash>psaautomountPID:3948TASK:f722ee30CPU:0COMMAND:"automount"ARG:/usr/sbin/automounttimeout=60/netprogram/etc/auto.netENV:SELINUX_INIT=YESCONSOLE=/dev/consoleTERM=linuxINIT_VERSION=sysvinit2.85PATH=/sbin:/usr/sbin:/bin:/usr/binLC_MESSAGES=en_USRUNLEVEL=3runlevel=3PWD=/LANG=ja_JP.UTF8PREVLEVEL=Nprevious=NHOME=/SHLVL=2_=/usr/sbin/automount

Note:Thisisinformationisonlyavailableifuserspacecontentsarenotfilteredfromkernelcrashdump.

Howtodetermineresourcelimits(rlimits)ofaprocess?

Theoption(r)of"ps"commanddisplaysresourcelimits(rlimits)ofselected,orall,tasks.

crash>set1PID:1COMMAND:"init"TASK:ffff880192d39500[THREAD_INFO:ffff880192d3a000]CPU:2STATE:TASK_INTERRUPTIBLE

crash>psr1PID:1TASK:ffff880192d39500CPU:2COMMAND:"init"RLIMITCURRENTMAXIMUMCPU(unlimited)(unlimited)FSIZE(unlimited)(unlimited)DATA(unlimited)(unlimited)STACK10485760(unlimited)CORE0(unlimited)RSS(unlimited)(unlimited)NPROC4533145331NOFILE10244096MEMLOCK6553665536AS(unlimited)(unlimited)LOCKS(unlimited)(unlimited)SIGPENDING4533145331MSGQUEUE819200819200NICE00RTPRIO00RTTIME(unlimited)(unlimited)

Howtodeterminetotalnumberoftasksindifferentstate?

Theoption(S)of"ps"commanddisplaysasummaryconsistingofthenumberoftasksinataskstate.

crash>psSRU:5IN:259

Howtodisplayonlyuserspaceprocess?

Theoption(u)of"ps"commanddisplaysonlyusertasks.

crash>psu|headPIDPPIDCPUTASKST%MEMVSZRSSCOMM102ffff880192d39500IN0.0193641500init48311ffff88018d76a040IN0.0112401320udevd133613ffff88014cf00aa0IN0.027640864auditd133710ffff88018e46f540IN0.027640864auditd136110ffff88018cde6080IN0.02490921628rsyslogd136213ffff880191be5540IN0.02490921628rs:mainQ:Reg136312ffff88018d49c080IN0.02490921628rsyslogd136413ffff88018e796aa0IN0.02490921628rsyslogd139612ffff88018d4d4080IN0.018976924rpcbind

Howtodisplayonlykernelthreads?

Theoption(k)of"ps"commanddisplaysonlykernelthreads.

crash>psk|headPIDPPIDCPUTASKST%MEMVSZRSSCOMM>000ffffffff81a8d020RU0.000[swapper]>001ffff880192d81540RU0.000[swapper]002ffff880192d8a040RU0.000[swapper]>003ffff880192dc2aa0RU0.000[swapper]201ffff880192d38aa0IN0.000[kthreadd]320ffff880192d38040IN0.000[migration/0]420ffff880192d67540IN0.000[ksoftirqd/0]520ffff880192d66ae0IN0.000[migration/0]620ffff880192d66080IN0.000[watchdog/0]

Howtocheckthetotaltimeofaprocessinaspecificstate?

Theoption(m)of"ps"commanddisplaysthetimestampintodays,hours,minutes,seconds,andmillisecondssincethetaskwaslastrunonacpu.

crash>psm|grepRU[000:00:00.000][RU]PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"[000:02:01.177][RU]PID:0TASK:ffffffff81a8d020CPU:0COMMAND:"swapper"[000:02:01.409][RU]PID:0TASK:ffff880192d81540CPU:1COMMAND:"swapper"[000:02:01.408][RU]PID:0TASK:ffff880192d8a040CPU:2COMMAND:"swapper"[000:02:01.408][RU]PID:0TASK:ffff880192dc2aa0CPU:3COMMAND:"swapper"

HowtocheckrunqueueofeachCPU?

"runq"commanddisplaysthetasksonaCFSrunqueue:

crash>runqCPU0RUNQUEUE:ffff880028216840CURRENT:PID:0TASK:ffffffff81a8d020COMMAND:"swapper"RTPRIO_ARRAY:ffff8800282169c8[notasksqueued]CFSRB_ROOT:ffff8800282168d8[notasksqueued]

CPU1RUNQUEUE:ffff880028256840CURRENT:PID:0TASK:ffff880192d81540COMMAND:"swapper"RTPRIO_ARRAY:ffff8800282569c8[notasksqueued]CFSRB_ROOT:ffff8800282568d8[notasksqueued]

CPU2RUNQUEUE:ffff880028296840CURRENT:PID:2187TASK:ffff88018d4e8ae0COMMAND:"bash"RTPRIO_ARRAY:ffff8800282969c8[notasksqueued]CFSRB_ROOT:ffff8800282968d8[notasksqueued]

CPU3RUNQUEUE:ffff8800282d6840CURRENT:PID:0TASK:ffff880192dc2aa0COMMAND:"swapper"RTPRIO_ARRAY:ffff8800282d69c8[notasksqueued]CFSRB_ROOT:ffff8800282d68d8[notasksqueued]

Howtodeterminethevaluesofsysctlparameterfromvmcore?

Checkthe".data"fieldofdesiredsysctlparameterinkernelsocurcesysctl.cfile.

SourceFile:kernel2.6.32431.el6/kernel/sysctl.c

{.ctl_name=VM_SWAPPINESS,.procname="swappiness",.data=&vm_swappiness,.maxlen=sizeof(vm_swappiness),.mode=0644,.proc_handler=&proc_dointvec_minmax,.strategy=&sysctl_intvec,.extra1=&zero,.extra2=&one_hundred,},

crash>vm_swappinessvm_swappiness=$1=60

SourceFile:kernel2.6.32431.el6/kernel/sysctl.c

{.ctl_name=VM_PANIC_ON_OOM,.procname="panic_on_oom",.data=&sysctl_panic_on_oom,.maxlen=sizeof(sysctl_panic_on_oom),.mode=0644,.proc_handler=&proc_dointvec,},

crash>sysctl_panic_on_oomsysctl_panic_on_oom=$2=1

InitialAnalysisofMemorySubsystem

Howtocheckoverallmemoryusageonthesystem?

"kmemi"commanddisplaysgeneralmemoryusageinformation:

crash>kmemiPAGESTOTALPERCENTAGETOTALMEM12058384.6GBFREE11025064.2GB91%ofTOTALMEMUSED103332403.6MB8%ofTOTALMEMSHARED1448456.6MB1%ofTOTALMEMBUFFERS637724.9MB0%ofTOTALMEMCACHED39960156.1MB3%ofTOTALMEMSLAB1772569.2MB1%ofTOTALMEM

TOTALSWAP1638264MBSWAPUSED000%ofTOTALSWAPSWAPFREE1638264MB100%ofTOTALSWAP

COMMITLIMIT6167412.4GBCOMMITTED59755233.4MB9%ofTOTALLIMIT

Howtocheckperzonememorystatistics?

"kmemz"commanddisplaysperzonememorystatistics.

crash>kmemzNODE:0ZONE:0ADDR:ffff880000010000NAME:"DMA"SIZE:4095PRESENT:3831MIN/LOW/HIGH:42/52/63VM_STAT:NR_FREE_PAGES:3930NR_INACTIVE_ANON:0NR_ACTIVE_ANON:0NR_INACTIVE_FILE:0NR_ACTIVE_FILE:0NR_UNEVICTABLE:0NR_MLOCK:0NR_ANON_PAGES:0NR_FILE_MAPPED:0NR_FILE_PAGES:0NR_FILE_DIRTY:0NR_WRITEBACK:0NR_SLAB_RECLAIMABLE:0NR_SLAB_UNRECLAIMABLE:0NR_PAGETABLE:0NR_KERNEL_STACK:0NR_UNSTABLE_NFS:0

Howtodeterminememoryusageinuserspace?

The"psGu"commandcanbeusedtodeterminetheRSSofuserspacetasks.

crash>psGu|sed's/>//g'|awk'{total+=$8}END{printf"TotalRSSofusermode:%.02fGiB\n",total/2^20}'TotalRSSofusermode:30.74GiB

Perprocessmemoryusage:

crash>psG|sortk8,8nr|headPIDPPIDCPUTASKST%MEMVSZRSSCOMM>28882201091ffff880831e0e040RU92.86527242832120772rsession>698569842ffff88082e934040RU0.06416644680coda>639213ffff88082e934ab0RU0.01673366324perfd612312ffff88082c194ab0IN0.123848045732splunkd694669451ffff88082f804ab0RU0.08337449188opcmona2738710ffff88082d288040RU0.01299526472bgsagent627510ffff88082f7eaab0UN0.0362845184scopeux209820973ffff88082dbe7520IN0.014543285172python698169801ffff880831017520RU0.09035483236opcmsga215113ffff88082daed520IN0.07063202748EracentEUAServi

Specificprocess'smemoryusage:

crash>psGursession|tailn+2|cutb2|gawk'{mem+=$8}END{print"Total"mem/1048576"GB"}'Total30.6334GB

Howtodisplayinformationforeachconfiguredswapdevice?

"swap"commanddisplaysinformationforeachconfiguredswapdevice.

crash>swapSWAP_INFO_STRUCTTYPESIZEUSEDPCTPRIFILENAMEffff88014be681c0FILE65528k0k0%1/swapfile

Howtodeterminememoryusageinkernelspace?

"kmems"commanddisplaysbasickmalloc()slabdata.

crash>kmems|grepeNAMEeanon_vma_chainCACHENAMEOBJSIZEALLOCATEDTOTALSLABSSSIZEffff88043bf20b00anon_vma_chain4830204272030204281839226344k

Formula:(OBJSIZE*ALLOCATED)/2^30

crash>!bcqscale=2(48*302042720)/2^3013.50

Memoryallocatedto"anon_vma_chain"slabobjectis~13.50GiB.

Howtocheckmemoryallocatedforhugepages?

RHEL6:

"kmemh"commanddisplaystheaddressofhugepagehstatearrayentries,alongwiththeirhugepagesize,totalandfreecounts,andname.

crash>kmemhHSTATESIZEFREETOTALNAMEffffffff81fbb8c02MB1010hugepages2048kB

RHEL5:

a)TotalNumberofHugepages.

crash>pdnr_huge_pagesnr_huge_pages=$1=35845

crash>pdnr_huge_pages*(1pdfree_huge_pagesfree_huge_pages=$2=0

HowtocheckmemoryallocatedtoVMwareballooningdriver(RHEL6+)?

Determinetheaddressofsymbolballoon.crash>symballoonffffffffa002b600(b)balloon[vmware_balloon]

Determinethevalueof"size"variableusingtheaddressofsymbolballoon.crash>structvmballoon.size0xffffffffa002b600$4=2721049

crash>!bcqscale=22721049*4/2^2010.37

2721049pages=2721049x4=10884196KiB=10.37GiB

Note:ThevalueofsizevariableistheamountofmemoryallocatedbyVMwareBallooningdriverinpages.

Determinethevalueof"target"variableusingtheaddressofsymbolballoon.crash>vmballoon.target0xffffffffa002b600$5=3177311

crash>!bcqscale=23177311*4/2^2012.12

3177311pages=3177311x4=12709244KiB=12.12GiB

Note:ThevalueoftargetvariableistheamountofmemoryneededbyVMwareBallooningdriverinpages.

Questions?