Linux Memory Basics for SysAdmins - ChinaNetCloud Training

ChinaNetCloudRunning the World's Internet Servers 管理全球服务器

Copyright 2013 ChinaNetcloud Training Program

ChinaNetCloud Training

Linux Memory BasicsBy ChinaNetCloudPioneers in OaaS – Operations-as-a-Service

October, 2013

www.ChinaNetCloud.com


Copyright 2013 ChinaNetCloud 2

Introduction

● Linux memory is complex and interesting● Class summarizes:

● Types of memory● How it's used● How to troubleshoot memory issues



Overview, Purpose & Use

● Linux memory is one of the most important areas for engineers to understand

● Used by everything● Often not well-understood● Often the cause of problems



Introduction

● Memory is simple, in theory● Complex in real use● Plenty of strange things, too● Important to understand● Important to monitor



Memory Types I

● Physical RAM – Fixed amount, real RAM● Virtual Memory – Virtual, can swap● Shared – Between processes

● Oracle & Postgres use this● Slab – Kernel memory

● /proc/slabinfo & slabtop utility● Includes big RAM users

– tcpmem & inode cache



Memory Types II

● Page Cache – File system cache● Dirty

● Changed file system data waiting to write to disks● Important for high write systems



/proc/meminfo

Very useful but complex ●These are items not found by 'free' or 'top'

●MemTotal: Total usable ram (RAM minus kernel binary code)●MemFree: Total free memory, same as free's 'free' output●SwapCached: Memory swapped out, then back in, but still also in the swapfile



/proc/meminfo

● Active – Recently used, not reclaimed unless necessary● Inactive – Less recently used, likely to be reclaimed ● Dirty – File / Page Cache waiting to be written to disk● Writeback – Memory which is now being written to the disk● Mapped – Memory Mapped files, such as libraries

● Includes Mongo, Varnish, and many others● Slab – Kernel memory, usually 256-512MB



Memory Page Size

● 4KB page size● Important as many stats are in pages

– Be careful of units ! KB vs. Pages

● sysctl items like tcpmem are mixed● Do not confuse with disk sector size of 0.5KB



Free -m

● Total – Physical in machine

● Used – by Apps plus Cache, Buffers and maybe Shared

● Free – Totally unused RAM, not important, usually very small

● Shared – by different processes like Oracle / Postgres

● Buffers – Raw block cache to/from disks, not impotrant

● Cached (Page Cache) – Used by disk files cached in RAM

● Swap – Total, Used, Free – Used should be small

● -/+ buffers/cache – Important numbers, inside box is key● PAY ATTENTION to the BOX number – it's all that matters

total used free shared buffers cachedMem: 3961 3901 60 0 121 1232-/+ buffers/cache: 2546 1414Swap: 4683 756 3927



Top, ps Memory Output

● VIRT – Virtual size● Includes mapped libraries, often very large● Not too useful – IGNORE THIS

● RSS – Resident Set Size● Most important !● Real RAM used by applications, not including swap

● SHR – Shared, such as Oracle / Postgres● Be careful to see processes not threads

● Threads will share RSS, see htop in thread mode



Swap

● Simple, but not simple● Managed by kswapd● Actual swap used calc

● SwapTotal – SwapCached – SwapFree● Goal is zero swap on servers

● Never let a system actively swap● But some systems will have small swap

● Common to see 100-200MB, more on big system● NUMA defaults can cause swapping



Problem with Swap

● Swapping FREEZES the swapping process ● So, a 32MB byte swap of MySQL RAM

● Freezes ALL of MySQL for many seconds● Very bad for applications

● Bad for any single process system● MySQL, Nginx, HAProxy, etc. (not Apache)

● Also uses valuable IO, slowing DBs, etc.● Goal is to NEVER swap real applications



Swappiness

● Tells kernel which is more important● File system Cache vs. App Memory

● Default is 60 – Means cache more important● Stupid for servers – will swap even with free RAM

● Set to 1, always (used to be 0, now use 1)● Won't swap until all RAM used by applications● Note some swap anyway (see next slide)

● Set by sysctl● Check in /proc/sys/vm/swappiness



Swap still happens

● Linux likes to swap a little ● Even with free RAM and swappiness = 0● Some kernel data wants to swap● Can pre-swap SwapCached data● Still a mystery, but it's okay

● Very large systems (64GB+ might swap 1-2GB)● Turning off can cause kswapd to go crazy● Watch vmstat swapin/out, make sure very small



NUMA - Can cause swap

● NUMA – Non-Uniform Memory Access● Each CPU has its own RAM● Slower to use 'other' CPU's RAM● Standard now on all Intel servers

● Causes RAM inbalance on big RAM processes● Like MySQL, MongoDB, Java

● Install & use numa-utils package, numactl app● Set all big processes to full interleave:

● numactrl –interleave all mysqld . . .



Dirty Memory & Ratios

● Causes strange problems & performance issues● See /proc/meminfo

● Dirty is the changed FileSystem cache data● Must be written to disk● Can cause big problems if too high● Two ratios control

● /proc/sys/vm/dirty_background_ratio / _bytes● /proc/sys/vm/dirty_ratio / _bytes

● Freezes all writing processes when dirty > dirty_ratio● Calculation not clear, but keep dirty data < 20% of RAM



OOM Killer

● Kills process when no more RAM or Swap● Chooses process to die via a score

● Biggest RAM user (MySQL) often dies first● See /proc/<pid>/oom_score

● Always a message in dmesg - /var/log/kernel● Can adjust score to control (advanced)● Best practice to monitor swap and log



Summary

● Memory important to understand● Especially basics & definitions● Understand swap & swappiness● Watch NUMA on big systems● Memory is cheap – buy more !



About ChinaNetCloud


ChinaNetCloud [email protected]

www.ChinaNetCloud.com

Beijing Office:

Lee World Business Building #305

57 Happiness Village Road,

Chaoyang District

Beijing, 100027 China

Silicon Valley Office:

California Avenue

Palo Alto, 94123 USA

Shanghai Headquarters:

X2 Space 1-601, 1238 Xietu Lu

Shanghai, 200032 China

T: +86-21-6422-1946 F: +86-21-

6422-4911

Technology

Linux Memory Basics for SysAdmins - ChinaNetCloud Training