Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab., Tempesta Technologies)

Как понять, что происходит на сервере?

Александр Крижановский

NatSys Lab., Tempesta [email protected]

http://www.brendangregg.com/linuxperf.html

Кейс

Nginx устанавливает мало соединений (=> /proc/net/tcp)# netstat -npt|grep 8080|grep -c ESTAB26946# netstat -npt|grep 8080|grep -c TIME_WAIT6736

Трафик небольшой (266Mbps, 384Kpps)# head -2 /proc/net/dev; grep eth0 /proc/net/dev; sleep 10; grep eth0 /proc/net/devInter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes … eth0: 84880897009 921337910 0 0 0 0 0 … eth0: 84531083661 917496243 0 0 0 0 0 …

toptop - 22:18:04 up 8 min, 1 user, load average: 1.56, 0.51, 0.19Tasks: 76 total, 2 running, 74 sleeping, 0 stopped, 0 zombie%Cpu(s): 6.8 us, 33.5 sy, 0.0 ni, 5.6 id, 1.7 wa, 0.0 hi, 52.3 si, 0.0 stKiB Mem: 1023200 total, 183452 used, 839748 free, 13180 buffersKiB Swap: 498684 total, 0 used, 498684 free, 120656 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3050 nobody 20 0 28868 1852 684 S 29.5 0.2 0:07.21 nginx 3044 nobody 20 0 28868 1852 684 S 21.7 0.2 0:05.82 nginx 3 root 20 0 0 0 0 S 28.3 0.0 0:12.36 ksoftirqd/0 3046 nobody 20 0 28868 1852 684 S 18.0 0.2 0:06.99 nginx 11 root 20 0 0 0 0 S 17.6 0.0 0:12.82 ksoftirqd/1 3049 nobody 20 0 28868 1852 684 S 12.5 0.2 0:06.91 nginx 3047 nobody 20 0 28868 1852 684 S 9.5 0.2 0:07.20 nginx 3052 nobody 20 0 29340 2312 684 R 7.5 0.2 0:06.30 nginx 3053 nobody 20 0 28868 1852 684 S 6.4 0.2 0:05.26 nginx 3048 nobody 20 0 28868 1852 684 S 5.4 0.2 0:05.94 nginx 9 root 20 0 0 0 0 S 4.7 0.0 0:03.70 rcu_sched 3045 nobody 20 0 28868 1852 684 S 3.4 0.2 0:06.55 nginx 3051 nobody 20 0 28868 1852 684 S 1.0 0.2 0:06.06 nginx

top, 1

%Cpu0 : 4.9 us, 25.6 sy, 0.0 ni, 20.7 id, 0.0 wa, 0.0 hi, 48.8 si, 0.0 st%Cpu1 : 7.4 us, 37.0 sy, 0.0 ni, 17.3 id, 0.0 wa, 0.0 hi, 38.3 si, 0.0 st

Основное время тратится в ядре

Процессоры загружены равномерно

GDB(профайлинг по-крестьянски)

# for p in `pidof nginx`; do \ gdb -batch -ex "set pagination 0" -ex "thread apply all bt" -p $p | grep "#[01]"; done

#0 0x00007fea49079df3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:82#1 0x000000000042074b in ngx_epoll_process_events (cycle=0x6a31d0, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:577

#0 0x00007fea49079df3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:82#1 0x000000000042074b in ngx_epoll_process_events (cycle=0x6a31d0, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:577

….....

Несколько раз повторяем и (часто) видим bottle-neck

Могли бы увидеть на верху стека прикладной код...

perf top

+ 36.18% nginx [nf_conntrack] [k] nf_ct_tuple_equal+ 9.95% swapper [kernel.kallsyms] [k] intel_idle+ 9.87% swapper [kernel.kallsyms] [k] cfq_set_request+ 5.83% nginx nginx [.] ngx_vslprintf+ 3.19% nginx nginx [.] ngx_http_parse_header_line

Матчинг соединений:$ grep -A 1 -i nf_conntrack linux/net/netfilter/Kconfigconfig NF_CONNTRACK tristate "Netfilter connection tracking support"

Статистика Conntracks:$ cat /proc/net/nf_conntrack | wc -l25164

Выключаем коннтреки

perf top (2)


Объявлена в linux/drivers/idle/intel_idle.c $ head -2 drivers/idle/intel_idle.c /* * intel_idle.c - native hardware idle loop for modern Intel processors

perf top → iostat


Disk IO scheduler:$ find . -name \*.[ch] |xargs grep --color '\<cfq_set_request\>'./block/cfq-iosched.c:cfq_set_request(struct request_queue *q, ….../block/cfq-iosched.c: .elevator_set_req_fn = cfq_set_request,

$ iostat -xkdz 1 3Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsda 0.00 0.00 0.00 471.00 0.00 141152.00 1024.00 35.89 93.35 0.00 93.35 0.75 15.40sda 0.00 0.00 0.00 496.00 0.00 153952.00 1024.00 53.14 91.37 0.00 91.37 0.76 17.90sda 0.00 38.00 0.00 602.00 0.00 207360.00 1021.13 60.00 112.66 0.00 112.66 0.79 27.50

iotop

# iotop -obn 1

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 3045 be/4 nobody 0.00 B/s 93.07 K/s 0.00 % 0.00 % nginx: worker process 3047 be/4 nobody 0.00 B/s 62.04 K/s 0.00 % 0.00 % nginx: worker process 3048 be/4 nobody 0.00 B/s 155.11 K/s 0.00 % 0.00 % nginx: worker process 3049 be/4 nobody 0.00 B/s 62.04 K/s 0.00 % 0.00 % nginx: worker process 3050 be/4 nobody 0.00 B/s 62.04 K/s 0.00 % 0.00 % nginx: worker process 3052 be/4 nobody 0.00 B/s 62.04 K/s 0.00 % 0.00 % nginx: worker process

strace

# strace -e write -p 3044

Process 3044 attached - interrupt to quitwrite(8, "172.16.0.1 - - [22/May/2015:00:4"..., 121) = 121write(8, "172.16.0.1 - - [22/May/2015:00:4"..., 121) = 121write(8, "172.16.0.1 - - [22/May/2015:00:4"..., 121) = 121…....

# ls -l /proc/3044/fd/8

l-wx------ 1 nobody nogroup 64 May 21 23:21 /proc/3044/fd/8 -> /opt/nginx-1.4.2/logs/access.log

Правильно пользоваться lsof

А если в Top все хорошо?

Попробуем запустить бенчмарк (lock-free queue – CPU eater):

$ g++ -std=c++0x -O2 -D DCACHE1_LINESIZE=64 lockfree_rb_q.cc -lpthread$ ./a.out

top, 1 (~130% CPU usage!):%Cpu0 : 12.0 us, 21.3 sy, 0.0 ni, 66.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu1 : 11.7 us, 22.7 sy, 0.0 ni, 65.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu2 : 13.3 us, 21.3 sy, 0.0 ni, 65.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu3 : 11.7 us, 22.7 sy, 0.0 ni, 65.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st

Lock contention => tracer(gdb, strace)

$ strace -c -p `pidof a.out`

Process 17940 attached% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ----------------100.00 26.238075 766 34271 futex 0.00 0.000000 0 3 write 0.00 0.000000 0 1 fstat 0.00 0.000000 0 1 mmap------ ----------- ----------- --------- --------- ----------------100.00 26.238075 34276 total

Спасибо!

[email protected]

Engineering

Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab., Tempesta Technologies)