Running FreeBSD on Linux KVM Virtualization

AsiaBSDCon 2012 Tokyo March 25, 2012 Takeshi HASEGAWA [ja] @hasegaw / [en] @hasegaw_en

©2003 Project TIMEMACHINE

Takeshi Hasegawa Interests

•  Virtualization (Xen, and KVM)

•  Storage

•  Snowboarding J

•  Poor at coding. I’m not a software developer.

Work Experience

•  April 2004- Systems Engineer at Software Research Associates, Inc.

•  June 2011- Sales Engineer at Fusion-io

In this winter I had great days with my colleges

2

My works – books and magazines I participated

3

Our sponsor, Gihyo Books!

The Best-seller UNIX admin guide with catoon characters. More than 15k copies are sold Best-seller Xen book in Japan.

More than 6k copies are sold

I discussed the benfit of Linux KVM.

And more… 3

Running articles on the Web

エンジニアなら知っておきたい仮想マシンのしくみ (Now you understand how virtual machine works)

http://gihyo.jp/dev/serial/01/vm_work/ 4 4

BSD and myself •  Met FreeBSD 2.2.5 in 1997

•  Open Source no matsuri 2000

o  Met JKH and Warner Losh

•  2001~2006

o  Contracted a dedicated Server with FreeBSD (4.x)

•  2010

o  Used V-IMAGE for the network basic session of new-hire training

o  Wrote virtio drivers for FreeBSD

•  2011-2012

o  Wrote KVM PVClock for FreeBSD

This picture is still my treasure! J

IP Networking amiation DEMO Powered by VIMAGE and HTML5

5

Running FreeBSD

on Linux KVM Virtualization

6

Why FreeBSD on Linux KVM? More opportunities to run FreeBSD

•  Many service providers are running KVM o  VPS (Virtual Private Server)

o  Cloud – such as Sakura Internet (#sacloud)

•  Linux KVM is surely handy

o  The free-of-charge Type-1 hypervisor

o  Guest OS doesn’t require device drivers for the latest hardware

Co-existence of FreeBSD allows leverage of its specialties

•  ZFS, BSD Network stacks, jail/vimage, …

•  and more!

7

However – FreeBSD on Linux KVM runs slow The biggest Bottlenecks seen in Virtual Machines is: •  I/O Bottlenecks

–  Disk I/O –  Network I/O

The solution is: •  I/O Para-virtualization

–  Disk para-virtualization –  Network para-virtualization

0"

10"

20"

30"

40"

50"

60"

70"

80"

90"

Na.ve" FreeBSD"VM"+"SCSI"Emu"

seq$write$MB/s$

seq"write"MB/s"

Especially FreeBSD is known as “low-performance guest ” on Linux KVM.

8

virtio – I/O interface for virtualized guests

•  Ring-Buffer Interface definition between Guest and Host

•  Independent to hypervisors –  lguest –  Linux KVM –  VirtualBox –  BHyVe –  Xen

last summer, personally I was also working the virtio support on Xen....

•  Works efficiently –  No overhead of H/W emulation –  Less Intel VT/AMD-V world

changes –  Less context switches

Host

Guest

Hardware

PCI Bus

virtio Ballon

virtio Block

virtio Block

virtio Net

virtio Ballon

virtio Block

virtio Block

virtio Net

9

virtio drivers for FreeBSD •  I wrote virtio drivers for FreeBSD – Fall of 2010

o  Rusty Russel, Father of virtio, made a presentation at LinuxCon 2010

“virtio-net guest driver can be implemented by less than 1,000 lines”

o  I actually tried on FreeBSD 8.1-R. It took more than 1,000 lines! J

o  Finally, I ported: •  virtio interface •  virtio-net (network) •  virtio-blk (block)

•  virtio driver for FreeBSD merged to -CURRENT – Winter of 2011 o  Congrats!

o  Please note - the merged one is a DIFFERENT work from mine!

10

virtio-net increases network performance

•  4.7x against rtl8139, 1.3x against e1000 bandwidth improvement!

1.3x

4.7x

Test Environment: HP ML115G1, 1.8G 1P2C, PC6400 DRAM (too poor!) Fedora 14 as the Host, and running FreeBSD 8.1-RELEASE as the Guest(2vCPUs) 11

virtio-blk increases block I/O performance …

0"

10"

20"

30"

40"

50"

60"

70"

80"

90"

Na.ve" FreeBSD"VM"+"SCSI"Emu"

seq$write$MB/s$

seq"write"MB/s"

12

virtio-blk increases block I/O performance a bit

0"10"20"30"40"50"60"70"80"90"

Na.ve" FreeBSD"VM"+"SCSI"Emu"

FreeBSD"VM"+"vir.o"

seq$write$MB/s$

seq"write"MB/s"OOPS!

13

Hey… what’s happening?

14

“Okay I’ll look into that…”

15

Latency Distribution – CentOS Guest w/virtio

1"

11"

21"

31"

41"

51"

61"

71"

81"

91"

0" 0.5" 1" 1.5" 2" 2.5" 3" 3.5" 4" 4.5"

CentOS"5"host"

CentOS"VM+Vir;o"

Linux guest performs native-level performance. Ideal!

>79% of IOs are done in <10ms

Low latency High

Distribution(%

)

16

Latency Distribution – FreeBSD Guest

1"

11"

21"

31"

41"

51"

61"

71"

81"

91"

0" 1" 2" 3" 4" 5" 6"

FreeBSD"VM+Vir6o"

Fedora"14"HDD"

FreeBSD"VM+SCSI"Emu"

FreeBSD guest has some latency bottleneck…

Only 38% of IOs are done in <10ms

Low latency High

Distribution(%

)

17

Latency Distribution – FreeBSD/CentOS Comparision

1"

11"

21"

31"

41"

51"

61"

71"

81"

91"

0" 1" 2" 3" 4" 5" 6"

Fedora"14"HDD"

CentOS5"VM"

FreeBSD"VM+Vir>o"

FreeBSD"VM+SCSI"Emu"

Low latency High

Distribution(%

)

18

I/O Characteristics between FreeBSD n’ Linux fio --filename=/dev/(virtio device) --direct=1 --rw=randwrite --bs=4k --size=2G --numjobs=1 --runtime=10

--group_reporting --name=a

CentOS 5.7 Guest on CentOS 5.7 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util

vdb 0.00 0.00 0.00 162.00 0.00 0.63 8.00 0.97 5.98 5.98 96.90

vdb 0.00 0.00 0.00 151.49 0.00 0.59 8.00 0.96 6.29 6.31 95.64

vdb 0.00 0.00 0.00 161.00 0.00 0.63 8.00 0.97 6.09 6.04 97.30

vdb 0.00 0.00 0.00 155.00 0.00 0.61 8.00 0.97 6.20 6.26 97.00

vdb 0.00 0.00 0.00 159.00 0.00 0.62 8.00 0.98 6.23 6.16 98.00

FreeBSD 8.1 Guest on Fedora 14 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util

dm-6 0.00 0.00 0.00 178.00 0.00 0.35 4.00 0.70 3.88 3.93 70.00

dm-6 0.00 0.00 0.00 160.00 0.00 0.31 4.00 0.73 4.64 4.58 73.30

dm-6 0.00 0.00 0.00 172.00 0.00 0.34 4.00 0.68 3.94 3.98 68.40

dm-6 0.00 0.00 0.00 164.00 0.00 0.32 4.00 0.69 4.20 4.21 69.00

dm-6 0.00 0.00 0.00 168.00 0.00 0.33 4.00 0.75 4.51 4.47 75.10

1.  IO reqs from FreeBSD is intermittent, and the disk is sometimes idle 2.  It seems that Linux queues 2x IO reqs/queue more than FreeBSD 19

iostat results in proliant2 (Fedora14) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util FreeBSD+SCSI Emu

dm-6 0.00 0.00 0.00 260.00 0.00 0.51 4.00 0.94 3.63 3.63 94.30

dm-6 0.00 0.00 0.00 264.00 0.00 0.52 4.00 0.94 3.54 3.55 93.80

dm-6 0.00 0.00 0.00 270.00 0.00 0.53 4.00 0.97 3.62 3.60 97.20

dm-6 0.00 0.00 0.00 270.00 0.00 0.53 4.00 0.97 3.58 3.59 96.90

dm-6 0.00 0.00 0.00 264.00 0.00 0.52 4.00 0.97 3.66 3.69 97.40

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util FreeBSD+VirtIO

dm-5 0.00 0.00 0.00 266.00 0.00 0.52 4.00 0.99 3.74 3.73 99.10

dm-5 0.00 0.00 0.00 278.00 0.00 0.54 3.97 0.99 3.55 3.56 99.00

dm-5 0.00 0.00 0.00 296.00 0.00 0.58 4.03 1.02 3.46 3.29 97.30

dm-5 0.00 0.00 0.00 272.00 0.00 0.53 4.00 1.03 3.79 3.62 98.40

dm-5 0.00 0.00 0.00 260.00 0.00 0.51 4.00 0.99 3.82 3.81 99.10

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util Linux guest

vdb 0.00 0.00 0.00 154.46 0.00 0.60 8.00 3.94 24.20 6.42 99.21

vdb 0.00 0.00 0.00 146.00 0.00 0.57 8.00 3.98 28.44 6.86 100.20

vdb 0.00 0.00 0.00 147.00 0.00 0.57 8.00 3.98 26.01 6.81 100.10

vdb 0.00 0.00 0.00 140.00 0.00 0.55 8.00 4.00 29.05 7.16 100.30

vdb 0.00 0.00 0.00 159.00 0.00 0.62 8.00 3.98 24.55 6.30 100.20

vdb 0.00 0.00 0.00 165.00 0.00 0.64 8.00 3.99 24.22 6.07 100.10

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util Linux native

dm-5 0.00 0.00 0.00 259.00 0.00 1.01 8.00 3.97 15.27 3.86 100.00

dm-5 0.00 0.00 0.00 259.00 0.00 1.01 8.00 3.97 15.37 3.86 100.00

dm-5 0.00 0.00 0.00 258.00 0.00 1.01 8.00 3.97 15.50 3.88 100.00

dm-5 0.00 0.00 0.00 250.00 0.00 0.98 8.00 3.99 15.88 4.00 100.00

dm-5 0.00 0.00 0.00 165.00 0.00 0.64 8.00 3.98 16.48 6.06 100.00

20

virtio-blk can make FreeBSD Guests faster? No!! Facts

•  I/O latency from virtual machine is more than twice of native I/O

•  The performance issue is seen in block layer

•  virtio-blk surely improved latency distribution (compared to SCSI

emu), but it is still ignorable impact

Then… what is the bottleneck in the FreeBSD Guests?? o  Interrupt delivery?

o  Queueing I/Os to the block device?

More deep investigation should be done J

21

My another work – KVM Para-virtualized Clock Motivation

•  I wanted more reliable timer needed to investigate performace issue

of FreeBSD on Linux KVM

What’s KVM PVClock?

•  clocksource which can be found in Linux KVM environment o  “Timecounter” source in FreeBSD world

o  More accurate VM time-keeping can be expected

•  Inspired by Xen o  Shadows TSC counter of the host processor, and calendar of the host.

o  The replica of timer virtualization in Xen

o  FreeBSD already has Xen clock support

22

The result of the early implementaiton

while true; do date; sleep 1; done on FreeBSD Guest

while true; do date; sleep 1; done on Linux KVM Host

The time-slip is less than one second -  The two OSes should be in synchronized

theoretically -  No any time-synchronization

method required -  tested for 120hours (5 days)

23

Conclusion •  Linux KVM will bring more opportunities to run FreeBSD

•  Linux KVM uses virtio for I/O para-virtualizaton

•  FreeBSD block layer does not fit well with Linux KVM

•  Porting Linux KVM PVClock for FreeBSD

•  Any comments? Questions? Suggestions?

•  please say hello to me after the session.

24

Thank you!

25