2012 年 5 月 31 日星期四 Xen400 渠道培训 XenServer Sunny Wang

2012 年 5 月 31 日星期四

Xen400渠道培训XenServer

Sunny Wang

© 2012 Citrix | Confidential – Do Not Distribute

1. XenServer Installation

2. Design Considerations

3. Practice Experience

4. XS Maintenance Rules

5. XE-Command Reference

Agenda


XenServer Installation


• 系统 BIOS 的检查及设置• Restore default BIOS setup• Enable VT or AMD/V,• disable C-state• Disable executive lock bit• Enable remote console(if IBM server)• Disable Multi-thread if possible• 本地磁盘设置成 RAID1 或 RAID 1/0• 如果磁盘以前安装过 VMware 或其它 OS, 重新初始化 RAID 10• If server total CPU core over 48, disable multi-core before installation, after

installation, enable it.• BIOS Power Regulator for Maximum Performance

XenServer安装 - step 1


CPU - BIOS setting


• 如果服务器不支持 UEFI 和 GPT 分区 , 安装 XS6.0 时可参照如下附件使用 MBR 分区安装，并根据业务需要将 Dom0 的文件系统扩展到 10G 或 20G

• 如果是安装 XS 5.6 sp2, 根据业务需要将 Dom0 的文件系统扩展到 10G 或 20G

XenServer 安装 - step 2

Citrix Confidential - Do Not Distribute

XenServer 6 Disable GPT and get a larger


• 安装 irqbalance service (only XS 5.6)• rpm –ivh irqbalance-0.55-15.el5.i386.rpm • yum --disablerepo=citrix --enablerepo=base,updates install -y irqbalance

• 参照 CTX124259, 将 Dom0 的内存上限扩展到 2940M

• 打最新的补丁和 driver 驱动• http://support.citrix.com/product/xens/v6.0/• http://support.citrix.com/product/xens/v5.6sp2/

• 设置 Openvswitch （可选）

• xe-switch-network-backend openvswitch （可选）



irqbalance-0.55-15.el5.i386.rpm

CTX124259 - Adjusting Dom0 and Xenheap Se

http://support.citrix.com/product/xens/v6.0/

http://support.citrix.com/product/xens/v5.6sp2/


• 启动存储的 Multipath• Check Storage support multipath type – DMP, MPP, DMP RDAC• If DMP, enable multipath at Xencenter• If MPP RDAC

• # /opt/xensource/libexec/mpp-rdac --enable• For ISCSI storage, refer to “xs-design-iSCSI-multipathing-config.Pdf”



xs-design-iSCSI-multipathing-config.pdf


• 创建资源池

• 设置网卡绑定• 检查交换机端口配置

• Configure Active/Standby bond for management NIC • xe pif-param-set uuid=<bond_pif_uuid> other-config:bond-mode=active-backup

• Configure SLB Bond for production NIC• If NIC is 10G, disable NIC LRO :ethtool –K eth# lro off

• 创建存储池



CTX123158 - Considerations for XenServer

CTX124421 - Understanding Network Interfa


• 创建虚拟机及模板• Windows Guest OS

• 安装 Xentools• 在创建模板前运行 sysprep 或 newid• Setup VM vif offload

• 在注册表中禁用 TCP 任务卸载，步骤如下：• (1) 单击开始、单击运行，键入 regedit ，然后单击确定。• (2) 找到并单击以下注册表子项： HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters• (3) 单击编辑，指向新建，然后单击 DWORD 值。• (4) 键入 DisableTaskOffload 作为输入的名称，然后按 ENTER 键。• (5) 用鼠标右键单击 DisableTaskOffload ，然后单击修改。• (6) 在数值数据框中键入值 1 ，然后单击确定。• (7) 退出注册表编辑器。• (8) 重新启动计算机。• 注意：在修改注册列表之前，要对注册列表进行备份• xe vif-param-set uuid=$VIF other-config:ethtool-tx="off“

xe vif-param-set uuid=$VIF other-config:ethtool-rx="off"

• Linux Guest OS• 安装 Xentools• Remove MAC address at Guest OS network configuration file

XenServer 安装 -step 6



安装过程中的常见问题及基本解决方法


服务器安装

• Alt-F3 显示 Linux login 标识符ᵒ Login: root (no password required)ᵒ Issue Linux commandsᵒ 技术支持压缩包

• support.sh (creates /tmp/support.tar.bz2)• Support tarball is created in /tmp on install error• Support tarball is saved in /root after successful install

• Alt-F2 显示安装的事件日志• Alt-F1 返回安装菜单


• RAID 卡不兼容• 升级 RAID 的 Firmware• DDK 编译最新的 Driver

• 升级 5.6 到 6.0 后，新创建虚机报错

其它问题

\Éý¼¶xenserverºó¶Ô´æ´¢²Ù×÷Ê±ºò±¨´í£ºError

Compiling the Realtek RTL8111 driver on XenServer.mht


• 如果安装完成后发现网卡有丢包或错包，而且网络传输较慢，可使用如下脚本或参照如下命令配置网卡的 offload ：• echo Setting checksum off on VIFs• VIFLIST=`xe vif-list | grep "uuid ( RO) " | awk '{print $5}'`• for VIF in $VIFLIST• do• echo Setting ethtool-tx=off and ethtool-rx=off on $VIF• xe vif-param-set uuid=$VIF other-config:ethtool-tx="off"• xe vif-param-set uuid=$VIF other-config:ethtool-rx="off"• done• echo Setting checksum off on PIFs• PIFLIST=`xe pif-list | grep "uuid ( RO) " | awk '{print $5}'`• for PIF in $PIFLIST• do• echo Setting ethtool-tx=off and ethtool-rx=off on $PIF• xe pif-param-set uuid=$PIF other-config:ethtool-tx="off"• xe pif-param-set uuid=$PIF other-config:ethtool-rx="off"• done

其它问题


Get better Network Performance on Citrix


• Run logrotate hourly instead of daily. Run the following command from the XenServer host console:# mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotateAdvanced cron scheduling can be used in some scenarios.

• Compress old logs. To compress the old log files use the VI to edit the logrotate.conf file. In the /etc/logrotate.conf file, remove the comment from the following line:#compress

• Retain fewer historical logs.By default XenServer retains 20 historical logs. This can be reduced by changing the numeric value in the following line:rotate 20

• Reboot XenServer host.

Refine the Log Retention Settings-CTX131619


Design Considerations


CPU Management – 2 Core System

Socket0

HT1HT0HT0 HT1

Core1

vCPU #2

vCPU #3

Core0

vCPU #0

vCPU #1

HT1HT0HT0 HT1

Core1

vCPU #6

vCPU #7

Core0

vCPU #4

vCPU #5

Socket1


CPU Management – 4 Core System

Socket0

HT1HT0HT0 HT1

Core1

vCPU #2

vCPU #3

Core0

vCPU #0

vCPU #1

HT1HT0HT0 HT1

Core3

vCPU #6

vCPU #7

Core2

vCPU #4

vCPU #5


• Xen schedules active vCPUs on physical cores• If active vcpus <= physical cores, vCPU per full core• If active vcpus > physical cores, vCPU share cores• Shared core effectively divides CPU Hz

• Remember dom0 is a VM!

Virtual CPUs and real CPUs


© 2012 Citrix | Confidential – Do Not DistributeCitrix Confidential - Do Not Distribute

• Start with 1 vCPU◦ All templates should have 1 vCPU

• Add more vCPUs if◦ Existing vCPUs show high utilization◦ Highly threaded workload

• Do not over allocate CPU cores◦ 4 physical CPU cores for dom0 & Xen◦ Stay under number of physical CPU cores◦ Constant Load Rule N (n-vcpu) – V (n-vms) <= P (n-cores)

vCPU Best Practices


• 10 VMs with 1 vCPU each◦ 10 vCPUs – 10 VMs <= 4 cores

• 4 VMs with 2 vCPUs each◦ 8 vCPUs – 4 VMs <= 4 cores

• 2 VMs with 4 vCPUs each◦ 8 vCPUs – 2 VMs > 4 cores◦ Better configuration: 3 vCPUs each

Constant Load Rule: Examples – 4 Core Server



• 30 VMs with 1 vCPU each◦ 30 vCPUs – 30 VMs <= 16 cores

• 16 VMs with 2 vCPUs each◦ 32 vCPUs – 16 VMs <= 16 cores

• 10 VMs with 4 vCPUs each◦ 40 vCPUs – 16 VMs > 16 cores◦ Better configuration: 3 vCPUs each

Constant Load Rule: Examples – 16 Core Server



• Up to 4 vCPUs by default• Improves system performance & scalability

◦ QEMU◦ Netback

• Manual Configuration via:/etc/sysconfig/unplug-vcpus*not officially supported*

Dom0 - vCPU


• CPU Pinning◦ xe vm-param-set uuid=<vmuuid> VCPUs-params:mask=1,2,3◦ (Limits VM to running on CPU number 1,2,3)

• CPU Weight◦ xe vm-param-set uuid=<vmuuid> VCPUs-params:weight=500◦ (512 will get twice as much as 256 (default ))

• CPU Cap◦ xe vm-param-set uuid=<vmuuid> VCPUs-params:cap=100◦ (Restricts percentage of CPU cores VM can use)◦ 50 = half a CPU core, 100 = 1 CPU core, 400 = 4 CPU cores

Enhanced Resource Control - CPU


• Do not over-allocate virtual memory◦ Most physical systems have more memory than they need

• Do not under-allocate virtual memory◦ Leads to excess use of in VM swap files

• Configure dynamic/static memory rage

Memory Best Practices


• 1 NIC for Server Traffic (eth0 by default)◦ Inter-server communication◦ XenMotion Traffic◦ Import/Export Traffic

• 1 NIC for VM (public) Traffic• 2 NICs bond for NFS Storage Traffic

◦ Multipath configuration for iSCSI

Network Best Practices – 4 pNIC Server


•部署 XenServer:建议


• Segregate host management, storage and VM network interfaces where possible• Enable Hyper-Threading on Intel Nehalem or newer CPUs• Tweak BIOS to optimize for performance

◦ Disable deep C-states◦ Disable executive lock bit◦ Disable hyper-threading

• Use NFS shared storage if space is an issue (thin provisioning)• Set up alerting and email notification• Utilise a centralised syslog server if available

Do

28


• Use the XenServer HCL to guide hardware selection◦ http://hcl.vmd.citrix.com/

• Keep XenServer hosts up to date with hotfixes• Ensure all XenServer hosts have sync’ed clocks (NTP recommended)• Enable serial console if the host has a usable serial port

◦ http://support.citrix.com/article/CTX121442◦ Especially useful if host crashes are seen and some form of serial port logging can be

configured

Do

29

http://hcl.vmd.citrix.com/

http://support.citrix.com/article/CTX121442

© 2012 Citrix | Confidential – Do Not Distribute 30Citrix Confidential – Do Not Distribute

• Storage Recommendations◦ Confirm storage fibre type - Tesco required to fix their HBA cards to “Point-to-Point”◦ Don’t exceed number of VDIs/SR or number of SRs/host & pool

Do


• Don’t leave crud in the dom0 filesystem (e.g. XVAs in /root) – it can reduce the disk space available for logs and ultimately lead to a full disk and unexpected behavior

• Use the active-active SLB bonding mode in complex L2 networks – we often find switch forwarding tables struggle with multiple frequent MAC movements between edge ports.

• Run more VMs on a host than official supported. Even if it looks like its working today it can push the host over the edge in a failure condition such as multipath failover of NFS outage. We are working on increasing VM density in future releases.

• Don’t try tweaking dom0 memory and vCPUs as a cheeky way to run more VMs. It puts the system in an untested configuration and unknown badness could ensue.

Don’t

31


Practice Experience


尽量不要用不在 HCL列表中的产品


• I have a customer who, after installing new HP NC550SFP 10Gb Emulex NICs, is getting duplicate NICs in XenCenter connected to a running XenServer 5.6 FP1 pool. Please see NICs 4 and 5 in the image below. When the customer runs ifconfig from the command line, he does not see any NICs with MAC addresses ending with 5F and 5A . Rescanning does not fix anything. He is trying to avoid rebooting, but he will if he has to.

尽量不要用不在 HCL列表中的产品


• Hotfix XS56E001

• HotFix XS56E006 + driver disk if needed

• Hotfix XS56E007

• Hotfix XS56E009 + driver disk if needed

• Hotfix XS56E010

• Hotfix XS56E012

• Some hotfixes may require an updated device driver

Apply hotfixes in sequential order


• The Xenserver 6.0 installation will in some cases come up with the following error: Could not parse sgdisk.

• You will get this error if the storage you try to install XenServer on contains a previous VMWare ESX installation.

• sgdisk cannot handle VMFS volumes.

• The solution is to remove the ESX partition from storage. After this XenServer setup will continue installing.

How to fix XenServer 6.0 error ‘could not parse sgdisk’ when installing XenServer


• If you do not see the storage properly attached, run the following command:# xe-toolstack-restart- Or -Disconnect and reconnect the pool in XenCenter.

xe-toolstack-restart


• We attempted to delete a virtual disk from the XS console and the options are greyed out. We tried to issue a number of xe commands but to no avail; instead we get messages stating the disk is in use by a VM. This is not true so how can we go about deleting this virtual disk?

• These are the commands we tried to run:

• xe vdi-destroy uuid=<UUID of VDI> --force

• xe vdi-unlock uuid=<UUID of VDI> --force

• We then figured out the UUID of the VBD that maps the VDI to the VM (the mysterious VM it thinks is in use) and then ran this command:

• xe vbd-destroy uuid=<UUID of VBD> --force

• We also tried this:

• xe vbd-unplug uuid=<UUID of VBD> --force >> this gave us an error saying the device is unpluggable.

• We really want to try to avoid rebooting the host or restarting XAPI. Please advise.

• Try xe vdi-forget uuid=<> and then rescan the SR where the VDIs are at. Then try remove them again.

Cannot delete a virtual disk image


• My customer want to know about “How to run custom script on Dom0 remotely except SSH or RSA Key connection method?”

• You can install/use the xe CLI on another Linux or Windows system.

• And any “xe” command that otherwise you would have executed in dom0, you can execute remotely against a XenServer pool.

• For Windows you’ll find the xe CLI in the same directory as XenCenter is installed.

• For Linux it is one of the RPMs on the XenServer ISO I believe (don’t have one in front of me).

• Once installed you can run your own bash scripts that execute xe commands against your XenServer pool.

• Or you use the PowerShell/.Net/java/C/Phyton SDKs to write your own scripts/programs. (See http://community.citrix.com/display/xs/Download+SDKs)

• All of these approaches will use an https connection to the XenServer pool.

question for excute custom script runs on Dom0 remotly.

http://community.citrix.com/display/xs/Download+SDKs


• There are some running VMs and xapi doesn’t allow you to unplug, forget,....

LVM over ISCSI SR options


• Admin guide for 5.6 fp1 tell that host log level can be set by host-loglevel-set level= but command (host-loglevel-set) is not a valid command …..

• Do you know correct way of modify host log level?

• References I have found are not clear …

• The command you want is xe log-set-output

• http://forums.citrix.com/thread.jspa?threadID=252628

host-loglevel-set

http://forums.citrix.com/thread.jspa?threadID=252628


• Anyone knows what happen if syslog server is not accesible by xenserver host?

• Are logs written back to local or dismissed

• Depends on how the syslog service is configured. By default it should have both destinations configured and will just continue to log locally instead…

syslog server


• Is needed to restart any daemon and or host after changes to be effective?

• Yes. Either a restart of the system logger service or a reboot of the host.

Changes in logrotate.conf


• Please ensure that no XenServer hosts are in maintenance mode. Garbage cleanup will not occur until all hosts are in sync.

• You should be able to use a vdi-delete command from CLI to get better feedback than from GUI

Problem with disk space on NFS


• Do they have antivirus running inside the Win7 VMs? We know that Symantec Endpoint protection causes a similar issue. • They do have McAfee running. We havent disabled yet, but can give it a try. Should

we be looking at something specific?

• The Symantec piece is called the filter driver, it gets in the way of the drivers fully loading during install. Only way around was to uninstall SEP and reinstall the PV tools, then put back SEP.

NICs stopped getting proper IP addresses and in some cases no IP addresses at all.



• One thing to do is to create a management interface for the Control VM (AKA “dom 0”) on the network that the VMs use. If dom 0 does not get a DHCP address, neither will the Guest VMs. That simple test would isolate it to the external network.

NICs stopped getting proper IP addresses and in some cases no IP addresses at all.



# ovs-appctl bond/set-active-slave bond0 eth2

done

# ovs-appctl bond/show bond0

bond_mode: active-backup

lacp: off

bond-detect-mode: carrier

updelay: 31000 ms

downdelay: 200 ms

slave eth1: enabled

slave eth2: enabled

active slave

DVS / active-passive bond / primary path


• Keep in mind that RAW LUN mapping and LUN-per-VDI (as iSL uses) are not necessarily the same thing. They result in a LUN on the array being mapped as a VDI to the VM,

• however we do not have generic RAW LUN mapping support at this moment (as in: taking an existing LUN and mapping it manually or through the XS architecture to a VM).

RAW LUN mapping vs LUN-per-VDI



• When I use CLI to import a VM, like:

• xe vm-import filename=blah.xva preserve=true

• …any networks that are need are created on the host.

• However,

• xe vm-import filename=blah.xva preserve=true metadata=true

• …it fails saying that it can’t find the networks needed.

• Anyone know of a way to make this work?

Importing VM metadata doesn't create networks needed


• When you import the metadata only, the networks are required to be there already. When you import the VM in full, it will create the network it was on previously.

• It cares b/c that is what is written into the code, always has been for the metadata process. You can add the `force’ option to make it import the metadata when the network doesn’t exist.

Importing VM metadata doesn't create networks needed


HVM is required for this operation


NFS SR Performance


• 安装 XS6.0 halted 在 46% 进度• 提示“ [Errno 5] Input/output error”

[Errno 5] Input/output error


• I would recommend disabling HA whenever you are doing maintenance/reconfiguration of XenServer, external network, and SAN.

• xe host-emergency-ha-disable --force

disabling HA


XS Maintenance Rules


XS Maintenance Rules

Citrix XenServer ÔËÎ¬×¢ÒâÊÂÏî_20120530_CN


XenServer 6.0 XE-Command Reference (1.1)




Work better. Live better.

Documents

2012 年 5 月 31 日星期四 Xen400 渠道培训 XenServer Sunny Wang