1.OpenStack + KVM = .com VPS OpenStack + KVM = .com VPS OSC 2012 Kyoto: 2012/08/03OpenStack User MTG#07: 2012/08/28 OSC 2012 Tokyo/Fall: 2012/09/08 http://www.slideshare.net/chroum/2012-osc-kyoto-openstack-vps-kvm 11294

2. OpenStack + KVM = .com VPS .com VPS VPS CentOS 6.x + griddynamics.net OpenStack Diablo VPS OpenStack OSS DevOps GlusterFS 3.3betanoVNCdisk resizeFreeBSD virtio 21294 3. OpenStack + KVM = .com VPS GMO (1) .com VPS VZ(2) GMOVirtuozzo(OpenVZ)OSswapBest Effort EL6, KVM .com VPS KVMLinux KVM KVMVPS webLB(Brocade ADX) (3) .com Windows Desktop Fusion-IOHyper-V, RDP GMO GMO VPS GMO Public (Xen, KVM) KVMPlesk API31294 4. OpenStack + KVM = .com VPS .com VPS KVM ( native KVM) ISO uploadOSLive CD boot http/https/ssh virtIO ON/OFF GMO( ) VIF(spoongIP) REST API OpenStack Diablo()(1)41294 5. OpenStack + KVM = .com VPS .com VPS KVM web (.Net + IIS) Windows Server DB (mysql) (Linux KVM) websocket-VNC(noVNC) ssh/console sftp rabbitmq dhcp ldap(dhcp, ssh) nova/keystone DB(mysql) nova Manage/API ISO upload storage (GlusterFS, NFS) Linux KVM (nova Compute) CentOS 6.xnova Compute, collectd, kvm51294 6. OpenStack + KVM = .com VPS .com VPS KVM InternetL3 SW L3 SWAPI / Manage web (.Net) API / Manage mysqldb API/ DBAPI / Manage API/nova Managenova Computenova Computenova Computenova Computenova Computenova Compute glancenova Computenova Computenova Computenova Compute sftpconsoleconsole consolesftpconsoledhcpvnc RAID 10 local diskRAID 10 local disk mqAPI / Manage mqldapmysqlmqdbnova DB Storagekeystone DBStorage ISO storage (NFS)61294 7. OpenStack + KVM = .com VPS .com VPS KVM .com VPS VZ(Virtuozzo) IN/OUTFirewall >> Port80 port, 22 port (php...) >> >> Linux bridge mail gateway(SPAM) >> >> VPS KVM FirewallLinux bridge(ebtables) 71294 8. OpenStack + KVM = .com VPS .com VPS KVM Flat DHCP + VLAN bridge (FirewallLinux bridge) GMONet() L3 SW(dhcp relay) L2 SW Rack L2 SW Linux bonding(act.-stan.) Linux VLAN Linux bridge VM VIF(tap - virtio NIC) 81294 9. OpenStack + KVM = .com VPS .com VPS KVM InternetGMO core backboneL3 SWL3 SW L3 SWL2 SWL3 SWdhcp relaydhcpeth0 eth1consolenova Compute dhcp bond0 bridge.VLAN1libvirt network lterconsole ldap bridge.VLAN2ebtalbes bridge.VLAN3tc htb (iproute) LDAPVMMACcollectd ADDRESSIPeth0VM eth0 eth0eth0 VMVM VM91294 10. OpenStack + KVM = .com VPS OpenStack griddynamics.net Diablo CentOS6 CentOS 6.xpythonpython 2.6back port libvirtCentOS http://yum.griddynamics.net/yum/diablo-centos/ => () src.rpm github.spec stable CentOS 6.3OSlibvirt diablo-centos (EPEL(6) OpenStack) 101294 11. OpenStack + KVM = .com VPS OpenStack griddynamics.net Diablo CentOS6 w VPS API (# ) => (CentOS 6.x) 111294 12. OpenStack + KVM = .com VPS OpenStack griddynamics.net Diablo CentOS6 ) griddynamics dashboard > glanceVMVM 121294 13. OpenStack + KVM = .com VPS OSVPS... ) VMTerminate : ... (virsh destroy )...>> ACPIshutdownw w python-nova-*.rpm /usr/lib/python2.6/site-packages/nova/compute/ /usr/lib/python2.6/site-packages/nova/virt/ ...131294 14. OpenStack + KVM = .com VPS shutoff(1)libvirt (nova/virt/driver.pyw) (destroy call) (Essexreboot(SOFT)) Terminate:OFF + VPSvmACPI shutdowncall shutoff diff -uNr ../usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py ./virt/libvirt/connection.py --- ../usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py 2012-07-31 12:01:28.890335240 +0900 +++ ./virt/libvirt/connection.py2012-07-31 10:15:55.186268025 +0900 @@ -1901,3 +1953,63 @@ def set_host_enabled(self, host, enabled): """Sets the specified hosts ability to accept new instances.""" pass + + + def shutoff(self, instance, network_info, flag):# GMO +instance_name = instance[name] + +try: + virt_dom = self._lookup_by_name(instance_name) +except exception.NotFound: + virt_dom = None141294 15. OpenStack + KVM = .com VPS shutoff(2) + + # If the instance is already terminated, were still happy + # Otherwise, destroy it + if virt_dom is not None: + try: +virt_dom.shutdown() + except libvirt.libvirtError as e: +is_okay = False +errcode = e.get_error_code() +if errcode == libvirt.VIR_ERR_OPERATION_INVALID: +# If the instance if already shut off, we get this: +# Code=55 Error=Requested operation is not valid: +# domain is not running +(state, _max_mem, _mem, _cpus, _t) = virt_dom.info() +if state == power_state.SHUTOFF: +is_okay = True + +if not is_okay: +LOG.warning(_("Error from libvirt during shutdown of " + "%(instance_name)s. Code=%(errcode)s " + "Error=%(e)s") % + locals()) +raise + + 151294 16. OpenStack + KVM = .com VPS shutoff(3) + """Called at an interval until the VM is gone.""" + timeout_count = range(FLAGS.shutoff_retry_count) + while timeout_count:# Y.Kawada +try: +state = self.get_info(instance_name)[state] +except exception.NotFound: +msg = _("During shutoff, %s disappeared.") % instance_name +LOG.info(msg) +break +if state == power_state.SHUTOFF: +msg = _("Instance %s shutoff successfully. stat:%s") % (instance_name, state) +LOG.info(msg) +break + + timeout_count.pop() + utils.my_logger(waiting... shutoff pw_state: %s % state) + if len(timeout_count) == 0: + utils.my_logger("Error shutoff: %s time over, try to destroy." % instance_name) + +""" try to destroy.""" +msg = _("try destroy. :%s") % (instance_name) +LOG.info(msg) +self.destroy(instance, network_info, cleanup=False) +break + time.sleep(1) + + return True161294 17. OpenStack + KVM = .com VPS ) dashboard : vnc(noVNC) >> noVNCEssex(python) >> noVNC (javascript) noVNC (html5 Canvas + WebSocketvnc(rfb)) OpenStackauthhttp://www.moongift.jp/2011/02/20110219-2/https://github.com/kanaka/noVNChttp://novnc.com/ python-nova-*.rpm /usr/lib/python2.6/site-packages/nova/vnc/ ... Essex (EssexnoVNCxvp) 171294 18. OpenStack + KVM = .com VPS noVNC ... 181294 19. OpenStack + KVM = .com VPS OSVPS... API Diablo(), Essex()() computevmlimit Diablolimithost(limit) vcpu, memory (Essex) try: except: VIF(VMattach) VMVIFIDvm 191294 20. OpenStack + KVM = .com VPS try:except: )API except catch diff -uNr ../usr/lib/python2.6/site-packages/nova/vnc/auth.py ./vnc/auth.py --- ../usr/lib/python2.6/site-packages/nova/vnc/auth.py 2012-07-31 12:01:28.900514682 +0900 +++ ./vnc/auth.py2012-07-31 10:15:55.190267491 +0900 @@ -52,7 +52,12 @@if not token:referrer = req.environ.get(HTTP_REFERER) -auth_params = urlparse.parse_qs(urlparse.urlparse(referrer).query) + try: + auth_params = urlparse.parse_qs(urlparse.urlparse(referrer).query) + except: + LOG.audit(_("Unauthorized Access: (%s)"), req.environ) + return webob.exc.HTTPForbidden(detail=Unauthorized) +if token in auth_params: token = auth_params[token][0] 201294 21. OpenStack + KVM = .com VPS VIFVPS) vm( iu1 + - + ID) iu1-000001c8ID(16)vneth0, eth1 ... >> sufx -0 , -1 ...vn000001c8-0 (tap) >> eth0 (VM)(collectd)(tc)ebtables 211294 22. OpenStack + KVM = .com VPS VIF ) diff -uNr ../usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py ./virt/libvirt/vif.py --- ../usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py 2012-07-31 12:01:28.890335240 +0900 +++ ./virt/libvirt/vif.py2012-07-31 10:15:55.186268025 +0900 @@ -38,7 +38,7 @@ class LibvirtBridgeDriver(VIFDriver): """VIF driver for Linux bridge.""" - def _get_configurations(self, network, mapping): + def _get_configurations(self, network, mapping, instance): """Get a dictionary of VIF configurations for bridge type.""" # Assume that the gateway also acts as the dhcp server. gateway6 = mapping.get(gateway6) @@ -59,6 +59,7 @@extra_params = "n" result = { + name: instance[name].split(-)[1], # Y.Kawada id: mac_id, bridge_name: network[bridge], mac_address: mapping[mac], @@ -90,7 +91,7 @@network[bridge],network[bridge_interface]) - return self._get_configurations(network, mapping) +return self._get_configurations(network, mapping, instance) def unplug(self, instance, network, mapping): """No manual unplugging required."""221294 23. OpenStack + KVM = .com VPS OpenStack: , , , OFF >> shutoff call >> shutoff > start API call(libvirt reboot kvm)231294 24. OpenStack + KVM = .com VPS Internet.com VPS : API L3 SWL3 SWAPI / Manage web (.Net)API / Manage mysqldb API/ DBAPI / Manage API/nova Managea Computeova Compute nova Computeglance nova Compute nova Computesftp consoleconsole console sftp consoledhcpvncRAID 10 local diskmq API / Manage mq ldap mysqlmqdbnova DB 241294Storage keystone DB 25. OpenStack + KVM = .com VPS OpenStack: html5 vnc (1) OpenStack dashboardvncnoVNC >> Top >>ajaxAPIw251294 26. OpenStack + KVM = .com VPS html5 vnc (2) noVNC websocketCanvas web browser (IE8 Chrome Frame)261294 27. / DBOpenStack + KVM = .com VPS API / Manage API/nova Managhtml5 vnc (3)g sftpconsoleconsoleconsolesftpconsoledhcp vnck mqAPI / Manage mqldapmysqlmqdb nova DBekeystone DBgee271294 28. OpenStack + KVM = .com VPS html5 vnc (3): nginxwebsocketreverse proxy + dnsnginx websocket reverse proxy patch(nginx 1.3)https://github.com/yaoweibin/nginx_tcp_proxy_modulePort 6080vncproxyPort 80/443[n-gohko@vnc1001 ~]$ ps axu | grep nova | grep -v greproot 15509 0.4 2.0 451048 81164 ? S Jul11 152:13 /usr/bin/python /usr/bin/nova-vncproxy --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-vncproxy.log --pidfile=/var/run/nova/nova-vncproxy.pid[n-gohko@vnc1001 ~]$ ps axu | grep nginx | grep -v greproot 1303 0.0 0.0 93188 1492 ? Ss Jun26 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.confnginx 1304 0.1 0.0 94100 3472 ? S Jun26 54:44 nginx: worker process 281294 29. OpenStack + KVM = .com VPS html5 vnc (3): nginx websocket reverse proxytcp {proxy_read_timeout 6000;proxy_send_timeout 6000;upstream websockets { server 172.20.113.31:6080;#check interval=3000 rise=2 fall=5 timeout=1000;check_http_send "GET / HTTP/1.0rnrn";}server { #listen 6081; server_name _; listen 443;access_log /var/log/nginx/access.ssl.log;ssl on;ssl_certificate /etc/nginx/ssl/ast.myvps.jp/ast.myvps.jp.pem;ssl_certificate_key /etc/nginx/ssl/ast.myvps.jp/ast.myvps.jp.key;ssl_protocolsSSLv3 TLSv1;ssl_ciphers HIGH:!ADH:!MD5;tcp_nodelay on;proxy_pass websockets;}} 291294 30. OpenStack + KVM = .com VPS : collectdcollectd-virtEPEL6collectd-virtcollectd-rrdtool(rrd) libvirt nova-computevmrrdrrd301294 31. OpenStack + KVM = .com VPS : collectd(rrdmysqlcsv)[root@cnode-a1001 novaclient]# rpm -qa | grep collectdcollectd-4.10.3-1.el6.x86_64collectd-web-4.10.3-1.el6.x86_64collectd-rrdtool-4.10.3-1.el6.x86_64collectd-virt-4.10.3-1.el6.x86_64311294 32. OpenStack + KVM = .com VPS html5 serial console: anyterm + conservernova-console(ajax-term)anyterm(OpenStack Diabloajax-term)http://anyterm.org/anytermweb (anyterm --> console)(OpenStack Essexajax-term)Pointpop-upURL [Enter `^Ec? for help]iu1-000007ff 321294 33. OpenStack + KVM = .com VPS html5 serial console: anyterm token https://console1001.myvps.jp/anyterm/proxy/11048/ anyterm.html?token=bb268280-8e4e-41c1-894f- ac325079530f&title=iu1-000007ff nova-consolehttpd apache reverse proxy (Port 80/443)[root@console1001 ~]# ps -aef | grep httpd | grep -v greproot 1523 1 0 Jul09 ? 00:00:38 /usr/sbin/httpdapache 3385 1523 0 Jul29 ?00:00:04 /usr/sbin/httpdapache 3386 1523 0 Jul29 ?00:00:04 /usr/sbin/httpdapache 3387 1523 0 Jul29 ?00:00:04 /usr/sbin/httpdapache 23084 1523 0 Jul31 ?00:00:03 /usr/sbin/httpdapache 26971 1523 0 Aug01 ? 00:00:02 /usr/sbin/httpd[root@console1001 ~]# netstat -antp | grep pythontcp0 0 172.20.113.36:33841 172.20.113.54:3306 ESTABLISHED 2712/pythontcp0 0 172.20.113.36:44081 172.20.113.32:5672 ESTABLISHED 2712/pythontcp0 0 172.20.113.36:44084 172.20.113.32:5672 ESTABLISHED 2712/python 331294 34. OpenStack + KVM = .com VPS ssh serial console: ldap ssh + conserverssh serial consolessh loginshellconsolesshvmanytermssh serial console [Enter `^Ec? for help]conservervirsh console 341294 35. OpenStack + KVM = .com VPS ISO upload: ldap ssh(sftp) chroot + NFSldap sshconsolesftp upload (openssh 5.4p1patch)NFS storagechroot(NFS chroot)>> proftpdmod_sftpquota 351294 36. OpenStack + KVM = .com VPS ISO upload: ldap ssh(sftp) chroot + NFSNFS serverGlusterFS () GlusterFS 3.3beta ()brick...GlusterFS quotaquota>> quota checkGlusterFSlockquotaISOGlusterFS...(GlusterFS 3.3 ) 361294 37. OpenStack + KVM = .com VPS OSS: OS Nagios 3.3.1 (EPEL6)Hardware IBM Director + HP SIM (Freenot OSS)HPg7OSagent()HPgen8age: cobbler (kickstart network install) (EPEL6)cobblerhostdhcp, dns, Nagioscobblertrigger(IBM Director, HP SIM) 371294 38. OpenStack + KVM = .com VPS cobbler triggernagios [n-gohko@v2cob1001 libvirt]$ find /var/lib/cobbler/triggers/ /var/lib/cobbler/triggers/ /var/lib/cobbler/triggers/sync /var/lib/cobbler/triggers/sync/post /var/lib/cobbler/triggers/sync/post/201-post-chmod-644-tftpboot-kick.sh /var/lib/cobbler/triggers/sync/post/nagios-add-hosts.sh /var/lib/cobbler/triggers/sync/post/301-post-chmod-644-tftpboot-kick.sh /var/lib/cobbler/triggers/sync/pre /var/lib/cobbler/triggers/delete /var/lib/cobbler/triggers/delete/file /var/lib/cobbler/triggers/delete/file/post /var/lib/cobbler/triggers/delete/file/pre /var/lib/cobbler/triggers/delete/distro /var/lib/cobbler/triggers/delete/distro/post /var/lib/cobbler/triggers/delete/distro/pre /var/lib/cobbler/triggers/delete/mgmtclass /var/lib/cobbler/triggers/delete/mgmtclass/post /var/lib/cobbler/triggers/delete/mgmtclass/pre /var/lib/cobbler/triggers/delete/package /var/lib/cobbler/triggers/delete/package/post /var/lib/cobbler/triggers/delete/package/pre /var/lib/cobbler/triggers/delete/system /var/lib/cobbler/triggers/delete/system/post /var/lib/cobbler/triggers/delete/system/pre /var/lib/cobbler/triggers/delete/profile ...381294 39. OpenStack + KVM = .com VPS DevOps: 1) GlusterFS 3.3betagluser.org>> >> 391294 40. OpenStack + KVM = .com VPS DevOps: 2) virt resizedisk image 20GB + disk>> GlanceOS(CentOS 6.2)>> GMO >> LVMdisk(10%) imageresizefsckOS OS >> image 401294 41. OpenStack + KVM = .com VPS DevOps: 3) vncGoogle chrome dev 20.xnoVNC(((())))noVNCupdatechrome 20.xnoVNC(html5) 411294 42. OpenStack + KVM = .com VPS DEMO: DNSname base DNSsub domain NS Internet sub domain NS MacBook-Air:$ dig ns d.myvps.jpwww ;d.myvps.jp.IN NS MacBook-Air:$ dig www.d.myvps.jp @d-ns01.myvps.jp ;; ANSWER SECTION:;; QUESTION SECTION: d.myvps.jp.5 INNS d-ns01.myvps.jp.;www.d.myvps.jp.IN A d.myvps.jp.5 INNS d-ns02.myvps.jp. ;; ANSWER SECTION: www.d.myvps.jp. 5 INA 157.7.129.212 ;; ADDITIONAL SECTION: d-ns01.myvps.jp. 86400 IN A 157.7.129.212 MacBook-Air:$ dig www.d.myvps.jp @d-ns02.myvps.jp d-ns02.myvps.jp. 86400 IN A 157.7.129.213 ;; QUESTION SECTION: ;www.d.myvps.jp.IN ADNS;; ANSWER SECTION: www.d.myvps.jp. 5 INA 157.7.129.213 DNS + DNS +web serverweb server L7(reverse proxy) L7(reverse proxy) d-ns01. d-ns02.w01.w02. 157.7.129.212 157.7.129.213157.7.129.214 157.7.129.215 reverse proxyw01, w02 421294 43. OpenStack + KVM = .com VPS (1)200 > compute nodeDiabloOpenStack DiabloVPSOpenStack Hypervisor Driver(Essex)VPSsimple(m(_;))m )IP(Floting IP)IPv6>> >> vipIPNIC 431294 44. OpenStack + KVM = .com VPS (2) http://b.hatena.ne.jp/articles/201206/9110 441294 45. OpenStack + KVM = .com VPS (2) vnc 451294 46. OpenStack + KVM = .com VPS (3)qemu-img convertCentOS 6.3, Essex? 2nd Unit(OpenStackversion upHosting)CentOS 6.x + EPEL Essex()API ==> VPS Stack ()(LB) 461294 47. OpenStack + KVM = .com VPS Special Thanks!GMO Internet Inc., System Div., SI Team and Cloud Team Yamada san 471294