CloudFoundry deploy on OpenStack (mailto:[email protected] ...p.primeton.com/uploads/file/201511/bca534d2bcf8.pdf · CloudFoundry deploy on OpenStack (mailto:[email protected]) 相关示例截图：

CloudFoundry deploy on OpenStack (mailto:[email protected])

VMWare

CloudFoundry


部署方案

CloudFoundry 支持 OpenStack、AWS、VMWare 等 IaaS 虚拟化平台，本文描述在

OpenStack 平台上如何部署 CloudFoundry，如何安装 OpenStack 不在本文范畴。

环境准备

准备好一个 OpenStack 环境，CloudFoundry 安装可以采用全程在线安装的方式，由于国

内网络环境不佳，我们先准备好离线安装包；资源下载地址 http://bosh.io/ ，下载 cf-

release-215.tgz （20151117 可下载最新版本 215，V3），Stemcell 有针对各个云平台的

不同操作系统的镜像文件，本文选择 OpenStack 云平台，Ubuntu 操作系统镜像（20151117

可下载最新版本 3139）。

安装过程

主要安装过程，可以参考 CloudFoundry 官方文档（英文版）

http://docs.cloudfoundry.org/deploying/openstack/

安装 BOSH CLI

使用 OpenStack 创建一台 ubuntu 虚拟机，如果没有镜像，在本地 VMWare Workstation

环境下安装 ubuntu-14.04-server-amd64 操作系统，安装完成后上传至 OpenStack 主机，

并转化镜像格式、导入 OpenStack 镜像库。

（1）安装基础类库

sudo apt-get install build-essential libxml2-dev libsqlite3-dev libxslt1-dev libpq-d

ev libmysqlclient-dev zlib1g-dev

http://bosh.io/

http://docs.cloudfoundry.org/deploying/openstack/


（2）安装 Ruby 环境（v2+）

apt-get 方式安装的 ruby 版本较低，可以从 ruby 官方下载 https://www.ruby-

lang.org/en/ 源码编译安装；本文使用 ruby-2.2.3。

（3）安装 BOSH_CLI

高版本 ruby 集成了 gem，不需要单独安装；因此直接安装 BOSH_CLI：

$ gem install bosh_cli --no-ri --no-rdoc

校验 OpenStack APIs

参考 http://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html

创建 ~/.fog 文件，并添加以下内容，修改 OpenStack 连接信息；

:openstack:

:openstack_auth_url: http://HOST_IP:5000/v2.0/tokens

:openstack_api_key: PASSWORD

:openstack_username: USERNAME

:openstack_tenant: PROJECT_NAME

:openstack_region: REGION # Optional

安装 fog 并测试 OpenStack APIs

$ gem install fog

$ fog openstack

>> Compute[:openstack].servers

[]

如果没有报错，则表明 API 调用正常。

测试虚拟机之间网络互通：

$ ping 30.10.3.2

PING 30.10.3.2 (30.10.3.2) 56(84) bytes of data.

64 bytes from 30.10.3.2: icmp_seq=1 ttl=64 time=0.095 ms

https://www.ruby-lang.org/en/

https://www.ruby-lang.org/en/

http://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html




...

API 调用压力测试：

$ gem install fog

$ fog openstack

>> 100.times { p Compute[:openstack].servers }

创建并挂载卷：登陆 OpenStack DashBoard 测试，此处不做演示；

测试 CloudFoundry stemcell 镜像上传和创建虚拟机：登陆 OpenStack DashBoard 测

试，此处不做演示。

（本节非必须，可以不做，有时间可以校验一下 OpenStack 环境是否可用）

创建 OpenStack 安全组

参考官方文档

http://docs.cloudfoundry.org/deploying/common/security_groups.html

ssh

Direction Ether Type IP Protocol Port Range Remote

Egress IPv4 Any - 0.0.0.0/0 (CIDR)

Ingress IPv4 UDP 68 0.0.0.0/0 (CIDR)

Ingress IPv4 ICMP - 0.0.0.0/0 (CIDR)

http://docs.cloudfoundry.org/deploying/common/security_groups.html



Egress IPv6 Any - ::/0 (CIDR)

Ingress IPv4 TCP 22 0.0.0.0/0 (CIDR)

cf-public

Direction Ether Type IP Protocol Port

Range Remote







cf-private





Ingress IPv4 TCP 1-65535 bosh

Ingress IPv4 UDP 3456-3457 0.0.0.0/0 (CIDR)


登陆 OpenStack DashBorad 进行配置，对于测试环境，可以配置较为开放的安全组，或直

接使用默认的安全组。

部署 MicroBosh

参考官方文档 http://bosh.io/docs/deploy-microbosh-to-openstack.html

（1）创建部署 manifest.yml 文件；

$ mkdir ~/cf

$ cd ~/cf

$ vi manifest.yml

拷贝以下文件内容并修改 OpenStack 连接参数；

---

name: microbosh

network:

type: manual

vip: FLOATING-IP # Replace with a floating IP address

ip: SUBNET-POOL-IP-ADDRESS # Replace with an address from the subnet IP address all

ocation pool of your OpenStack internal network

cloud_properties:

net_id: NETWORK-UUID # Replace with your OpenStack internal network UUID

resources:

persistent_disk: 20000

cloud_properties:

http://bosh.io/docs/deploy-microbosh-to-openstack.html


instance_type: m1.xlarge

cloud:

plugin: openstack

properties:

openstack:

auth_url: IDENTITY-API-ENDPOINT # Replace with your OpenStack Identity API endp

oint

tenant: OPENSTACK-TENANT # Replace with OpenStack tenant name

username: OPENSTACK-USERNAME # Replace with OpenStack username

api_key: OPENSTACK-PASSWORD # Replace with your OpenStack password

default_key_name: microbosh # OpenStack Keypair name

private_key: microbosh.pem # Path to OpenStack Keypair private key

default_security_groups: [bosh]

apply_spec:

properties:

director: {max_threads: 3}

hm: {resurrector_enabled: true}

ntp: [0.north-america.pool.ntp.org, 1.north-america.pool.ntp.org]

示例：


（2）准备 OpenStack 环境

环境验证、创建 Keypair、配置安全组、分配浮动 IP，参考官方文档截图，此处不再介绍；

（3）准备 Stemcell

如果已经下载了 stemcell，则直接上传至 BOSH_CLI 机器上，否则从官方下载，下载地址：

https://bosh.io/stemcells

（4）部署 MicroBOSH

https://bosh.io/stemcells


打开终端，输入 bosh micro deployment manifest.yml 设置部署文件；

$ cd ~/cf

$ bosh micro deployment manifest.yml

WARNING! Your target has been changed to https://192.168.0.7:25555!

Deployment set to ~/my-micro-deployment/manifest.yml

执行 bosh micro deploy STEMCELL-NAME 部署 MicroBOSH；

$ bosh micro deploy bosh-stemcell-3139-openstack-kvm-ubuntu-trusty-go_agent.tgz

No 'bosh-deployments.yml' file found in current directory.

Is ~/my-micro-deployment a directory where you can save state? (type 'yes' to continu

e): yes

Deploying new micro BOSH instance ~/my-micro-deployment/manifest.yml to 'https://19

2.168.0.7:25555' (type 'yes' to continue): yes

Started deploy micro bosh ...

Done deploy micro bosh

Deployed '~/my-micro-deployment/manifest.yml' to 'https://192.168.0.7:25555', took 0

0:04:51 to complete

登陆 MicroBOSH Server，使用默认用户密码 admin/admin；

$ bosh target https://192.168.0.7:25555

Target set to 'microbosh'

Your username: admin

Enter password: *****

Logged in as 'admin'

$ bosh vms

No deployments


相关示例截图：

使用 BOSH 部署 CloudFoundry

（1）配置 OpenStack 套餐

OpenStack Flavor Name CPUs RAM (GB) Disk (GB) Ephemeral Disk

(GB)

m1.small 1 2 10 20

m1.medium 2 4 10 40

m1.large 4 8 10 80

m1.xlarge 8 16 10 160

登陆 OpenStack DashBoard 配置，此处不做介绍。

（2）登陆 BOSH Server

$ bosh target https://192.168.0.7:25555


Target set to `bosh'

Your username: admin

Enter password: *****

Logged in as 'admin'

（3）记录 BOSH Director UUID

$ bosh status

Config

/root/.bosh_config

Director

Name microbosh

URL https://192.168.0.7:25555

Version 1.3137.0 (00000000)

User admin

UUID abcdef12-3456-7890-abcd-ef1234567890

CPI openstack

dns enabled (domain_name: bosh)

compiled_package_cache disabled

snapshots disabled

Deployment

not set

（4）上传 stemcell

$ bosh upload stemcell ./bosh-stemcell-3139-openstack-

kvm-ubuntu-trusty-go_agent.tgz

（5）上传 cf-release

下载地址 http://bosh.io/releases/github.com/cloudfoundry/cf-

release?version=225

可以在线上传，

$ bosh upload release https://bosh.io/d/github.com/cloudfoundry/cf-release?v=225

http://bosh.io/releases/github.com/cloudfoundry/cf-release?version=225

http://bosh.io/releases/github.com/cloudfoundry/cf-release?version=225


由于国内网络较差，建议离线下载后再上传；

$ bosh upload release ./cf-release-225.tgz

如果上传失败，出现磁盘不足情况，请检查部署 MicroBOSH 配置文件 manifest.yml 中指

定的 OpenStack 虚拟机套餐。上传成功后，可以查看，在终端输入 bosh releases；

（6）配置 OpenStack 部署 manifest stub

参考官方文档： http://docs.cloudfoundry.org/deploying/openstack/cf-stub-

openstack.html

新建 cf-stub.yml 文件，并加入下面内容：

---

director_uuid: DIRECTOR_UUID

meta:

openstack:

net_id: net_id

auth_url: auth_url

tenant: openstack_tenant

username: openstack_username

api_key: openstack_api_key

security_groups: []

floating_static_ips:

- 0.0.0.0

networks:

- name: cf1

subnets:

- cloud_properties:

static:

- 0.0.0.0 - 0.0.0.26

http://docs.cloudfoundry.org/deploying/openstack/cf-stub-openstack.html

http://docs.cloudfoundry.org/deploying/openstack/cf-stub-openstack.html


properties:

cc:

droplets:

droplet_directory_key: the_key

buildpacks:

buildpack_directory_key: bd_key

staging_upload_user: username

staging_upload_password: password

bulk_api_password: password

db_encryption_key: the_key

ccdb:

roles:

- name: ccadmin

password: admin_password

tag: admin

databases:

roles:

- name: ccadmin

password: ccadmin_password

- name: uaaadmin

password: uaaadmin_password

dea_next:

disk_mb: 2048

memory_mb: 1024

domain: DOMAIN

loggregator_endpoint:

shared_secret: loggregator_endpoint_secret

nats:

user: nats_user

password: nats_password

router:

logrotate:

freq_min: 20

rotate: 5

size: 3M

enable_ssl: true

ssl_cert: |

-----BEGIN CERTIFICATE-----

MIIDBjCCAe4CCQCz3nn1SWrDdTANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJB

VTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0

cyBQdHkgTHRkMB4XDTE1MDMwMzE4NTMyNloXDTE2MDMwMjE4NTMyNlowRTELMAkG

A1UEBhMCQVUxEzARBgNVBAgTClNvbWUtU3RhdGUxITAfBgNVBAoTGEludGVybmV0

IFdpZGdpdHMgUHR5IEx0ZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB

AKtTK9xq/ycRO3fWbk1abunYf9CY6sl0Wlqm9UPMkI4j0itY2OyGyn1YuCCiEdM3


b8guGSWB0XSL5PBq33e7ioiaH98UEe+Ai+TBxnJsro5WQ/TMywzRDhZ4E7gxDBav

88ZY+y7ts0HznfxqEIn0Gu/UK+s6ajYcIy7d9L988+hA3K1FSdes8MavXhrI4xA1

fY21gESfFkD4SsqvrkISC012pa7oVw1f94slIVcAG+l9MMAkatBGxgWAQO6kxk5o

oH1Z5q2m0afeQBfFqzu5lCITLfgTWCUZUmbF6UpRhmD850/LqNtryAPrLLqXxdig

OHiWqvFpCusOu/4z1uGC5xECAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAV5RAFVQy

8Krs5c9ebYRseXO6czL9/Rfrt/weiC1XLcDkE2i2yYsBXazMYr58o4hACJwe2hoC

bihBZ9XnVpASEYHDLwDj3zxFP/bTuKs7tLhP7wz0lo8i6k5VSPAGBq2kjc/cO9a3

TMmLPks/Xm42MCSWGDnCEX1854B3+JK3CNEGqSY7FYXU4W9pZtHPZ3gBoy0ymSpg

mpleiY1Tbn5I2X7vviMW7jeviB5ivkZaXtObjyM3vtPLB+ILpa15ZhDSE5o71sjA

jXqrE1n5o/GXHX+1M8v3aJc30Az7QAqWohW/tw5SoiSmVQZWd7gFht9vSzaH2WgO

LwcpBC7+cUJEww==

-----END CERTIFICATE-----

ssl_key: |

-----BEGIN RSA PRIVATE KEY-----

MIIEpAIBAAKCAQEAq1Mr3Gr/JxE7d9ZuTVpu6dh/0JjqyXRaWqb1Q8yQjiPSK1jY

7IbKfVi4IKIR0zdvyC4ZJYHRdIvk8Grfd7uKiJof3xQR74CL5MHGcmyujlZD9MzL

DNEOFngTuDEMFq/zxlj7Lu2zQfOd/GoQifQa79Qr6zpqNhwjLt30v3zz6EDcrUVJ

16zwxq9eGsjjEDV9jbWARJ8WQPhKyq+uQhILTXalruhXDV/3iyUhVwAb6X0wwCRq

0EbGBYBA7qTGTmigfVnmrabRp95AF8WrO7mUIhMt+BNYJRlSZsXpSlGGYPznT8uo

22vIA+ssupfF2KA4eJaq8WkK6w67/jPW4YLnEQIDAQABAoIBAQCDVqpcOoZKK9K8

Bt3eXQKEMJ2ji2cKczFFJ5MEm9EBtoJLCryZbqfSue3Fzpj9pBUEkBpk/4VT5F7o

0/Vmc5Y7LHRcbqVlRtV30/lPBPQ4V/eWtly/AZDcNsdfP/J1fgPSvaoqCr2ORLWL

qL/vEfyIeM4GcWy0+JMcPbmABslw9O6Ptc5RGiP98vCLHQh/++sOtj6PH1pt+2X/

Uecv3b1Hk/3Oe+M8ySorJD3KA94QTRnKX+zubkxRg/zCAki+as8rQc/d+BfVG698

ylUT5LVLNuwbWnffY2Zt5x5CDqH01mJnHmxzQEfn68rb3bGFaYPEn9EP+maQijv6

SsUM9A3lAoGBAODRDRn4gEIxjPICp6aawRrMDlRc+k6IWDF7wudjxJlaxFr2t7FF

rFYm+jrcG6qMTyq+teR8uHpcKm9X8ax0L6N6gw5rVzIeIOGma/ZuYIYXX2XJx5SW

SOas1xW6qEIbOMv+Xu9w2SWbhTgyRmtlxxjr2e7gQLz9z/vuTReJpInnAoGBAMMW

sq5lqUfAQzqxlhTobQ7tnB48rUQvkGPE92SlDj2TUt9phek2/TgRJT6mdcozvimt

JPhxKg3ioxG8NPmN0EytjpSiKqlxS1R2po0fb75vputfpw16Z8/2Vik+xYqNMTLo

SpeVkHu7fbtNYEK2qcU44OyOZ/V+5Oo9TuBIFRhHAoGACkqHhwDRHjaWdR2Z/w5m

eIuOvF3lN2MWZm175ouynDKDeoaAsiS2VttB6R/aRFxX42UHfoYXC8LcTmyAK5zF

8X3SMf7H5wtqBepQVt+Gm5zGSSqLcEnQ3H5c+impOh105CGoxt0rk4Ui/AeRIalv

C70AJOcvD3eu5aFq9gDe/1ECgYBAhkVbASzYGnMh+pKVH7rScSxto8v6/XBYT1Ez

7JOlMhD667/qvtFJtgIHkq7qzepbhnTv5x3tscQVnZY34/u9ILpD1s8dc+dibEvx

6S/gYLVorB5ois/DLMqaobRcew6Gs+XX9RPwmLahOJpZ9mh4XrOmCgPAYtP71YM9

ExpHCQKBgQCMMDDWGMRdFMJgXbx1uMere7OoniBdZaOexjbglRh1rMVSXqzBoU8+

yhEuHGAsHGWQdSBHnqRe9O0Bj/Vlw2VVEaJeL1ewRHb+jXSnuKclZOJgMsJAvgGm

SOWIahDrATA4g1T6yLBWQPhj3ZXD3eCMxT1Q3DvpG1DjgvXwmXQJAA==

-----END RSA PRIVATE KEY-----

cipher_suites: TLS_RSA_WITH_RC4_128_SHA:TLS_RSA_WITH_AES_128_CBC_SHA

status:

user: router_user

password: router_password


login:

logout:

redirect:

url: "/login?template=openstack"

uaa:

admin:

client_secret: admin_secret

cc:

client_secret: cc_client_secret

clients:

app-direct:

secret: app-direct_secret

developer_console:

secret: developer_console_secret

notifications:

secret: notification_secret

login:

secret: login_client_secret

doppler:

secret: doppler_secret

cloud_controller_username_lookup:

secret: cloud_controller_username_lookup_secret

cc_routing:

secret: cc_routing_secret

gorouter:

secret: gorouter_secret

database:

max_connections: 100

max_idle_connections: 10

remove_abandoned: true

abandoned_timeout: 600

log_abandoned: true

jwt:

verification_key: vk

signing_key: sk

scim:

users:

-

admin|fakepassword|scim.write,scim.read,openid,cloud_controller.admin,doppler.fireh

ose

groups: additionalgroup1,additionalgroup2

uaadb:

roles:

- name: uaaadmin



tag: admin

配置项解释：

Deployment Manifest Stub Contents Editing Instructions

director_uuid: DIRECTOR_UUID

Replace DIRECTOR_UUID with the BOSH Director

UUID. Use bosh status to view the BOSH Director

UUID.

floating_static_ips:

- 0.0.0.0

Replace the floating_static_ips: 0.0.0.0 with

an existing static IP address for your OpenStack

floating network. This is assigned to the ha_proxy

job to receive incoming traffic.

networks:

- name: cf1

subnets:

- cloud_properties:

static:

- 0.0.0.0 - 0.0.0.25

Replace the cf1 subnets: cloud_properties:

static:0.0.0.0 - 0.0.0.25 IP address range with

a range of at least 26 consecutive IP addresses on

your private network and generally complete this

network section with net_id mapped to an

existing OpenStack network UUID, and

security_groups, similarly to the bosh/micro-bosh

configuration.

properties:

cc:

droplets:

droplet_directory_key:

the_key

buildpacks:

buildpack_directory_key:

bd_key

staging_upload_user:

username

staging_upload_password:

password

bulk_api_password:

password

db_encryption_key: the_key

Replace the droplet_directory_key:the key with

the directory (bucket) used to store droplets.

Replace thebuildpack_directory_key: bd_keywith

the directory (bucket) used to store buildpacks.

Replace the staging_upload_user:username with

the account user name used to upload files to the

Cloud Controller.

Replace thestaging_upload_password: passwordwith

the password of the account used to upload files to

the Cloud Controller.

Replace the bulk_api_password: password with

the password used to access the bulk_api.

Replace the db_encryption_key: the_key with a


secure key you generate to encrypt sensitive values

in the Cloud Controller database.

ccdb:

roles:

- name: ccadmin


tag: admin

Replace the roles: name: ccadminwith the admin

user name used to connect to the Cloud Controller

database.

Replace ccadmin_password with the Cloud

Controller database admin password.

databases:

roles:

- name: ccadmin


- name: uaaadmin

password:

uaaadmin_password

Replace ccadmin with the admin user name used to

connect to the Cloud Controller database.

Replace the ccadmin_password with the Cloud

Controller database admin password.

Replace uaaadmin with the admin user name used

to connect to the UAA database.

Replace uaaadmin_password with the UAA

database admin password.

dea_next:

disk_mb: 2048

memory_mb: 1024

Do not change these values.

domain: example.com

Replace example.com with your domain.

loggregator_endpoint:

shared_secret:

loggregator_endpoint_secret

Replace

the shared_secret:loggregator_endpoint_secret

with a secure secret.

nats:

user: nats_user

password: nats_password

Replace nats_user and nats_passwordwith a

secure user name and password for NATS access.

router:

status:

Replace router_user and router_password wi

th a secure user name and password for router

access.


user: router_user

password: router_password

uaa:

admin:

client_secret:

admin_secret

batch:

username: batch_username

password: batch_password

cc:

client_secret:

cc_client_secret

clients:

app-direct:

secret: app-

direct_secret

developer_console:

secret:

developer_console_secret

login:

secret:

login_client_secret

notifications:

secret:

notification_secret

servicesmgmt:

secret:

service_mgmt_secret

space-mail:

secret: space-

mail_secret

support-services:

secret: support-

services_secret

Replace batch_username andbatch_password wi

th a secure user name and password.

Generate secure keys for each secret and replace

them.

jwt:

verification_key: vk

signing_key: sk

Replace vk with an RSA Public Key.

Replace sk with an RSA Private Key.

Replace fakepassword with an admin password.


scim:

users:

-

admin|fakepassword|scim.write,

scim.read,

openid,cloud_controller.admin

uaadb:

roles:

- name: uaaadmin


tag: admin

Replace uaaadmin with the admin user name used

to connect to the UAA database.

Replace admin_password with the UAA database

admin password.

示例配置，如下图所示：






克隆 cf-release GitHub 仓库：

$ git clone https://github.com/cloudfoundry/cf-

release.git

$ cd cf-release

$ ./scripts/update

安装 spiff：

离线安装包下载地址：https://github.com/cloudfoundry-incubator/spiff/releases

解压安装包，拷贝 spiff 文件至 /usr/bin 目录下，如果该文件没有执行权限，请赋予执行权

限，chmod +x /usr/bin/spiff；

https://github.com/cloudfoundry-incubator/spiff/releases


生成 CloudFoundry 部署文件：

$ ./scripts/generate_deployment_manifest openstack cf-

stub.yml > cf-deployment.yml

检查 cf-deployment.yml 部署配置文件，修改部分配置，官方方式生成的配置缺失一部分，

需要自己添加，各个版本可能有部分差异，需要尝试多次部署反复修改配置文件来解决问题；

登陆 MicroBOSH：

$ bosh target

Current target is https://192.168.0.7:25555

部署 CloudFoundry：

$ bosh deployment /root/cf/cf-deployment.yml

$ bosh deploy

参考下面截图：


部署完成（耗时较长，一般需要 1.5h 以上）


查看部署的虚拟机：

$ bosh vms

至此，CloudFoundry 平台部署完成。


Documents

CloudFoundry deploy on OpenStack (mailto:[email protected] ...p.primeton.com/uploads/file/201511/bca534d2bcf8.pdf · CloudFoundry deploy on OpenStack (mailto:[email protected]) 相关示例截图：