1

Orchestrating Virtualized Network FunctionsMd. Faizul Bari∗, Shihabur Rahman Chowdhury∗, Reaz Ahmed∗, Raouf Boutaba∗, and Otto Carlos Muniz

Bandeira Duarte†∗David R. Cheriton School of Computer Science, University of Waterloo

mfbari | sr2chowdhury | r5ahmed | rboutaba@uwaterloo.ca†Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil

otto@gta.ufrj.br

Abstract—Middleboxes or network appliances like firewalls,proxies and WAN optimizers have become an integral part oftoday’s ISP and enterprise networks. Middlebox functionalitiesare usually deployed on expensive and proprietary hardwarethat require trained personnel for deployment and maintenance.Middleboxes contribute significantly to a network’s capital andoperation costs. In addition, organizations often require theirtraffic to pass through a specific sequence of middleboxes forcompliance with security and performance policies. This makesthe middlebox deployment and maintenance tasks even morecomplicated. Network Function Virtualization (NFV) is an emerg-ing and promising technology that is envisioned to overcome thesechallenges. It proposes to move packet processing from dedi-cated hardware middleboxes to software running on commodityservers. In NFV terminology, software middleboxes are referredto as Virtualized Network Functions (VNFs). It is a challengingproblem to determine the required number and placement ofVNFs that optimizes network operational costs and utilization,without violating service level agreements. We call this the VNFOrchestration Problem (VNF-OP) and provide an Integer LinearProgramming (ILP) formulation with implementation in CPLEX.We also provide a dynamic programming based heuristic tosolve larger instances of VNF-OP. Trace driven simulations onreal-world network topologies demonstrate that the heuristic canprovide solutions that are within 1.3 times of the optimal solution.Our experiments suggest that a VNF based approach can providemore than 4× reduction in the operational cost of a network.

I. INTRODUCTION

Today’s enterprise networks ubiquitously deploy verticallyintegrated proprietary middleboxes or network appliances tooffer various network services. Examples of such middleboxesinclude firewalls, proxies, WAN optimizers, Intrusion Detec-tion Systems (IDSs), and Intrusion Prevention Systems (IPSs).These middleboxes are used for realizing various performanceand security objectives [1], [2]. A recent study shows that thenumber of different middleboxes is comparable to the numberof routers in enterprise and data center networks [2], [3]. Eventhough middleboxes have become an integral part of modernnetworks, they come with high Capital Expenditure (CAPEX)and Operational Expenditure (OPEX). They are usually vendorspecific, vertically integrated, expensive, and require speciallytrained personnel for deployment and maintenance. Moreover,it is often impossible to add new functionality to an existingmiddlebox, which makes it very difficult and cumbersome forthe network operator to deploy new services. In many cases,the network operator is compelled to upgrade or purchase newhardware for introducing new network services.

Another set of problems arise from the fact that mostoften traffic flows are required to pass through multiple stages

of middlebox processing in a particular order, e.g., a trafficflow may be required to go through a firewall, then anIDS, and finally through a proxy [4]. This phenomenon isvery common for middleboxes and is typically referred toas Service Function Chaining (SFC) [5]. The IETF Networkand Service Chaining Working Group has several IETF draftsdemonstrating middlebox chaining use-cases in operator net-works [6], mobile networks [7], and data center networks [8].The task of sequencing these in-network middlebox processingis commonly referred to as middlebox orchestration. Currently,middleboxes are placed at fixed locations within a network.Traffic flows are routed through the required sequence ofmiddleboxes by manually crafting the routing table entries. Itis a cumbersome and error-prone process. Moreover, the fixedlocation of middleboxes cannot be optimal for all possibletraffic patterns in the long run.

An emerging and promising technology that can ad-dress these limitations is Network Function Virtualization(NFV) [9], [10]. It proposes to move packet processing fromhardware middleboxes to software middleboxes or VirtualNetwork Functions (VNFs) running on commodity (e.g.,x86 based systems) servers. This approach will not hamperperformance as many state-of-the-art software middleboxeshave already shown the potential to achieve near-hardwareperformance [11], [12]. NFV provides ample opportunities fornetwork optimization and cost reduction. Previously, middle-boxes were hardware appliances placed at fixed locations, butnow we can deploy a VNF on any server in the network. VNFlocations can be determined intelligently to ensure efficienttraffic routing. NFV opens-up the opportunity to simultane-ously optimize VNF locations and traffic routing paths, whichcan significantly reduce the network OPEX.

VNF chains can be orchestrated by dynamically deployinga composition of VNFs either on a single server or on a clusterof servers. This approach can significantly reduce the OPEXof a network. However, several issues need to be consideredbefore provisioning VNFs: (i) the cost of deploying a newVNF, (ii) energy cost for running a VNF, (iii) the cost offorwarding traffic to and from a VNF, and (iv) fragmentationof the underlying physical resource pool. Placing just enoughVNFs to match traffic processing requirements may yield thelowest deployment and energy cost, but steering traffic throughthese VNFs will increase traffic forwarding cost and mayeventually lead to Service Level Objective (SLO) violations.On the other hand, one may try to always forward trafficthrough the shortest possible path by deploying VNFs in allpossible locations. This approach may avoid SLO violation

2

penalty, but will surely lead to huge deployment and energycosts. An optimal VNF orchestration strategy must addressthese issues during the cost minimization process. Moreover,it must avoid Service Level Objective (SLO) violations andsatisfy the capacity constraints of the physical servers andphysical links. We refer to this problem as the VirtualizedNetwork Function Orchestration Problem (VNF-OP). Thispaper builds on our earlier work [13] and makes the followingcontributions:• We provide the first quantifiable results showing that

dynamic VNF orchestration can have more than 4×reduction in OPEX compared to hardware middleboxes.

• The problem is formulated as an Integer Liner Program(ILP) and implemented in CPLEX1 to find optimal solu-tions for small scale networks.

• We prove the NP-hardness of VNF-OP by a reductionfrom the Capacitated Plant Location Problem with SingleSource constraints.

• We propose a fast heuristic algorithm that can findsolutions within 1.3 times of the optimal.

• The heuristic’s performance in terms of solution qualityand scalability is evaluated using both real-world andsynthetic topologies and traffic traces.

• Finally, we compared the performance of our heuristicwith related work from literature [14].

The rest of the paper is organized as follows: we start byexplaining the mathematical model used for our system and byformally defining the VNF Orchestration Problem (Section II).Then the problem formulation is presented (Section III). Next,a heuristic is proposed to obtain near-optimal solutions (Sec-tion IV). We validate our solution through trace driven sim-ulations on real-world network topologies (Section V). Then,we provide a literature review (Section VI) and finally, weconclude with some future research directions (Section VII).

II. MATHEMATICAL MODEL AND PROBLEM DEFINITION

In this section we introduce the mathematical model for oursystem and formally define the VNF Orchestration Problem.

A. Physical Network

We represent the physical network as an undirected graphG = (S, L), where S and L denote the set of switches andlinks, respectively. We assume that VNFs can be deployed oncommodity servers located within the network. These networklocations are traditionally known as Point-of-Presences orPoPs. The set N represents these servers and the binaryvariable hns ∈ 0, 1 indicates whether server n ∈ N isattached to switch s ∈ S.

hns =

1 if server n ∈ N is attached to switch s ∈ S,0 otherwise.

Let, R denote the set of resources (CPU, memory, disk, etc.)offered by each server. The resource capacity of server n isdenoted by crn ∈ R+, ∀ r ∈ R. The bandwidth capacity and

1http://www-01.ibm.com/software/commerce/optimization/cplex-optimizer/

propagation delay of a physical link (u, v) ∈ L is representedby βuv ∈ R+ and δuv ∈ R+, respectively. We also define η(u)as the set of neighbors for switch u.

η(u) = v | (u, v) ∈ L or (v, u) ∈ L, u, v ∈ S

B. Virtualized Network Functions (VNFs)

Different types of VNFs (e.g., firewall, IDS, IPS, proxy, etc.)can be provisioned in a network. The possible VNF types arerepresented by the set P . Each VNF type p has a specificdeployment cost, resource requirements, processing capacity,and processing delay represented by D+

p , κrp ∈ R+(∀r ∈ R),cp (in Mbps), and δp (in ms), respectively. These quantitiesare explained below:• Deployment Cost (D+

p ) includes the cost of imagetransfer and booting a VNF of type p on a server.

• Resource Requirement (κrp) is the amount of resourceof category r that must be allocated to a type p VNF.

• Processing Capacity (cp) represents the amount of traffic(in Mbps) a type p VNF can process.

• Processing Delay (δp) is the average delay (in ms)experienced by a packet when traversing through a VNFof type p.

The actual values of the above mentioned quantities arehighly implementation specific and depend on a lot of fac-tors. Here, we have assumed an approximate value for theseproperties to simplify the mathematical model.

There can be certain hardware requirements (e.g., hardware-accelerated encryption for Deep Packet Inspection (DPI))that may prevent a server from running a particular type ofVNF. Furthermore, the network manager may have preferencesregarding provisioning a particular type of VNF on a particularset of servers, e.g., Firewalls should be deployed close to thenetwork edge. So, we assume that for each VNF type there isa set of servers on which it can be provisioned. The followingbinary variable represents this relationship:

dnp =

1 if VNF type p ∈ P can be provisioned on n,0 otherwise.

C. Traffic Request

We assume that the network operator is receiving requestsfor setting up paths for different kinds of traffic (e.g., VPNsetup, security features, new application or service in a datacenter, etc.). A traffic request is represented by a 6-tuplet = 〈ut, vt,Ψt, βt, δt, ωt〉, where ut, vt ∈ S denote the ingressand egress switches, respectively. βt ∈ R+ is the bandwidthdemand of the traffic. δt is the maximum allowed propaga-tion delay according to Service Level Agreement (SLA). Ψt

represents the ordered VNF sequence the traffic must passthrough (e.g., Firewall IDS Proxy). lΨt denotes thelength of Ψt and ωt denotes the policy to determine SLOviolation penalties.

In our mathematical model, we transform a VNF sequenceΨt into a directed graph Gt = (N t, Lt), where N t representsthe set of traffic nodes (switches and VNFs) and Lt denotes the

3

Ingress

Firewall IDS Proxy

Egress

s1 s2

Fig. 1. VNF Chain

links between them. Fig. 1 shows a sample VNF chains. Here,traffic flows through the chain Firewall IDS Proxy.Modeling the traffic flow in this way makes it easy for theprovisioning process to ensure that it passes though the correctsequence of VNFs. We also define ηt(n1) to represent theneighbors of n1 ∈ N t:

ηt(n1) = n2 | (n1, n2) ∈ Lt, n1, n2 ∈ N t

Next, we define a binary variable gtnp ∈ 0, 1 to indicatethe type of a node n ∈ N t

gtnp =

1 if node n ∈ N t is of type p ∈ P,0 otherwise.

D. VNF Orchestration Problem (VNF-OP)

We consider a scenario where an operational network isserving a set of traffics T . It has a set of VNFs alreadydeployed and the routing paths for the traffics in T arealso provisioned. Now, the network operator is receiving newtraffic requests and wants to provision the required VNFs androuting paths for them. The network operator can choose toprovision resources for one traffic request at a time or leveragea lookahead interval by accumulating a number of trafficrequests and provision resources in batches. Determining theoptimal number or volume of traffic or the length of thelookahead interval for each batch is an interesting researchchallenge that is beyond the scope of this work and we planto pursue it in the future. In the rest of this paper, we denotea new traffic batch by T . Based on the operator’s choice, abatch may contain just one or multiple traffic requests.

In the VNF-OP, we are given a physical network topology,VNF specifications, current network status and a set of newtraffic requests. Our objective is to minimize the overallnetwork OPEX and physical resource fragmentation by (i)provisioning an optimal number of VNFs, (ii) placing themat the optimal locations, and (iii) finding the optimal routingpaths for each traffic request, while respecting the capacityconstraints (e.g., physical servers, links, and VNFs) and en-suring that traffic passes through the proper VNF sequence.

OPEX: In this work, we consider the network OPEX to becomposed of the following four cost components:• VNF deployment cost: we need to complete tasks like

transferring a VM image, booting it and attaching it todevices before deploying a VNF. We associate a cost (indollars) with these operations.

• Energy cost: it represents the cost of energy consump-tion by the active servers. A server is considered activeif it has at least one active VNF. Servers consume powerbased on the amount of resources (e.g., CPU, memory,disk, etc.) under use. A server is assumed to be in theidle state if it does not have any active VNFs [15].

• Traffic forwarding cost: traffic forwarding cost may in-cur from two sources: (i) leasing cost of transit links [16]and (ii) energy consumption of the network devices (e.g.,switches, routers, etc.).

• Penalty for SLO violation: this cost component repre-sents the penalty that must be paid to the customer forSLO violations, e.g., if a traffic experienced more that themaximum allowed propagation delay.

Resource Fragmentation: We compute physical resourcefragmentation by measuring the percentage of idle resourcesfor the active servers and links. We want to minimize frag-mentation as it eventually increases the possibility of accom-modating more traffic on the same physical resources.

III. PROBLEM FORMULATION AND COMPLEXITYANALYSIS

VNF-OP is a considerably harder problem to solve thantraditional Virtual Network (VN) embedding problems [17].There is no node ordering requirement in VN embedding,while in VNF-OP we need to preserve the ordering of VNFs.Moreover, in VNF-OP we need to respect the processingcapacity constraints of servers and the VNFs to be deployed.How many VNFs are to be deployed is not known in advance,rather it is an outcome of the optimization process. Multi-dimensional Bin Packing [18] can also be used to solveVNF-OP, but here we will end-up with a nested bin packingproblem. In the first layer traffics need to be packed into VNFsand in the next layer VNFs need to be packed into the physicalservers. The fact that the number and locations of VNFs is notknown in advance, results in quadratic constraints for resourcecapacity and renders the problem unsolvable even for verysmall instances by existing optimization solvers. In this work,we address these challenges by judiciously augmenting thephysical network, as explained in the rest of the section.

A. Physical Network Transformation

We transform the physical network to generate an aug-mented pseudo-network that reduces the complexity involvedin solving the VNF-OP. The transformation process is per-formed in the following two steps as explained by Fig. 2:

1) VNF Enumeration: A part of a physical network topol-ogy is shown in Fig. 2(a). Here, we have three switches (s1, s2and s3) and a server n2 connected to switch s2. We enumerateall possible VNFs in this step by finding the maximum numberfor each VNF type that can be deployed on each server. Wecalculate this number based on the resource capacity of theserver and the resource requirement of a type of VNF. Forexample, if a server has 16 cores, and CPU requirement forFirewall and IDS are 4 and 8 cores, respectively, we can deploy4 Firewalls and 2 IDSs on it. In Fig. 2(b) we show enumeratedVNFs for server n2.

We denote the set of these VNFs (called pseudo-VNFs) byM. Each VNF m ∈ M is implicitly attached to a servern ∈ N . We use the function ζ(m) to denote this mapping.

ζ(m) = n if VNF m is attached to server n

4

s1 s2

s3

n2

Switch

Server

(a) Original Network

s1 s2

s3

n2

Firewall

Proxy

IDS

(b) VNF Enumeration

s1 s2

s3

Pseudo

Switch

(c) Adding pseudo-switches

Fig. 2. Network Transformation

We also define a function Ω(n) to represent this mappingin the opposite direction:

Ω(n) = m | ζ(m) = n, m ∈M, n ∈ N

Next, we define qmp ∈ 0, 1 to indicate the type of a VNF:

qmp =

1 if VNF m is of type p ∈ P,0 otherwise.

As discussed earlier, a given type of VNF can be deployedon a specific set of servers. To ensure this we must have:

qmp = dζ(m)p (1)

We should note that pseudo-VNFs simply represent wherea particular type of VNF can be provisioned. ym ∈ 0, 1indicates whether a pseudo-VNF is active or not.

ym =

1 if pseudo-VNF m ∈M is active,0 otherwise.

2) Adding Pseudo-Switches: Next, we augment the physi-cal topology again by adding a pseudo-switch between eachpseudo-VNF and the original switch to which it was con-nected. This process is shown in Fig. 2(c). We performthis step to simplify the expressions of the network flowconservation constraint in the ILP formulation. This processdoes not increase the size of the solution space as we considerthem only for the flow conservation constraint.

B. ILP Formulation

We define the decision variable xtnm to represent the map-ping of a traffic node to a pseudo-VNF:

xtnm =

1 if node n ∈ N t is provisioned on m ∈M,0 otherwise.

Next, we define another variable to represent the mappingbetween a traffic node and a switch in the physical network.

ztns =

1 if node n ∈ N t is attached to switch s,0 otherwise.

ztns is not a decision variable as it can be derived from xtnm:

ztns = 1 if xtnm = 1 and hζ(m)s = 1

We can also derive the variable ym from xtnm as follows:

TABLE IGLOSSARY OF SYMBOLS

Physical NetworkG(S, L) Physical network G with switches S and links LN Set of servershns ∈ 0, 1 If server n ∈ N is attached to switch s ∈ SR Set of resources offered by serverscrn ∈ R+ Resource capacity of server n, ∀r ∈ Rβuv , δuv ∈ R+ Bandwidth, propagation delay of link (u, v) ∈ Lη(u) Neighbors of switch uan ∈ 0, 1 an = 1 if Server n is activefuv ∈ 0, 1 fuv = 1 if physical link (u, v) is active

Virtualized Network Functions (VNFs)P Set of possible VNF typesD+

p , κrp, cp, δp Deployment cost, resource requirement, processing

capacity and processing delay of VNF type p ∈ Pdnp ∈ 0, 1 dnp = 1 if VNF type p can be provisioned on server n

Trafficut, vt,Ψt Ingress, egress and VNF sequence for traffic tβt, δt, ωt Bandwidth, expected delay, SLA penalty for tNt ut, vt,ΨtLt (ut,Ψt

1), . . . , (Ψt|Ψt|−1

,Ψt|Ψt|), (Ψ

t|Ψt|, v

t)ηt(n) Neighbors of n ∈ Nt

gtnp ∈ 0, 1 gtnp = 1 if node n ∈ Nt is of type p ∈ PM Set of pseudo-VNFsζ(m) ζ(m) = n if VNF m ∈M is attached to server nΩ(n) m | ζ(m) = n, m ∈M, n ∈ Nqmp ∈ 0, 1 qmp = 1 if VNF m ∈M is of type p ∈ P

Decision Variables*xtnm ∈ 0, 1 xtnm = 1 if node n ∈ Nt is provisioned on m ∈M*wtn1n2

uv ∈ 0, 1 wtn1n2uv = 1 if (n1, n2) ∈ Lt uses physical link (u, v) ∈ L

Derived Variables*ym ∈ 0, 1 ym = 1 if VNF m ∈M is activeztns ∈ 0, 1 ztns = 1 if node n ∈ Nt is attached to switch s*xtnm, w

tn1n2uv , ym denote value from the previous iteration

ym = 1 iff∑t∈T

∑n∈Nt

xtnm > 0

We assume that xtnm represents the value of xtnm at thelast traffic provisioning event. To ensure that resources forpreviously provisioned traffic are not deallocated we must havextnm ≥ xtnm, ∀ t ∈ T , n ∈ N t,m ∈ M. Now, we defineym ∈ 0, 1 that represents the value of ym at the last trafficprovisioning event as follows:

ym = 1 iff∑t∈T

∑n∈Nt

xtnm > 0

Again, to ensure that resources for previously provisionedtraffics are not deallocated we must have ym ≥ ym, ∀ m ∈

5

M. Next, we need to ensure that VNF capacities are not over-committed. The processing capacity of an active VNF mustbe greater than or equal to the total amount of traffic passingthrough it. We express this constraint as follows:∑

t∈T

∑n∈Nt

xtnm × βt ≤ cm, ∀ m ∈M|ym = 1 (2)

We also need to make sure that physical server capacity con-straints are not violated by the deployed VNFs. We representthis constraint as follows:∑

m∈Ω(n)

ym × κrm ≤ crn, ∀ n ∈ N , r ∈ R (3)

Each node of a traffic must be mapped to a proper VNFtype. This constraint is represented as follows:

xtnm × gtnp = qmp, ∀ t ∈ T, n ∈ N t,m ∈M, p ∈ P (4)

Next, we need to ensure that every traffic node is provi-sioned and to exactly one VNF.∑

t∈T

∑n∈Nt

xtnm = 1, ∀ m ∈M (5)

Now, we define our second decision variable to representthe mapping between links in the traffic model (Fig. 1) to thelinks in the physical network.

wtn1n2uv =

1 if (n1, n2) ∈ Lt uses physical link (u, v),0 otherwise.

We also assume that wtn1n2uv represents the value of wtn1n2

uv

at the last traffic provisioning event. To ensure that resourcesfor previously provisioned traffics are not deallocated in thecurrent iteration we must have

wtn1n2uv ≥ wtn1n2

uv ,∀ t ∈ T , n1, n2 ∈ N t|n2 ∈ ηt(n1)

and n2 > n1, u, v ∈ S (6)

To ensure that each directed link in a traffic request is notmapped to both directions of a physical link, we must have:

wtn1n2uv + wtn1n2

vu ≤ 1,∀ t ∈ T,n1, n2 ∈ N t|n2 ∈ ηt(n1) and n2 > n1, u, v ∈ S (7)

Now, we present the capacity constraint for physical links:∑u∈S

∑v∈S

(wtn1n2uv + wtn1n2

vu )× βt ≤ βuu,

∀ t ∈ T, n1, n2 ∈ N t|n2 ∈ ηt(n1) and n2 > n1 (8)

Next, we present the flow constraint that makes sure that thein-flow and out-flow of each switch in the physical networkis equal except at the ingress and egress switches:∑

v∈η(u)

(wtn1n2uv − wtn1n2

vu

)= ztn1u − z

tn2u,

∀ t ∈ T, n1, n2 ∈ N t|n2 ∈ ηt(n1) and n2 > n1, u ∈ S (9)

Finally, we need to ensure that every link in a traffic requestis provisioned on a path in the physical network:

∑u∈S

∑v∈S

(wtn1n2uv + wtn1n2

vu ) ≥ 0,

∀ t ∈ T, n1, n2 ∈ N t|n2 ∈ ηt(n1) and n2 > n1 (10)

Our objective is to find the optimal number and placementof VNFs that minimizes OPEX and physical resource frag-mentation in the network. We formulate them in detail below:

OPEX: We consider four cost components to contribute toOPEX. These are as follows:

1. VNF Deployment Cost: the VNF deployment cost can beexpressed as follows:

D =∑

m∈M|ym=1

D+p × qmp × (ym − ym) (11)

2. Energy Cost: Without loss of generality we assume thatthe energy consumption of a server is proportional to theamount of resources being used. However, a server usuallyconsumes power even in the idle state. So, we compute thepower consumption of a server as follows:

En =∑m∈Ωn

ym × qmp × er(crn, κrp)

where

er(rt, rc) = (ermax − eridle)×rcrt

+ eridle

Here, rt and rc denote the total and consumed resource,respectively. eridle and ermax denote the energy cost in the idleand peak consumption states for resource r, respectively.

Hence, the total energy cost is

E =∑n∈N

∑m∈Ωn

ym × qmp × er(crn, κrp) (12)

3. Cost of Forwarding Traffic: Let us assume that the cost offorwarding 1 Mbit data through one link in the network is σ (indollars). We can compute the total cost of traffic forwardingas follows:

F =∑t∈T

∑n1∈Nt

∑n2∈ηt(n1)and n2>n1

∑u∈S

∑v∈η(u)

((wtn1n2

uv −

wtn1n2uv )× βt × σ

)(13)

4. Penalty for SLO violation: We can compute the actualpropagation delay experienced by a traffic as follows:

δat =∑n1∈Nt

∑n2∈ηt(n1)and n2>n1

∑u∈S

∑v∈η(u)

wtn1n2uv δuv

Let ρt(ωt, δt, δta) be a function that computes the penaltyfor SLO violation given the policy for determining penalty(ωt), expected propagation delay (δt) and actual propagationdelay (δta) for traffic t. So, the total cost for SLO violationscan be expressed as follows:

P =∑t∈T

ρt(ωt, δt, δta) (14)

– Resource Fragmentation: Our second objective is tominimize resource (e.g., server and links) fragmentation ofactive servers and links. We express it using the same unit as

6

the above mentioned costs. For this purpose, we assume thatpr denotes the price of unit resource of type r ∈ R. We alsodenote ρβ as the price of unit bandwidth.

A physical server n is considered active if it hosts at leastone active pseudo-VNF. The binary variable an captures thisproperty:

an =

1 if

∑m∈Ω(n)

ym > 0,

0 otherwise.Similarly, a physical link (u, v) is considered active if it is

hosting at least one traffic flow. We use the binary variablefuv to represent this:

fuv =

1 if

∑t∈T,

(n1,n2)∈Lt

wtn1n2uv > 0,

0 otherwise.Now, we can compute the total cost for resource fragmen-

tation as follows:

C =∑n∈N

an∑r∈R

(crn −

∑m∈Ω(n)

(κrp × qmpym))pr+

∑u∈S

∑v∈η(u)

fuv

(βuv −

∑t∈T

∑n1∈Nt

∑n2∈ηt(n1)and n2>n1

(wtn1n2uv × βt)

)ρβ

(15)

Here, the first term represents the cost of server resourcefragmentation (e.g., CPU, memory, disk, etc.) and the secondterm represents the cost of link bandwidth fragmentation.

Our objective is to minimize the total network operationalcost and resource fragmentation that can be expressed as aweighted sum of the aforementioned costs.

minimize(αD + βE + γF + λP + µC

)(16)

Here, α, β, γ, λ and µ are weighting factors that are usedto adjust the relative importance of the cost components.

VNF-OP is NP-Hard. We reduce the NP-Hard Capaci-tated Plant Location Problem with Single Source constraints(CPLPSS) [19] to the VNF-OP. In CPLPSS, we are givena set of potential locations for production plants with fixedcosts and capacities. A commodity produced by these plantsis to be supplied to a set of customers with fixed demands andassociated transportation costs. Moreover, each customer mustbe served by a single plant. The objective is to find a subsetof the plats that should be operated to minimize cost withoutviolating capacity and demand constraints.

Given an instance of the CPLPSS we can transform it to aninstance of VNF-OP in the following manner: (i) for each cus-tomer we create the chain DS → plant → customer,where DS is a dummy ingress switch, customer is theegress switch, and plant is a VNF, (ii) set the bandwidthof the chain to be equal to the customer demand, (iii) use thetransportation cost as the traffic forwarding cost, (iv) configureeach physical machine to deploy a single VNF of type plant,and (V) set the processing capacity of each plant to be equalto its production capacity. These operation can be performedin polynomial time of the problem size. Now, if we can solvethis instance of VNF-OP, we will also get a solution for the

CPLPSS. However, CPLPSS is NP-hard, so the VNF-OP isNP-hard as well.

IV. HEURISTIC SOLUTION

In this section, we present a heuristic to solve the VNF-OP.Given a network topology, a set of middlebox specificationsand a batch of traffic requests, the heuristic finds the numberand locations of different types of VNFs required to operatethe network with minimal OPEX. We did not explicitlyconsider resource fragmentation to keep the heuristic simpleand fast. However, our experimental results show that evenwith this simplification, the heuristic produces solutions thatare very close to the optimal. The heuristic runs in twosteps. First, we model the VNF-OP as a multi-stage directedgraph with associated costs. Then we find a near-optimal VNFplacement from the multi-stage graph by running the Viterbialgorithm [20]. In the following, we first describe the modelingof VNF-OP using multi-stage graph (Section IV-A), followedby the solution using Viterbi algorithm (Section IV-C). Adetailed discussion of the heuristic along with an illustrativeexample is provided in the Appendix.

A. Modeling with Multi-Stage Graph

For a given traffic request, t = 〈ut, vt,Ψt, βt, δt, ωt〉, werepresent t as a multi-stage graph with lΨt + 2 stages. Thefirst and the last (i.e., lΨt + 2) stages represent the ingressand egresses switches, respectively. These two stages containonly one node representing ut and vt, respectively. Stage i(∀i ∈ 2, . . . (lΨt + 1)), represents the (i − 1)-th VNF inthe traffic request and the node(s) within this stage representthe possible server locations where that type of VNFs canbe placed. Each node is associated with a VNF deploymentcost (Eq. 11) and an energy cost (Eq. 12) as described inSection III-B.

An edge (vi, vj) in this multi-stage graph represents theplacement of a VNF at a server attached to switch vj , giventhat the previous VNF in the sequence is deployed on a serverattached to switch vi. We put a directed edge between all pairsof nodes in stage i and i+ 1 (∀i ∈ 1, 2, . . . (lΨt + 1)). Weassociate two costs with each edge: the cost for forwardingtraffic (Eq. 13) and the penalty for SLO violations (Eq. 14).The traffic forwarding cost is proportional to the weightedshortest path (in terms of latency) between the switches.The penalty for SLO violations is obtained by the followingprocess: (i) we equally divide the maximum allowed delaybetween the stages, (ii) we assign a SLO violation cost fora transition between two successive stages in the multi-stagegraph whenever we incur more than the allocated delay dueto traffic transport and processing at the nodes. The total costof a transition between two successive stages is calculated bysumming the node and edge costs following Eq. 16. Finally,a path from the node in the first stage to the node in the laststage represents a placement of the VNFs. Our goal is to finda path in the multi-stage graph that yields minimal OPEX.

7

B. Heuristic Algorithm

Algorithm 1 gives the pseudcode of the heuristic solution.The procedure ProvisionTraffic takes as input a trafficrequest t and the network topology graph G annotated withthe resource capacities at each switch. We keep two tables,cost and π, to keep track of the cost and the sequence ofmiddlebox placements, respectively. costi,j represents the costof deploying the j-th middlebox in the middlebox sequenceΨt to a server attached with switch i. The cost computationprocedure is the same as described in Section IV-C. We use anumber of helper procedures for the ease of implementation.The first helper procedure, IsResourceAvailable checks ifa middlebox mbox for a traffic request t can be placed atswitch i, satisfying the minimum bandwidth and resourcerequirements. The second helper, GetCost, computes the costof placing middlebox mbox for a traffic request t at a serverattached to switch j. The previous node k that yields theminimum cost for the current node in consideration j, istracked by the entry πk,j . Finally, we backtrace using entriesin π to obtain the desired middlebox sequence.

Running Time: Let the number of switches and themaximum length of a middlebox sequence be n and m,respectively. Algorithm 1 performs Θ(nm) computations atthe beginning to initialize the cost matrix. Then for eachelement in the traffic sequence, the algorithm takes all possiblepairs of nodes u, v and computes the cost of deploying amiddlebox at the server attached to switch v given that theprevious middlebox in the sequence was deployed at a serverconnected to switch u. Therefore, there is a total of Θ(n2m)operations involved. With some pre-computation steps thecosts can be calculated and resource availability can be queriedin O(1) time. Therefore, Algorithm 1 runs in Θ(n2m).

Algorithm 1 ProvisionTraffic(t, G)1: ∀(i, j) ∈ 1 . . . |Ψt| × 1 . . . |S| : costi,j ←∞, πi,j ←NIL

2: ∀i ∈ |S| :3: if IsResourceAvailable(ut, i,Ψt

1, t) then4: cost1,n ← GetCost(ut, i,Ψt

1, t), π1,n ← n5: end if6: ∀(i, j, k) ∈ 2 . . . |Ψt| × 1 . . . |S| × 1 . . . |S| :7: if IsResourceAvailable(k, j,Ψt

i, t) then8: costi,j ← mincosti,j , costi−1,k +

GetCost(k, j,Ψti, t)

9: πi,j ← i yielding minimum costi,j10: end if11: Π← NIL, C ←∞, ψ ←<>12: ∀i ∈ |S| :13: C ← minC, cost|Ψt|,i + ForwardingCost(i, vt)+

SLOV iolationCost(i, vt, t)14: Π← i yielding minimum cost|Ψt|,i15: ∀i ∈< |Ψt|, |Ψt| − 1 . . . 1 > : Append Π to ψ, Π← πi,Π16: return Reverse(ψ)

C. Finding a Near-Optimal Solution

Viterbi algorithm is a widely used method for findingthe most likely sequence of states from a set of observedstates. To find such a sequence, Viterbi algorithm first modelsthe states and their relationships as a multi-stage graph.Each stage consists of the possible states and a transitioncost is assigned between all pairs of states in successivestages. Once the multi-stage graph is constructed, Viterbialgorithm proceeds by computing a per node cumulative cost,costu. This cost is computed recursively as the minimum ofcostv + transition cost(v, u), for all v in the previous stageas of u’s stage. costu represents the cost of including node u inthe final solution. This computation proceeds in the increasingorder of stages. After finishing the computation at the finalstage, the most likely sequence of states is constructed bytracing back a path from the final stage back to the first thatyields the minimum cost.

We borrow the idea of how costs are computed fromViterbi Algorithm and propose a traffic provisioning algorithm,ProvisionTraffic (Algorithm 1). It takes a traffic request tand a network topology G as input and returns a placementof Ψt in G. For each node u in each stage i, we find a nodev in stage i − 1 that yields the minimum total cost costv,u(costs are defined according to the discussion in Section IV-A).We keep track of the minimum cost path using the table π.After finishing computation for the final stage, we constructthe desired VNF placement by back tracing from the final stageto the first stage, using the entries in π. During this processwe update residual resource capacities of the servers and theresidual bandwidth of the links after each path is allocated.For each traffic request, the heuristic solution runs in Θ(n2m)time, where n is the number of switches in the network andm is the VNF sequence length.

D. Heuristic in Action

Fig. 3(a) shows an example network topology with sixswitches, where the servers are connected to switch 2, 3 and4. We need to find the path for a traffic which is going fromswitch 1 to 6 and must pass through a firewall, then an IDSand finally through a proxy.

First, we generate a multi-stage graph as shown in Fig. 3(b).Here, we are assuming that the firewall and proxy can bedeployed on any server, but the IDS can only be deployedon servers connected to switches 3 and 4. Each node in themulti-stage graph represents a decision about where to placea VNF. For example, if we select node 4 in the stage labeled“IDS”, it means that a VNF corresponding to an IDS will bedeployed on the server connected to switch 4. As explainedearlier, there is a cost associated with each node selection.

Now, we traverse this graph starting at node 1. The firststage is trivial, we just compute the cost of deploying andrunning (energy cost) a firewall at node 2, 3 and 4 and addthe cost of routing traffic from node 1 to each node. There is noadditional computation as there is just one incoming link foreach node. However, the operations for the subsequent stagesinvolve comparing the cost of reaching a particular node fromdifferent nodes. For example, node 3 in stage “IDS” can be

8

1

2

4

3

5 6

20ms

12ms

(a) Example Network Topolog

2

4

3

2

4

3 4

3

Firewall IDS Proxy

1 6

(b) Multi-Stage Graph

2

4

3 3

Firewall IDS

2

4

3 3

Firewall IDS

38

15

21

38

15

21

back_ptr

(c) A Single Stage

Fig. 3. Modeling with Multi-Stage Graph

reached from three different nodes. The operation performedin this is stage is explained in Fig. 3(c).

We need to compute the cost of transition from nodes 2,3 and 4 to node 3. These costs are shown on the left sideof Fig. 3(c). Now, if we select the link between node 4 andnode 3 then the Firewall will be deployed on node 4 and theIDS will be deployed on node 3 and cost of deploying the IDSwill be 38. However, we have links with lower costs than thisone and at each stage we select the incoming link with theminimal cost. So, here we will select the link between node2 and 3 as it has the lowest cost of 15. We will also save apointer (back_ptr) to mark the node that was selected. Wecontinue in this manner until we reach the destination node(node 6 in this example), then we follow the back_ptrs tore-construct the solution.

V. PERFORMANCE EVALUATION

We perform trace driven simulations on real-world net-work topologies to gain a deeper insight, and to evaluatethe effectiveness of the proposed solution. Our simulationis focused on the following aspects: (i) demonstrating thebenefits of dynamic VNF orchestration over hardware mid-dleboxes (Section V-C), (ii) comparing the performance ofthe heuristic solution with that of the CPLEX based optimalsolution (Section V-D), (iii) comparing the performance of ourheuristic with state-of-the art (Section V-E), (iv) demonstratingthe scalability of our heuristic (Section V-F), and (iv) Analyz-ing the behavior of the proposed solution for different trafficvolume (Section V-G). Before presenting the results, we brieflydescribe the simulation setup (Section V-A) and the evaluationmetrics (Section V-B). Implementations of both CPLEX andheuristic are available at http://goo.gl/Da7EZu.

A. Simulation Setup

1) Topology Dataset: We have used a wide range of net-work topologies: (i) Internet2 research network (12 nodes, 15links) [21], (ii) A university data center network (23 nodes,42 links) [22] and (iii) Autonomous System 3967 (AS-3967)from Rocketfuel topology dataset (79 nodes, 147 links) [23].

2) Traffic Dataset: We use both real traces and syntheticallygenerated traffic for the evaluation. We use traffic matrix tracesfrom [21] to generate time varying traffic for the Internet2topology. This trace contains a snapshot of a 12 × 12 trafficmatrix and demonstrates significant variation in traffic volume.For the data center network, we use the traces available

TABLE IISERVER AND MIDDLEBOX DATA USED IN EVALUATION

Server Data [15]Physical CPU Cores Idle Energy Peak Energy

16 80.5W 2735WHardware Middlebox Data

Idle Energy Peak Energy Processing Capacity1100W 1700W 40Gbps

VNF Data [11], [27]Network Function CPU Required Processing Capacity

Firewall 4 900MbpsProxy 4 900MbpsNat 2 900MbpsIDS 8 600Mbps

from [22], and replay the traffic between random source-destination pairs. Finally, for the Rocketfuel topology, wegenerated a synthetic time-varying traffic matrix using theFNSS tool [24]. It follows the distribution from [25] andexhibits time-of-day effect.

3) Middlebox and Cost Data: We have generated a 3-length middlebox sequence for each traffic based on the dataprovided in [26] and [4]. We have used publicly availabledata sheets from manufacturers and service providers to selectand infer values for server energy cost, SLO violation cost(for violating maximum latency), resource requirements forsoftware middleboxes and their processing capacities. We alsoobtained energy consumption data for hardware middleboxesfrom a popular network equipment manufacturer. Table II liststhe parameters used for servers, VNFs and middleboxes. In therest of this section we use the term “middlebox” to refer toboth hardware middlebox and VNF.

B. Evaluation Metrics

1) Operational Expenditure (OPEX): We measure OPEXaccording to Eq. 16, and compare CPLEX and heuristicsolutions by plotting the ratio of OPEX and its components.

2) Execution Time: It is the time required to find middleboxplacement for a given traffic batch and network topology.

3) System Utilization: We compute it as the fraction of usedCPU for a server. We also report the number of active servers.

4) Topological Properties of Solution: We report two topo-logical properties of the middlebox locations: (i) percentageof middleboxes placed withing k-hops from the ingress/egressswitches and (ii) path stretch, i.e., the ratio of path lengthobtained by CPLEX or the heuristic to the shortest path lengthfor the traffic. The first metric gives us an insight into thelocation of middleboxes with respect to the ingress/egressswitches, and the second one shows how many additional links

9

0

0.25

0.5

0.75

1

0 2000 4000 6000 8000 10000

Frac

tion

of P

eak

Traf

fic

Time (min)

(a) Internet2

0

0.25

0.5

0.75

1

0 5 10 15 20 25 30 35 40

Frac

tion

of P

eak

Traf

fic

Time (min)

(b) Data Center

0.75

1

0 200 400 600 800 1000 1200

Frac

tion

of P

eak

Traf

fic

Time (min)

(c) RocketfuelFig. 4. Traffic Distribution over Time for Different Scenarios

3.7

5 4

.5

0 2 4 6 8 10

Time (x103 min)

Total Cost

10

15

Energy Cost

1 1

.2 1

.4

Hardware vs. VNF

Transit Cost

(a) Hardware vs. VNF (Internet2)

1 1

.1

0 2 4 6 8 10

Time (x103 min)

Total Cost

0.5

1 Energy Cost 1

1.2

1.4

Heuristic vs. CPLEX

Transit Cost

(b) Internet2

1 1

.2 1

.4

0 5 10 15 20 25 30 35 40

Time (min)

Total Cost

1 1

.5

Energy Cost

1 1

.2 1

.4

Heuristic vs. CPLEX

Transit Cost

(c) Data Center

Fig. 5. Time vs. Cost Ratio

(hence more bandwidth) are required to steer traffic throughmiddlebox sequences.

C. VNFs vs. Hardware Middleboxes

One of the driving forces behind NFV is that VNFs cansignificantly reduce a network’s OPEX. Here, we providequantifiable results to validate this claim. Fig. 5(a) showsthe ratio of OPEX for hardware middleboxes to VNFs forincoming traffic provisioning requests (about 132 requests perbatch) over a period of 10000 minutes. We show two compo-nents of OPEX: energy and transit cost. There is no publiclyavailable data that can be used to estimate the deployment costof hardware middleboxes. So, for this experiment, we do notconsider deployment cost as a component of OPEX to makethe comparison fair. The SLO violation penalty is not shownas it is zero for all time-instances. We implemented a differentCPLEX program to peak provision the hardware middleboxes(peak traffic occurs at time-instance 7665). VNFs are provi-sioned at each time-instance by our CPLEX implementationcorresponding to the formulation provided in Section III.

The bottom part of Fig. 5(a) shows that VNFs provide morethan 4× reduction in OPEX. The individual reductions inenergy and transit costs are also shown in the same figure.The reduction in energy cost is much higher than that of thetransit cost. This is due to the fact that hardware middleboxesconsume considerably higher energy than commodity servers.From Fig. 5(a) and Fig. 4(a), we can also see that withthe increase in traffic volume (after time-instance 4000) thetotal cost ratio decreases. Interestingly, the energy cost ratio

decreases, but the transit cost ratio increases. Handling highertraffic volume requires higher number of VNFs to be de-ployed, which increases the energy consumption of commodityservers, thus decreasing the energy cost ratio. However, VNFsare provisioned at optimal locations by CPLEX, which causesthe transit cost to decrease and increases the transit costratio. The cost ratio relationship between VNFs and hardwaremiddleboxes depends on a number of factors like processingcapacity, traffic volume, idle and peak energy consumption.

The topological properties of VNF and hardware middleboxplacement locations are reported in Fig. 6. The CDF of hopdistance between the ingress switch and middlebox is shownin Fig. 6(a). Higher percentage of VNFs are located within 2hops of the ingress switch (mostly withing 1 hop), comparedto hardware middleboxes. Some VNFs are also located at 4hop distance. This only occurs when placing a VNF fartheraway reduces the OPEX by decreasing the energy cost. Similarresults are obtained for the hop distance between middleboxand egress switch (Fig. 6(b)). These two figures also demon-strate the fact that CPLEX places middleboxes in a morebalanced (symmetric) way on the path between the ingress andegress switch. The path stretch for both hardware middleboxesand VNFs are shown in Fig. 6(c). VNFs consistently achievea lower path stretch than hardware middleboxes, as VNFlocations are not static like the hardware middleboxes. Theycan be provisioned on any server to reduce OPEX.

10

0

0.2

0.4

0.6

0.8

1

0 0.5 1 1.5 2 2.5 3 3.5 4

CD

F

Hop Distance of Middlebox from Ingress Switch

VNF-Internet2 Hardware-Internet2

(a) Distance between Middlebox and Ingress

0

0.2

0.4

0.6

0.8

1

0 0.5 1 1.5 2 2.5 3 3.5 4

CD

F

Hop Distance of Middlebox from Egress Switch

VNF-Internet2 Hardware-Internet2

(b) Distance between Middlebox and Egress

0

0.25

0.5

0.75

1

1 2 3 4 5 6 7

CD

F

Stretch

VNF-Internet2 Hardware-Internet2

(c) Path Stretch.

Fig. 6. Topological Property Comparison between Hardware middlebox and VNF deployment (Internet2).

0.94

0.95

0.96

0.97

0.98

0.99

1

0 2000 4000 6000 8000 10000

Mea

n U

tiliz

atio

n

Time (min)

Heuristic Solution CPLEX Solution

(a) Mean Server Utilization (Internet2)

0.5

1

1.5

2

2.5

0 2000 4000 6000 8000 10000

Num

ber o

f Act

ive

Ser

vers

Time (min)

Heuristic Solution CPLEX Solution

(b) Number of Active Servers (Internet2)

0

0.2

0.4

0.6

0.8

1

Serve

r-1

Serve

r-3

Serve

r-4

Mea

n U

tiliz

atio

n

Server ID

Heuristic Solution CPLEX Solution

(c) Per Server Utilization (Internet2)

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30 35 40

Mea

n U

tiliz

atio

n

Time (min)

Heuristic Solution CPLEX Solution

(d) Mean Server Utilization (Data Center)

0

2

4

6

8

10

0 5 10 15 20 25 30 35 40

Num

ber o

f Act

ive

Ser

vers

Time (min)

Heuristic Solution CPLEX Solution

(e) Number of Active Servers (Data Center)

0

0.2

0.4 0.6

0.8

1

Serve

r-6

Serve

r-11

Serve

r-12

Serve

r-13

Serve

r-14

Serve

r-20

Serve

r-13

Serve

r-14

Serve

r-16

Serve

r-17

Serve

r-20

Mea

n U

tiliz

atio

n

Server ID

Heuristic Solution CPLEX Solution

(f) Per Server Utilization (Data Center).

Fig. 7. Resource Utilization

.

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5 6

CD

F

Hop Distance of Middlebox from Ingress Switch

Heuristic-Internet2CPLEX-Internet2

Heuristic-DCCPLEX-DC

(a) Distance between middlebox and ingress

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5 6

CD

F

Hop Distance of Middlebox from Egress Switch

Heuristic-Internet2CPLEX-Internet2

Heuristic-DCCPLEX-DC

(b) Distance between middlebox and egress

0

0.25

0.5

0.75

1

1 2 3 4 5 6 7

CD

F

Stretch

Heuristic-Internet2CPLEX-Internet2

Heuristic-DCCPLEX-DC

(c) Path stretch

Fig. 8. Topological properties of solution

D. Performance Comparison Between CPLEX and Heuristic

Now, we compare the performance of our heuristic with thatof the optimal solution. Fig. 5(b) and Fig. 5(c) show the costratios for Internet2 and data center networks, respectively. Thetraffic patterns for these two topologies are shown in Fig. 4(a)and Fig. 4(b), respectively. The deployment cost and penaltyfor SLO violation are not shown, as the deployment cost isequal in both cases and the SLO violation penalty is zero forall time-instances. From Fig. 5(b), we can see that the heuristicfinds solutions that are within 1.1 times of the optimal solution.During peak traffic periods, the ratio of energy cost goesbelow 1, but the ratio of transit cost increases. The optimalsolution adapts to high traffic volumes by deploying moreVNFs (increasing energy cost) and placing them at locationsthat decrease the transit cost. As a result, the ratio of energycost decreases and the ratio of transit cost increases. However,the total cost ratio stays almost the same (varying between

TABLE IIIAVERAGE EXECUTION TIME

Topology CPLEX HeuristicInternet2 (12 nodes, 15 links) 34.99s 0.535s

Data Center (23 nodes, 43 links) 1595.12s 0.442sAS-3967 (79 nodes, 147 links) ∞ 2.54s

1 and 1.1). Similar results are obtained for the data centernetwork (Fig. 5(c)), where the cost ratio is also very close to1 and varies between 1.1 and 1.3.

The average execution times of the heuristic and CPLEXare shown in Table III. They were run on a machine with10 × 16-Core 2.40GHz Intel Xeon E7-8870 CPUs and 1TBmemory. As we can see, our heuristic provides solutions thatare very close to the optimal one and its execution time isseveral order of magnitude faster than CPLEX.

Fig. 7 shows results related to server resource utilizationfor Internet2 and data center networks. Fig. 7(a) and Fig. 7(b)

11

120

140

160

180

0 200 400 600 800 1000 1200 1

1.25

1.5

Tran

sit C

ost (

$)

Ene

rgy

Cos

t ($)

Time (min)

Transit Cost Energy Cost

.Fig. 9. OPEX Components for AS-3967

.

show the mean utilization and the total number of activeservers, respectively, for the Internet2 topology. Fig. 7(c)shows the average utilization per server over all time-instances.The mean utilization of the heuristic is less than that ofCPLEX, as CPLEX uses more servers than the heuristic(Fig. 7(b)). CPLEX achieves lower OPEX by deploying moreVNFs during higher traffic periods to route traffic throughshorter paths. However, the solutions provided by the heuristicare within 1.1 times the optimal results (Fig. 5(b)). In caseof the data center network, CPLEX uses less servers thanthe heuristic (Fig. 7(e)) and the utilization is also higher(Fig. 7(d)). The solution provided by the heuristic has higherresource fragmentation than the CPLEX one (Fig. 7(f)). Thedata center topology offers higher number of locations todeploy VNFs compared to Internet2. Hence, the heuristic fallsa little short of the optimal placement as it explores a smallersolution space. CPLEX finds the optimal value, but at the costof much higher execution time (Table III).

The topological properties for middlebox deployment forInternet2 and data center networks are shown in Fig. 8. TheCDF of hop distance from the ingress switch to a VNF isshown in Fig. 8(a). The hop distances for the heuristic is veryclose to that of the optimal solution. In case of the data centernetwork, there is a relatively larger gap. This occurs due tothe higher path diversity offered by a data center network.Each pair of nodes has more than one equal cost path. CPLEXfinds the optimal solution by exploring all of them. However,the heuristic always picks the first shortest path. It does notexplore the alternate paths to keep the execution time withinpractical limits (Table III). Similar results are observed for theegress case (Fig. 8(b)). From Fig. 8(a) and Fig. 8(b) we canalso see that the CDFs are quite similar, which means thatboth CPLEX and heuristic place VNFs uniformly on the pathbetween the ingress and egress switches. The path stretch isshown in Fig. 8(c). As before, the heuristic’s performance isclose to that of the optimal solution. In case of the data centernetwork, the heuristic has a larger stretch, which is a result ofthe path diversity issue discussed earlier.

The results for the AS-3967 topology are shown in Fig. 9and Fig. 10. The traffic for this topology is show in Fig. 4(c).As mentioned earlier, this traffic was generated using theFNSS tool [24] and it exhibits time-of-day effect. We cannotprovide a comparison with the optimal solution as the CPLEXprogram was not able to solve the problem for this topology. Itfailed to fit the optimization model in its memory even thoughthe physical machine had 1TB of memory. The programcrashes after the total memory usage reaches around 300 GB.We observed similar behavior when experimenting with hightraffic volumes. CPLEX was not able to solve the problem for

the Internet2 topology when traffic was increased to utilizethe network by more than 40%. We tuned different parameters(e.g., solving the dual problem, storing branch and bound treedata on disk, reducing the number of threads, etc.) of theCPLEX solver according to the guidelines provided by IBM2,but could not solve the problem. We plan to investigate thisissue further in the future. However, the heuristic solution wasable to solve the same problem in less than 3 seconds.

The transit and energy cost for the AS-3967 topology isreported in Fig. 9. The transit cost is two order-of-magnitudehigher than the energy cost, which is expected for a largernetwork with large amount of traffic. From Fig. 4(c) and Fig. 9,we can see that our dynamic VNF orchestration approachadapts nicely with the changing traffic conditions. It candynamically scale-up or scale-down the number of activeVNFs (demonstrated by the rise and fall of the energy cost).It can also adapt the location of the VNFs according to thevariation in the traffic volume.

The results for system resource utilization and topologicalproperties for middlebox locations are shown in Fig. 10.From Fig. 10(a) we can see that the mean utilization and num-ber of active servers vary with fluctuation in traffic volume.The mean utilization of the servers is around 80%, but there isa small number of servers that are underutilized (Fig. 10(b)).The CDF of percentage of middleboxes deployed within k-hopdistance from the ingress switch is reported in Fig. 10(c). Morethan 90% middleboxes are deployed within 5 hops, which isquite reasonable for a network with 79 switches and 147 links.Similar results are obtained for the egress case as shown in thesame figure. Finally, the path stretch is shown in Fig. 10(d). Wecan observe that 20% traffic passes through the shortest patheven after going though the VNF sequence. So, in 20% of thecases VNFs are provisioned on the shortest path between theingress and egress switches that the traffic is passing through.

E. Performance Comparison with Previous Work

0 10 20 30 40 50 60 70 80

0 50 100 150 200 250 300

Dep

loye

d VN

Fs

Requested VNFs

NFO-DP NFO-BS

(a) VNFs Deployed

0.1

1

10

100

1000

10000

10 20 30 40 50 60 70 80 90 100Exec

utio

n Ti

me

(s) [

Log

Scal

e]

Number of Chaining Requests

NFO-DP NFO-BS

(b) Execution Time

Fig. 11. Performance comparison with [14]2http://www-01.ibm.com/support/docview.wss?uid=swg21399933

12

0.7

0.75

0.8

0.85

0 200 400 600 800 1000 1200 11

12

13

14

15

16

Mea

n U

tiliz

atio

n

Num

ber o

f Act

ive

Ser

vers

Time (min)

Utilization Active Servers

(a) Mean Server Utilization

0

0.2

0.4 0.6

0.8

1

Serve

r-9

Serve

r-12

Serve

r-13

Serve

r-15

Serve

r-18

Serve

r-21

Serve

r-31

Serve

r-36

Serve

r-48

Serve

r-51

Serve

r-66

Serve

r-69

Serve

r-72

Serve

r-75

Serve

r-76

Serve

r-77

Mea

n U

tiliz

atio

n

Server ID

Heuristic Solution

(b) Per Server Utilization

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10 12 14

CD

F

Hop Distance

Heuristic-AS3967-IngressHeuristic-AS3967-Egress

(c) Distance to middlebox

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

CD

F

Stretch

Heuristic-AS3967

(d) Path stretch.

Fig. 10. Results for Rocketfuel Topology (AS-3967)

We demonstrate the effectiveness of our proposed heuristic(NFO-DP) over prior work by comparing with a very recentand relevant proposal. We implemented the binary searchbased heuristic proposed in [14] (NFO-BS). We adjustedthe heuristic parameters according to the provided guidelinein the paper. We experimented with a moderate sized ISPnetwork topology with 79 nodes and 147 links (AS3967 fromRocketFuel topologies [23]). We varied the number of VNFchaining requests from 10 to 100 and measured the executiontime along with the number of deployed VNFs. The resultsare reported in Fig. 11. NFO-BS could not find a feasiblesolution for more than 60 traffic requests within a time limitof 24 hours. Moreover, the solution quality is not consistent, asshown by the irregular line in Fig. 11. Our findings show thaton similar problem instances NFO-DP outperforms NFO-BSin both solution quality and execution time.

F. Scalability of Heuristic

1

4

16

64

256

10 20 30 40 50 60 70 80 90 100Exec

utio

n Ti

me

(s) [

Log

Scal

e]

Number of Chaining Requests

AS1239 28-port Fat Tree

Fig. 12. Scalability of Heuristic

In this scenario, we test the scalability of our proposedheuristic by running it on larger network topologies and reportthe execution time. For larger network topologies, we useda 28-port fat tree [28] with around 1000 nodes and 10Klinks as a data center network and an ISP network topologywith 315 nodes and 972 links (AS1239 from RocketFuelISP topologies [23]). For each of these topologies we variedthe number of VNF chaining requests from 10 to 100 andreported the execution time. Fig. 12 shows the results of thisexperiment. As we can see, even for a very large data centernetwork, our proposed heuristic could embed 100 requestsin under 6 minutes. It is worth mentioning that the heuristicproposed in [14] could embed only less than 30 VNF chainingrequest withing 6 minutes on a much smaller network.

G. Effect of High Traffic Volume

Now, we show the impact of higher traffic volume on our so-lution. We perform this experiment by increasing the original

0.75

1

1.25

10% 20% 30% 40%

Cos

t Rat

io (H

euris

tic /

CP

LEX

)

Network Utilization (%)

Total CostEnergy CostTransit Cost

(a) Cost Ratio

0

0.1

0.2

0.3

0.4

0.5

10 20 30 40M

ean

Ser

ver U

tiliz

atio

n (%

)

Network Utilization (%)

HeuristicCPLEX

(b) Server Utilization

Fig. 13. Cost Ratio (Heuristic / CPLEX) with Varying Load

traffic by 10% to 40% (in increments of 10%) for the Internet2topology (Fig. 13). We observed a linear relationship betweenOPEX and network utilization for both of our solutions. Thecost also grows almost at the same rate for both CPLEX andheuristic as evident from Fig. 13(a). The heuristic is able tofollow the optimal solution very closely. Although it mightseem a bit unintuitive by looking at the ratio of the individualcost components, it occurs as the transit cost is two order-of-magnitude larger than the energy cost.

The server utilization increases sub-linearly with increasingnetwork load (Fig. 13(b)). The number of used servers remainsthe same for different network loads, but more cores wereused since more VNFs were deployed. The larger error barfor CPLEX indicates the deployment of more VNFs, whichincreases the energy cost. However, more VNFs eventuallydecreased the transit cost, which is the major contributor toOPEX in this case.

VI. RELATED WORK

The initial drive for NFV was from several telecommunica-tion operators back in 2013 [9]. The motivation behind NFVis to break the barrier of proprietary hardwares and have moreflexibility in the network in terms of the placement of servicepoints, introducing new services and vendor independence. Tothis date, research efforts have been made in different aspectsof NFV. In this section, we first discuss about state-of-the-artNFV management and orchestration proposals (Section VI-A),

13

then we describe some placement algorithms for VNFs andVNF chains (Section VI-B), followed by some enabling tech-nologies for NFV (Section VI-C).

A. Management and Orchestration of Network Functions

Some of the early works on managing Network Functions(NFs), propose to outsource them to a cloud service [2], [29].Such outsourcing is motivated in the literature by studyingexperiences of different network operators. [2], [29] showhow the management complexities arising in today’s enterprisenetworks can be mitigated by outsourcing.

A more formal management approach towards NFV istaken by projects like Stratos [30], and OpenNF [31]. Stratosproposes an architecture for orchestrating VNFs outsourced toa remote cloud by taking care of traffic engineering, horizontalscaling of VNFs etc. On the other hand, OpenNF proposes aconverged control plane for VNFs and network forwardingplane by extending the centralized SDN paradigm.

Some recent works on managing NFs focus on trafficengineering issues such as steering the traffic through somepredefined sequence of NFs. This problem becomes morechallenging when NFs along the sequence modify the packetheaders, thus changing the traffic signature. [32] and [4]propose SDN based solutions to the traffic steering problem.They propose tagging based mechanisms to identify a trafficduring its lifetime and also to keep track of the visitedsequence of middleboxes. The global network view of SDNmakes it easier to manage and assign tags to different trafficsand to ensure different policy enforcement on NFs.

B. VNF and VNF Chain Placement

Authors in [33] proposed a grammar for specifying VNFchains and then provided a mathematical formulation for VNFchain placement. Their formulation is quadratic and does notallow VNF sharing between multiple tenants. In contrast,we provide a linear formulation and allow for VNF sharing.In [34], authors provided a LP-relaxation based approachfor finding inter-data center VNF chain placement. However,due to LP-relaxation their solution has the flaw of violatingphysical resource capacities by a factor of at most 16. Oursolutions do not have such flaws. A genetic algorithm for VNFchain placement is proposed in [35], but it does not addressthe issue of dynamically adjusting the placement of VNFs tobalance between network operating cost and performance.

C. Enabling Technologies for NFV

NFV proposes to run VNFs on commodity hardware asvirtual appliances. This flexibility raises the question of perfor-mance. In recent years, a number of research efforts have beentargeted to achieve near line speed network I/O throughputwith commodity servers [36], [37]. Apart from acceleratingthe packets along the network I/O stack, more recent workshave proposed changes to virtualization technologies to sup-port the deployment of modular software NFs on lightweightVMs [11]. Hundreds of these VMs can be instantiated on asingle physical machine within miliseconds to run different

VNFs. Substantial research efforts are also being put towardsprogramming models and deployment architecture for VNFsas well. CoMb [38] and xOMB [39] propose an extensible andconsolidated framework for incrementally developing scalablemiddleboxes. Both of these works leverage the idea of reusablenetwork processing pipelines for middlebox composition.

VII. CONCLUSION

Virtualized network functions provide a flexible way todeploy, operate and orchestrate network services with muchless capital and operational expenses. Software middleboxes(e.g., ClickOS) are rapidly catching up with hardware mid-dlebox performance. Network operators are already opting forNFV based solutions. We believe that our model for dynamicVNF orchestration will have significant impact on middleboxmanagement in the near future. Our model can be used todetermine the optimal number of VNFs and to place themat the optimal locations to optimize network operational costand resource utilization. Our trace driven simulations on theInternet2 research network demonstrate that network OPEXcan be reduced by a factor of 4 over hardware middleboxesthrough proper VNF orchestration.

In this paper, we presented two solutions to the VNForchestration problem: CPLEX based optimal solution forsmall networks and a heuristic for larger networks. We foundthat the heuristic produces solutions that are within 1.3 timesof the optimal solution, yet the execution-time is about 65 to3500 times faster than that of the CPLEX solution. We intendto extend this work in a number of ways. We plan to extend ourmodel for supporting both hardware and software middleboxesin the same network. We want to explore the possibility ofintroducing failure-resilience by deploying backup VNFs thatcan take over the traffic processing tasks from failed VNFs.We also plan to enhance the physical network transformationprocess to further reduce the solution space and speed-up therunning time of the optimal solution.

REFERENCES

[1] V. Sekar, S. Ratnasamy, M. K. Reiter, N. Egi, and G. Shi, “Themiddlebox manifesto: Enabling innovation in middlebox deployment,”in ACM HotNets-X, 2011, pp. 1–6.

[2] J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, andV. Sekar, “Making middleboxes someone else’s problem: network pro-cessing as a cloud service,” ACM CCR, vol. 42, no. 4, pp. 13–24, 2012.

[3] D. A. Joseph, A. Tavakoli, and I. Stoica, “A policy-aware switchinglayer for data centers,” in ACM SIGCOMM, 2008, pp. 51–62.

[4] Z. A. Qazi, C.-C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu,“SIMPLE-fying middlebox policy enforcement using SDN,” in Proc.of ACM SIGCOMM ’13, pp. 27–38.

[5] P. Quinn and T. Nadeau, “Service Function Chaining Problem State-ment,” draft-quinn-sfc-problem-statement-10 (work in progress), 2014.

[6] W. Liu, H. Li, O. Huang, M. Boucadair, N. Leymann, Q. Fu, Q. Sun,C. Pham, C. Huang, J. Zhu, and P. He, “Service Function ChainingProblem Statement,” draft-liu-sfc-use-cases-08 (work in progress), 2014.

[7] W. Haeffner, J. Napper, M. Stiemerling, D. Lopez, and J. Uttaro,“Service Function Chaining Use Cases in Mobile Networks,” draft-ietf-sfc-use-case-mobility-01 (work in progress), 2014.

[8] S. Surendra, M. Tufail, S. Majee, C. Captari, and S. Homma, “ServiceFunction Chaining Use Cases in Mobile Networks,” draft-ietf-sfc-dc-case-01 (work in progress), 2014.

[9] G. ETSI, “Network Functions Virtualisation (NFV): ArchitecturalFramework,” ETSI GS NFV, vol. 2, p. V1, 2013.

14

[10] R. Mijumbi, J. Serrat, J. Gorricho, N. Bouten, F. De Turck, andR. Boutaba, “Network function virtualization: State-of-the-art and re-search challenges,” IEEE CST, 2015.

[11] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Honda, R. Bifulco,and F. Huici, “ClickOS and the art of network function virtualization,”in Proc. of USENIX NSDI ’14, pp. 459–473.

[12] J. Hwang, K. K. Ramakrishnan, and T. Wood, “NetVM: High Perfor-mance and Flexible Networking Using Virtualization on CommodityPlatforms,” in Proc. of USENIX NSDI ’14, pp. 445–458.

[13] M. F. Bari, S. R. Chowdhury, R. Ahmed, and R. Boutaba, “On orches-trating virtual network functions,” in CNSM 2015, 2015, pp. 50–56.

[14] M. C. Luizelli, L. R. Bays, L. S. Buriol, M. P. Barcellos, and L. P. Gas-pary, “Piecing together the nfv provisioning puzzle: Efficient placementand chaining of virtual network functions,” in IFIP/IEEE IM, 2015.

[15] “Comparison of enterprise class power enclosure,”http://www.dell.com/downloads/global/products/pedge/en/bladepower-studywhitepaper 08112010 final.pdf.

[16] “Internet Transit Pricing,” http://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php.

[17] N. M. K. Chowdhury and R. Boutaba, “A survey of network virtualiza-tion,” Computer Networks, vol. 54, no. 5, pp. 862 – 876, 2010.

[18] L. T. Kou and G. Markowsky, “Multidimensional bin packing algo-rithms,” IBM Journal of Research and development, vol. 21, no. 5, pp.443–448, 1977.

[19] R. Sridharan, “A lagrangian heuristic for the capacitated plant locationproblem with single source constraints,” European journal of operationalresearch, vol. 66, no. 3, pp. 305–312, 1993.

[20] G. D. Forney Jr, “The Viterbi Algorithm,” Proc. of the IEEE, vol. 61,no. 3, pp. 268–278, 1973.

[21] “Internet2 Research Network Topology and Traffic Matrix,”http://www.cs.utexas.edu/˜yzhang/research/AbileneTM/.

[22] T. Benson, A. Akella, and D. A. Maltz, “Network traffic characteristicsof data centers in the wild,” in Proc. of ACM IMC ’10, pp. 267–280.

[23] N. Spring, R. Mahajan, and D. Wetherall, “Measuring ISP topologieswith Rocketfuel,” vol. 32, no. 4, pp. 133–145, 2002.

[24] L. Saino, C. Cocora, and G. Pavlou, “A Toolchain for SimplifyingNetwork Simulation Setup,” in Proc. of SIMUTOOLS ’13.

[25] A. Nucci, A. Sridharan, and N. Taft, “The problem of syntheticallygenerating ip traffic matrices: initial recommendations,” ACM CCR,vol. 35, no. 3, pp. 19–32, 2005.

[26] “https://datatracker.ietf.org/doc/draft-ietf-sfc-dc-use-cases/.”[27] “pfSense Hardware Sizing Guide,”

https://www.pfsense.org/hardware/#sizing.[28] M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commodity data

center network architecture,” vol. 38, no. 4, pp. 63–74, 2008.[29] G. Gibb, H. Zeng, and N. McKeown, “Outsourcing network functional-

ity,” in Proc. of ACM HotSDN ’12, pp. 73–78.[30] A. Gember, A. Krishnamurthy, S. S. John, R. Grandl, X. Gao, A. Anand,

T. Benson, V. Sekar, and A. Akella, “Stratos: A Network-Aware Or-chestration Layer for Virtual Middleboxes in Clouds,” arXiv preprintarXiv:1305.0209, 2013.

[31] A. Gember-Jacobson, R. Viswanathan, C. Prakash, R. Grandl, J. Khalid,S. Das, and A. Akella, “OpenNF: enabling innovation in networkfunction control,” in Proc. of ACM SIGCOMM ’14, pp. 163–174.

[32] S. K. Fayazbakhsh, L. Chiang, V. Sekar, M. Yu, and J. C. Mogul,“Enforcing network-wide policies in the presence of dynamic middleboxactions using flowtags,” in Proc. of USENIX NSDI ’14.

[33] S. Mehraghdam, M. Keller, and H. Karl, “Specifying and Placing Chainsof Virtual Network Functions,” arXiv preprint arXiv:1406.1058, 2014.

[34] R. Cohen, L. Lewin-Eytan, J. Naor, and D. Raz, “Near optimal place-ment of virtual network functions,” in IEEE INFOCOM 2015, 2015, pp.1346–1354.

[35] W. Rankothge, J. Ma, F. Le, A. Russo, and J. Lobo, “Towards mak-ing network function virtualization a cloud computing service,” inIFIP/IEEE IM, May 2015, pp. 89–97.

[36] “Intel DPDK,” http://dpdk.org/.[37] L. Rizzo, “netmap: A Novel Framework for Fast Packet I/O.” in Proc.

of USENIX ATC ’12, pp. 101–112.[38] V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi, “Design and

Implementation of a Consolidated Middlebox Architecture.” in Proc. ofUSENIX NSDI ’12, pp. 323–336.

[39] J. W. Anderson, R. Braud, R. Kapoor, G. Porter, and A. Vahdat, “xOMB:Extensible open middleboxes with commodity servers,” in Proc. ofACM/IEEE ANCS ’12, pp. 49–60.

PLACEPHOTOHERE

Md. Faizul Bari is a Ph.D. candidate at the Schoolof Computer Science, University of Waterloo. Hereceived M.Sc. and B.Sc. degrees in computer sci-ence and engineering from BUET. His researchinterest includes future Internet architecture, networkvirtualization, and cloud computing. He has receivedthe Ontario Graduate Scholarship, President’s Grad-uate Scholarship, and David R. Cheriton GraduateScholarship at the University of Waterloo.

PLACEPHOTOHERE

Shihabur Rahman Chowdhury is a PhD student atthe School of Computer Science, University of Wa-terloo. He received his B.Sc. degree in computer sci-ence and engineering from Bangladesh University ofEngineering and Technology (BUET). His researchinterests include virtualization and softwarization ofcomputer networks. He is a recipient of the OntarioGraduate Scholarship, Presidents Graduate Scholar-ship, and GoBell Scholarship at the University ofWaterloo.

PLACEPHOTOHERE

Reaz Ahmed received his Ph.D. degree in computerscience from the University of Waterloo, in 2007.He received M.Sc. and BSc. degrees in computerscience from BUET in 2002 and 2000, respectively.He received the IEEE Fred W. Ellersick award in2008. His research interests include future Internetarchitectures, Information-Centric Networks, Net-work Virtualization and content sharing peer-to-peernetworks with focus on search flexibility, efficiencyand robustness.

PLACEPHOTOHERE

Raouf Boutaba received the M.Sc. and Ph.D. de-grees in computer science from the University Pierre& Marie Curie, Paris, in 1990 and 1994, respectively.He is currently a professor of computer science atthe University of Waterloo (Canada). His researchinterests include resource and service managementin networks and distributed systems. He is the found-ing editor in chief of the IEEE Transactions onNetwork and Service Management (2007-2010) andon the editorial boards of other journals. He receivedseveral best paper awards and recognitions including

the Premiers Research Excellence Award, the IEEE ComSoc Hal Sobol, FredW. Ellersick, Joe LociCero, Dan Stokesbury, Salah Aidarous Awards, and theIEEE Canada McNaughton Gold Medal. He is a fellow of the IEEE, theEngineering Institute of Canada, and the Canadian Academy of Engineering.

PLACEPHOTOHERE

Otto Carlos Muniz Bandeira Duarte Otto CarlosMuniz Bandeira Duarte received the Electronic En-gineer degree and the M. Sc. degree in ElectricalEngineering from Universidade Federal do Rio deJaneiro, Brazil, in 1976 and 1981, respectively, andthe Dr. Ing. degree from ENST/Paris, France, in1985. Since 1978 he is Professor at UniversidadeFederal do Rio de Janeiro. In 1992/1993 he hasworked at Paris 6 University, in 1995, at Interna-tional Computer Science Institute (ICSI) associatedto the University of California at Berkeley, and in

2014 at University of California at Berkeley. He was invited professor atParis 6 University for several times.