View
220
Download
0
Category
Preview:
Citation preview
Convex Functions
Let f : Rn → R be a convex function if ∀x , y ∈ dom f and0 ≤ θ ≤ 1 such that
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y).
It is also known as the Jensen’s inequality.
if f is concave then −f is convex.Lam Tran Convex Optimization
Strictly Convex Functions
Let f : Rn → R be a convex function if ∀x , y ∈ dom f and0 < θ < 1 such that
f (θx + (1− θ)y) < θf (x) + (1− θ)f (y).
Lam Tran Convex Optimization
Extended Convex Functions
Let f̂ : Rn → R ∪ {∞} be an extended convex function if∀x , y ∈ dom f and 0 ≤ θ ≤ 1 such that
f̂ (x) =
{f (x) x ∈ dom f∞ otherwise
andf̂ (θx + (1− θ)y) ≤ θf̂ (x) + (1− θ)f̂ (y).
Lam Tran Convex Optimization
First-order conditions
f is differentiable then f is convex iff dom f is convex and
f (y) > f (x) +∇f (x)T (y − x)
hold ∀x , y ∈ dom f and
∇f (x) =(∂f (x)∂x1
, . . . ,∂f (x)∂xn
).
Lam Tran Convex Optimization
Second-order conditions
f is twice differentiable then f is convex if
∇2f (x) ≥ 0
hold ∀x ∈ dom f and each element is defined as
∇2f (x)ij =∂f (x)∂xixj
.
Lam Tran Convex Optimization
Sublevel Sets
The a-sublevel set of a function f : Rn → R is defined as
Ca = {x ∈ dom f | f (x) ≤ a}.
f is a convex function if ∀x , y ∈ Ca and 0 ≤ θ ≤ 1 such thatf (x) ≤ a and f (y) ≤ a then
f (θx + (1− θ)y) < θf (x) + (1− θ)f (y).
hold ∀x , y .
Lam Tran Convex Optimization
Epigraph
The Ephigraph is defined as
epi f = {(x , t)|x ∈ dom f , f (x) ≤ t}
From a-sublevel sets, epigraph is convex.
Lam Tran Convex Optimization
Preserve Convexity Operations: Nonnegativeweighted sums
For i = 1, . . . ,n such that wi ≥ 0 and fi is a convex function.Then the function
f (x) =n∑
i=1
wi fi(x).
is convex.Proof: Let x1, x2 ∈ dom g, 0 ≤ θ ≤ 1, and θ̂ = 1− θ. We have
g(θx1 + θ̂x2) =n∑
i=1
wi fi(θx1 + θ̂x2) ≤n∑
i=1
wi
[θfi(θx1) + θ̂fi(x2)
]=
n∑i=1
wiθfi(θx1) +n∑
i=1
wi θ̂fi(x2)
= θg(x1) + θ̂g(x2).
Lam Tran Convex Optimization
Preserve Convexity Operations: Composition with anaffine mapping
Let x ∈ dom g the affine mapping function is defined as
g(x) = f (Ax + b).
If f is convex function then g is a convex function..Proof: Let x1, x2 ∈ dom g and Ax + b ∈ dom f , 0 ≤ θ ≤ 1, andθ̂ = 1− θ. We have
g(θx1 + θ̂x2) = f (A(θx1 + θ̂x2) + b)
= f (Aθx1 + Aθ̂x2 + b)
= f (θ(Ax1 + b1) + θ̂(Ax2 + b2))
≤ θf (Ax1 + b1) + θ̂f (Ax2 + b2)
= g(x1) + g(x2),
where b = θb1 + θ̂b2.Lam Tran Convex Optimization
Preserve Convexity Operations: Pointwise maximumand supremum
The perspective function: Let x ∈ dom f and f1, f2 be convexfunctions such as
f (x) = max{f1(x), f2(x)}
then f is a convex function.Proof: Let x1, x2 ∈ dom f , 0 ≤ θ ≤ 1, and θ̂ = 1− θ. We have
f (θx1 + θ̂x2) = max(f1(θx1 + θ̂x2), f2(θx1 + θ̂x2))
= max(θf1(x1) + θ̂f (x2), θf2(x1) + θ̂f2(x2))
= max(θf1(x1), θf2(x1)) + max(θ̂f1(x2) + θ̂f2(x2))
= θf (x1) + θ̂f (x2).
Lam Tran Convex Optimization
Preserve Convexity Operations: Minimization
If f is convex in (x , y), and C is a convex nonempty setthen the function
h(x) = infy∈C
f (x , y)
is a convex function. The domain of g is
dom g = {x |(x , y) ∈ dom f for some y ∈ C}.
Lam Tran Convex Optimization
Preserve Convexity Operations: Minimization
Proof: Let x1, x2 ∈ dom h, y1, y2 ∈ C, and 0 ≤ θ ≤ 1, andθ̂ = 1− θ. We have
h(θx1 + θ̂x2) = infy∈C
f (θx1 + θ̂x2, y)
≤ f (θx1 + θ̂x2, θy1 + θ̂y2)
≤ θf (x1, y1) + θ̂f (x2, y2)
≤ θg(x1) + θ̂g(x2) + ε
where ε > 0.
Lam Tran Convex Optimization
Preserve Convexity Operations: Composition
Let x ∈ dom h be a composition function
h(x) = g(f (x)).
1 g is convex & non-decreasing and f is convex, then h isconvex.
2 g is convex & non-increasing and f is concave, then h isconvex.
3 g is concave & non-increasing and f is convex, then h isconcave.
4 g is concave & non-decreasing and f is concave, then h isconcave.
Proof: Let x1, x2 ∈ dom h, 0 ≤ θ ≤ 1, and θ̂ = 1− θ
h(θx1 + θ̂x2) = g(h(θx1 + θ̂x2)) = g(θh(x1) + θ̂h(x2))
= θg(h(x1)) + θ̂g(h(x2)).
Lam Tran Convex Optimization
Preserve Convexity Operations: Perspective
The perspective function: Let x , t ∈ dom g
g(x , t) = t × f (x/t).
The perspective function g is convex if f is convex.Proof: Let x1, x2, t1, t2 ∈ dom g, 0 ≤ θ ≤ 1, and θ̂ = 1− θ. Wehave
g(θx1 + θ̂x2, θt1 + θ̂t2) =
= (θt1 + θ̂t2)× f
(θx1 + θ̂x2
θt1 + θ̂t2
)
= (θt1 + θ̂t2)× f
(θ t1
t1x1 + θ̂ t2t2x2
(θt1 + θ̂t2)
)
≤ (θt1 + θ̂t2)[
θ1t1θt1 + θ̂t2
f (θx1/t1) +θ2t2
θt1 + θ̂t2f(θ̂x2/t2
)]= t1g(x1, t) + t2g(x2, t2).
Lam Tran Convex Optimization
Practical Methods to Establish Convexity of a Function
Verify definition (often simplified by restricting to a line).Twice differentiable functions, show
∇2f (x) ≥ 0.
Show that f is obtained from simple convex functions byoperations that preserve convexity.
Lam Tran Convex Optimization
The Conjugate Function
Let f : Rn → R. The conjugate function of f is f ∗ : Rn → R is
f ∗(x) = supx∈ dom f
(y tx − f (x))
f ∗ is convex (even if f is not convex)
Lam Tran Convex Optimization
Negative logarithm
f (x) = − log(x)
The deriviate of f (x) w.r.t x is
∂
∂xyx − f (x) = y +
1x= 0 =⇒ x∗ = − 1
y .
We have
yx∗ − f (x∗) = −yy+ log
(−1
y
)= −1− log(−y).
f ∗(y) = supx>0
(xy + log(x)) =
{−1− log(−y) y < 0∞ otherwise
Lam Tran Convex Optimization
Strictly convex quadratic
f (x) = (1/2)xT Qx
with Q ∈ Sn++. The deriviate of f (x) w.r.t x is
∂
∂xyT x − f (x) = yT − xT Q = 0 =⇒ x∗ = Q−1y .
Plug x∗ into x , we get
yT x∗ + f (x∗) = yT Q−1y − 12
yT Q−1y =12
yT Q−1y .
Thus, we have
f ∗(y) = supx
(y tx − (1/2)x tQ) = 1/2yT Q−1y
Lam Tran Convex Optimization
Quasiconvex Functions
A function f : Rn → R is quasiconvex if dom f is convex and thesublevel sets
Sa = {x ∈ dom f | f (x) ≤ a}are convex for all a.
-f is quasiconcave.f is quasilinear then every level set {x |f (x) = a} is convex.Any monotonically increasing (decreasing) f is quasilinear.
sublevel of α is Sα = [a,b] and sublevel of β Sβ = [−∞, c].Lam Tran Convex Optimization
Quasiconvex Proof
If f is convex then f is quasiconvex.Proof: Let f be a convex function such as
f (θx + (1− θ)y) ≤ θf (x) + (1− x)f (y)
and the set Sa = {x ∈ dom f |f (x) ≤ a} are convex for all a.Suppose that 0 ≤ θ ≤ 1 and x , y ∈ Sa implies that f (x) ≤ a andf (y) ≤ a. We have
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y)≤ θa + (1− θ)a = a.
Thus, θa + (1− θ)a ∈ Sa and f is quasiconvex.
Lam Tran Convex Optimization
Quasiconcave Proof
If f is convex then f is quasiconcave.Proof: Let f be a concave function such as
f (θx + (1− θ)y) ≥ θf (x) + (1− x)f (y)
and the set Sa = {x ∈ dom f |f (x) ≥ a} are convex for all a.Suppose that 0 ≤ θ ≤ 1 and x , y ∈ Sa implies that f (x) ≥ a andf (y) ≥ a. We have
f (θx + (1− θ)y) ≥ θf (x) + (1− θ)f (y)≥ θa + (1− θ)a = a.
Thus, θa + (1− θ)a ∈ Sa and f is quasiconcave.
Lam Tran Convex Optimization
Quasiconvex and Quasiconcave Function Example
Let f : R → R and dom f ∈ R++ defined as
f (x) = log(x)
The 2nd deriviate is
f ′′(x) = − 1x2 < 0
The function f is concave implies that it is quasiconcave (proofnext page).
Lam Tran Convex Optimization
Quasiconvex and Quasiconcave Function Example
The function f is quasiconcave if the set
sa = {x ∈ R++| log(x) ≥ a}
is convex for all a. Let x , y ∈ Sa and 0 ≤ θ ≤ 1, implies thatlog(x) ≥ a and log(y) ≥ a. Then we have we have
log(θx + (1− θ)y) ≥ θ log(x) + (1− θ) log(y)≥ θa + (1− θ)a = a.
It implies thatlog(θx + (1− θ)y) ≥ a.
Thus, we have θx + (1− θ)y ∈ Sa and log(x) and the functionis quasiconcave.
Lam Tran Convex Optimization
Quasiconvex and Quasiconcave Function Example
The function f is quasiconvex if the set
sa = {x ∈ R++| log(x) ≤ a}
is convex for all a. Let x , y ∈ Sa and 0 ≤ θ ≤ 1, then we havelog(x) ≤ a =⇒ x ≤ exp(a) so as y ≤ exp(a).
θx + (1− θ)y ≤ θ exp(a) + (1− θ)exp(a) = exp(a).
Take the log of both side and we get
log(θx + (1− θ)y) ≤ a.
Thus, we have θx + (1− θ)y ∈ Sa and log(x) is quasiconvex.The above showed that the function log(x) is quasilinear.
Lam Tran Convex Optimization
Quasiconvex and Quasiconcave Function Example
Let f : R2 → R and dom f ∈ R+ defined as
f (x1, x2) = x1x2
The Hessian Matrix is
∇f ′′(x1, x2) =
[0 11 0
]
and the eigenvector is[−11
]. The function f is not convex nor
concave. But f is quasiconcave and quasiconvex.
Lam Tran Convex Optimization
Quasiconvex and Quasiconcave Function Example
Proof: Sa = {x ∈ R2+|x1x2 ≥ a}, x , y ∈ Sa, 0 ≤ θ ≤ 1 θ̂ = 1− θ.
(θx1 + θ̂x2)(θy1 + θ̂y2) = θ2x1y1 + θ̂2x2y2 + θθ̂(x1y2 + x2y1)
= θ2x1y1 + θ̂2x2y2 + θθ̂
(x1
x2x2y2 +
x2
x1x1y1
)≥ θ2a + θ̂2a + θθ̂
(x1
x2a +
x2
x1a)
= a[θ2 + θ̂2 + θθ̂
(x1
x2+
x2
x1+ 2− 2
√x1
x2
√x2
x1
)]= a
[θ2 + θ̂2 + θθ̂
{(√x1
x2−√
x2
x1
)2
+ 2
}]≥ a
[θ2 + θ̂2 + 2θθ̂
]≥ a(θ2 + θ2 − 2θ + 1− 2θ2 + 2θ) = a. = a.
Thus, we have θx + (1− θ)y ∈ Sa and Sa is a convex for all a.The function xy is quasiconcave.
Lam Tran Convex Optimization
Quasiconvex and Quasiconcave Function Example
Proof: Let Sa = {x ∈ R2+|x1x2 ≤ a}, x , y ∈ Sa, 0 ≤ θ ≤ 1, and
θ̂ = 1− θ.
(θx1 + θ̂x2)(θy1 + θ̂y2) = θ2x1y1 + θ̂2x2y2 + θθ̂(x1y2 + x2y1)
≤ θ2x1y1 + θ̂2x2y2
≤ θx1y1 + θ̂x2y2
≤ θa + (1− θ)a = a.
Thus, we have θx + (1− θ)y ∈ Sa and Sa is a convex for all a.The function xy is quasiconvex. Since f is both quasiconvexand quasiconcave, f is quasilinear.
Lam Tran Convex Optimization
Quasiconvex Functions
A function f is quasiconvex iff the dom f is convex and for anyx , y ∈ dom f and 0 ≤ θ ≤ 1 such that
f (θx + (1− θ)y) ≤ max{f (x), f (y)}.
It is known as the Jensens inequality for quasiconvex functions.
The value of the function on a segment does not exceed themaximum of its values at the endpoints.
Lam Tran Convex Optimization
Quasiconvex Functions
A function f is quasiconvex, that is Sa = {x |f (x) ≤ a} is convexfor all a ∈ R =⇒ f (θx + (1− θ)y) ≤ max{f (x), f (y)}.Proof: WLOG let max(f (x), f (y)) = f (x) = a. Let x , y ∈ Sa suchthat
f (x) ≤ a and f (y) ≤ a
and let 0 ≤ θ ≤ 1. Since Sa is convex, we haveθx + (1− θ)y ∈ Sa, thus we have
f (θx + (1− θ)y) ≤ a = max(f (x), f (y)).
Thus, f is quasiconvex.
Lam Tran Convex Optimization
Quasiconvex Functions
f(θx + (1− θ)y) ≤ max(f (x), f (y)) =⇒ Sa = {x |f (x) ≤ a} isconvex for al a ∈ R.
Proof: WLOG Let max(f (x), f (y)) = f (x) = a. Let x , y ∈ Sa and0 ≤ θ ≤ 1. We have f (x) ≤ a and f (y) ≤ a. Since
f (θx + (1− θ)y) ≤ max(f (x), f (y)) = a,
we have f (θx + (1− θ)y) ≤ a. This implies thatθx + (1− θ)y ∈ Sa and Sa is convex.
Quasiconcave can be proved in a similar fashion.
Lam Tran Convex Optimization
Quasiconvex Functions
The function f : R→ R is quasiconvex, if and only if at least oneof the following conditions holds
f is nondecreasingf is nonincreasingThere exist a point c ∈ dom f such that for t ≤ c (andt ∈ dom f ), f is nonincreasing, and for t ≥ c (and t ∈ domf ), f is nondecreasing.
Lam Tran Convex Optimization
Differentiable Quasiconvex Functions
First-order conditions:Suppose f : Rn → R is differentiable. Then f is quasiconvex iffdom f is convex and for all x , y ∈ dom f
f (y) ≤ f (x)⇒ ∇f (x)t(y − x) ≤ 0. (1)
Lam Tran Convex Optimization
Differentiable Quasiconvex Functions
Proof: We assume that f (y) ≤ f (x)⇒ f ′(x)t(y − x) ≤ 0 is true.Let the function f : Rn → R, x , y ∈ dom f such that y > x andf (y) ≤ f (x). Let z ∈ [x , y ] and f (z) ≤ f (x).
Suppose for contradiction that f (z) ≥ f (x), since f isdifferentiable, we have f ′(z) < 0. This implies that
f (x) ≤ f (z) =⇒ f ′(z)(x − z) ≤ 0.
Since (x − z) ≤ 0 and f ′(z) < 0, which contradict the (1).
Lam Tran Convex Optimization
Differentiable Quasiconvex Functions
Need to show the other direction as well.
Proof: We assume that f (y) ≤ f (x)⇒ f ′(x)t(y − x) ≤ 0 is true.Let the function f : Rn → R, x , y ∈ dom f such that y < x andf (y) ≤ f (x). Let z ∈ [y , x ] and f (z) ≤ f (x).
Suppose for contradiction that f (z) ≥ f (x), since f isdifferentiable, we have f ′(z) > 0. This implies that
f (x) ≤ f (z) =⇒ f ′(z)(x − z) ≤ 0.
Since (x − z) ≥ 0 and f ′(z) > 0, which contradict the (1). Thus,we have
f (y) ≤ f (x)⇒ ∇f (x)t(y − x) ≤ 0.
Lam Tran Convex Optimization
Quasiconvex Functions
Second-order conditions:If f is quasiconvex, then for all x ∈ dom f , and all y ∈ Rn , wehave
y t∇f (x) = 0⇒ y t∇2f (x)y ≥ 0.
Lam Tran Convex Optimization
Preserve Quasiconvex Operations
Scalar: If f (x) is quasiconvex and w > 0, theng(x) = wf (x) is also quasiconvex.
Proof: Let Sa = {x |g(x) ≤ a}, x1, x2 ∈ Sa, and 0 ≤ θ ≤ 1.
g(θx + (1− θ)y) = wf (θx + (1− θ)y) ≤ wθf (x) + w(1− θ)f (y)≤ θa + (1− θ)a = a.
Lam Tran Convex Optimization
Preserve Quasiconvex Operations
Pointwise supremum: if f1 and f2 are quasiconvex then
g(x) = supi∈{1,2}
(fi(x))
is quasiconvex.
Proof: Let Sa = {x |g(x) ≤ a}, x1, x2 ∈ Sa, and 0 ≤ θ ≤ 1.
g(θx + (1− θ)y) = supi∈{1,2}
(fi(θx + (1− θ)y))
≤ supi∈{1,2}
(θfi(x) + (1− θ)fi(y))
≤ supi∈{1,2}
(θa + (1− θ)a) = a
Lam Tran Convex Optimization
Preserve Quasiconvex Operations
Pointwise supremum: if f1 and f2 are quasiconvex then
g(x) = supi∈{1,2}
(fi(x))
is quasiconvex.
Lam Tran Convex Optimization
Preserve Quasiconvex Operations
Compositiong(x) = h(f (x))
is quasiconvex if f : Rn → R is quasiconvex and h : R → Ris nondecreasing.
Proof: Let Sa = {x |g(x) ≤ a}, x1, x2 ∈ Sa, and 0 ≤ θ ≤ 1.
g(f (θx + (1− θ)y)) = h(f (θx + (1− θ)y))≤ h(θf (x) + (1− θ)f (y))≤ θh(f (x)) + (1− θ)h(f (y)) ≤ a.
Thus, Sa is convex and g(x) is quasiconvex.
Lam Tran Convex Optimization
Preserve Quasiconvex Operations
Minimization If f (x , y) is quasiconvex jointly in x and y andC is a convex set, then the function
g(x) = infy∈C
f (x , y)
is quasiconvex.
Show Sa = {x |g(x) ≤ a} is convex. Let x1, x2 ∈ Sa, y1, y2 ∈ Cand ε > 0 such that f (x1, y1) ≤ a + ε and f (x2, y2) ≤ a + ε.Since f is quasiconvex, we have
f (θx1 + (1− θ)x2, θy1 + (1− θ)y2) ≤ a + ε
Thus, g(θx1 + (1− θ)x2) ≤ a and Sa is convex.
Lam Tran Convex Optimization
Log-concave and Log-convex Functions
A log-concave function f : Rn → R if f (x) > 0 for all x ∈ dom fand log f (x) is concave. For x , y ∈ dom f and 0 ≤ θ ≤ 1 then
f (θx + (1− θ)y) = f (x)θf (y)1−θ.
Lam Tran Convex Optimization
Recommended