*[scale=1.

25]esahubble

CPSC 542F KIREX LUSA

Convex Analysis and Optimization

Lecture Notes
S UMMER R ESEARCH I NTERNSHIP, U NIVERSITY OF W ESTERN O NTARIO

GITHUB . COM /L AURETH T E X/C LUSTERING

This research was done under the supervision of Dr. Pauline Barmby with the financial support of
the MITACS Globalink Research Internship Award within a total of 12 weeks, from June 16th to
September 5th of 2014.

First release, August 2014

head1.png
Contents

head2.png

3
 Chapter 1

Convex Sets

1.1 Convexity
1.1.1 Cone
[Cone] A set K Rn , when x K implies x K. A non convex cone can be hyper-plane.
For convex cone x + y K, x, y K.
Cone dont need to be "pointed". e.g.
Direct sums of cones C1 +C2 = {x = x1 + x2 |x1 C1 , x2 C2 }.
S1n {X|X = X n , (x) 0}
A matrix with positive eigenvalues.

Operations preserving convexity

Intersection C iI Ci

Linear map Let A : Rn Rn be a linear map. If C Rn is convex, so is A(C) = {Axx C}

Inverse image A1 (D) = {x R|Ax D}

Operations that induce convexity

Convex hull on S = {C|S C,Cisconvex}
Co{x1 , x2 , , xm } = {m
i=1 i xi | m } For a convex set x C x = i xi . [Carathodorys
theorem] If a point x Rd lies in the convex hull of a set P, there is a subset P0 of P consisting of
d + 1 or fewer points such that x lies in the convex hull of P0 . Equivalently, x lies in an r-simplex
with vertices in P.

1.2 Convex Functions

[Convex function] Let C Rn be convex, f : C R is convex on f if x, y C C. (0, 1),
f (x + (1 )y) f (x) + f ((1 )y)
[Strictly Convex function] Let C Rn be convex, f : C R is strictly convex on f if x, y C C.
(0, 1), f (x + (1 )y)h f (x) + f ((1 )y)

5
 [Strongly convex] f : C R is strongly convex with modules u 0 if f 12 u|| ||2 is convex.
Interpretation: There is a convex quadratic 12 u|| ||2 that lower bounds f. minxC f (x) min f(x)
Useful to turn this into an unconstrained problem.

f(x) = { f (x) i f x C elsewhere

A function f : Rn R R is convex if x, y Rn Rn , x, y, f(x + (1 )y) f (x) + f ((1

)y) Definition 1 is equivalent to definition 2 if f (x) = . f (x) = sup jJ f j (x)

1.2.1 Epigraph
[Epigraph] For f : Rn R, its epigraph epi( f ) Rn+1 isthesetepi( f ){(x, )| f (x) } Next: a
function is convex i.f.f. its epigraph is convex.
A function f : C R,C Rn is convex if x, y C, f (ax+(1a)x) a f (x)+(1a) f (x) a
(0, 1).
Strict convex: x 6= y f (ax + (1 a)x) a f (x) + (1 a) f (x) f is convex f is concave.
Level set: S f = {x| f (x) }.
S f is convex f is convex.
[Strongly convex] f : C R is strongly convex with modules if x, y C, (0, 1), f (ax +
1
(1 a)x) a f (x) + (1 a) f (x) 2 (1 )kx yk2 .

f is 2nd-differentiable, f ix convex 2 f (x)i0.

f is strongly convex 2 f (x)iI x

[2] f : Rn R is convexif x, y R, (0, 1), f (ax + (1 a)x) a f (x) + (1 a) f (x). The effective
domain of f is dom f = {x| f (x)h+} [ludcator function] c (x) = { 0 x C
+ elsewhere.
domc (x) = C [Epigraph] The epigraph of f is epi f = {(x, )| f (x) } The graph of epi f is
{(x, f (x)|x dom f }. [III] A function f : Rn R is f : Rn R is convex x, y Rn ,
(0, 1), f (ax + (1 a)x) a f (x) + (1 a) f (x).

Proof. take x, y dom f , (x, f (x)) epi f ,(y, f (y)) epi f .

[Distance] Distance to a convexset dc (x) = inf{kzxk|z C}. Take any two sequence {yk }and{yk }
C s.t. kyk xk dc (x), kyk xk dc (x). zk = yk + (1 )yk . dc (x + (1 )x) kzk x
(1 )xk
= k(yk x) + (1 )(yk x)k
kyk xk + (1 )kyk xk Take k , dc (x + (1 )x) d(x) + (1 )d(x) [Eigenval-
ues] Let X Sn := {n nsymmetricmatrix}. 1 (x) 2 (X) . . . n (x).
fk (x) = n1 i (x).
Equivalent characterization
fk (x) = max{i vTi Xvi |vi v j , i 6= j}
= max{tr(V T XV |V T V = Ik }
max{tr(VV T X)}bycircularity Note hA, Bi = tr(A, B) is true for symmetric matrix.
hA, Ai = |A|2F = i A2ii
1.3 Support Function
Take a set C Rn , not necessarily convex.The support function is C = Rn R. C (x) = {hx, ui|u
C}. [scale=0.5]11 .pngT hesupport f unctionbindsthesupportinghyper plane.
Supporting functions are

Positively homogeneous
C (x) = C (x)i0
C (x) = supuC hx, ui = supuC hx, ui = C (x)

Sub-linear( a special case of convex, linear combination holds .

C (x + (1 )y) = supuC hx + (1 )y, ui supuC hx, ui + (1 ) supuC hy, ui

[L2-norm] kxk = supuC {hx, ui, u Rn }.

kxk p = sup{hx, ui, u Bq } where 1p + 1q = 1. Bq = {kxkq 1}.
The norm is

Positive homogeneous

sub-linear

If 0 C, C is non-negative.

If C is central-symmetric, C (0) = 0 and C (x) = C (x)

[Epigraph of a support function] epiC = {(x,t)|C (x) t}. Suppose (x,t) epiC . Take any
> 0. (x,t) = (x, t).
C (x) = C (x) t. (x, c) epiC
[]12

1.4 Operations Preserve Convexity of Functions

Positive affine transformation
f1 , f2 , . . . , fk cvxRn .
f = 1 f1 + 2 f2 + . . . + k fk

Supremum of functions. Let { fi }iI be arbitrary family of functions. If x sup jJ f j (x) <
f (x) = sup jJ f j (x)
[]13Compositionwithlinearmap.
f cvxRn , A : Rn Rm is a linear map. f A(x) = f (Ax) cvxRn
f A(x) = f (A(x + (1 )y))
= f (Ax + (1 )Ay)
f (Ax) + (a ) f (Ay)