Lecture Note 1: Introduction to optimization

Xiaoqun Zhang
Shanghai Jiao Tong University

Last updated: September 23, 2017

Lecture note 1 Introduction to Optimization

1.1 Introduction
1. Optimization is an important tool in daily life, business and engineer:
• Running a business: to maximize profit, minimize loss, maximize effi-
ciency, or minimize risk.
• Design: minimize the weight of a bridge, and maximize the strength,
within the design constrains; airplane engineering such as shape design
and material selection; packing transistors in a computer chip in a func-
tional way.
• Planning: select a flight route to minimize time or fuel consumption of
an airplane
• Supermarket pricing and logistic
2. Formal definition: to minimize (or maximize) a real function by deciding the
values of free variables from within an allowed set.
3. Status of optimization: in last few decades, astonishing improvements in com-
puter hardware and software motivated optimization modeling, algorithm de-
signs, and implementation. Solving certain optimization problems has become
standard techniques and everyday practice in businesses, science and engineer-
ing. It is now possible to solve certain optimization problems with thousands,
millions and even thousands of millions of variables; optimization (along with
statistics) has been the foundation of machine learning and big-data analytic.
4. Ingredients of successful optimization:
• modeling (turn a problem into one of the typical optimization formula-
tions);
• algorithms an (iterative) procedure that leads you toward a solution
(most optimization problems do not have a closed-form solution);
• software and hardware implementation: realize the algorithms and return
numerical solutions.
5. Optimization formulation:

mininizex (maximize) f (x)

subject to x∈C

• ’minimize’ is often abbreviated as ’min’;

• decision variable is typically stated under ’minimize’, unless obvious;
• ’subject to’ is often shorten as ’s.t.’;
• in linear and nonlinear optimization, feasible set C is represented by

hi (x) = bi , i ∈ E(equality constraints)

gj (x) ≤ b′j , j ∈ I(inequality constrains)

2
Lecture note 1 Introduction to Optimization

6. First examples:

• An office furniture manufacturer wants to maximize the daily profit. De-

cide the daily number of desks and chairs that should be manufactured
such that the profit is maximum. The required material and profit of
manufacturer desks and chair is listed as followed:

Each piece Required material Required labor Profit

desk 10 5 20
chair 6 2 10
Total available 240 100
Problem: let the daily manufactured number of desks and chairs be x1 ,
x2 , solve

max 20x1 + 10x2

x1 ,x2

s.t. 10x1 + 6x2 ≤ 240

5x1 + 2x2 ≤ 100
x1 , x2 ≥ 0

• Find two nonnegative numbers whose sum is up to 6 so that their product

is a maximum. ( find the largest area of a rectangular region provided
that its perimeter is no greater than 12). Problem: let the two numbers
be x, y, solve

max xy
x,y

s.t. x + y = 6
x, y ≥ 0

• Find a line that ”best” fit three given points (x1 , y1 ) = (2, 1), (x2 , y2 ) =
(3, 6) and (x3 , y3 ) = (5, 4). Problem: let the line equation be y = ax + b,
in the following least square sense:

X
3
min (axi + b − yi )2 = (2a + b − 1)2 + (3a + b − 6)2 + (5a + b − 4)2
a,b
i=1

• Traveling sales man problem: A salesman needs to visit a number (n)

of cities, denoted by city 1, 2, · · · , n. The distances between cities are
known, i.e. dij denotes the distance between city i and city j. The
salesman wants to travel each city once in turn and in the meantime

3
Lecture note 1 Introduction to Optimization

minimize the total travel distance. What order should be used? Problem:
denote the order of travelling by (i1 , · · · , in ), model
X
n−1
min dik ,ik+1
k=1
s.t. (i1 , i2 , · · · , in ) is a permutation of (1, 2, · · · , n)
7. Classification of optimization problems
Continuous vs Discrete
Constrained vs Unconstrained
Global vs Local
Stochastic vs Deterministic
Convex vs Nonconvex
8. Convex programming:
min f (x) s.t. X ∈ C
both f and C are convex. Special cases:
• linear programming
min cT x s.t. Ax ≤ b; Bx = d;
• least square problems (quadratic program):
min xT Qx + cT x s.t. Ax ≤ b; Bx = d;
9. Global vs local solution: ”solutions means ”optimal solution”.
• Global solution x∗ : f (x∗ ) ≤ f (x) for all x ∈ C.
• Local solutions x∗ : ∃δ > 0 such that f (x∗ ) ≤ f (x) for all x ∈ C and
kx − x∗ k ≤ δ.
• A (global or local) solution x∗ is unique if ≤ holds strictly as <.
• In general, it is difficult to tell if a local solution is global because algo-
rithms can only check ”nearby points” and have not clue of behaviors
”father way”. Hence a solution may refer to a local solution.
• A local solution to a convex program if globally optimal. A LP is convex.
• A ”stationary point” (where the derivatives is zero is also known as a
solution, but it can be a maximization, minimization
10. Nonlinear program:
min f (x) = −(x1 + x2 )2
s.t.x1 x2 ≥ 0
− 2 ≤ x1 ≤ 1
− 2 ≤ x2 ≤ 1
Decision variables, feasible set, objective, (box) constraints,global minimizer
(x∗ = (−2, −2)) vs local minimizer (x′ = (1, 1)). See the figure 10.

4
Lecture note 1 Introduction to Optimization

bA 1 b H bD
x′(Local solution)

b F b E b I
−4 −3 −2 −1 1 2

−1

bB g
−2 bG bC
x∗ (Global solution)

−3

−4

−5

Figure 1.1: Global solution vs. local solution

11. Optimization algorithms requirement

• Robustness: perform well on a wide variety of problem
• Efficiency: reasonable computer time or storage
• Accuracy: precision, not being sensitive to norms
• Using convex optimization problem: often difficult to recognize, many
tricks for transforming problems into convex form, surprisingly many
problems can be solved via convex optimization.
12. This course focuses on finding local solutions and, for convex programs, global
solutions.
• Asking for global solutions is computationally intractable, in general.
• Many useful problems are convex, that is, a local solution is global
• In some applications, a local solution is an improvement from an existing
point. Local solutions are OK.
13. We do not cover discrete optimization: Discrete variables often take binary
or integer values, or values from a discrete set. Those problems are called
discrete optimization.