John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001

Engineering design
optimization
Lecture 8: Direct minimization of multi-dimensional
problems with constraints
Husk!
- at forelsning 9/10 er udsat.
Den skemalgges af Grethe/Gitte senere.
PThe conjugate gradient algorithm revisited
PLinear programming
<Properties of LP problems
<The Simplex method
PNonlinear programming
<Feasible directions
PIntro to a penalty algorithm
Lecture 8
Constrained multiple-dimension optimization
To read:
PArora: chapter [4 - 4.5.1] + [6.8 - 6.8.1]
The conjugate gradient algorithm
The searchdir function
void SearchDir(long itnr, double *Grad, double *OldGrad, double *Oldss, double *ss)
{
bool Conjugate=true; // Set to false for steepest descent
// Always the opposite gradient
for (long j=0;j<nDV;j++) ss[j] = -Grad[j];
// Normalize
normalize(ss);
// If an old search direction is present, modify the new direction
if (itnr>0 && Conjugate) {
double beta = norm2(Grad)/norm2(OldGrad);
for (j=0;j<nDV;j++) ss[j] += beta*Oldss[j];
}
else return; // Don't normalize again unless we have modified ss
// Normalize length after modification
normalize(ss);
return;
}
The conjugate gradient algorithm
The main minimization function
double CJGmin() {
double alpha=1000.0;
double B = 4.0; // Search length. Fits the banana function
long count=0;
double df[MAXDES],OldGrad[MAXDES],OldS[MAXDES];
// Compute current objective
double Fxx = f(x);
// Main loop. Keep minimizing until the change is small enough
while (alpha > 1.0e-4) {
// Store old gradient and search direction
if (count>0) {
for (long j=0;j<nDV;j++) {
OldGrad[j]=df[j];
OldS[j]=s[j];
}
}
The conjugate gradient algorithm
The main minimization function (contd)
// Calculate gradient by finite difference
Gradient(x,Fxx,df);
// Find conjugate gradient search direction
SearchDir(count,df,OldGrad,OldS,s);
// Minimize along s by the golden section method
Fxx=GoldenSectionSearch(0,B,&alpha);
// Set next search interval according to the present
B=2*alpha;
// Update design variables
for (long j=0;j<nDV;j++) x[j] = x[j] + alpha*s[j];
count++;
}
printf("\nCJGmin: %d iterations\n",count);
return Fxx;
}
John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001
Minimize
subject to
g
g G
i i
0
( )
( )
x
x
Minimize / Maximize
subject to
g c x
a x b
i i
i
n
ij j i
j
n
0
1
1
( ) x =

=
=

a x b
a x x b
x
ij j i
j
n
ij j n i i
j
n
n

+ =

=
+
=
+

1
1
1
0
is converted to
If the functions g
i
, i=0..n are linear, then the problem
is linear. We call the solution of such problems
linear programming.
Mathematical programming
Usual formulation
Linear programs can be written like this:
Linear programming
PLinear programs are convex. They usually only have one optimum. If they
have several optimal, then they are all equally good.
Linear programming
Properties
PPlanning of traffic and transportation
PScheduling
PProduction planning
PEconomics
PEletrical circuits
PSupply networks
PBiomechanics
PTrusses
Linear programs
Where do they occur?
PThe feasible domain is a hyper-polyhedron, a so-called simplex, in n
dimensions.
PThe optimumis always in a corner of the polyhedron.
PA linear programcan only have asmany equality constraints, m, as it has
variables, n. If n=m, then it degenerates to a systemof equations with
usually one solution.
PIf n <m, then it usually has no solutions.
PIf n >m, then the feasible domain has infinitelymany points, and we have
an optimization problem.
PThere is no limit on the number of inequality costraints.
Linear programs
- properties
So by increasing the number of variables, we can get rid of all inequality
constraints except the trivial ones that require the slack variables to be
positive.
Linear programming
Slack variables can convert inequalities to equalities
John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001
Minimize / Maximize
subject to
j=1
g c x
a x b
x
i i
i
n
ij j i
n
j
0
1
0
( ) x =
=

Minimize / Maximize
subject to
c x
Ax b
x 0
T
=

For reasons of tradition and efficiency, linear

programs are usually formulated in a certain way that
is a little different from nonlinear optimization
problems:
Linear programming
Standard formulation
This formulation is the basis of theSimplexmethod, a classical andvery
efficient way of solvingmathematical programming problems.
Linear programming
- or on matrix form:
The basic idea of the simplexmethod is to use the knowledge that the
solution is always in a corner of the simplex.
The Simplex method
Basic idea
Suppose we have 2 variables and 5
inequality constraints as in this picture.
We add slack variables and get 7
variables and 5 equality constraints.
Each corner is now the solution to a 5-
subset of the 7 equalities. The subset is
called the basic set.
The simplex method steps from one
corner to another by replacing the
equations in the basic set one by one
with equations from the outside.
PInvestigate the coefficients of the equations in the active and non-active sets to
see if any neighbouring corner is better than the one we are in
PIf there is a better neighbour, exchange an equation in the active set to step to
it.
PRepeat until convergence.
A Simplex algorithm
Find a basic feasible solution. Notice that far from all basic solutions are
feasible. Then,
PThe simplexmethod is verymuch like solving equations. Thismeans that
it is very numerically efficient and finishes in a finite number of steps.
PSimplex is themost reliable andmost used of all optimization algorithms.
PProblems up to a fewthousandvariables and constraints are solved in a
short time on a small computer
PThe time consumption does growvery quickly with the number of
constraints andvariables (due to the combinatorics of corner formation in
the simplex and the cost of solving lienar equations)
PFor very large problems, so-called interior point methods can be better.
The Simplex Method
- properties
Example: biomechanics
Biceps Brachii
Brachialis
Brachioradialis
The body is statically
indeterminate.
Solution:
Distribute muscle force by an
optimality criterion
John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001
[ ]
Cf r
f f f
f
M
=
=

( ) ( )
( )
,
M R
0
G
f
N
i
i
p
i
n
( ) f =
|
\

|
.
|
=

1
G
f
N
i
i
p
i
n
( ) f =
|
\

|
.
|
=

1
min minmax
f
N
f
N
p
i
i
p
i
n
i
i
|
\

|
.
|
|
\

|
.
|

=

1
for
Physical problem
Forces
Reactions
Mathematical problem
A statically indeterminate problem has more variables than
equations
Equilibrium. C is
rectangular
Muscles can only
pull
Traditionally, the following types of objective functions have been used:
Choosing an Optimality Criterion
Our problem has infinitely many solutions. Just like an
optimization poblem usually has infinitely many feasible
points. We need to find the best solution.
Biceps Brachii
Brachialis
Brachioradial is
Popular values:
p = 1, p = 2, p = 3.
N
j
= 1, N
j
= PCSA
Conclusion:
Best results if N
j
= actual strength (Hill model)
and p = 100!
The significance of exponent p
Pp = 1 does not work because it usually only makes one of
the muscles spanning a joint active
Pp = 2..3 does not work because some muscles will be
loaded above their physiological strength
Pp $ 4 is difficult to handle numerically
It is well-known that
Exponent p (contd)
What happens when p becomes large?
PConsistent with test results
P Tends to distribute relative
muscle load evenly
postpones fatigue as far as
possible
PMuscles loaded above their
strength never occur (unless
the load is too high to be
carried)
PRecruitment simple from the
CNS point of view
Why min/max?
Good arguments for the min/max criterion
Biceps Brachii
Brachialis
Brachioradialis
John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001
Minimize
Subject to
max
( )
( )
f
N
i
M
i
M
|
\

|
.
|
=

Cf r
f 0
Minimize
Subject to

f
N
i
M
i
(M)
( )

Cf r
f 0
Allowable direction
Reducing direction
d g d 0
d g d 0
0
:
:


i
Muscle distribution problem
Mathematical formulation
Minimization problem
Dynamic equilibrium
Muscles cannot push
Min/max problems are not
differentiable
A non-differentiable objective function is very difficult to
handle mathematically
F
i
(x)
x
max(F
i
(x))
Bound formulation
The differentiability problem can be solved very elegantly by
a simple reformulation
This problem has the same
solution as the original, but it
is linear and differentiable and
can be solved very efficiently.
In biomechanics, we typically
solve problems with a few
hundred variables thousands
of times in succession.
This works well on a small
PC.
We have seen that penaltymethods as a cheap and dirty
solution can handle constrained optimization problems.
Direct methods are much more robust and flexible. There are
two types:
- Search methods
- Sequential programming
Today, well look at the former. Sequential programming is
treated in lecture 9/10.
Nonlinear programming
Direct methods
PFind a search direction
PMinimize in one dimension along the search direction
PRepeat until convergence.
Search methods
- work just like the conjugate gradient method.
The only differences for constrained algorithms are
- that when picking a search direction, we check that it does not
collide right away with a constraint
- when performing 1-D line search, we stop when we hit a
constraint.
Feasible directions
- a very robust search method for constrained problems
x
x
1
2
Reducing
directions
Allowable
directions
Feasible
directions
John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001
Minimize
Minimize


g d
g d
0
i
( ) Minimize max g d g d
0 i
,
Minimize
Subject to

g d
g d
0
i
j
d 1 1
Choice of feasible direction is a
compromise between
- quick reduction
- avoiding constraint violation
At the same time, we want to
Choice of feasible direction
x
x
1
2
Reducing
directions
Allowable
directions
Feasible
directions
This can be done by bound formulation:
Choice of feasible direction
- as an optimization problem
This is a linear problem and can be solved by the simplex
method
PCompletely robust. When started froma feasible point, it will never exit
the feasible domain again.
PEvery newdesign is both feasible and better than the previous.
PDepending on derivatives only in the direction choice. The line search can
be done with a slightlymodified golden section search. Thismeans that
the algorithmis good if you do not have an efficient method of computing
sensitivities.
PErroneous direction choices will slowthe convergence but not will not kill
the algorithm.
PProgramming is a little complicated because the direction choice problem
requires another optimizer, for instance simplex.
The method of feasible directions
- properties
PUse the conjugate gradient code fromlast lecture.
PModify it as described on the following pages.
PCompile and test with different 2-D functions, for instance
example 6.14. Drawa picture of the problembyMaple or by
hand and follow the convergence of the algorithm. Try to
understand eventual problems.
Assignment
Program your own penalty algorithm
Skeleton of a penalty algorithm
It can be developed from the conjugate gradient algorithm
by a simple modification of the objective function.
First, we need another global variable:
// --------------------------------------------------------------
// These variables are global because it simplifies the code.
// Global variables are not recommended for larger programs
double x[MAXDES]; // Design variable vector
double s[MAXDES]; // Search direction
long nDV; // Number of design variables
long nCon; // Number of constraints
Penalty algorithm (contd)
Then we need a function to evaluate the constraints
// --------------------------------------------------------------
// This function evaluates all constraints. Add constraints to the
// problemby including them in this function.
void EvalConstraints(double *xx, double *gi) {
// Set the number of constraints here
nCon=7;
// Evaluate the left hand side of constraints of the
// form: gi(x) - Gi <= 0. In the interest of having
// non-dimensional constraints, it might be a good idea
// to use the equivalent formulations gi(x)/Gi - 1 <= 0.
// Example 6.14
gi[0] = 2.40e+7/(xx[0]*xx[1]*xx[1]) - 1.0;
gi[1] = 1.125E+5/(xx[0]*xx[1]) - 1.0;
gi[2] = xx[1]/(2*xx[0]) - 1.0;
gi[3] = 1.0 - xx[0]/10.0;
gi[4] = 1.0 - xx[1]/10.0;
gi[5] = xx[0]/1000.0 - 1.0;
gi[6] = xx[1]/1000.0 - 1.0;
return;
} // EvalConstraints
John Rasmussen, Institute of Mechanical Engineering, AalborgUniversity, 2001
Penalty algorithm (contd)
Finally, we replace the objective by the pseudo-objective
// --------------------------------------------------------------
// This is the PSEUDOobjective function. It calls the objective
// function and constraint evaluation and sums up the penalized
// pseudo objective function. The onlyuser-interactionwith this
// function is to set the value of the penalty parameter.
double Fpseudo(double *xx) {
double gi[MAXCON];
// Set penalty factor here. This is problem-depedent. The factor
// should have a considerable size compared to the objective
// function
double r =1.0e+1;
// Call the objective function
double res = f(xx);
// Evaluate constraints
EvalConstraints(xx,gi);
// Add contributions fromviolated constraints
for (long i=0;i<nCon;i++) if (gi[i]>0) res += r*gi[i]*gi[i];
// Return the result to caller
return res;
} // Fpseudo
Thats all there is to it. If we replace all calls to F in the code
by a call to Fpseudo, then we are solving a constrained
problem.
But remember: The exterior penaltymethod is not a very
reliable partner. It may return the wrong optimumor not
converge at all.
Penalty algorithm (contd)
Concluding remarks