1

Experimental Design,
Response Surface
Analysis, and
Optimization

2
Outline
Motivation and Terminology
Difficulties in Solving the Basic Problem
Examples of Factors and Responses
Types/Examples of Experimental Design
Full Factorial Designs
Randomness of Effects
Example: Full Factorial Design
Situations with Many Factors
Response Surfaces and Metamodels
Regression Analysis
Response Surface Methodology

3
Motivation and Terminology
Useful when there are many alternatives to
consider (e.g., numerous capacity levels of
various types, numerous parameters for a
proposed inventory system)
Two basic types of variables: factors and
responses
Factors: input parameters:
-controllable or uncontrollable
-quantitative or qualitative
Responses: outputs from the simulation model:
-uncertain in nature
Basic problem: find the best levels (or values
of the parameters) in terms of the responses
Experimental Design can tell you which
alternatives to simulate so that you obtain the
desired information with the least amount of
simulation

4
Multiple Responses

Uncertain Responses
Difficulties in Solving
the Basic Problem

5
Examples of Factors
1. Mean interarrival time (uncontrollable,
quantitative)
2. Mean service time (controllable or
uncontrollable, quantitative)
3. Number of servers (controllable, quantitative)
4. Queuing discipline (controllable, qualitative)
5. Reorder point (controllable, quantitative)
6. Mean interdemand time (uncontrollable,
quantitative)
7. Distribution of interdemand time
(uncontrollable, qualitative)


6
1. Mean daily production rate
2. Mean time in the system for patients
3. Mean inventory level
4. Number of customers who wait for more than 5
minutes
Examples of Responses

7
Types/Examples of
Experimental Designs
Completely Randomized Designs

Randomized Complete Block Designs

Nested Factorial Designs

Split Plot Type Designs

Latin Square Type Designs

Full Factorial Designs

Fractional Factorial Designs

8
2
k
Factorial Designs
Suppose that we have k (k > 2) factors.
A 2
k
factorial design would require that
two levels be chosen for each factor,
and that n simulation runs (replications)
be made at each of the 2
k
possible
factor-level combinations (design
points). For 3 factors, this yields a
Design Matrix:



9
Design Matrix for 3 Factors


Factor 1 Factor 2 Factor 3
Design Point Level Level Level
Response
1 - - - O
1

2 + - - O
2
3 - + - O
3
4 + + - O
4
5 - - + O
5
6 + - + O
6
7 - + + O
7
8 + + + O
8


+ refers to one level of a factor and - refers to
the other level. Normally, for quantitative factors,
the smallest and largest levels for each factor are
chosen.
2
k
Factorial Designs

10
2
k
Factorial Designs:
Estimating Main Effects
The main effect of factor 1 is the change in the
response variable as a result of the change in
the level of the factor, averaged over all levels
of all of the other factors



If the effect of some factor depends on the
level of another factor, these factors are said to
interact.
The degree of interaction (two-factor
interaction effect) between two factors i and j is
defined as half the difference between the
average effect of factor i when factor j is at its
+ level ( and all factors other than i and j are
held constant) and the average effect of i when
j is at its - level; for example,

( ) ( ) ( ) ( )
e
O O O O O O O O
1
2 1 4 3 6 5 8 7
4
=
+ + +
e
O O O O O O O O
12
4 3 8 7 2 1 6 5
1
2 2 2
=
+

(
( ) ( ) ( ) ( )

11
Example: Full Factorial (2
k
)
Design
Consider a simulation model of reorder point,
reorder quantity inventory system. The two
decision variables, or factors, to consider are
the reorder point (P) and the reorder quantity
(Q) for the inventory system. The maximum
and minimum allowable values for each are
given below:





Suppose that the response variable output by
the model is the long-run average monthly cost
(composed of three components: holding cost,
shortage costs, and ordering costs) in
thousands of dollars.

Factor Minimum
Value
Maximum
Value
P 20 40
Q 15 50

12
Example: Full Factorial (2
k
)
Design
A 2
2
factorial design matrix with simulation results
(for 10 replications at each design point) might be
given by:






where a factor level of - indicates the minimum
possible value for that factor, and a factor level of
+ indicates the maximum possible level; for
example, design point 2 has P=40 and Q=15.
Design Point Factor Level for
P Q
Response
1
2
3
4
- -
+ -
- +
+ +
135.6
128.2
121.7
131.5

13
The response given is the average cost over
the 10 replications. Now, the main effects are
given by:




The interaction effect (e
PQ
) is given by:



Therefore, the average effect of increasing P
from 20 to 40 is to increase monthly cost by
1.2, and the average effect of increasing Q
from 15 to 50 is to decrease monthly cost by
5.3. Hence, it would be advisable to set P as
low as possible and set Q as high as possible.

e
O O O O
Q
=
+
=
+
=
( ) ( ) ( . . ) ( . . )
.
3 1 4 2
2
1217 1356 1315 1282
2
53
e
O O O O
P
=
+
=
+
=
( ) ( ) ( . . ) ( . . )
.
2 1 4 3
2
1282 1356 1315 1217
2
12
e
PQ
=
+
=
1356 1282 1217 1315
2
86
. . . .
.
Example: Full Factorial (2
k
)
Design

14
Also since the interaction effect is positive,
it would seem advisable to set P and Q at
opposite levels. (Of course, all of the
above could be inferred from a cursory
analysis of the responses for the various
design points.) Note also that the literal
interpretation of main effects assumes no
interaction effects (pages 669 and 670 of
Law and Kelton, 1991).
Example: Full Factorial (2
k
)
Design

15
Randomness of Effects
Note that the main and interaction effects
computed in the previous examples are
just random variables. To determine if
the effects are significant or real, and
not due to random fluctuations, one could
compute values for the main and
interaction effects 10 times (once for each
replication) and form confidence intervals
for each of the main effects, and the
interactions effect. If the confidence
interval contains 0, then the effect is not
statistically significant. (Note that
statistical significance does not
necessarily imply practical significance).

16
Situations with Many Factors
When there are many factors to consider,
full factorial, or even fractional factorial
designs may not be feasible from a
computational standpoint.
Other types of design (e.g., Plackett-
Burman designs or supersaturated
designs) may be appropriate (Mauro,
1986).
Another approach is to reduce the number
of factors to consider via factor screening
techniques, involving, for example, group
screening in which a whole group is
treated as a factor.

17
Response Surfaces and
Metamodels
A response surface is a graph of a
response variable as a function of the
various factors.
A metamodel (literally, model of the
simulation model), is an algebraic
representation of the simulation model,
with the factors as independent variables
and a response as the dependent
variable. It represents an approximation
of the response surface.
The typical metamodel used in a
simulation application is a regression
model.



18
A metamodel through the use of response
surface methodology can be used to find
optimal values for a set of factors. It can
also be used to answer what if questions.
(Experimentation with a metamodel is
typically much less expensive than using a
simulation model directly).
An experimental design process assumes
a particular metamodel, e.g.,
E C P Q B B P B Q B PQ [ ( , )] = + + +
0 1 2 12
Response Surfaces and
Metamodels

19
Basic Concepts of
Regression Analysis
Regression is used to determine the
best functional relation among
variables.
Suppose that the functional relationship
is represented as:

E(Y) = f (X
1
, ..., X
p
/ B
1
, ..., B
E
)

where E(Y) is the expected value of the
response variable Y; the X
1
, ..., X
p

are factors; and the B
1
, ..., B
E
are
function parameters; e.g.,

E(Y) = B
1
+ B
2
X
1
+ B
3
X
2
+ B
4
X
1
X
2


20
The observed value for Y, for a given
set of X s, is assumed to be a random
variable, given by:

Y = f (X
1
, ..., X
p
/B
1
, ..., B
E
) +

Where is a random variable with
mean equal to 0 and variance . The
values for B
1
,...,B
E
are obtained by
minimizing the sum of squares of the
deviations.
c
c
o
E
2
Basic Concepts of
Regression Analysis

21
Response Surface
Methodology
Source: (Fu, 1994)
Response surface methodology (RSM)
involves a combination of
metamodeling (i.e., regression) and
sequential procedures (iterative
optimization).



22
RSM involves two phases:
Fit a linear regression model to some initial
data points in the search space (through
replications of the simulation model).
Estimate a steepest descent direction from
the linear regressions model, and a step
size to find a new (and better) solution in
the search space. Repeat this process
until the linear regression model becomes
inadequate (indicated by when the slope of
the linear response surface is
approximately 0; i.e., when the interaction
effects become larger than than the main
effects).
Fit a nonlinear quadratic regression
equation to this new area of the search
space. Then find the optimum of this
equation.
Response Surface
Methodology

23
Terminology in Experimental
Design
Source:(Ostle, 1963)
Replication - the repetition of the basic
experiment
Treatment - a specific combination of several
factor levels
Experimental Unit - the unit to which a single
treatment is applied to one replication of the
basic experiment
Experimental Error - the failure of two
identically treated experimental units to yield
identical results
Confounding - the mixing up of two or more
factors so that its impossible to separate the
effects

24
Randomization - randomly assigning
treatments to experimental units (assures
independent distribution of errors)
Main Effect (of a factor) - a measure of the
change in a response variable to changes in
the level of the factor averaged over all levels
of all the other factors
Interaction is an additional effect (on the
response) due to the combined influence of
two or more factors
Terminology in Experimental
Design