450

Linear and Nonlinear Regression Analysis

Chapter 7

According to Himmelblau and Bischoff [I]: "Process analysis is the application of scientific methods to the recognition and definition of problems and the development of procedures for their solution. In more detail, this means (1) mathematical specification of the problem for the given physical solution, (2) detailed analysis to obtain mathematical models, and (3) synthesis and presentation of results to ensure full comprehension." In the heart of successful process analysis is the step of mathematical modeling. The objective of modeling is to construct, from theoretical and empirical knowledge of a process, a mathematical formulation that can be used to predict the behavior of this process. Complete understanding of the mechanism of the chemical, physical, or biological aspects of the process under investigation is not usually possible. However, some information on the mechanism of the system may be available; therefore, a combination of empirical and theoretical methods can be used. According to Box and Hunter [2]: "No model can give a precise description of what happens. A working theoretical model, however, supplies information on the system under study over important ranges of the variables by means of equations which reflect at least the major features of the mechanism." The engineer in the process industries is usually concerned with the operation of existing plants and the development of new processes. In the first case, the control, improvement, and optimization of the operation are the engineer's main objectives. In order to achieve this, a quantitative representation of the process, a model, is needed that would give the relationship between the various parts of the system. In the design of new processes, the engineer draws information from theory and the literature to construct mathematical models that may be used to simulate the process (see Fig. 7.1). The development of mathematical models oftenrequires the implementation of an experimental program in order to obtain the necessary information for the verification of the models. The experimental program is originally designed based on the theoretical considerations coupled with a priori knowledge of the process and is subsequently modified based on the results of regression analysis. Regression analysis is the application of mathematical and statistical methods for the analysis of the experimental data and the fitting of the mathematical models to these data by the estimation of the unknown parameters of the models. The series of statistical tests, which normally accompany regression analysis, serve in model identification, model verification, and efficient design of the experimental program. Strictly speaking, a mathematical model of a dynamic system is a set of equations that can be used to calculate how the state of the system evolves through time under the action of the control variables, given the state of the system at some initial time. The state of the system is described by a set of variables known as state variables. The first stage in the development of a mathematical model is to identify the state and control variables. The control variables are those that can be directly controlled by the experimenter and that influence the way the system changes from its initial state to that of any later time. Examples of control variables in a chemical reaction system may be the temperature, pressure, and/or concentration of some of the components. The state variables are those that describe the state of the system and that are not under direct control. The concentrations of reactants and products are state variables in chemical systems. The distinction between state and control

452

Linear and Nonlinear Regression Analysis

Chapter 7

where

x = independent variable y = vector of state (dependent) variables I3 = vector of control variables b = vector of parameters whose values must be determined

In this chapter, we concern ourselves with the methods of estimating the parameter vector b using regression analysis. For this purpose, we assume that the vector of control variables 8 is fixed; therefore, the mathematical model simplifies to

In their integrated form, the above set of performance equations convert to

For regression analysis, mathematical models are classified as linear or nonlinear with respect to the unknown parameters. For example, the following differential equation:

which we classified earlier as linear with respect to the dependent variable (see Chap. 5 ) , is nonlinear with respect to the parameter k. This is clearly shown by the integrated form of Eq. (7.4):

where y is highly nonlinear with respect to k. Most mathematical models encountered in engineering and the sciences are nonlinear in the parameters. Attempts at linearizing these models, by rearranging the equations and regrouping the variables, were common practice in the precomputer era, when graph paper and the straightedge were the tools for fitting models to experimental data. Such primitive techniques have been replaced by the implementation of linear and nonlinear regression methods on the computer. The theory of linearregression has been expounded by statisticians and econometricians, and a rigorous statistical analysis of the regression results has been developed. Nonlinear regression is an extension of the linear regression methods used iteratively to anive at the values of the parameters of the nonlinear models. The statistical analysis of the nonlinear regression results is also an extension of that applied in linear analysis but does not possess the rigorous theoretical basis of the latter. In this chapter, after giving a brief review of statistical terminology, we develop the basic algorithm of linear regression and then show how this is extended to nonlinear regression. We develop the methods in matrix notation so that the algorithms are equally applicable to fitting single or multiple variables and to using single or multiple sets of experimental data.

7.2 Review of Statistical Terminology Used in Regression Analysis

453

It is assumed that the reader has a rudimentary knowledge of statistics. This section serves as a review of the statistical definitions and terminology needed for understanding the application of linear and nonlinear regression analysis and the statistical treatment of the results of this analysis. For amorecomplete discussion of statistics, the reader should consult a standard text on statistics, such as Bethea [31 and Ostle et al. 141.

7.2.1 Population and Sample Statistics

A population is defined as a group of similar items, or events, from which a sample is drawn for test purposes; the population is usually assumed to be very large, sometimes infinite. A sample is a random selection of items from a population, usually made for evaluating a variable of that population. The variable under investigation is a characteristic property of the population. A random variable is defined as a variable that can assume any value from a set of possible values. A statistic or statistical parameter is any quantity computed from a sample; it is characteristic of the sample, and it is used to estimate the characteristics of the population variable. Degrees offreedom can be defined as the number of observations made in excess of the minimum theoretically necessary to estimate a statistical parameter or any unknown quantity. Let us use the notation N to designate the total number of items in the population under study, where 0 s N 5 m, and n to specify the number of items contained in the sample taken from that population, where 0 i n 5 N. The variable being investigated will be designated as X; it may have discrete values, or it may be a continuous function, in the range - m < x < m. For specific populations, these ranges may be more limited, as will be mentioned below. For the sake of example and in order to clarify these terms, let us consider for study the entire human population and examine the age of this population. The value of N, in this case, would be approximately 6 billion. The age variable would range from 0 to possibly 150 years. Age can he considered either as a continuous variable, because all ages are possible, or more commonly as a discrete variable, because ages are usually grouped by year. In the continuous case, the age variable takes an infinite number of values in the range 0 < x s 150, and in the discrete case it takes a finite number of values xj,where j = 1, 2, 3, . . . , M and M i 150. Assume that a random sample of n persons is chosen from the total population (say n = 1 million) and the age of each person in the sample is recorded.

454

Linear and Nonlinear Regression Analysis

Chapter 7

The frequency at which each value of the variable (age, in the above example) may occur in the population is not the same; some values (ages) will occur more frequently than others. Designating m, as the number of times the value of x, occurs, we can define the concept of probability of occurrence as number of occurrences of xJ total number of observations

For a discrete random variable, p(xj) is called the probubility function, and it has the following properties:

0 < p(xl) < 1

The shape of a typical probability function is shown in Fig. 7 . 2 ~ . For a continuous random variable, the probability of occurrence is measured by the continuous function p(x), which is called the probubility densityfunction, so that P r { x < X F- x
+

dx)

p(x)dx

(7.8)

The probability density function has the following properties:

is called the continuous The smooth cutlie obtained from plottingp(x) versus x (Fig. 7 . 3 ~ ) probability density distribution. The clrmulative distributionfunction is defined as the probability that a random variable X will not exceed a given value x, that is:

7.2 Review of Statistical Terminology Used in Regression Analysis

455

The equivalent of Eq. (7.10) for a discrete random variable is

The cumulative distribution functions for discrete and continuous random variables are illustrated in Figs. 7.2b and 7.3b, respectively. It is obvious from the integral of Eq. (7.10)that the cumulative distribution function is obtained from calculating the area under the density distribution function. The three area segments shown in Fig. 7.3a correspond to the following three probabilities:
P r { X s xa]
=

[p(x)dx

(7.12)

Figure 7.2 (a) Probability function and (b)cumulative distribution function for discrete random variable.

456

Linear and Nonlinear Regression Analysis

Pr{X>xnJ

p(x)dx 1
X1

Chapter 7

(7.14)

The population mean, or expected value, of a discrete random variable i s defined as

and that of a continuous random variable as p

=

xp(x)dx S ~-

E[X]

Figure 7.3 (a) Probability density function and (b)cumulative distribution function for a continuous random variable.

7.2 Review of Statistical Terminology Used in Regression Analysis

457

The usefulness of the concept of expectation, as defined above, is that it corresponds to our intuitive idea of average, or equivalently to the center of gravity of the probability density distribution along then-axis. It is easy to show that combining Eqs. (7.15)and (7.6)yields the arithmetic average of the random variable for the entire population:

In addition, the integral of Eq. (7.16)can be recognized from the field of mechanics as thefirst noncentral moment of X. The sample mean, or arithmetic average, of a sample of observations is the value obtained by dividing the sum of observations by their total number:

The expected value of the sample mean is given by

that is, the sample mean is an unbiased estimate of the population mean. In MATLAB the built-in function mean(x) calculates the mean value of the vector x [Eq. (7.18)]. If x is a matrix, mean(x) returns a vector of mean values of each column. The population variance is defined as the expected value of the square of the deviation of the random variable X from its expectation:

For a discrete random variable, Eq. (7.20) is equivalent to

oZ =

C (x, - Ld2p(x,)
j=l

458

Linear and Nonlinear Regression Analysis

Chapter 7

When combined with Eq. (7.6), Eq. (7.21) becomes

- ;=I
-

C (xi
N

yI2

which is the arithmetic average of the square of the deviations of the random variable from its mean. For a continuous random variable, Eq. (7.20) is equivalent to

which is the second central moment of X about the mean. It is interesting and useful to note that Eq. (7.20) expands as follows:'

V[X]

E[(X - E[X])']

E[X'

(E[x])'

- 2XE[X]]

The positive square root of the population variance is called the population standard deviation:

The sample variance is defined as the arithmetic average of the square of the deviations of x, from the population mean y:
n
s2

C (xi
i=l

WI2
(7.26)
and the sample variance

n However, since p is not usually known, f is used as an estimate of is calculated from

C (xi - x ) ~
sZ

,2 = l

n - 1

(7.27)

' 'lbe expected value of aconstant is that constant. The expected value of X i s a constant; therefore, E[E[.Y] = E[,Y.