LINEAR AND NONLINEAR PROGRAMMING

Stephen G. Nash and Ariela Sofer

George Mason University

The McGraw-Hill Companies, Inc.

New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico City Milan Montreal New Delhi San Juan Singapore Sydney Tokyo Toronto

Contents

Preface Acknowledgments

xv xxv

Part I Basics
Optimization Models
1.1 1.2 1.3 1.4 1.5 Introduction Linear Equations Linear Programming Least Squares Data Fitting Nonlinear Programming 3 3 4 6 9 11 15 15 16 20 23 25 29 33 37 43 47 47 52 56 56 59 ix

Fundamentals of Optimization
2.1 2.2 2.3 2.4 2.5 2.6 2.7 Introduction Feasibility and Optimality Convexity 2.3.1 Derivatives and Convexity The General Optimization Algorithm Rates of Convergence Taylor Series Newton's Method for Nonlinear Equations 2.7.1 Systems of Nonlinear Equations

Representation of Linear Constraints

3.1 3.2 3.3 Basic Concepts Null and Range Spaces Generating Null-Space Matrices 3.3.1 Variable Reduction Method 3.3.2 Orthogonal Projection Matrix

CONTENTS

3.3.3 3.3.4

Other Projections The QR Factorization

61 61

Part II Linear Programming

4 Geometry of Linear Programming
4.1 4.2 4.3 4.4 Introduction Standard Form Basic Solutions and Extreme Points Representation of Solutions; Optimality

67
67 70 76 86

5 The Simplex Method

5.1 5.2 Introduction The Simplex Method 5.2.1 General Formulas 5.2.2 Unbounded Problems The Simplex Method (Tableaus) 5.3.1 The Fll Tableau 5.3.2 Deficiencies of the Fll Tableau 5.3.3 The Revised Simplex Tableau 5.3.4 The Revised Tableau Versus the Fll Tableau The Simplex Method (Details) 5.4.1 Multiple Solutions 5.4.2 Feasible Directions and Edge Directions Getting StartedArtificial Variables 5.5.1 The Two-Phase Method 5.5.2 The Big-M Method Degeneracy and Termination 5.6.1 Resolving Degeneracy Using Perturbation

94
94 95 98 103 107 107 110 111 114 117 117 118 121 123 128 134 139

5.3

5.4

5.5

5.6

6 Duality and Sensitivity

6.1 6.2 The Dual Problem Duality Theory 6.2.1 Complementary Slackness 6.2.2 Interpretation of the Dual The Dual Simplex Method Sensitivity Parametric Linear Programming

144
144 150 153 155 159 165 173

6.3 6.4 6.5

7 Enhancements of the Simplex Method

7.1 7.2 7.3 7.4 7.5 7.6 Introduction The Simplex Method Using the Product Form of the Inverse Problems with Upper Bounds Column Generation The Decomposition Principle Numerical Stability and Computational Efficiency 7.6.1 Representation of the Basis 7.6.2 Pricing 7.6.3 The Initial Basis

181
181 182 191 200 204 216 218 222 227

XI

7.6.4 7.6.5 7.6.6 7.6.7

Tolerances; Degeneracy Scaling Preprocessing Model Formats

227 228 229 230

8 Network Problems
8.1 8.2 8.3 8.4 8.5 Introduction Basic Concepts and Examples Representation of the Basis The Network Simplex Method Resolving Degeneracy

234
234 234 244 251 259

9 Computational Complexity of Linear Programming

9.1 9.2 9.3 9.4 9.5 9.6 Introduction Computational Complexity Worst-Case Behavior of the Simplex Method The Ellipsoid Method The Average Case Behavior of the Simplex Method The Primal-Dual Interior-Point Method

265
265 265 269 272 278 280

Part III Unconstrained Optimization

10 Basics of Unconstrained Optimization
10.1 10.2 10.3 10.4 10.5 10.6 Introduction Optimality Conditions Newton's Method for Minimization Guaranteeing Descent Guaranteeing Convergence: Line-Search Methods 10.5.1 Other Line Searches Guaranteeing Convergence: Trust-Region Methods

295
295 295 302 308 312 318 327

11 Methods for Unconstrained Optimization

11.1 11.2 11.3 11.4 Introduction Steepest Descent Quasi-Newton Methods Automating and Avoiding Derivative Calculations 11.4.1 Finite Difference Derivative Estimates 11.4.2 Automatic Differentiation 11.4.3 Methods That Do Not Require Derivatives Termination Rules Historical Background

338
338 339 347 358 358 362 365 373 377

11.5 11.6

12 Low-Storage Methods For Unconstrained Problems

12.1 12.2 12.3 12.4 12.5 12.6 Introduction Iterative Methods for Solving Linear Equations Truncated-Newton Methods Nonlinear Conjugate-Gradient Methods Limited-Memory Quasi-Newton Methods Preconditioning

382
382 383 391 396 400 404

Xll

CONTENTS

13 Nonlinear Least-Squares Data Fitting

13.1 13.2 13.3 13.4 Introduction Nonlinear Least-Squares Data Fitting Statistical Tests Orthogonal Distance Regression

409
409 411 418 420

Part IV Nonlinear Programming

14 Optimality Conditions for Constrained Problems
14.1 14.2 14.3 14.4 14.5 14.6 14.7 14.8 Introduction Optimality Conditions for Linear Equality Constraints The Lagrange Multipliers and the Lagrangian Function Optimality Conditions for Linear Inequality Constraints Optimality Conditions for Nonlinear Constraints 14.5.1 Statement of Optimality Conditions Preview of Methods Derivation of Optimality Conditions For Nonlinear Constraints Duality 14.8.1 Games and Min-Max Duality 14.8.2 Lagrangian Duality 14.8.3 More on the Dual Function Historical Background

427
427 428 435 438 447 447 454 457 464 465 468 475 481

14.9

15 Feasible-Point Methods
15.1 15.2 15.3 15.4 15.5 15.6 16.1 16.2 Introduction Linear Equality Constraints Computing the Lagrange Multipliers Linear Inequality Constraints 15.4.1 Linear Programming Sequential Quadratic Programming Reduced-Gradient Methods Introduction Classical Penalty and Barrier Methods 16.2.1 Barrier Methods 16.2.2 Penalty Methods 16.2.3 Convergence 111 Conditioning Stabilized Penalty and Barrier Methods Exact Penalty Methods Multiplier-Based Methods 16.6.1 Dual Interpretation

486
486 486 494 500 508 511 519

16 Penalty and Barrier Methods

527
527 528 529 537 539 544 545 549 552 561

16.3 16.4 16.5 16.6

17 Interior-Point Methods for Linear and Convex Programming

17.1 17.2 17.3 Introduction Interior-Point Methods for Linear Programming Affine Scaling Methods

567
567 567 569

Xlll

17.4 17.5 17.6

17.7

Path-Following Methods Karmarkar's Projective Scaling Method Interior-Point Methods for Convex Programming 17.6.1 Basic Ideas of Seif Concordance 17.6.2 The Newton Decrement 17.6.3 Convergence of the Damped Newton Method The Path-Following Method 17.7.1 Convergence and Complexity

578 587 593 594 598 599 604 606

Appendices
A Topics from Linear Algebra
A.l A.2 A.3 A.4 A.5 A.6 Introduction Vector and Matrix Norms Systems of Linear Equations Solving Systems of Linear Equations by Elimination Gaussian Elimination as a Matrix Factorization A.5.1 Sparse Matrix Storage Other Matrix Factorizations A.6.1 Positive-Definite Matrices A.6.2 The LDLT and Cholesky Factorizations A.6.3 An Orthogonal Matrix Factorization Sensitivity (Conditioning) A.7.1 Eigenvalues and Sensitivity The Sherman-Morrison Formula Introduction Computer Arithmetic Big-0 Notation, O(-) The Gradient, Hessian, and Jacobian Gradient and Hessian of a Quadratic Function Derivatives of a Product The Chain Rule Continuous Functions; Closed and Bounded Sets The Implicit Function Theorem Software

617
617 617 619 621 624 631 632 632 634 636 638 642 643

A.7 A.8

Other Fundamentals
B.l B.2 B.3 B.4 B.5 B.6 B.7 B.8 B.9

646
646 646 648 649 651 652 653 654 655

C D

Software
C.l

658
658

Bibliography Index

661 679