14b-1/2 CHE 5703 FALL 2014 Rhinehart

Comparing Optimizers or any options

Which optimizer is best? Here the objective function is to choose the algorithm that minimizes
all of the bad aspects and maximizes all of the good aspects. Your DV is the optimizer type. This is
optimizing the choice of optimizer.
Min J = undesirable aspects desired aspects
{optimizer choices}
A question is, How to get measures of the OF aspects? Here we need to work in Blooms
Analysis Cognitive Level.
Suppose that you decide that algorithm complexity is a bad thing, because simplicity is a good
thing. K.I.S.S. is a fundamental value for engineering practice. But, complexity is an abstract
concept. You might rank from simplest to worst R3 Heuristic, Golden Section, Secant, Newtons,
Successive quadratic. The ranking 1
st
, 2
nd
, 3
rd
, etc. would be your personal assessment (the numerical
values, metrics, provide a measure of the abstract concept). But any assessment should have a
defensible basis and a universally verifiable value. For complexity, appropriate metrics might be lines of
code, education level required to understand the algorithm, number of user inputs, or number of
conditional statements. The problem with a rank, such as 1
st
, 2
nd
, 3
rd
, etc. is that 1
st
and 2
nd
may be very
close and 3
rd
may be a distant 3
rd
. Rank does not provide relative position. Further, if you learned
Newtons first, because it is familiar to you, you may think it simpler than Golden Section. Rank is OK to
use as a measure of goodness, but it is not a universal specific truth.
Suppose you also decide that computer work is a bad thing. Work is an abstract concept of
badness. The number of iterations to converge is a metric that you could use to assess work. However,
some optimizers need more function evaluations per iteration than others, so perhaps number of
function evaluations (NOFE) to converge could be a better metric to use as the assessment of work. But,
successive quadratic, in the form of my algorithm in the line search examples, only uses one function
evaluation at each iteration, but it does all of the computer work to solve for the a, b, and c, coefficients
in the linear equation set. So, perhaps execution time is a better measure of work than NOFE. But
computer execution time depends on the computer (core speed and how it handles particular
operations between devices), on the operating system (how it manages operations), and on the
algorithm programmer (how much display overhead is integrated in the basic code). So, perhaps a
count of the essential read-write, get-put, and mathematical operations is a better measure of work
than either NOFE or execution time. Lets say you choose NOFE. Whatever you decide as a measure of
work, you do trials to get that metric for each DV (optimizer).
Suppose you decide that probability of finding the global optimum is a good thing. There are
two issues with this. First, within multi-optima the probability of finding the global optimum is a yes
you did or no that was not the global. But, second, within constraints, or poor surface
characterization or flat spots, etc., the optimizer might stop in the vicinity of the global but not exactly
on it. You could run each optimizer 1000 times, chose the best result from all optimizers to represent
perfection, and then count the number of results for each optimizer that are in the vicinity of the
ultimate, and of those measure the variance from perfection.
The initial trial solution (starting point) will define where the algorithm goes and how many
function evaluations it needs to converge. One algorithm might converge rapidly from one point, but
14b-2/2 CHE 5703 FALL 2014 Rhinehart

not another because of local surface features. However, the other optimizer might be designed to cope
with the features and have an alternate record. So random initialization might be a good way to see the
breadth of responses, but replicate trials from random initializations will not produce exact duplicate
results. So, statistical comparisons need to be made on replicate results, such as a t-test of differences
in NOFE. And the number of trials needs to be high enough to permit differences to be statistically
significant.
Further, one optimizer (Newtons, successive quadratic) might jump to the solution in one step if
the function is a simple quadratic response, but it might become hopelessly lost on a different function.
How do you compare? You need to choose a set of test functions that represent a diversity of features.
You could combine several OF aspects to create one overall OF: min (NOFE + Rank - fraction
near ultimate). But, how would you scale complexity (rank) with NOFE with fraction? You could use the
EC approach. Defend your choice of EC values.
Alternately, you could plot each DV (optimizer choice) result on an NOFE vs. Rank graph, and
then see if any optimizers dominate the others. See which are Pareto Optimal.
Whatever you do to evaluate best, the outcome will depend on: 1) Your interpretation best,
your concepts of goodness or badness, desirability or undesirability. 2) Your choice of metrics to assess
that aspect. 3) Your choice of experiments to gather assessment data (Only use function 12? Use ten
functions to cover an inclusive range of surface aberrations? Only evaluate each function once? Run
the optimizer 10,000 times from random initializations from a range of parameter values for each
function to be able to see the truth through the statistical vagaries?) 4) Your choice of parameters
(expansion coefficients, attenuation factors, starting range, number of players, convergence criteria,
number of replicate trials, EC values, etc. for each algorithm. And 5) How you decide to combine and
interpret the assessment data.
Life is like that. Personal and business decisions are like that. To be appreciated, to be valued,
to have your recommendations accepted, you need to be sure that all stakeholders accept your choices
in the five steps above (agree that your analysis is fair, comprehensive, complete, not infected by chance
or personal bias). So, you need to reveal choices in Steps 1-5.
For the assignment, Ill accept any of any number of choices for steps 1), 2), 3), 4), and 5), except
choices representing a trivial, token analysis. But dont be excessive either, dont strive for perfection in
completeness of comprehensiveness. Balance perfection with sufficiency.