Design and Analysis of Algorithms

Copyright Li Zimao @ 2007-2008-1 SCUEC

Algorithm: A brief History

Muhammad ibn Musa al-Khwarizmi, one of the most influential mathematicians in 9th century in Baghdad, wrote a textbook in Arabic
about adding, multiplying, dividing numbers, and extracting square roots and computing .
www.lib.virginia.edu/science/parshall/khwariz.html

Many centuries later, decimal system was adopted in Europe, and the procedures in Al Khwarizmis book were named after him as Algorithms.

(image of Al Khwarizmi from http://jeff560.tripod.com/)

Copyright Li Zimao @ 2007-2008-1 SCUEC

Expected Outcomes
Define algorithm formally and informally. Explain the idea of different algorithms for computing the greatest common devisor Describe the process of algorithm design and analysis Design recursive and non-recursive algorithms for computing a Fibonacci number

Copyright Li Zimao @ 2007-2008-1 SCUEC

Notion: Algorithm
You all have learned the course DATA STRUCTURE or DISCRETE MATHEMATICS, and have your own sense on algorithm.
to your opinion, what is algorithm?

Copyright Li Zimao @ 2007-2008-1 SCUEC

Notion: Algorithms
An algorithm is a sequence of unambiguous instructions for solving a computational problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

problem

algorithm

input

computer
Copyright Li Zimao @ 2007-2008-1 SCUEC

output

Notion: Algorithm
More precisely, an algorithm is a method or process to solve a problem satisfying the following properties:

Finiteness
terminates after a finite number of steps

Definiteness
Each step must be rigorously and unambiguously specified.
-e.g., stir until lumpy

Input
Valid inputs must be clearly specified.

Output
can be proved to produce the correct output given a valid can be proved to produce the correct output given a valid input.

Effectiveness
Steps must be sufficiently simple and basic.
-e.g., check if 2 is the largest integer n for which there is a solution to the equation xn + yn = zn in positive integers x, y, and z
Copyright Li Zimao @ 2007-2008-1 SCUEC

Examples
Is the following a legitimate algorithm?
i 1 While (i <= 10) do a i+1 Print the value of a End of loop Stop

Copyright Li Zimao @ 2007-2008-1 SCUEC

Notion: Algorithm*
The exact definition of algorithm is given by Alonzo Church and Alan Turing in 1936.
-calculus system of Church Deterministic Turning machine model Equivalent definition

For Turning machine model, see

Michael Sipser, Introduction to the Theory of Computation, China Machine Press. John E. Hopcroft, Rajeev Motwani, Introduction to Automata Theory, Languages, and Computation (Second Edition), Tsinghua University Press.

Copyright Li Zimao @ 2007-2008-1 SCUEC

Notion: Algorithm*
According to the definition of Algorithms, almost all problems are unsolvable. For example:
Determine if a polynomial with n (n > 1) variables has an integral solution. Determine if there is a 1111111 pattern in s decimal format. Halting Problem:
Determine if a program halts on an input

Copyright Li Zimao @ 2007-2008-1 SCUEC

Examples of Algorithms Computing the Greatest Common Divisor of Two Integers

gcd(m, n): the largest integer that divides both m and n.
First try -- Euclids algorithm: gcd(m, n) = gcd(n, m mod n) Step1: If n = 0, return the value of m as the answer and stop; otherwise, proceed to Step 2. Step2: Divide m by n and assign the value of the remainder to r. Step 3: Assign the value of n to m and the value of r to n. Go to Step 1.

Copyright Li Zimao @ 2007-2008-1 SCUEC

Methods of Specifying an Algorithm

Natural language
Ambiguous
Mike ate the sandwich on a bed.

Pseudocode
A mixture of a natural language and programming language-like structures Precise and succinct. Pseudocode in this course
omits declarations of variables use indentation to show the scope of such statements as for, if, and while. use for assignment

Copyright Li Zimao @ 2007-2008-1 SCUEC

Pseudocode of Euclids Algorithm

Algorithm Euclid(m, n) //Computes gcd(m, n) by Euclids algorithm //Input: Two nonnegative, not-both-zero integers m and n //Output: Greatest common divisor of m and n while n 0 do r m mod n mn nr return m Questions:
Finiteness: how do we know that Euclids algorithm actually comes to a stop? Definiteness: nonambiguity Effectiveness: effectively computable.
Copyright Li Zimao @ 2007-2008-1 SCUEC

Second Try for gcd(m, n)

Consecutive Integer Algorithm Step1: Assign the value of min{m, n} to t. Step2: Divide m by t. If the remainder of this division is 0, go to Step3;otherwise, go to Step 4. Step3: Divide n by t. If the remainder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to Step4. Step4: Decrease the value of t by 1. Go to Step2. Questions
Finiteness Definiteness Effectiveness Which algorithm is faster, the Euclids or this one?
Copyright Li Zimao @ 2007-2008-1 SCUEC

Third try for gcd(m, n)

Middle-school procedure
Step1: Find the prime factors of m. Step2: Find the prime factors of n. Step3: Identify all the common factors in the two prime expansions found in Step1 and Step2. (If p is a common factor occurring Pm and Pn times in m and n, respectively, it should be repeated in min{Pm, Pn} times.) Step4: Compute the product of all the common factors and return it as the gcd of the numbers given.

Question
Is this a legitimate algorithm?

Copyright Li Zimao @ 2007-2008-1 SCUEC

Sieve of Eratosthenes
Input: Integer n 2 Output: List of primes less than or equal to n for p 2 to n do A[p] p for p 2 to sqrt(n) do if A[p] 0 //p hasnt been previously eliminated from the list j p* p while j n do A[j] 0 //mark element as eliminated jj+p Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Copyright Li Zimao @ 2007-2008-1 SCUEC

What can we learn from the previous 3 examples?

Each step of an algorithm must be unambiguous. The same algorithm can be represented in several different ways. (different pseudocodes) There might exists more than one algorithm for a certain problem. Algorithms for the same problem can be based on very different ideas and can solve the problem with dramatically different speeds.

Copyright Li Zimao @ 2007-2008-1 SCUEC

Fundamentals of Algorithmic Problem Solving

Understanding the problem
Asking questions, do a few examples by hand, think about special cases, etc.

Deciding on
Exact vs. approximate problem solving Appropriate data structure

Design an algorithm Proving correctness Analyzing an algorithm

Time efficiency : how fast the algorithm runs Space efficiency: how much extra memory the algorithm needs.

Coding an algorithm
Copyright Li Zimao @ 2007-2008-1 SCUEC

Algorithm Design and Analysis Process

Copyright Li Zimao @ 2007-2008-1 SCUEC

Two main issues related to algorithms

How to design algorithms

How to analyze algorithm efficiency

Copyright Li Zimao @ 2007-2008-1 SCUEC

Algorithm Design Techniques/Strategies

Brute force Divide and conquer Decrease and conquer Transform and conquer Space and time tradeoffs Greedy approach Dynamic programming Backtracking Branch and bound

Copyright Li Zimao @ 2007-2008-1 SCUEC

Analysis of Algorithms
How good is the algorithm? time efficiency space efficiency Does there exist a better algorithm? lower bounds optimality

Copyright Li Zimao @ 2007-2008-1 SCUEC

Recurrence Relation of Fibonacci Number fib(n):

{0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, }

Copyright Li Zimao @ 2007-2008-1 SCUEC

Algorithm: Fibonacci
Problem: What is fib(200)? What about fib(n), where n is any positive integer?
Algorithm 1 fib(n) if n = 0 then return (0) if n = 1then return (1) return (fib(n 1) + fib(n 2))

Questions that we should ask ourselves. 1. Is the algorithm correct? 2. What is the running time of our algorithm? 3. Can we do better?

Copyright Li Zimao @ 2007-2008-1 SCUEC

Answer of Questions
Is the algorithm correct?
Yes, we simply follow the definition of Fibonacci numbers

How fast is the algorithm?

If we let the run time of fib(n) be T(n), then we can formulate T(200) 2139 The world fastest computer BlueGene/L, which can run 248 instructions per second, will take 291 seconds to compute. (291 seconds = 7.85 1019 years ) Can Mooses law, which predicts that CPU get 1.6 times faster each year, solve our problem? No, because the time needed to compute fib(n) also have the same growth rate
if we can compute fib(100) in exactly a year, then in the next year, we will still spend a year to compute fib(101) if we want to compute fib(200) within a year, we need to wait for 100 years. T(n) = T(n 1) + T(n 2) + 3 1.6n

Copyright Li Zimao @ 2007-2008-1 SCUEC

Improvement
Can we do better?
Yes, because many computations in the previous algorithm arerepeated.

Algorithm 2: fib(n) comment: Initially we create an array A[0: n] A[0] 0,A[1] 1 for i = 2 to n do A[i] = A[i 1] + A[i 2] return (A[n])
Copyright Li Zimao @ 2007-2008-1 SCUEC

Important problem types

sorting searching string processing graph problems combinatorial problems geometric problems numerical problems
Copyright Li Zimao @ 2007-2008-1 SCUEC