Design and Analysis of Computer Algorithm Unit 1: Priya Gupta Asst. Prof. Department of Information Technology

Design and Analysis of Computer
Algorithm
UNIT 1
Priya Gupta
Asst. Prof.
Department of Information Technology
2
Design and Analysis of Thursday,

Computer Algorithm August 1, 2019
What is Algorithm?
• Algorithm
▫ is any well-defined computational procedure that
takes some value, or set of values, as input and
produces some value, or set of values, as output.
▫ is thus a sequence of computational steps that
transform the input into the output.
▫ is a tool for solving a well - specified
computational problem.
▫ Any special method of solving a certain kind of
problem (Webster Dictionary)
3

What is a program?
• A program is the expression of an algorithm in a
programming language
• a set of instructions which the computer will
follow to solve a problem
4
Where We're Going (1/2)

• Learn general approaches to algorithm design

▫ Divide and conquer
▫ Greedy method
▫ Dynamic Programming
▫ Basic Search and Traversal Technique
▫ Graph Theory
▫ Linear Programming
▫ Approximation Algorithm
▫ NP Problem
5

Where We're Going(2/2)

• Examine methods of analyzing algorithm
correctness and efficiency
▫ Recursion equations
▫ Lower bound techniques
▫ O,Omega and Theta notations for best/worst/average case
analysis
• Decide whether some problems have no solution
in reasonable time
▫ List all permutations of n objects (takes n! steps)
▫ Travelling salesman problem
• Investigate memory usage as a different measure
of efficiency
6

Some Application
• Study problems these techniques can be applied
to
▫ sorting
▫ data retrieval
▫ network routing
▫ Games
▫ etc
7

The study of Algorithm

• How to devise algorithms
• How to express algorithms
• How to validate algorithms
• How to analyze algorithms
• How to test a program
8

Importance of Analyze Algorithm

• Need to recognize limitations of various algorithms
for solving a problem
• Need to understand relationship between problem
size and running time
▫ When is a running program not good enough?
• Need to learn how to analyze an algorithm's running
time without coding it
• Need to learn techniques for writing more efficient
code
• Need to recognize bottlenecks in code as well as
which parts of code are easiest to optimize
9

Why do we analyze about them?

• understand their behavior, and (Job -- Selection,
performance, modify)
• improve them. (Research)
10

What do we analyze about them?

• Correctness
▫ Does the input/output relation match algorithm
requirement?
• Amount of work done (aka complexity)
▫ Basic operations to do task
• Amount of space used
▫ Memory used
11

What do we analyze about them?

• Simplicity, clarity
▫ Verification and implementation.
• Optimality
▫ Is it impossible to do better?
12

Complexity
• The complexity of an algorithm is simply the
amount of work the algorithm performs to
complete its task.
13

RAM model
• has one processor
• executes one instruction
at a time
• each instruction takes
"unit time“
• has fixed-size operands,
and
• has fixed size storage
(RAM and disk).
14

What’s more important than performance?
• Modularity • Reliability
• Correctness
• Maintainability
• Functionality
• Robustness
• User-friendliness
• Programmer time
• Simplicity
• Extensibility
15

Why study algorithms and performance?
• Algorithms help us to understand scalability.

• Performance often draws the line between what
is feasible and what is impossible.
• Algorithmic mathematics provides a language
for talking about program behavior.
• Performance is the currency of computing.
• The lessons of program performance generalize
to other computing resources.
• Speed is fun!
Example Of Algorithm
pp 17

What is the running time of this algorithm?
PUZZLE(x)
while x != 1
if x is even
then x = x / 2
else x = 3x + 1
Sample run: 7, 22, 11, 34, 17, 52, 26, 13, 40, 20,
10, 5, 16, 8, 4, 2, 1
18

The Selection Problem (1/2)

• Problem: given a group of n numbers, determine
the kth largest
• Algorithm 1
▫ Store numbers in an array
▫ Sort the array in descending order
▫ Return the number in position k
19

The Selection Problem(2/2)

• Algorithm 2
▫ Store first k numbers in an array
▫ Sort the array in descending order
▫ For each remaining number, if the number is larger than the kth
number, insert the number in the correct position of the array
▫ Return the number in position k
Which algorithm is better?

20

Example: What is an Algorithm?

Problem: Input is a sequence of integers stored in an array.
Output the minimum.
INPUT Algorithm
OUTPUT
instance
m:= a[1];
25, 90, 53, 23, 11, 34 for I:=2 to size of 11
input
if m > a[I] then
m:=a[I];
return s
m
Data-Structure
21

Define Problem
• Problem:
▫ Description of Input-Output relationship
• Algorithm:
▫ A sequence of computational step that transform the
input into the output.
• Data Structure:
▫ An organized method of storing and retrieving data.
• Our task:
▫ Given a problem, design a correct and good
algorithm that solves it.
22

Example Algorithm A
Problem: The input is a sequence of integers stored in array.
Output the minimum.
Algorithm A
23
Example Algorithm B
This algorithm uses two temporary arrays.
1. copy the input a to array t1;

assign n  size of input;
2. While n > 1
For i  1 to n /2
t2[ i ]  min (t1 [ 2*i ], t1[ 2*i + 1] );
copy array t2 to t1;
n n/2;
3. Output t2[1];
24
Visualize Algorithm B
34 6 5 9 20 8 11 7
Loop 1
6 5 8 7
Loop 2
5 7
Loop 3
5
25

Which algorithm is better?

The algorithms are
correct, but which is
the best?
• Measure the running
time (number of
operations needed).
• Measure the amount of
memory used.
• Note that the running
time of the algorithms
increase as the size of the
input increases.
26

What do we need?
Correctness: Whether the algorithm computes
the correct solution for all instances
Efficiency: Resources needed by the algorithm
1. Time: Number of steps.

2. Space: amount of memory used.
Measurement “model”: Worst case, Average case

and Best case.
27
Time vs. Size of Input

Measurement
parameterized by the
size of the input.
4
Td(n)
The algorihtms A,B,C are Tc (n)
implemented and run in
Running time
(second)
a PC. 2
Tb (n)
Algorithms D is
Ta (n)
implemented and run in
a supercomputer.
0
Let Tk( n ) be the 500 1000
amount of time taken by Input Size
the Algorithm
28

Methods of Proof
• Proof by Contradiction
▫ Assume a theorem is false; show that this assumption implies a
property known to be true is false -- therefore original hypothesis
must be true
• Proof by Counterexample
▫ Use a concrete example to show an inequality cannot hold
• Mathematical Induction
▫ Prove a trivial base case, assume true for k, then show hypothesis
is true for k+1
▫ Used to prove recursive algorithms
29

Review: Induction
• Suppose
▫ S(k) is true for fixed constant k
 Often k = 0
▫ S(n) S(n+1) for all n >= k
• Then S(n) is true for all n >= k
30

Proof By Induction
• Claim:S(n) is true for all n >= k
• Basis:
▫ Show formula is true when n = k
• Inductive hypothesis:
▫ Assume formula is true for an arbitrary n
• Step:
▫ Show that formula is then true for n+1
31

Induction Example:
Gaussian Closed Form
• Prove 1 + 2 + 3 + … + n = n(n+1) / 2
▫ Basis:
 If n = 0, then 0 = 0(0+1) / 2
▫ Inductive hypothesis:
 Assume 1 + 2 + 3 + … + n = n(n+1) / 2
▫ Step (show true for n+1):
1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1)
= n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2
= (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2
32

Induction Example:
Geometric Closed Form
• Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a
1
▫ Basis: show that a0 = (a0+1 - 1)/(a - 1)
a0 = 1 = (a1 - 1)/(a - 1)
▫ Inductive hypothesis:
 Assume a0 + a1 + … + an = (an+1 - 1)/(a - 1)
▫ Step (show true for n+1):
a0 + a1 + … + an+1 = a0 + a1 + … + an + an+1
= (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)
33

Induction
• We’ve been using weak induction
• Strong induction also holds
▫ Basis: show S(0)
▫ Hypothesis: assume S(k) holds for arbitrary k <=
n
▫ Step: Show S(n+1) follows
• Another variation:
▫ Basis: show S(0), S(1)
▫ Hypothesis: assume S(n) and S(n+1) are true
▫ Step: show S(n+2) follows
34

Basic Recursion
• Base case: value for which function can be
evaluated without recursion
• Two fundamental rules
▫ Must always have a base case
▫ Each recursive call must be to a case that eventually leads toward
a base case
35

Bad Example of Recursion

Example of non-terminating recursive
program (let n=1)
int bad(unsigned int n)
{
if(n == 0)
return 0;
else
return(bad(n/3 + 1) + n - 1);
}
36

Recursion(1/2)
Problem: write an algorithm that will strip
digits from an integer and print them out one
by one
void print_out(int n)
{
if(n < 10)
print_digit(n); /*outputs single-digit to terminal*/
else {
print_out(n/10); /*print the quotient*/
print_digit(n%10); /*print the remainder*/
}
}
37

Recursion(2/2)
Prove by induction that the recursive printing
program works:
▫ basis: If n has one digit, then program is correct
▫ hypothesis: Print_out works for all numbers of k or fewer
digits
▫ case k+1: k+1 digits can be written as the first k digits
followed by the least significant digit
The number expressed by the first k digits is exactly floor(

n/10 )? which by hypothesis prints correctly; the last digit is
n%10; so the (k+1)-digit is printed correctly
By induction, all numbers are correctly printed
38

Recursion
• Don't need to know how recursion is being managed
• Recursion is expensive in terms of space
requirement; avoid recursion if simple loop will do
• Last two rules

▫ Assume all recursive calls work
▫ Do not duplicate work by solving identical problem in separated recursive
calls
• Evaluate fib(4) -- use a recursion tree
fib(n) = fib(n-1) + fib(n-2)
pp 39

What is Algorithm Analysis?

• How to estimate the time required for an
algorithm
• Techniques that drastically reduce the running
time of an algorithm
• A mathemactical framwork that more rigorously
describes the running time of an algorithm
pp 40

Running time for small inputs

Running time for moderate inputs
Thursday, August 1, 2019 Design and Analysis of Computer Algorithm 41

42

Important Question
• Is it always important to be on the most
preferred curve?
• How much better is one curve than another?
• How do we decide which curve a particular
algorithm lies on?
• How do we design algorithms that avoid being
on the bad curves?
43

Algorithm Analysis(1/5)
• Measures the efficiency of an algorithm or its
implementation as a program as the input size
becomes very large
• We evaluate a new algorithm by comparing its
performance with that of previous approaches
▫ Comparisons are asymtotic analyses of classes of
algorithms
• We usually analyze the time required for an
algorithm and the space required for a
datastructure
44

Algorithm Analysis (2/5)

• Many criteria affect the running time of an
algorithm, including
▫ speed of CPU, bus and peripheral hardware
▫ design think time, programming time and
debugging time
▫ language used and coding efficiency of the
programmer
▫ quality of input (good, bad or average)
45


• Programs derived from two algorithms for
solving the same problem should both be
▫ Machine independent
▫ Language independent
▫ Environment independent (load on the system,...)
▫ Amenable to mathematical study
▫ Realistic
46


• In lieu of some standard benchmark conditions
under which two programs can be run, we estimate
the algorithm's performance based on the number of
key and basic operations it requires to process an
input of a given size
• For a given input size n we express the time T to run
the algorithm as a function T(n)
• Concept of growth rate allows us to compare
running time of two algorithms without writing two
programs and running them on the same computer
47


• Formally, let T(A,L,M) be total run time for
algorithm A if it were implemented with
language L on machine M. Then the complexity
class of algorithm A is
O(T(A,L1,M1) U O(T(A,L2,M2)) U O(T(A,L3,M3)) U ...
• Call the complexity class V; then the complexity
of A is said to be f if V = O(f)
• The class of algorithms to which A belongs is
said to be of at most linear/quadratic/ etc.
growth in best case if the function TA best(n) is
such (the same also for average and worst case).
48

Asymptotic Performance
• In this course, we care most about asymptotic
performance
▫ How does the algorithm behave as the problem
size gets very large?
 Running time
 Memory/storage requirements
 Bandwidth/power requirements/logic gates/etc.
49

Asymptotic Notation
• By now you should have an intuitive feel for
asymptotic (big-O) notation:
▫ What does O(n) running time mean? O(n2)?
O(n lg n)?
▫ How does asymptotic running time relate to
asymptotic memory usage?
• Our first task is to define this notation more
formally and completely
50

Analysis of Algorithms
• Analysis is performed with respect to a
computational model
• We will usually use a generic uniprocessor
random-access machine (RAM)
▫ All memory equally expensive to access
▫ No concurrent operations
▫ All reasonable instructions take unit time
 Except, of course, function calls
▫ Constant word size
 Unless we are explicitly manipulating bits
51

Input Size
• Time and space complexity
▫ This is generally a function of the input size
 E.g., sorting, multiplication
▫ How we characterize input size depends:
 Sorting: number of input items
 Multiplication: total number of bits
 Graph algorithms: number of nodes & edges
 Etc
52

Running Time
• Number of primitive steps that are executed
▫ Except for time of executing a function call most
statements roughly require the same amount of
time
 y=m*x+b
 c = 5 / 9 * (t - 32 )
 z = f(x) + g(y)
• We can be more exact if need be
53

Analysis
• Worst case
▫ Provides an upper bound on running time
▫ An absolute guarantee
• Average case
▫ Provides the expected running time
▫ Very useful, but treat with care: what is “average”?
 Random (equally likely) inputs
 Real-life inputs
54
Function of Growth rate

Divide-and-Conquer
The most-well known algorithm design strategy:
1. Divide instance of problem into two or more
smaller instances
2. Solve smaller instances recursively
3. Obtain solution to original (larger) instance by

combining these solutions
Divide-and-Conquer Technique (cont.)
a problem of size n
(instance)
subproblem 1 subproblem 2
of size n/2 of size n/2
a solution to a solution to
subproblem 1 subproblem 2
a solution to It general leads to a

the original problem recursive algorithm!
• Sorting: mergesort and quicksort
Divide-and-Conquer Examples
• Binary tree traversals
• Binary search (?)
• Multiplication of large integers
• Matrix multiplication: Strassen’s algorithm
• Closest-pair and convex-hull algorithms

General Divide-and-Conquer
Recurrence
T(n) = aT(n/b) + f (n) where f(n)  (nd), d  0
Master Theorem: If a < bd, T(n)  (nd)

If a = bd, T(n)  (nd log n)
If a > bd, T(n)  (nlog b

a)
Note: The same results hold with O instead of .
Examples: T(n) = 4T(n/2) + n  T(n)  ? (n^2)

T(n) = 4T(n/2) + n2  T(n)  ? (n^2log n)
T(n) = 4T(n/2) + n3  T(n)  ?
(n^3)
• Split array A[0..n-1] into about equal halves and
Mergesort
make copies of each half in arrays B and C
• Sort arrays B and C recursively
• Merge sorted arrays B and C into array A as
follows:
▫ Repeat the following until no elements remain in one
of the arrays:
 compare the first elements in the remaining
unprocessed portions of the arrays
 copy the smaller of the two into A, while
incrementing the index indicating the unprocessed
portion of that array
▫ Once all elements in one of the arrays are processed,
copy the remaining unprocessed elements from the
other array into A.
Pseudocode of Mergesort
Pseudocode of Merge
Time complexity: Θ(p+q) = Θ(n) comparisons

Mergesort Example
8 3 2 9 7 1 5 4
8 3 2 9 7 1 5 4
8 3 2 9 71 5 4
8 3 2 9 7 1 5 4 The non-recursive
version of Mergesort
starts from merging
3 8 2 9 1 7 4 5 single elements into
sorted pairs.
2 3 8 9 1 4 5 7
1 2 3 4 5 7 8 9
Analysis of Mergesort
T(n) = 2T(n/2) + Θ(n), T(1) = 0
• All cases have same efficiency: Θ(n log n)
• Number of comparisons in the worst case is close to

theoretical minimum for comparison-based sorting:
log2 n! ≈ n log2 n - 1.44n
• Space requirement: Θ(n) (not in-place)
• Can be implemented without recursion (bottom-up)

• Select a pivot (partitioning element) – here, the first
Quicksort
element
• Rearrange the list so that all the elements in the first s
positions are smaller than or equal to the pivot and all the
elements in the remaining n-s positions are larger than or
equal to the pivot (see next slide for an algorithm)
p
A[i]p A[i]p
• Exchange the pivot with the last element in the first (i.e.,
) subarray — the pivot is now in its final position
• Sort the two subarrays recursively
Partitioning Algorithm
or i > r
or j = l
<
Time complexity: Θ(r-l) comparisons

Quicksort Example
5 3 1 9 2 83 1 2
4 5 8 9 7
4 7
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
Analysis
• Best case:of Quicksort
split in the middle — T(n)
Θ(n= log n)+ Θ(n)
T(n-1)
• Worst case: sorted array! — Θ(n ) 2
• Average case: random arrays — Θ(n log n)
• Improvements:
▫ better pivot selection: median of three partitioning
▫ switch to insertion sort on small subfiles
▫ elimination of recursion
These combine to 20-25% improvement
• Considered the method of choice for internal

sorting of large files (n ≥ 10000)
Very efficient
Binary Search algorithm for searching in sorted
array:
K
vs
A[0] . . . A[m] . . . A[n-1]
If K = A[m], stop (successful search); otherwise,
continue
searching by the same method in A[0..m-1] if K <
A[m]
and in A[m+1..n-1] if K > A[m]
l  0; r  n-1
while l  r do
m  (l+r)/2
if K = A[m] return m
else if K < A[m] r  m-1
else l  m+1
Analysis ofrecurrence:
▫ worst-case Binary CSearch
• Time efficiency
(n) = 1 + C w w( n/2 ), Cw (1) =
1
solution: Cw(n) = log2(n+1)
This is VERY fast: e.g., Cw(106) = 20
• Optimal for searching a sorted array
• Limitations: must be a sorted array (not linked list)
• Bad (degenerate) example of divide-and-conquer

because only one of the sub-instances is solved
• Has a continuous counterpart called bisection method for solving

equations in one unknown f(x) = 0 (see Sec. 12.4)
Binary tree is a divide-and-conquer ready structure!
Binary Tree Algorithms
Ex. 1: Classic traversals (preorder, inorder,
postorder)
Algorithm Inorder(T)
if T   a a
Inorder(Tleft) b c b c
print(root of T) d e • • d
e
Inorder(Tright) ••
Each node is visited/printed once.
••
Binary Tree Algorithms (cont.)
Ex. 2: Computing the height of a binary tree
TL TR
h(T) = max{h(TL), h(TR)} + 1 if T   and h() = -1
Efficiency: Θ(n). Why?

Multiplication of Large Integers
Consider the problem of multiplying two (large) n-
digit integers represented by arrays of their digits
such as:
A = 12345678901357986429 B =
87654321284820912836
The grade-school algorithm:

a1 a2 … an
b1 b2 … bn
(d10) d11d12 … d1n
(d20) d21d22 … d2n
…………………
(dn0) dn1dn2 … dnn
First Divide-and-Conquer Algorithm
A small example: A  B where A = 2135 and B = 4014
A = (21·102 + 35), B = (40 ·102 + 14)
So, A  B = (21 ·102 + 35)  (40 ·102 + 14)
= 21  40 ·104 + (21  14 + 35  40) ·102 + 35  14
In general, if A = A1A2 and B = B1B2 (where A and B

are n-digit,
A1, A2, B1, B2 are n/2-digit numbers),
A  B = A1  B1·10n + (A1  B2 + A2  B1) ·10n/2 + A2  B2
Recurrence for the number of one-digit

multiplications M(n):
M(n) = 4M(n/2), M(1) = 1
Second Divide-and-Conquer Algorithm
A  B = A1  B1·10n + (A1  B2 + A2  B1) ·10n/2 + A2  B2

The idea is to decrease the number of multiplications
from 4 to 3:
(A1 + A2 )  (B1 + B2 ) = A1  B1 + (A1  B2 + A2  B1) +

A2  B2,
I.e., (A1  B2 + A2  B1) = (A1 + A2 )  (B1 + B2 ) - A1  B1 -

A2  B2,
which requires only 3 multiplications at the expense of
(4-1) extra add/sub. What if we count
both
multiplications
Recurrence for the number of multiplications M(n):
and additions?
Example of Large-Integer Multiplication
2135  4014
= (21*10^2 + 35) * (40*10^2 + 14)

= (21*40)*10^4 + c1*10^2 + 35*14
where c1 = (21+35)*(40+14) - 21*40 - 35*14, and
21*40 = (2*10 + 1) * (4*10 + 0)
= (2*4)*10^2 + c2*10 + 1*0
where c2 = (2+1)*(4+0) - 2*4 - 1*0, etc.
This process requires 9 digit multiplications as opposed to 16.

Conventional Matrix Multiplication
• Brute-force algorithm
c00 c01 a00 a01 b00 b01
= *
c10 c11 a10 a11 b10 b11
a00 * b00 + a01 * b10 a00 * b01 + a01 * b11

=
a10 * b00 + a11 * b10 a10 * b01 + a11 * b11
8 multiplications Efficiency class in general:  (n3)

4 additions
• Strassen’s algorithm for two 2x2 matrices (1969):
Strassen’s
c00 c01 Matrix a00 Multiplication
a01 b00 b01
= *
c10 c11 a10 a11 b10 b11
m1 + m4 - m5 + m7 m3 + m5
=
m2 + m4 m1 + m3 - m2 + m6
• m1 = (a00 + a11) * (b00 + b11)
• m2 = (a10 + a11) * b00
• m3 = a00 * (b01 - b11)
• m4 = a11 * (b10 - b00)
• m5 = (a00 + a01) * b11
• m6 = (a10 - a00) * (b00 + b01)
• m7 = (a01 - a11) * (b10 + b11)
7 multiplications
18 additions
Strassen observed [1969] that the product of two
Strassen’s Matrix Multiplication
matrices can be computed in general as follows:
C00 C01 A00 A01 B00 B01

= *
C10 C11 A10 A11 B10 B11
M1 + M4 - M5 + M7 M3 + M5
=
M2 + M4 M1 + M3 - M2 + M6
Formulas for Strassen’s Algorithm
M1 = (A00 + A11)  (B00 + B11)
M2 = (A10 + A11)  B00
M3 = A00  (B01 - B11)
M4 = A11  (B10 - B00)
M5 = (A00 + A01)  B11
M6 = (A10 - A00)  (B00 + B01)
M7 = (A01 - A11)  (B10 + B11)

Analysis of Strassen’s
If n is not a power Algorithm
of 2, matrices can be padded with
zeros. What if we count both
multiplications and
additions?
Number of multiplications:
M(n) = 7M(n/2), M(1) = 1
Solution: M(n) = 7log 2n = nlog 27 ≈ n2.807 vs. n3 of
brute-force alg.
Algorithms with better asymptotic efficiency are

known but they
are even more complex and not used in practice.
81
The Greedy Method
The Greedy Method

82
The Greedy Method
Outline and Reading
• The Greedy Method Technique (§5.1)

• Fractional Knapsack Problem (§5.1.1)
• Task Scheduling (§5.1.2)
• Minimum Spanning Trees (§7.3) [future lecture]
83
The Greedy Method The Greedy Method
Technique
• The greedy method is a general algorithm design
paradigm, built on the following elements:
▫ configurations: different choices, collections, or values to find
▫ objective function: a score assigned to configurations, which
we want to either maximize or minimize
• It works best when applied to problems with the
greedy-choice property:
▫ a globally-optimal solution can always be found by a series of
local improvements from a starting configuration.
84
The Greedy Method
Making Change
• Problem: Accept n dollars, to return a collection of coins with a total
value of n dollars.
• Configuration: A collection of coins with a total value of n
• Objective function: Minimize number of coins returned.
• Greedy solution: Always return the largest coin you can
• Example 1: Coins are valued $.32, $.08, $.01
▫ Has the greedy-choice property, since no amount over $.32 can be made
with a minimum number of coins by omitting a $.32 coin (similarly for
amounts over $.08, but under $.32).
▫ For a certain amount (y) over a coin dimension (x), if we can reach it
using coins (<x) with a total n coins, then we can also use coin x with ≤ n
coins.
• Example 2: Coins are valued $.30, $.20, $.05, $.01
▫ Does not have greedy-choice property, since $.40 is best made with two
$.20’s, but the greedy solution will pick three coins (which ones?)
85
The Fractional Knapsack Problem

The Greedy Method
• Given: A set S of n items, with each item i having

▫ bi - a positive benefit
▫ wi - a positive weight
• Goal: Choose items with maximum total benefit but with
weight at most W.
• If we are allowed to take fractional amounts, then this is
the fractional knapsack problem.
▫ In this case, we let xi denote the amount we take of item i
▫ Objective: maximize
b (x / w )
iS
i i i
x
▫ Constraint:
i W
iS
86
Example
• Given: A set S of n items, with each item i having
▫ bi - a positive benefit
▫ wi - a positive weight
• Goal: Choose items with maximum total benefit but with
weight at most W.
“knapsack”
Solution:
• 1 ml of 5
Items:
1 2 3 4 5
• 2 ml of 3
• 6 ml of 4
Weight: 4 ml 8 ml 2 ml 6 ml 1 ml • 1 ml of 2
Benefit: $12 $32 $40 $30 $50 10 ml
Value: 3 4 20 5 50
The Greedy Method
($ per ml)
87
The Fractional Knapsack Algorithm

The Greedy Method
• Greedy choice: Keep taking

item with highest value Algorithm fractionalKnapsack(S, W)
(benefit to weight ratio) Input: set S of items w/ benefit bi
▫ Use a heap-based priority and weight wi; max. weight W
queue to store the items, then Output: amount xi of each item i
the time complexity is O(n log to maximize benefit with
n). weight at most W
• Correctness: Suppose there for each item i in S
is a better solution xi  0
▫ there is an item i with higher vi  bi / wi {value}
value than a chosen item j (i.e.,
vj<vi) , if we replace some j with w0 {current total weight}
i, we get a better solution while w < W
▫ Thus, there is no better solution remove item i with highest vi
than the greedy one xi  min{wi , W - w}
w  w + min{wi , W - w}
88
The Greedy Method
Task Scheduling
• Given: a set T of n tasks, each having:
▫ A start time, si
▫ A finish time, fi (where si < fi)
• Goal: Perform all the tasks using a minimum number of
“machines.”
Machine 3
Machine 2
Machine 1
1 2 3 4 5 6 7 8 9
89
Task Scheduling Algorithm

The Greedy Method
• Greedy choice: consider tasks by

their start time and use as few
machines as possible with this
order. Algorithm taskSchedule(T)
▫ Run time: O(n log n). Input: set T of tasks w/ start time si
• Correctness: Suppose there is a and finish time fi
better schedule. Output: non-conflicting schedule
▫ We can use k-1 machines with minimum number of machines
▫ The algorithm uses k m0 {no. of machines}
▫ Let i be first task scheduled on while T is not empty
machine k remove task i w/ smallest si
▫ Task i must conflict with k-1
if there’s a machine j for i then
other tasks
▫ K mutually conflict tasks schedule i on machine j
▫ But that means there is no non- else
conflicting schedule using k-1 mm+1
machines schedule i on machine m
90
The Greedy Method
Example
• Given: a set T of n tasks, each having:
▫ A start time, si
▫ A finish time, fi (where si < fi)
▫ [1,4], [1,3], [2,5], [3,7], [4,7], [6,9], [7,8] (ordered by start)
• Goal: Perform all tasks on min. number of machines
Machine 3
Machine 2
Machine 1
1 2 3 4 5 6 7 8 9
Optimal 2-way Merge patterns and Huffman Codes:
Example. Suppose there are 3 sorted lists L1, L2, and L3, of sizes 30, 20,
and 10, respectively, which need to be merged into a combined sorted list,
but we can merge only two at a time. We intend to find an optimal merge
pattern which minimizes the total number of comparisons. For example,
we can merge L1 and L2, which uses 30 + 20 = 50 comparisons resulting in
a list of size 50. We can then merge this list with list L3, using another 50 +
10 = 60 comparisons, so the total number of comparisons is 50 + 60 = 110.
Alternatively, we can merge lists L2 and L3, using 20 + 10 = 30
comparisons, the resulting list (size 30) can then be merged with list L1, for
another 30 + 30 = 60 comparisons. So the total number of comparisons is
30 + 60 = 90. It doesn’t take long to see that this latter merge pattern is
the optimal one.
Binary Merge Trees: We can depict the merge patterns using a binary
tree, built from the leaf nodes (the initial lists) towards the root in which
each merge of two nodes creates a parent node whose size is the sum of the
sizes of the two children. For example, the two previous merge patterns
are depicted in the following two figures:
Cost = 30*2 Cost = 30*1

60 + 20*2 + 60 + 20*2 +
10*1 = 110 10*2 = 90
50 10 30 30
30 20 20 10
Merge L1 and L2, then with Merge L2 and L3, then with
L3 L1
merge cost = sum of all weighted external path lengths
Optimal Binary Merge Tree Algorithm:
Input: n leaf nodes each have an integer size, n  2.
Output: a binary tree with the given leaf nodes which
has a minimum total weighted external path lengths
Algorithm:
(1) create a min-heap T[1..n ] based on the n initial sizes. (2) while
(the heap size  2) do
(2.1) delete from the heap two smallest values, call
them a and b, create a parent node of size a + b
for the nodes corresponding to these two values
(2.2) insert the value (a + b) into the heap which
corresponds to the node created in Step (2.1)
When the algorithm terminates, there is a single value left in the heap
whose corresponding node is the root of the optimal binary merge tree.
The algorithm’s time complexity is O(n lgn) because Step (1) takes O(n)
time; Step (2) runs O(n) iterations, in which each iteration takes O(lgn)
time.
Example of the optimal merge tree algorithm:
2 3 5 7 9 Initially, 5 leaf nodes with

sizes
5
2 3 5 7 9
Iteration 1: merge 2 and 3
into 5
10
Iteration 2:
5 5 16 Iteration 3: merge 7
merge 5 and
and 9 (chosen among
5 into 10
2 3 7 9 7, 9, and 10) into 16
2
6 Iteration 4: merge
16
10 10 and 16 into 26
5 5 7 9
Cost = 2*3 + 3*3 + 5*2 +
2 3 7*2 + 9*2 = 57.
Proof of optimality of the binary merge tree algorithm:
We use induction on n  2 to show that the binary merge tree is optimal in that
it gives the minimum total weighted external path lengths (among all possible
ways to merge the given leaf nodes into a binary tree).
(Basis) When n = 2. There is only one way to merge two
nodes. (Induction Hypothesis) Suppose the merge tree is
optimal when there are k leaf nodes, for some k  2.
(Induction) Consider (k + 1) leaf nodes. Call them
a1, a2, …, and ak+1. We may assume nodes a1, a2 are of the smallest values,
which are merged in the first step of the merge algorithm into node b. We call
the merge tree T, the part excluding a1, a2 T’ (see figure). Suppose an optimal
binary merge tree is S. We make two observations. (1) If node x of S is a
deepest internal node, we may swap its two children with nodes a1, a2 in S
without increasing the total weighted external path lengths. Thus, we may
assume tree S has a subtree S’ with leaf nodes x, a2, …, and ak+1.
(2) The tree S’ must be an optimal merge
tree for k nodes x, a2, …, and ak+1. By
induction hypothesis, tree S’ has a T S
total weighted external path lengths equal T’ S’
b x
to that of tree T’. Therefore, the total
weighted external path lengths of T equalsato 1 a a1 a
that of tree S, proving the optimality of T.
2 2
Huffman Codes:
Suppose we wish to save a text (ASCII) file on the disk or to transmit it
though a network using an encoding scheme that minimizes the number of
bits required. Without compression, characters are typically encoded by
their ASCII codes with 8 bits per character. We can do better if we have the
freedom to design our own encoding.
Example. Given a text file that uses only 5 different letters (a, e, i, s, t), the
space character, and the newline character. Since there are 7 different
characters, we could use 3 bits per character because that allows 8 bit
patterns ranging from 000 through 111 (so we still one pattern to spare).
The following table shows the encoding of characters, their frequencies, and
the size of encoded (compressed) file.
Character Frequency Code Total bits Code Total bits
a 10 000 001 30
30 e 15 01 30
001 45 i 10 24
12 010 36 00000 15
s 3 011 9 0001 16 11
t 4 100 12 26
space 13 101 39 00001 5
newline 1 110 3
146
Total 58 174
Fixed-length encoding Variable-length encoding
If we can use variable lengths for the codes, we can actually compress
more as shown in the above. However, the codes must satisfy the
property that no code is the prefix of another code; such code is called a
prefix code.
How to design an optimal prefix code (i.e., with minimum total
length) for a given file?
We can depict the codes for the given collection of characters using a
binary tree as follows: reading each code from left to right, we construct a
binary tree from the root following the left branch when encountering a
‘0’, right branch when encountering a ‘1’. We do this for all the codes by
constructing a single combined binary tree. For example,
0 1
0 0
0 0 1 0 1 0
1 1 1
Codes 001, 01, 10,
00000, 0001, 11, and
00001
Code Codes 001 Codes Note: each code terminates at
001 and 01 001, 01, a leaf node, by the prefix
and 10 property.
We note that the encoded file size is equal to the total weighted external
path lengths if we assign the frequency to each leaf node. For example,
‘e’
15 12 13 Total file size = 3*5 + 1*5 + 4*4 + 10*3 +
10 ‘i’ ‘’ 15*2 + 12*2 + 13*2 = 146, which is exactly
‘a’ the total weighted external path lengths.
4
‘t’
3 1
‘s’ ‘\n’
We also note that in an optimal prefix
code, each node in the tree has either x x
no children or has two. Thus, the
optimal binary merge tree algorithm y
finds the optimal code (Huffman code).
Merge x and y,
Node x has reducing total
only one child size
y
Greedy Strategies Applied to Graph problems:
We first review some notations and terms about graphs. A graph
consists of vertices (nodes) and edges (arcs, links), in which each edge
“connects” two vertices (not necessarily distinct). More formally, a
graph G = (V, E), where V and E denote the sets of vertices and edges,
respectively.
In this example, V = {1, 2, 3, 4}, E = {a,

1
a b b, c, d, e}. Edges c and d are parallel
edges; edge e is a self-loop. A path is a
2 3 sequence of “adjacent” edges, e.g., path
d abeb, path acdab.
c
e
4
Directed graphs vs. (un-directed) graphs:
If every edge has an orientation, e.g., an edge starting from node x
terminating at node y, the graph is called a directed graph, or digraph for
short. If all edges have no orientation, the graph is called an undirected
graph, or simply, a graph. When there are no parallel edges (two edges
that have identical end points), we could identify an edge with its two end
points, such as edge (1,2), or edge (3,3). In an undirected graph, edge (1,2)
is the same as edge (2,1). We will assume no parallel edges unless
otherwise stated.
1 A directed graph. Edges c and d are

a b
parallel (directed) edges. Some directed
2 3 paths are ad, ebac.
d
c
e
4
Both directed and undirected graphs appear often and naturally in many
scientific (call graphs in program analysis), business (query trees, entity-
relation diagrams in databases), and engineering (CAD design) applications.
The simplest data structure for representing graphs and digraphs is using 2-
dimensional arrays. Suppose G = (V, E), and |V| = n. Declare an array
T[1..n][1..n] so that T[i][j] = 1 if there is an edge (i, j)  E; 0 otherwise. (Note
that in an undirected graph, edges (i, j) and (j, i) refer to the same edge.)
j
1 2 3 4
1 A 2-dimensional array
0 1 0 0
1 0 0 0 1 for the digraph, called
2 3 i 2   the adjacency matrix.
3 1 0 1 0
 
4 0 1 0 0
4
Sometimes, edges of a graph or digraph are given a positive weight or
cost value. In that case, the adjacency matrix can easily modified so
that T[i][j] = the weight of edge (i, j); 0 if there is no edge (i, j). Since
the adjacency matrix may contain many zeros (when the graph has
few edges, known as sparse), a space-efficient representation uses
linked lists representing the edges, known as the adjacency list
representation.
1 2
1
2 4
2 3 3 3 1
4 2
4 The adjacency lists for the digraph, which can
store edge weights by adding another field in the
list nodes.
The Minimum Spanning Tree (MST) Problem:
Given a weighted (undirected) graph G = (V, E), where each edge e has a
positive weight w(e). A spanning tree of G is a tree (connected graph
without cycles, or circuits) which has V as its vertex set, i.e., the tree
connects all vertices of the graph G. If |V| = n, then the tree has n – 1
edges (this is a fact which can be proved by induction). A minimum
spanning tree of G is a spanning tree that has the minimum total edge
weight.
1 1
3 6 3 6
8 A minimum spanning
2 3 2 3 tree (of 4 edges),
5 4 4 weight = 3 + 2 + 4 + 6
7 = 15.
5 4 5 4
2 2
A weighted graph of no parallel
edges or self-loops
Prim’s Algorithm for the Minimum Spanning Tree problem:
Create an array B[1..n] to store the nodes of the MST, and an array T[1..n –1]
to store the edges of the MST. Starting with node 1 (actually, any node can be
the starting node), put node 1 in B[1], find a node that is the closest (i.e., an
edge connected to node 1 that has the minimum weight, ties broken
arbitrarily). Put this node as B[2], and the edge as T[1]. Next look for a node
connected from either B[1] or B[2] that is the closest, store the node as B[3],
and the corresponding edge as T[2]. In general, in the kth iteration, look for a
node not already in B[1..k] that is the closest to any node in B[1..k]. Put this
node as B[k+1], the corresponding edge as T[k]. Repeat this process for n –1
iterations (k = 1 to n –1). This is a greedy strategy because in each iteration,
the algorithm looks for the minimum weight edge to include next while
maintaining the tree property (i.e., avoiding cycles). At the end there are
exactly n –1 edges without cycles, which must be a spanning tree.
Example: Prim’s MST Algorithm.
Step Next edge selected Partial tree

3 1
Initially 1
8 6
2 3 3 1
5 4 1 (1,5), weight=3
7 5
5 4
2 1
2 (5,4),
A weighted graph weight=2 5 4
2
1
3 (4,2), weight=4 5 2
4
4
1
6
4 (1,3), weight=6 5 2
3
4
An adjacency matrix implementation of Prim’s algorithm:
Input: W[1..n][1..n] with W[i, j] = weight of edge (i, j); set W[i, j] =  if
no edge Output: an MST with tree edges stored in T[1..n –1]
Algorithm:
(1) declare nearest[2..n], minDist[2..n] such that minDistt[i] = the
minimum edge weight connecting node i to any node in partial tree T, and
nearest[i]=the node in T that gives minimum distance for node i.
(2) for i = 2 to n do
nearest[i]=1; minDist[i]=W[i, 1]
(3) for p = 1 to (n –1) do
(3.1) min =  Tree T nearest[i
(3.2) for j = 2 to n do ]
if 0  minDist[j] < min then i
min = minDist[j]; k = j
(3.3) T[p] = edge (nearest[k], k) minDist[i
// selected
the nest edge (3.4) minDist[k] = –1 //] a negative
value means node k is “in” (3.5) for j = 2 to n do //
update minDist and nearest values if W[j, k]
< minDist[j] then
minDist[j] = W[j, k]; nearest[j] = k
The time complexity is O(n2) because Step (3) runs O(n) iterations, each
iteration runs O(n) time in Steps (3.2) and (3.5).

Design and Analysis of Computer Algorithm Unit 1: Priya Gupta Asst. Prof. Department of Information Technology

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Design and Analysis of Computer Algorithm Unit 1: Priya Gupta Asst. Prof. Department of Information Technology

Transféré par

Droits d'auteur :

Formats disponibles

Design and Analysis of Computer

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Where We're Going (1/2)

• Learn general approaches to algorithm design

Design and Analysis of Thursday,

Where We're Going(2/2)

Design and Analysis of Thursday,

Design and Analysis of Thursday,

The study of Algorithm

Design and Analysis of Thursday,

Importance of Analyze Algorithm

Design and Analysis of Thursday,

Why do we analyze about them?

Design and Analysis of Thursday,

What do we analyze about them?

Design and Analysis of Thursday,

What do we analyze about them?

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

What’s more important than performance?

Design and Analysis of Thursday,

Why study algorithms and performance?

• Algorithms help us to understand scalability.

Design and Analysis of Thursday,

What is the running time of this algorithm?

Design and Analysis of Thursday,

The Selection Problem (1/2)

Design and Analysis of Thursday,

The Selection Problem(2/2)

Which algorithm is better?

Design and Analysis of Thursday,

Example: What is an Algorithm?

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

This algorithm uses two temporary arrays.

1. copy the input a to array t1;

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Which algorithm is better?

Design and Analysis of Thursday,

Efficiency: Resources needed by the algorithm

1. Time: Number of steps.

Measurement “model”: Worst case, Average case

Design and Analysis of Thursday,

Time vs. Size of Input

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Design and Analysis of Thursday,

Bad Example of Recursion

Design and Analysis of Thursday,

Design and Analysis of Thursday,

The number expressed by the first k digits is exactly floor(

Design and Analysis of Thursday,

= (2110^2 + 35) (40*10^2 + 14)