mlbaker presents

lambertw.wordpress.com)
CS 341
Algorithms 1
Instructor: Timothy Chan
Term: Spring 2012 (1125)
University of Waterloo
July 23, 2012
Disclaimer: These notes are provided as-is, and may be incomplete or contain errors.
Contents
1 Introduction 2
1.1 Bentleys problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 3SUM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Math review 7
2.1 Asymptotic notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Algorithm design technique 1: divide and conquer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Closest pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.2 Multiplying large numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.3 Matrix multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.4 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Algorithm design technique 2: greedy algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.1 Disjoint intervals (activity selection) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.2 Fractional Knapsack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.3 Stable marriage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4 Algorithm design technique 3: dynamic programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.1 Longest common subsequence (LCS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.2 Min-length triangulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5 Topological Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6 Strongly Connected Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1
Tuesday, May 1
Administrative
The course website is www.student.cs.uwaterloo.ca/~cs341.
Email: tmchan@cs.uwaterloo.ca.
Oce: DC 2107, x36941.
Oce hours: Tuesday 4:005:00, Friday 1:002:30.
Textbook: CLRS (3rd edition).
Course work: 5 assignments (30%), a midterm (20%), and a nal exam (50%).
Midterm: Wednesday June 13, 79pm.
No late assignments will be accepted.
1 Introduction
This course is about problem solving. You are usually handed a problem, and then in order to solve it you write a
program. However, the point is to think before simply writing the program. So we have
problem design algorithm analyze algorithm program.
At each step, after doing the (mathematical) analysis we check again whether we could have done better. Sometimes
it is possible that a problem is inherently dicult, so nding a good algorithm may be impossible. In this case, we
attempt to prove a lower bound on the complexity of the problem. This is considerably harder, since in this case we
are trying to prove a property that holds for all possible algorithms that solve the problem. Also, if we successfully
prove a lower bound, we sometimes go back and change the problem itself (perhaps we only, in practice, need to solve
a special case of the problem). In this course we will deal always with problems which have already been formulated
in a mathematically precise fashion.
1.1 Bentleys problem
Given numbers a
1
, . . . , a
n
, nd a block, that is, a (contiguous) subsequence a
i
, a
i+1
, . . . , a
j
with the largest sum.
Example 1.1. Consider the input
1, 6, 3, 1, 4, 2, 3, 2.
The optimal block is underlined. The sum of this block is 3 1 + 4 + 2 = 8.
If we think of the numbers as representing changes, one possible (bogus
1
) application of this is determining when to
buy and sell a stock in order to maximize prot. Another application is found in bioinformatics, where the numbers
can be thought of as representing information about DNA, and we want to maximize some given quantity.
We now attempt to design an algorithm to solve this problem.
1
This is bogus simply because when buying and selling stocks, one has no information about what will happen in the future.
2
Algorithm 0 brute force
The rst algorithm we show is the naive (brute force) method. Here, we literally simply compute the sum of every
block and return the highest one.
1 ans = 0
2 for i = 1 to n do
3 for j = i to n do
4 sum = 0
5 for k = i to j do
6 sum = sum + a_k
7 if sum > ans then ans = sum
8 return ans
We now perform the analysis, that is, we ask: How fast is it? Lines 56 take time
c(j i + 1)
for some constant c which depends on the machine/computer. However c(j i +1) cn, so lines 56 run in what we
call linear time. Now lines 37 cost
(cn +c

)(n i + 1) cn
2
+c

n.
So for the whole algorithm: note that line 1 takes c

and lines 27 take (cn

2
+c

n) per iteration and that loop has n

iterations, so we have
c

+ (cn
2
+c

n)n = cn
3
+c

n
2
+c

O(n
3
).
This is a polynomial time algorithm, so its not too bad in the bigger scheme of things. However, this running time
still leaves something to be desired. To get some perspective, suppose the input size is n = 10
6
. Then n
3
= 10
18
. If
we can do 10
9
operations per second, then this means 10
9
seconds will be required, in other words, 3 years! So O(n
3
)
is not ideal, however, it is still better than something like exponential time O(2
n
). For example if we take n = 1000,
we see that
2
n
10
300
.
When the computation would take that long, we might as well just forget about it, so exponential time is horrible.
Algorithm 1 keep the previous sum
In this version of the algorithm we use the previous sums to help us compute the current sum, instead of computing
it from scratch each time.
1 ans = 0
2 for i = 1 to n do {
3 sum = 0
4 for j = i to n do
5 sum = sum + a_j
6 if sum > ans then ans = sum
7 return ans
If we do a similar analysis, we see that this algorithm runs in O(n
2
).
3
Algorithm 2 divide and conquer
We divide the input into two equally sized chunks. There are two cases: either the optimal solution is made up of an
optimal solution from the left and an optimal solution from the right, or it straddles the middle line between the two
chunks, in which case we must do more work.
To handle the latter situation, we want to compute the best sux of the rst chunk, and the best prex of the second
chunk, and put them together. This gives us the best block (of the latter kind).
1 solve(a_1 , ..., a_n):
2 if n = 0 return 0
3 if n = 1 return max { a_1 , 0 }
4 ans = max{ solve(a_1 , ..., a_{floor(n/2)}), solve(a_{floor(n/2)+1}, ..., a_n) }
5 ans_L = sum = 0
6 for i = floor(n/2) to 1 do {
7 sum = sum + a_i
8 if sum > ans_L then ans_L = sum
9 }
10 ans_R = sum = 0
11 for i = floor(n/2) + 1 to n do {
12 sum = sum + a_i
13 if sum > ans_R then ans_R = sum
14 }
15 return ans = max { ans , ans_L + ans_R }
Consider again our example:
1,
4
..
6, 3,
1
..
1
. .
2
. .
3
,

3
..
4
..
4 , 2
. .
6
, 3, 2
. .
5
So the optimal block is 3, 1, 4, 2 which has a sum of 8 (we maximized the left and right sides).
Analysis: we write a recurrence, because the algorithm calls itself. For simplicity, say n is even. If we denote the
running time by T, we observe that
T(n) =
_

_
O(1) if n 1
2 T(
n
2
)
. .
line 4
+ O(n)
. .
lines 614
if n > 1.
This is just the mergesort recurrence. So the time is O(nlog n).
Algorithm 3 dynamic programming
We want to solve an optimization problem, i.e. solve intermediate subproblems. Original problem asks us to nd the
best overall block. The idea is to dene subproblems that concentrate on blocks of certain forms. Given an index j,
let b
j
be the maximum sum over all blocks that end at index j. Going back to our old example, we had
1, 6, 3, 1, 4
. .
6
, 2, 3, 2.
4
For j = 5 we can see that b
5
= 6. If we simply compute every b
j
(which takes O(n) time) we note there are n possible
choices of j, making for a running time overall of O(n
2
). If we do this, then, we are not making any progress. However,
the idea is to nd a way to use the values b
k
(for k < j), i.e. the previous values, to help you compute b
j
. Find a
formula to compute b
j
from b
j1
. The nal answer will be given by
max
j
b
j
.
Thursday, May 3
Note that any block ending at a
j
can be decomposed into a block ending at a
j1
, and adding in a
j
. However, this
initial block could be empty. So in fact, the correct formula is
b
j
= maxb
j1
+a
j
, a
j
= maxb
j1
, 0 +a
j
.
If we trace through the algorithm on the example, we get
b
1
= 1, b
2
= 5, b
3
= 3, b
4
= 2, b
5
= 6, b
6
= 8, b
7
= 5, b
8
= 7.
1 ans = 0, b = 0
2 for j = 1 to n do {
3 b = max{b,0} + a_j
4 if b > ans then ans = b
5 }
Analysis: theres a single for loop, so its obviously O(n). However, this is the best possible, that is we can prove an
(n) lower bound. This is because you have to read all n input items in order to have a correct algorithm roughly
speaking, if there is some number you havent read, that number could be innity and this would change the nal
answer.
Remark 1.2. In general, analyze growth rate of runtime as a function of n (input size). In most algorithms the
runtime depends not only on the input size but on the input itself.
To get a function that depends on n alone, we just take the maximum over all inputs of size n. This is called worst-case
analysis.
Disclaimer: This model is not perfect in practice, constant factors do matter. Also, worst-case analysis is overly
pessimistic.
1.2 3SUM
Given numbers a
1
, . . . , a
n
, target t, do there exist indices i, j, k, not necessarily distinct, with a
i
+a
j
+a
k
= t?
Example 1.3. Consider the input
30, 12, 8, 37, 33, 82
and target value t = 50. Output: yes.
Algorithm 0 brute force
5
1 for i = 1 to n
2 for j = 1 to n
3 for k = 1 to n
4 if a_i + a_j + a_k = t
5 return yes
6 return no
This clearly takes O(n
3
).
Algorithm 1
1 let A = { a_i + a_j | i,j = 1, ..., n }
2 let B = { t - a_k | k = 1, ..., n }
3 if A and B have a common element
4 return yes
5 else return no
Note that set A has O(n
2
) elements and set B has O(n) elements. Line 1 requires O(n
2
) time. Line 2 requires O(n)
time. To do line 3, the idea is to sort A B. Then do a scan. We see that this takes
O(n
2
log(n
2
)) = O(2n
2
log n) = O(n
2
log n).
So the whole algorithm runs in total time O(n
2
log n).
Note that A is not just any set it was constructed from an input of size n. One question we could ask is as follows.
Can we sort A B, well, in particular, sort the set
A = a
i
+a
j
: i, j = 1, . . . , n
in time O(n
2
)? This is an open problem, actually.
Algorithm 2
1 let B = { t - a_k | k = 1, ..., n }
2 for i = 1 to n do {
3 A_i = { a_i + a_j | j = 1, ..., n }
4 if A_i and B have a common element
5 yes
6 }
7 return no
Analysis: Line 1 takes O(n) time. Line 3 takes O(n) time. Line 4: Sort A
i
(O(nlog n)) and sort B (O(nlog n))
separately, then merge two sorted lists (O(n)), and then do a scan (O(n)). Now, note that we can move the step of
sorting B out of the loop we dont need to sort it at each iteration since it remains xed. Also, we dont need to
sort each A
i
. The idea is to pre-sort
a
1
a
2
. . . a
n
.
6
Then A
i
is already sorted:
a
i
+a
1
a
i
+a
2
. . . a
i
+a
n
.
This reduces the total running time from O(n
2
log n) down to
O(nlog n +n n) = O(n
2
).
Can we do better? Nobody knows. In the case of the integers, the complexity is known to be about
O
_
n
2
log
2
n
_
.
2 Math review
2.1 Asymptotic notation
(tight bound), O (upper bound), (lower bound), o (loose upper bound), (loose lower bound).
Denition 2.1. For f, g : N R, we say f(n) O(g(n)) if there are constants c > 0 and n
0
such that
f(n) cg(n)
for all n n
0
.
We say f(n) (g(n)) if there are constants c > 0 and n
0
such that
f(n) cg(n)
for all n n
0
.
We say f(n) (g(n)) if f(n) O(g(n)) and f(n) (g(n)).
Example 2.2. Let f(n) = 2n
2
+ 8n 10. f(n) O(n
2
) because
2n
2
+ 8n 10 2n
2
+ 8n 10n
2
for all n 1. Set c = 10, n
0
= 1.
Also, f (n
2
), because
2n
2
+ 8n 10 2n
2
10 n
2
for all n 10, i.e. n

10. Hence f(n) (n

2
).
Tuesday, May 8
Denition 2.3. f(n) o(g(n)) if for all
2
c > 0 there is n
0
such that for all n n
0
,
f(n) cg(n).
Similarly, f(n) (g(n)) if for all c > 0 there is n
0
such that for all n n
0
,
f(n) cg(n).
Example 2.4. 3n + 2 o(n
2
) because
3n + 2 5n cn
2
as long as n 1 and n 5/c. Set
n
0
= max1, 5/c.
2
In the denition of little o, think of c as arbitrarily small; in the denition of little , think of c as arbitrarily large.
7
Proposition 2.5 (connection with limits). f(n) o(g(n)) if and only if
lim
n
f(n)
g(n)
= 0.
f(n) (g(n)) if and only if
lim
n
f(n)
g(n)
= .
f(n) O(g(n)) if and only if
limsup
n
f(n)
g(n)
< .
Example 2.6. To show 2n
2
+ 8n 10 (n
2
), we just note
lim
n
2n
2
+ 8n 10
n
2
= lim
n
(2 +
8
n

10
n
2
) = 2.
Example 2.7. nlog n o(n
2
), because by LHopitals Rule,
lim
n
nlog n
n
2
= lim
n
log n
n
= lim
n
1
nln2
= 0.
Example 2.8. 2
n
(n
2
) because
lim
n
2
n
n
2
= lim
n
2
n
ln2
2n
= lim
n
2
n
(ln2)
2
2
= .
Example 2.9. We have the following examples: (choose from , o, )
1. Compare 1.01
n
against n
100
: .
2. Compare log
100
n against n
0.01
: o.
3. Compare n
3
+ 2
n
/10 against n
2
+ 3
n
/100: o.
4. Compare 2
n
against 2
n+1
: .
5. Compare n! against (n + 1)!: o.
6. Compare log
5
n against log
6
n: .
7. Compare 2
log
5
n
against 2
log
6
n
: .
8. Compare n against n
1+sin n
: not comparable.
Example 2.10.
1, . . . , log log n, log n, . . . , log
100
n, n
0.001
,

n, n, nlog n, n
2
, n
3
, . . . , 1.01
n
, . . . , 100
n
, . . . , n!, n
n
, 2
2
n
, . . .
There are other examples like log(n!), or 2

log n
. For example to compare 2

log n
with n, rewrite n = 2
log n
and then
compare the exponents. By the same logic we can rewrite

n = 2
1
2
log n
and do a similar comparison.
Remark 2.11. Asymptotic notation may be used in equations, and is read left to right. For example,
n
2
= O(n
3
) = O(n
4
).
Another example is
2n
3
+ 3n + 4 = 2n
3
+O(n) = 2n
3
+o(n
3
) = (n
3
).
n
O(1)
is shorthand for polynomial running time. So we could have a statement like n
O(1)
= o(2
n
).
8
Proposition 2.12 (summations). We have:
1.
n

i=1
i
d
= (n
d+1
) for any constant d 0.
2.
n

i=1
c
i
=
c
n+1
1
c 1
= (c
n
) for any constant c > 1. For c < 1, its (1).
3. The harmonic numbers,
n

i=1
1
i
= lnn +O(1).
To prove the upper bound,
n

i=1
1
i

_
n
1
1
x
dx + 1.
4. log(n!) =
n

i=1
log i = nlog n (n). Stirling formula.
Example 2.13. Consider
1 for i = 1 to n do {
2 for j = 1 to n do ...
3 ...
4 for j = 1 to i do
5 for k = 1 to i do
6 ...
7 }
Analysis: lines 2-5 we have (n + 1 +i
2
) = (n +i
2
). Total time

_
n

i=1
[n +i
2
]
_
=
_
n n +
n

i=1
i
2
_
= (n
2
+n
3
) = (n
3
).
Example 2.14 (build-heap). Consider
1 for i = n down to 1 do {
2 j = i
3 while j <= n do {
4 ...
5 j = 2j
6 }
7 }
Analysis: consider while loop. It has log(n/i) iterations. After kth iteration, j = 2
k
i. If we let t be the number of
iterations,
2
t1
i n < 2
t+1
i
thus
log
n
i
< t log
n
i
+ 1
hence t = (log
n
i
).
9
Thursday, May 10
Lines 35 are (log
n
i
). For the total time, we obtain the rough upper bound
O(nlog n).
Note that
total time =
_
n

i=1
log
n
i
_
=
_
n

i=1
(log n log i)
_
=
_
nlog n
n

i=1
log i
. .
nlog n(n)
_
= (n)
We now continue with more math review: solving recurrences.
Example 2.15.
T(n) =
_
2T(
n
2
) +n if n > 1
1 if n = 1.
How do we get a general technique to solve dierent kinds of recurrences? That is, we want to obtain something like
T(n) = (nlog n)
or alternatively, we could consider the example
T(n) =
_
2T(
n
2
) +n
2
if n > 1
1 if n = 1.
There are 3 methods.
1. Recursion tree method
expand for k iterations to get a tree of items
set k to reach base case
sum across rows, then levels
Example 2.16. Revisiting our example above
T(n) =
_
2T(
n
2
) +n
2
if n > 1
1 if n = 1.
[diagrams Im currently too lazy to include, etc.]
Tuesday, May 15
Solving recurrences:
1. recursion tree
2. master method
3. guess-and-check (substitution method)
guess form of solution
T(n) . . .
verify by induction
10
ll in constants
Example 2.17. We have
T(n) =
_
2T(n/2) +n
2
if n > 1
1 if n = 1.
Guess T(n) cn
2
. Verify by induction. Base case n = 1:
T(n) = 1 c 1
2
as long as c 1. Induction step: assume T(n/2) c(n/2)
2
. Then
T(n) = 2T(n/2) +n
2
2c(n/2)
2
+n
2
= (
c
2
+ 1)n
2
cn
2
as long as
c
2
+ 1 c, i.e.
c
2
+ 1 c = 1
c
2
= c 2.
So as long as c 2.
Pick c = 2. Conclude T(n) 2n
2
= O(n
2
).
It is clear from the denition of T(n) that it is (n
2
) since it contains a +n
2
term. Therefore we conclude (n
2
).
Remark 2.18. Dont use big O notation inside an induction proof and make sure you get the same constant c as you
started o with.
Example 2.19. Consider
T(n) =
_
3T(
_
n
2
_
) + 4T(
_
n
4
_
) + 1 if n > 4
1 if n 4.
Guess T(n) cn
2
. Verify by induction. Base case n 4:
T(n) = 1 cn
2
as long as c 1. Induction step: assume
T(n/2|) cn/2|
2
, T(n/4|) cn/4|
2
then
T(n) = 3T(n/2|) + 4T(n/4|) + 1
3[cn/2|
2
] + 4[cn/4|
2
] + 1
3(c(n/2)
2
) + 4(c(n/4)
2
) + 1
= (3c/4 +c/4)n
2
+ 1
= cn
2
+ 1.
We are o because of the +1 term.
Counter-intuitive part: try to prove
T(n) cn
2
c

instead, where c

is another constant.
11
Verify by induction. Base case n 4: T(n) = 1 cn
2
c

as long as c 1 +c

. Induction step: assume

T(n/2|) cn/2|
2
c

, T(n/4|) cn/4|
2
c

thus
T(n) = 3T(n/2|) + 4T(n/4|) + 1
3[cn/2|
2
c

] + 4[cn/4|
2
c

] + 1
3(c(n/2)
2
c

) + 4(c(n/4)
2
c

) + 1
= (3c/4 +c/4)n
2
+ 1 7c

cn
2
c

as long as 1 7c

i.e. 6c

1, i.e. c

1/6. So pick c

=
1
6
, c =
7
6
. Then we have shown
T(n)
7
6
n
2

1
6
= O(n
2
).
bound is similar.
End of math review.
2.2 Algorithm design technique 1: divide and conquer
divide into subproblems of the same type
recurse
combine
2.2.1 Closest pair
Given a set P of n points in 2D [bunch of points drawn on board],
p
1
= (x
1
, y
1
), . . . , p
n
= (x
n
, y
n
)
nd a pair p, q P (p ,= q) with smallest distance
d(p, q) =
_
(p.x q.x)
2
+ (p.y q.y)
2
Brute force algorithm
1 delta = infinity
2 for each p in P
3 for each q in P
4 if p =/= q
5 if d(p,q) < delta
6 delta = d(p,q)
7 return delta
Consider the 1-dimensional problem. We can solve this in O(nlog n) by sorting and scanning.
2D: ??? Idea 0: divide-and-conquer. divide vertically.
12
Shamos Algorithm (1975)
The pseudocode begins as follows:
1 ClosestPoint(P):
2 if n <= 3 ...
3 x_m = median x-coordinate
4 P_L = { p in P : p.x <= x_m }
5 P_R = { p in P : p.x > x_m }
6 delta_L = ClosestPair( P_L )
7 delta_R = ClosestPair( P_R )
8 delta = min( delta_L , delta_R )
Case 1: closest pair is on the left. Case 2: closest pair is on the right. Case 3: closest pair straddles the middle line.
Idea 1: Only need to check pairs inside strip of width 2 [see camera 1].
x
m
x x
m
+.
Idea 2: Only need to check pairs inside some rectangle of height , width 2 [see notebook labelled second pic].
x
m
x x
m
+, t y t +
Thursday, May 17
Proposition 2.20. Each rectangle contains 8 points.
(We are going to scan by moving the rectangle from bottom to top).
Proof. [diagram of rectangle]. The left square contains 4 points because the closest pair distance of P
L
=
L
.
Worst case: 4 points, each placed on a corner of the square. Similarly for the right square.
Now we continue the pseudocode: [see notebook, numbering starts at line 8].
Analysis: denote by T(n) the runtime of ClosestPair for [P[ = n.
By the claim, the number of iterations of the for loop at line 10 is 8 1 = 7 (we do not have to check the
point against itself, hence the 1).
Lines 9-11 has time O(n 7) = O(n).
Line 8 requires sorting on the y coordinates, so O(nlog n) time.
Line 5-6: 2T(n/2) time.
Line 2-4: O(nlog n).
The recurrence we obtain is therefore
T(n) =
_
2T(
n
2
) +O(nlog n) if n > 4
(1) otherwise.
The masters theorem doesnt work because nlog n doesnt t into any of the cases. So we use the recursion tree
method. [see notebook]. Therefore the total

_
n
log n

i=1
i
_
= (nlog
2
n).
13
Renement: assume we have pre-sorted P in x and in y. Then line 8 will take O(n) time. Also line 2 takes O(n) time.
P
L
, P
R
would be sorted as well.
T(n) = 2T(n/2) + (n) = (nlog n) + (nlog n) for initial pre-sorting.
So the total time is (nlog n).
2.2.2 Multiplying large numbers
Given two n-bit numbers in binary,
A = a
n1
a
n2
a
0
B = b
n1
b
n2
b
0
we wish to compute
AB = c
2n1
c
2n2
c
0
.
Elementary school algorithm
We have to do n shifts and n additions. Each of these costs O(n) so the total time of this school algorithm is
(n
2
).
Karatsuba-Ofman algorithm (1962)
First approach: divide
A =
A

..
a
n1
a
n/2

..
a
n/21
a
0
, B =
B

..
b
n1
b
n/2

..
b
n/21
b
0
numerically we have
A = 2
n/2
A

+A

, B = 2
n/2
B

+B

.
So that
AB = (A

2
n/2
+A

)(B

2
n/2
+B

)
= A

. .
2
n
+ (A

. .
+A

. .
)2
n/2
+A

. .
where we compute the underbraced terms recursively. Also, multiplying by 2
n
is merely a shift. So we need 2 shifts,
3 adds, 4 recursive calls to multiply (n/2)-bit numbers.
T(n) = 4T(
n
2
) + (n)
whereby the master theorem yields a = 4, b = 2, d = 2 so were in Case 1: (n
2
). No improvement!
14
A clever idea:
A

+A

= (A

+A

)(B

+B

) A

so we only need to perform 3 multiplications.

1 mult(A,B):
2 divide A into A and A
3 divide B into B and B
4 C1 = mult(A,B)
5 C2 = mult(A,B)
6 C3 = mult(A+A,B+B)
7 return C1*2^n + (C3-C1-C2)*2^(n/2) + C2
So we require 2 shifts, 4 additions/subtractions, 3 recursive calls to multiply two numbers of (n/2) +1 bits. We get
the recurrence
T(n) = 3T(
n
2
+ 1) + (n)
(the +1 will not matter). The solution to this is
(n
log
2
3
) (n
1.59
)
Thursday, May 24
On Tuesday May 29, the lecture will be held in RCH 309. Also, Assignment 2 is available.
We imagine that perhaps we can reduce the number 3 above to something less, by rewriting the formula in a dierent
way. However, there is no way to do this; we must always compute 3 products.
To rene the algorithm, we could try a 3-way divide and conquer. The naive way of doing it would give
T(n) = 9T(n/3) +O(n) = O(n
log
3
9
) = O(n
2
)
so this is not good. It turns out that we can actually reduce the 9 above to a 5. Hence we obtain
(n
log
3
5
) = O(n
1.47
)
If we consider 4-way divide and conquer, we get
T(n) = 7T(n/4) +O(n) = (n
log
4
7
) = O(n
1.41
)
5-way gives
T(n) = 9T(n/5) +O(n) = (n
log
5
9
) = O(n
1.37
)
and performing this a bunch of times, we nd out that we can actually get the complexity to be
O(n
1+
)
for any constant > 0 (however, the constant gets very large the closer we take to 0).
Schnhage, Strassen 71
Obtained an algorithm that runs in
O(nlog nlog log n).
15
Frer 2007
Obtained an algorithm that runs in
O(nlog nlog log log n).
2.2.3 Matrix multiplication
Given n n matrices A, B R
nn
, compute n n matrix C = AB. For example,
_
_
1 2 3
4 5 6
7 8 9
_
_
_
_
2 3 4
5 6 7
8 9 10
_
_
=
_
_
36 42 . . .
_
_
If we use the standard method (by denition) we will get (n
3
).
1 for i = 1 to n do
2 for j = 1 to n do
3 C_ij = sum from k = 1 to n of a_ik b_kj
Surprisingly, we can do better than this.
Strassens algorithm 69
Write
A =
_
A
1
A
2
A
3
A
4
_
, B =
_
B
1
B
2
B
3
B
4
_
.
where A
i
, B
j
are
n
2

n
2
matrices. First idea is
AB =
_
A
1
B
1
+A
2
B
3
A
1
B
2
+A
2
B
4
A
3
B
1
+A
4
B
3
A
3
B
2
+A
4
B
4
_
.
If we look at this, we observe we will need 8 recursive calls. Hence we get the recurrence
T(n) = 8T(n/2) + (n
2
)
. .
4 adds, copying
so by the Master theorem, Case 1 we get
(n
log
2
8
) = (n
3
).
So we get no improvement by using naive divide-and-conquer.
An ingenious idea: recursively compute
C
1
= A
1
(B
1
B
3
), C
2
= (A
2
+A
1
)B
3
, C
3
= A
4
(B
4
B
2
), C
4
= (A
3
+A
4
)B
2
also
C
5
= (A
1
+A
4
)(B
2
+B
3
), C
6
= (A
2
A
4
)(B
3
+B
4
), C
7
= (A
3
A
1
)(B
1
+B
2
)
Combine
AB =
_
C
1
+C
2
C
5
+C
6
+C
3
C
2
C
5
+C
7
+C
1
C
4
C
3
+C
4
_
.
16
Analysis:
T(n) = 7T(n/2) + (n
2
)
. .
8 adds/subs, copying
implies (n
log
2
7
) = O(n
2.81
).
Remark 2.21. Applications to other matrix operations, e.g. solve Ax = b, inverse, determinant, etc.) and graph
problems.
Renement?
Pan, 1980
T(n) = 143640T(n/70) + (n
2
).
This is O(n
2.796
).
After a lot of work, Coppersmith and Winograd in 1990 showed we can achieve O(n
2.376
).
2010: Stothers showed O(n
2.374
), 2011: Vassilevska Williams showed O(n
2.373
)
2.2.4 Selection
Given n numbers (unsorted) a
1
, . . . , a
n
and k, nd the kth smallest.
Example 2.22. 60, 52, 83, 41, 32, 94, 75, 43 and k = 3. The answer is 43.
Median: n/2|. Method 0: sort and look up index k. This runs in O(nlog n) time.
Method 1 (selection sort variant): nd smallest, remove, repeat. This gives O(nk) time.
Method 2 (heapsort variant): has time
O(n +k log n)
(the n comes from build-heap, and k log n comes from k delete-min operations). If k is ssmall (k < n/ log n) then this
is O(n) time.
O(n) for all k?
Tuesday, May 29
Method 3 (quicksort variant quickselect):
1 Select( {a1, ..., an}, k ):
2 if n = 1 return a1
3 pick "pivot" x from {a1, ..., an}
4 L = { ai : ai <= x }, l = |L|
5 R = { ai : ai > x }
6 if k <= l Select( L, k ) else Select( R, k-l )
Analysis attempt:
T(n) T(max, n ) +O(n)
(assume all numbers are distinct and x is not the largest). Worst-case: = 1 or = n 1
T(n) T(n 1) +O(n) = O(n
2
).
17
Ideal case (if you are very lucky), = n/2
T(n) T(n/2) +O(n) = O(n)
Suppose now we pick the pivot at random. Randomized analysis: is equally likely to be 1, 2, . . . , n.
P(
n
4

3n
4
) =
1
2
.
This is because
3n
4

n
4
=
1
2
n. Expect that after 2 iterations we have
n
4

3n
4
i.e. max, n
3
4
n. Therefore,
T(n) T
_
3
4
n
_
+ 2O(n) = O(n) expected time for any input.
Algorithm (1973) by Blum, Floyd, Pratt, Rivest, Tarjan
First idea: nd a good pivot x deterministically, e.g. ideal choice x is the median. However, this is as dicult to do as
solving the original problem. Need some approximate median. Ingenious idea: pick x to be the median of a subset
(we nd it by performing another recursive call).
1 Select( {a1, ..., an}, k ):
2 if n = 1 return a1.
3 split {a1, ..., an} into groups G_1 , ..., G_{n/5} each of 5 elements
4 for i = 1 to n/5 do x_i = median of G_i
5 x = Select( {x1, ..., x_{n/5}}, n/10 )
6 L = { ai : ai <= x }, l = |L|
7 R = { ai : ai > x }
8 if k <= l Select( L,k ) else Select( R,k-l )
Example 2.23.
1, 10, 5, 8, 21
. .
G
1
,x
1
=8
, 34, 6, 7, 12, 23
. .
G
2
,x
2
=12
, 2, 4, 30, 11, 25
. .
G
3
,x
1
3=11

Analysis: we claim
3n
10

7n
10
Proof: There are about
n
10
groups G
i
with x
i
x. Each such group G
i
contains 3 numbers x
i
, so that at least
3n
10
elements are x. Therefore is approximately
3n
10
(similarly,
7n
10
).
T(n) T(
n
5
) +T(
7n
10
) +O(n)
See notebook for recursion tree analysis.
Thursday, May 31
Selection: randomized quicksort has O(n) expected time. Blum, Floyd, Pratt, Rivest, Tarjan has O(n) worst-case
time. There is an obvious (n) lower bound, so in fact this is the best we can do.
18
Renements? We are concerned with how many comparisons are required of the input numbers. It is known that
sorting has complexity
1 nlog n +O(n)
so the constant is actually 1, assuming the logarithm is taken to base 2. It turns out that BFPRT 73 takes about
16n comparisons, but it can be reduced to about 5.43n. Three years later in 1976, Schnhage, Paterson, Pippenger
obtained an algorithm that only needs 3n comparisons. Dor, Zwick 95 obtained 2.95n.
For lower bounds: BFPRT 73 showed the lower bound of 1.5n. After a lot of subsequent work, Bent and John 85
showed 2n. Dor and Zwick in 96 showed 2.0000000000000000000000001n.
2.3 Algorithm design technique 2: greedy algorithms
This design technique is usually used for optimization problems (nd solution minimizing or maximizing some quan-
tity/quantities).
incrementally build the solution
at each step, choose what looks the best at the moment.
The advantages are that usually, greedy algorithms are simple and fast. The disadvantage is that it may not be
correct. This is due to a dierence between locally optimal versus globally optimal (greedy algorithms try to be
locally optimal). We need to prove correctness which can be dicult.
Example 2.24 (coin changing). Given n coin values c
1
, . . . , c
n
and a number W, nd the minimum number of coins
summing to W (assume unlimited # of coins for each c
i
).
For example, consider 1, 5, 10, 25, 100, 200 and W = 83. Then we write
83 = 25 + 25 + 25 + 5 + 1 + 1 + 1.
Here is the pseudocode for what we are doing; it is a greedy algorithm.
1 while W =/= 0 {
2 pick largest c_i such that c_i <= w
3 print c_i
4 W = W - c_i
5 }
Counterexample: 1, 3, 6, 12, 24, 30 and W = 48.
48 = 30 + 12 + 6.
However, this is bad because 48 = 24 + 24, so we have failed to nd the minimal solution.
2.3.1 Disjoint intervals (activity selection)
Given n intervals [a
1
, b
1
], . . . , [a
n
, b
n
], nd the largest subset of disjoint intervals. [diagram 1] Application: schedule
maximum number of activities/jobs without time conicts.
Idea 1 (greedy): pick shortest duration: this obviously will fail.
Idea 2 (greedy): pick an interval that conicts with the fewest other intervals. It turns out that this, too, does not
always work. [diagram 2 (of counterexample)]
Idea 3: pick the interval with lowest upper bound: this turns out to work.
19
1 repeat {
2 pick interval [a_i , b_i] with smallest b_i
3 remove [a_i , b_i] and all intervals intersecting it
4 }
This runs in O(n
2
) and can be improved to O(nlog n) time by sorting and then scanning.
Correctness: Want to show there exists an optimal solution containing all intervals chosen by greedy.
Proof. Let I

be an optimal solution. Let [a

, b

] be the leftmost interval in I

. Let [a, b] be the rst interval chosen

by greedy. We know b b

. [diagram 3]. Then

I

[a

, b

] [a, b]
is a feasible solution with same number of intervals. Hence there exists an optimal solution containing [a, b]. Repeat
argument.
2.3.2 Fractional Knapsack
Given values v
1
, . . . , v
n
> 0 and weights w
1
, . . . , w
n
, W > 0, maximize
n

i=1
v
i
x
i
subject to
n

i=1
w
i
x
i
W
over variables 0 x
1
, . . . , x
n
1, real numbers.
Example: values 50, 90, 20, 10 and weights 50, 60, 10, 75 with W = 100.
x = (1/2, 0, 0, 1) = value 35
x = (4/5, 1, 0, 0) = value 130
x = (3/5, . . .) = value 140.
Tuesday, June 5
Greedy idea 1: pick largest value at each step. We get
x = (
4
5
, 1, 0, 0)
which has value 130. This is not the best, so this strategy fails!
Greedy idea 2: pick smallest weight. We get
x = (1,
2
3
, 1, 0)
which has value 130.
Greedy idea 3: pick largest value per unit weight. We get
x = (3/5, 1, 1, 0)
which is the optimal solution.
The third idea indeed works, but we require a proof.
20
1 repeat {
2 find j with largest ratio v_j/w_j
3 x_j = min{ 1, W/w_j }
4 W = W - w_j x_j , remove j
5 }
This algorithm runs in time O(n
2
). This can easily be improved to O(nlog n) by sorting.
Correctness: Let (x
1
, . . . , x
n
) be the greedy solution. We claim there exists an optimal solution (x

1
, . . . , x

n
) with
x

j
= x
j
for all iterations.
Proof. We assume all the ratios are distinct. Consider the rst iteration. Suppose x

j
,= x
j
. We know that x

j
x
j
and therefore x

j
< x
j
(by our supposition). We nd another item k with x

k
> 0. This exists, because otherwise
n

i=1
v
i
x

i
= v
j
x

j
< v
j
x
j

n

i=1
v
i
x
i
,
contradicting that (x

1
, . . . , x

n
) is optimal.
We now increase x

j
by /w
j
and decrease x

k
by /w
k
for some suciently small > 0. Then observe that
n

i=1
w
i
x

i
changes by the value
w
j

w
j
w
k

w
k
= 0.
So it is still feasible. We also see that
n

i=1
v
i
x

i
changes by
v
j

w
j
v
k

w
k
=
_
v
j
w
j

v
k
w
k
_
> 0
by our greedy choice. This is a contradiction: the original optimal solution is not optimal! Repeat argument.
2.3.3 Stable marriage
Given n candidates and n employers, and a preference list (a permutation of the employers) for each candidate, and a
preference list (a permutation of the candidates) for each employer, nd a matching between candidates and employers
that is stable, i.e. for any two matched pairs (C, E) and (C

, E

) we cant have both that C prefers E

over E, and
E

prefers C over C

.
Example 2.25. We may have
C
1
: E
4
, E
3
, E
1
, E
2
E
1
: C
1
, C
2
, C
3
, C
4
C
2
: E
1
, E
3
, E
2
, E
4
E
2
: C
1
, C
3
, C
2
, C
4
C
3
: E
4
, E
3
, E
2
, E
1
E
3
: C
3
, C
2
, C
4
, C
1
C
4
: E
3
, E
1
, E
4
, E
2
E
4
: C
2
, C
3
, C
1
, C
4
21
Is the matching (C
1
, E
1
), (C
2
, E
2
), (C
3
, E
3
) and (C
4
, E
4
) stable? No, because C
1
prefers E
4
over E
1
and E
4
prefers
C
1
over C
4
.
Remark 2.26. The solution may not be unique but does it always exist?
Brute force algorithm: try all matchings (there are n! matchings). Therefore O(n! n
2
) time (SLOW!)
The natural algorithm:
1. Start with any matching matching
2. while there exist two unstable pairs (C, E) and (C

, E

) do
3. re-match (C, E

) and match (C

, E).
This can also be slow (may not terminate!)
Gale Shapleys Greedy Algorithm (1962)
1 while there is an unmatched employer E do {
2 pick C = next best candidate in Es pref. list
3 // E makes an offer to C
4 if C is unmatched or C prefers E over Cs current employer E_0
5 then unmatch(C,E_0) and match(C,E)
6 }
Example 2.27. An example is given below.
E
1
: C
1
E
2
: C
1
, C
3
, C
2
, C
4
E
3
: C
3
, C
2
E
4
: C
2
, C
3
Analysis: each employer makes n oers, therefore n
2
oers, hence n
2
iterations, so the algorithm runs in O(n
2
)
time.
Correctness:
Claim 1: In line 2, C exists.
Proof. There must be an unmatched candidate. All candidates previously considered by E are currently matched.
Therefore there exists an unmatched candidate not yet considered by E.
Claim 2: Matching is stable.
Proof. By contradiction. There exist matched pairs (C, E) and (C

, E

) such that C prefers E

over E, and E

prefers
C over C

. The second implies that E

makes an oer to C before C

. Now, C has received oers from E

and also E.
Thus C prefers E over E

, but this is a contradiction to the rst condition.

Thursday, June 7 (I think this is where we started)
22
2.4 Algorithm design technique 3: dynamic programming
This is used for optimization problems, usually.
form subproblems
get recursive formula to explore all choices at each step
evaluate formula bottom-up using a table
Works when total number of subproblems is not too big.
Example 2.28. Evaluate (for 0 k n)
C(n, k) =
_
1 if k = 0 or k = n
C(n 1, k 1) +C(n 1, k) else.
The divide-and-conquer (recursion):
T(n) = 2T(n 1) +O(1) = T(n) = O(2
n
)
which is exponential. There is subproblem sharing occurring [diagram of tree]. The number of subproblems is
1 + 2 + 3 +. . . +n = (n
2
).
[table].
Example 2.29. Coin changing. Given coin values c
1
, . . . , c
n
, target W, nd the minimum number of coins summing
to W. Dene subproblems (i = 0, . . . , n, j = 0, . . . , W)
C[i, j] = min # of coins from c
1
, . . . , c
i
summing to j.
Answer: C[n, W]. Base cases: C[i, 0] = 0 (i = 0, . . . , n) and C[0, j] = (j = 1, . . . , W). Recursive formula:
C[i, j] =
_
minC[i 1, j], C[i, j c
i
] + 1 if j c
i
C[i 1, j] else.
(2 choices: not use c
i
, or use c
i
). [example, table].
1 for i = 0 to n do C[i,0] = 0
2 for j = 1 to W do C[0,j] = inf
3 for i = 1 to n do
4 for j = 1 to W do
5 if j < c_i or C[i-1,j] < C[i,j-c_i] + 1
6 then C[i,j] = C[i-1,j], PI[i,j] = "not use"
7 else C[i,j] = C[i,j-c_i] + 1, PI[i,j] = "use"
8 return C[n,W]
Time is O(nW), as is the space. However we dont need to keep track of all the rows, only the previous and current
one. So the space can be reduced to O(W) (however we can do this only if we dont need to recover the solution; if
we only want the optimal value).
1 PrintCoints(i,j):
2 if i = 0 or j = 0 return
3 if PI[i,j] = "not use"
4 PrintCoins(i-1,j)
5 else print c_i
6 PrintCoins(i, j-c_i)
23
this takes O(n +W) time.
Second DP method:
Dene subproblems:
Tuesday, June 12
2.4.1 Longest common subsequence (LCS)
Denition 2.30. Given sequence a
1
, . . . , a
n
, a subsequence is a
i
1
, a
i
2
, . . . , a
i
k
for some i
1
< . . . < i
k
.
For example, for the sequence ABCADECF, subsequences are ABCD and ABDC. However ABFC is not a subsequence.
Given two sequences
a = a
1
a
2
a
m
b = b
1
b
2
b
n
Find the max-length subsequence of a that is also a subsequence of b.
Example 2.31. Consider ALGORITHM and LOGARITHM. We have LGRITHM, LORITHM. Both are of length 7.
Application: Dierence of 2 strings/les (UNIX di, DNA sequencing, etc.). How many lines you have to delete from
one le and insert to go from one le to the other.
Brute force takes O(2
n
) time. A greedy solution seems to not work. So we look at a dynamic programming solution.
Dene subproblems: C[i, j] = length of LCS of a
1
a
i
and b
1
b
j
. The answer will be C[m, n].
Base case: C[i, 0] = 0 (i = 0, . . . , m), and C[0, j] = 0 (j = 0, . . . , n).
Recursive formula: 3 choices for last character of optimal subsequence for C[i, j]:
1. If the last character is not a
i
, then we get C[i 1, j].
2. If the last character is not b
j
, then we get C[i, j 1].
3. Last character is a
i
= b
j
, then we get C[i 1, j 1] + 1.
So
C[i, j] =
_
maxC[i 1, j], C[i, j 1] if a
i
,= b
j
maxC[i 1, j], C[i, j 1], C[i 1, j 1] + 1 if a
i
= b
j
However, we can remove the expressions in red, because they are always dominated by C[i 1, j 1] + 1.
i \j 0 1 2 3 4 5
0 0 0 0 0 0 0
1 0 0 0 0 1 1
2 0 1 1 1 1 1
3 0 1 1 2 2 2
4 0 1 2 2 2 2
5 0 1 2 2 2 3
Evaluation order: increas. i for some i, increas. j
Analysis: # entries is O(mn). Time per entry is O(1). Total time O(mn). Space is O(mn) but if we dont need to
retrieve the optimal solution then we can reduce space to O(n).
24
2.4.2 Min-length triangulation
Denition 2.32. A polygon is convex if all angles are < . [diagram]. A chord is a line segment between 2
non-adjacent vertices that lies inside the polygon. A triangulation is a set of non-intersecting chords that divide P
into triangles.
Note that all triangulations have n 2 triangles, and n 3 chords.
Problem: Given a convex polygon with vertices v
1
v
2
. . . v
n
v
1
(in counterclockwise order), nd a triangulation with
minimum total length (i.e. one that minimizes the sum of the lengths of its chords).
Idea 0: brute force. How many triangulations? [examples]
The number of ways is given by
1
n 1
_
2n 4
n 2
_
the Catalan numbers. It is
(
4
n
n
3/2
)
DP algorithm: Dene subproblems (1 i < j n): C[i, j] = length of min triangulation for the polygon v
i
v
i+1
v
j
v
i
.
Answer: C[1, n].
Base cases: C[i, i + 1] = 0, C[i, i + 2] = 0.
Recursive formula: [diagram] choices over which triangle containing v
i
v
j
. So v
i
v
j
v
k
for some v
k
. So
C[i, j] = min
k{i+1,...,j1}
C[i, k] +C[k, j] +d(v
i
, d
k
) +d(v
k
, v
j
)
Evaluation order: increas. j i
Runtime: there are O(n
2
) entries. Time per entry is O(j i) = O(n). Total is O(n
3
) time, and O(n
2
) space.
Midterm information: June 13, 7pm 9pm.
MC 4021 (last names A through L), MC 4059 (last names M through Z). Covers everything up to greedy, but not
dynamic programming. No calculations or cheat sheets allowed. Sample midterm on the course webpage. There
will be short questions to make sure you pay attention during class. Be familiar with specic problems or examples
discussed in class. There will be at least one long question on divide and conquer, one long question on greedy,
and one question about solving recurrences by one of the three methods, one question on analyzing pseudocode
(like in the rst assignment) and comparing growth rates of functions (like in the rst assignment).
Extra oce hour: tomorrow 1pm 2pm, today 4pm 5pm.
Thursday, June 14 missed rst 15 minutes
[rst few boards on camera, one was erased when i arrived]
Basic concepts: adjacency, incidence, path, connectedness, cycles, trees.
Note:
n 1 m
. .
if G is connected
n
2
Representation:
25
adjacency matrix
A[u, v] =
_
1 if (u, v) is an edge
0 else
which takes (n
2
) space. This is good for dense graphs when m is close to n
2
.
adjacency lists. For each u V , store linked list
Adj[u] = v : (u, v) E
space

_
n +

uV
[Adj[u][
_
=
_
n +

uV
(out)deg(u)
_
= (n +m)
Basic problems: is there a path from s to t? Basic algorithms: breadth-rst search (BFS), depth-rst search (DFS).
Example 2.33. Trees. Denitions of BFS, DFS (discovery order) pre-order (nish order) post-order.
Example 2.34. Graph
Remark: No forward edges in BFS. For undirected graphs, no cross edges in DFS.
Implementation:
1 BFS(G,s): // idea : use a queue Q
2 for each v in V do mark v "undiscovered"
3 insert s to Q, mark s "discovered", d[s]=0
4 while Q is not empty {
5 remove head u of Q
6 for each v in Adj[u] do
7 if v is "undiscovered" {
8 insert v to tail of Q
9 mark v "discovered", d[v]=d[u]+1
10 }
11 }
12 }
Analysis: lines 58 take O(out deg(u)) time. Each node u is inserted once to Q, removed once from Q. Total time

_
n +

uV
out deg(u)
_
= (n +m)
1 DFS(G,s): // idea - use recursion (implicitly , a stack)
2 mark s "discovered"
3 for each v in Adj[s] do
4 if v is "undiscovered" {
5 DFS(G,v), pi[v] = s
6 }
7 mark s "finished"
26
1 DFS(G):
2 for each v in V do mark v "undiscovered"
3 for each v in V do
4 if v is undiscovered
5 DFS(G,v)
Analysis:

_
n +

uV
out deg(u)
_
= (n +m)
Tuesday - cancelled
Thursday, June 21
We talked about graph algorithms BFS and DFS, which run in (n +m) time.
Applications of BFS/DFS.
Example 2.35 (example 0). Is undirected graph connected?
Solution. Run DFS/BFS from any vertex s.
Example 2.36 (example 1). Does a graph contain a cycle?
Solution. Undirected: run DFS/BFS and test for non-tree edge.
Directed: run DFS and test for back edge.
Example 2.37 (example 2). Unweighted shortest path. Given directed/undirected graph G = (V, E) and s, t V ,
nd path from s to t that has shortest length. [diagram]
Solution. Run BFS, take path in the BFS tree.
Example 2.38 (example 3: bipartiteness/2-coloring). Given undirected graph G = (V, E), decide whether V can be
partitioned into V
1
and V
2
so that for all edges uv E, u V
1
and v V
2
or vice versa [diagrams* explaining
what a bipartite graph is (...)].
Solution. BFS tree: [diagram]. Color all even-level vertices red, odd-level vertices blue. Check all non-tree edges.
Takes (n +m) time.
If and only if condition: graph is bipartite if and only if the graph has no cycles of odd length.
2.5 Topological Sort
Given directed graph G = (V, E) return a vertex order such that for all uv E, u appears before v (or report no
answer).
Example 2.39. [diagram]
Algorithm: Assume G is acyclic (has no cycles called a DAG).
1. run DFS(G).
2. return reverse of nish order. (n +m) time.
27
Application: job scheduling.
Example 2.40 (Prof. Bumsteads Schedule). shorts pants belt jacket; tie shirt belt. watch. socks
shoes. [more in notebook]
Correctness: for all (u, v) E,
Case A: u discovered rst before v
DFS trees: [diagrams]
Case B: v discovered rst before u.
DFS trees: [diagrams]
Theorem 2.41. A topological sort exists i G is a dag.
2.6 Strongly Connected Components
Given a directed graph G = (V, E), partition V into components such that u, v is in same component if and only if
there is a path u > v and a path v > u [diagram]
Application: condense directed graph
[b]->[ace]->[df]
Kosaraju-Sharir algorithm (1981)
1. run DFS(G)
2. form transposed graph G
T
, run DFS(G
T
), return DFS trees.
midterm. highest mark: 100. average: 56.5. median: 63.
Tuesday, June 26 - NOTEBOOK
Thursday, June 28 - NOTEBOOK
Tuesday, July 3 - MISSED - notes due to Aamir Mansoor
Thursday, July 5 - NOTEBOOK
Tuesday, July 10 - CUMC day - notes due to Aamir Mansoor
NP-Completeness. We dene P to be the set of all decision problems solvable in polytime, and NP to be the set of
all decision problems of the form:
Input: object x
Output: yes if there exists something polysize y satisfying some condition R(x, y) veriable in polytime
Proposition 2.42. P NP EXPTIME, where EXPTIME consists of those problems solvable in 2
n

time for
constant .
Proof. (P NP) ignore certicate
(NP EXPTIME) brute-force, i.e. try all certicates of length p(n), total number of certicates 2
p(n)
.
The Class NPC
Dene the hardest problem in NP.
To dene hardest, we must gure out how to dene L
1
is easier than L
2
(is at least as easy as).
28
One way: L
1
reduces to L
2
(e.g. selection reduces to sorting (polytime))
Denition 2.43. Given decision problems L
1
, L
2
a (Karp) reduction from L
1
to L
2
is an algorithm that runs in
polynomial time such that the output of L
1
on x is yes if and only if the output of L
2
on f(x) is yes.
We write L
1

p
L
2
when there is a polytime reduction from L
1
to L
2
.
Solver for L
1
:
x f
f(x)
solver for L
2
yes or no
Prop A: If L
1

p
L
2
and L
2
P, then L
1
P.
Prop B: If L
1

p
L
2
and L
2

p
L
3
, then L
1

p
L
3
.
Denition 2.44. L is NP-complete if
1. L NP.
2. For all L

NP, L

p
L (that is, L is NP-hard).
Prop C: Let L be an NP-complete problem, then L / P i P ,= NP
Proof. Suppose L / P. Then L NP by (1), so NP ,= P.
Suppose L P, then for all L

NP, we have that L

p
L by (2). Hence L

P by Prop A. Thus NP P.
The world assuming P ,= NP: (diagram: P and NPC disjoint, sitting inside larger bubble)
The rst NPC problem: satisability (SAT)
Input: Boolean formula in n variables F(x
1
, . . . , x
n
).
Output: yes if there exists an assignment of Boolean values to variables such that F(x
1
, . . . , x
n
) evaluates to
true.
Example 2.45. F(x
1
, x
2
, x
3
) = (x
1
(x
2
x
3
)) (x
1
(x
2
x
3
)) x
2
.
The answer here is yes (x
2
= 0, x
3
= 0 or 1, x
1
= 1).
Theorem 2.46 (Cook, 1971). SAT is NP-complete.
Proof sketch. We have:
1. SAT NP: certicate is an assignment (n bits, polysize). To verify: F evaluates to true given this assignment
(polytime).
2. Need to give a polytime reduction from every L NP to SAT.
Say L is:
Input: z
Output: yes i there exists y such that R(z, y) is true (verication algorithm in p(n) time).
General idea: simulate the algorithm for R by a Boolean formula.
Reduction: Given z of size n, construct a boolean formula F as follows. [UNFINISHED]
Thursday, July 12 - CUMC day - notes due to Aamir Mansoor
Tuesday, July 17 - NOTEBOOK
Thursday, July 19 - NOTEBOOK
29