Divide-and-Conquer

Reading: CLRS Sections 2.3, 4.1, 4.2, 4.3, 28.2, 33.4.

CSE 6331 Algorithms

Steve Lai
Divide and Conquer
• Given an instance x of a problem, the divide-and-conquer
method works as follows:
function DAC(x)
if x is sufficiently small then
solve it directly
else
divide x into smaller subinstances x1 , x2 , … , xk ;
yi ← DAC(xi ), for 1 ≤ i ≤ k ;
y ← combine ( y1 , y2 ,  , yk ) ;
return(y )
2
Analysis of Divide-and-Conquer
• Typically, x1 , … , xk are of the same size, say  n b  .
• In that case, the time complexity of DAC, T (n), satisfies
a recurrence:

 c if n ≤ n0
T ( n) = 
 kT (  n b  ) + f (n) if n > n0
• Where f (n) is the running time of dividing x and
combining yi 's.
• What is c ?
• What is n0 ?
3
Mergesort: Sort an array A[1..n ]

• procedure mergesort ( A[i.. j ])

// Sort A[i.. j ]//
if i = j then return // base case //
m ← (i + j ) 2  

mergesort ( A[i..m]) 
 divide and conquer
mergesort ( A[m + 1.. j ]) 
merge ( A[i..m ], A[m + 1.. j ]) 

• Initial call: mergesort ( A[1..n ])

4
Analysis of Mergesort
• Let T (n) denote the running time of mergesorting an
array of size n.
• T (n) satisfies the recurrence:

 c if n ≤ 1
T ( n) = 
 T (  n 2  ) + T (  n 2  ) + Θ(n) if n > 1
• Solving the recurrence yields:
T (n) = Θ(n log n)
• We will learn how to solve such recurrences.

5
Linked-List Version of Mergesort
• function mergesort ( i, j )
// Sort A[i.. j ]. Initially, link[k ] = 0, 1 ≤ k ≤ n.//
global A[1..n], link[1..n]
if i = j then return (i ) // base case //
m ← (i + j ) 2  

ptr1 ← mergesort ( i, m ) 

ptr 2 ← mergesort ( m + 1, j )  divide and conquer

ptr ← merge ( ptr1, ptr 2 ) 
return ( ptr ) 


6
Solving Recurrences
• Suppose a function T ( n ) satisfies the recurrence

 c if n ≤ 1
T (n ) = 
 3T (  n 4  ) + n if n > 1
where c is a positive constant.

• Wish to obtain a function g ( n ) such that T ( n ) = Θ ( g ( n ) ) .

• Will solve it using various methods: Iteration Method,

Recurrence Tree, Guess and Prove, and Master Method.

7
Iteration Method
Assume n is a power of 4. Say, n = 4m. Then,
T (n=
) n + 3T (n / 4)
= n + 3 [ n / 4 + 3T (n /16) ]
= n + 3(n / 4) + 9 [ (n /16) + 3T (n / 64) ]
= n + (3 / 4)n + (3 / 4) 2 n + 33 T (n / 43 )
 3  
3
2
  
3
m −1
m  n 
= n 1 + +   + ⋅⋅⋅ +    + 3 T  m 
 4 4  4   4 
= n Θ(1) + O(n) = Θ(n)

So, T (n) =
Θ(n | n a power of 4) ⇒ T (n) =
Θ(n). (Why?)
8
Remark
• We have applied Theorem 7 to conclude T (n) = Θ(n)
from T (n) = Θ(n | n a power of 4).
• In order to apply Theorem 7, T (n) needs to be
nondecreasing.
• It will be a homework question for you to prove that
T (n) is indeed nondecreasing.

9
Recurrence Tree
solving problems time needed
1 of size n
↓ ↑ n
3 of size n / 4
↓ ↑ 3⋅ n / 4
32 of size n / 42
↓ ↑ 32 ⋅ n / 42
33 of size n / 43
↓ ↑ 33 ⋅ n / 43
 
3m −1 of size n / 4m −1
↓ ↑ 3m −1 ⋅ n / 4m −1
3m of size n / 4m 3m ⋅ Θ(1)
10
Guess and Prove
 c if n ≤ 1
• Solve T (n) = 
 3T (  n 4  ) + Θ(n) if n > 1
• First, guess T (n) = Θ(n), and then try to prove it.
• Sufficient to consider n = 4m , m = 0, 1, 2, . . .
• Need to prove: c1 4m ≤ T (4m ) ≤ c2 4m for some c1 , c2
and all m ≥ m0 for some m0 . We choose m0 =
0 and prove
by induction on m.
• IB: When m = 0, c1 40 ≤ T (40 ) ≤ c2 40 if c1 ≤ c ≤ c2 .
• IH: Assume c1 4m −1 ≤ T (4m −1 ) ≤ c2 4m −1 for some c1 , c2 .

11
• IS: T (4m ) = 3T (4m −1 ) + Θ(4m )
≤ 3c2 4m −1 + c2′ 4m for some constant c2′
= ( 3c2 4 + c2′ ) 4m
≤ c2 4m if c2′ ≤ c2 4

T (4m ) = 3T (4m −1 ) + Θ(4m )

≥ 3c1 4m −1 + c1′ 4m for some constant c1′
= ( 1
3c 4 + c1)
′ 4 m

≥ c1 4m if c1′ ≥ c1 4
• Let c1 , c2 be such that c1 ≤ c ≤ c2 , c2′ ≤ c2 4, c1 4 ≤ c1′.
Then, c1 4m ≤ T ( 4m ) ≤ c2 4m for all m ≥ 0.

12
The Master Theorem
• Definition: f ( n ) is polynomially smaller than g ( n ), denoted
as= ( g ( n )n −ε ), or f ( n )n ε O ( g ( n )),
f ( n )  g ( n ), iff f ( n ) O=
for some ε > 0.

• For example, 1  n  n 0.99  n  n 2 .

• Is 1  log n ? Or n  n log n ?

• To answer these, ask yourself whether or not n ε = O (log n ).

Θ ( g (n) ) .
• For convenience, write f ( n ) ≈ g ( n ) iff f ( n ) =

• Note: the notations  and ≈ are good only for this class.
13
The Master Theorem
=
If T ( n ) satisfies the recurrence T ( n ) aT ( n / b) + f ( n ),
then T ( n ) is bounded asymptotically as follows.
1. If f ( n )  n logb a , then T ( n ) = Θ ( n logb a ) .
2. If f ( n )  n logb a , then T ( n ) = Θ ( f ( n ) ) .
Θ ( f ( n )log n ) .
3. If f ( n ) ≈ n logb a , then T ( n ) =
Θ ( f ( n )log n ) .
4. If f ( n ) ≈ n logb a log k n, then T ( n ) =

In case 2, it is required that af ( n / b) ≤ cf ( n ) for some c < 1,

which is satisfied by most f ( n ) that we shall encounter.

In the theorem, n / b should be interpreted as  n / b  or  n / b  .

14
Examples: solve these recurrences
• T (n) = 3T (n / 4) + n.
=
• T ( n) 9T (n / 3) + n.

=
• T ( n) T (2n / 3) + 1.
=
• T ( n) 3T (n / 4) + n log n.

=
• T ( n) 7T (n / 2) + Θ(n 2 ).
=
• T ( n) 2T (n / 2) + n log n.
• T (n) = T (n / 3) + T (2n / 3) + n.

15
T (n) = aT (n / b) + f (n)
solving problems time needed
1 of size n
↓ ↑ f (n )
a of size n / b
↓ ↑ a ⋅ f (n / b)
a 2 of size n / b2
↓ ↑ a 2 ⋅ f (n / b2 )
 
a log n −1 of size n / blog n −1
b b

↓ ↑ a log n −1 ⋅ f (n / blog n −1 )
b b

a log n of size n / blog n

b b
a log n ⋅ Θ(1)
b

16
T (n) aT (n / b) + f (n)
1 of size n
↓ ↑ f (n )
a of size n / b
↓ ↑ a ⋅ f (n / b)
a 2 of size n / b2
↓ ↑ a 2 ⋅ f (n / b2 )
 
a log n −1 of size n / blog n −1
b b

↓ ↑ a log n −1 ⋅ f (n / blog n −1 )
b b

a log n of size n / blog n

b b
a log n ⋅ Θ(1)
b

logb n −1
a i f ( n / bi ) + n logb a
log a n
T (n) = ∑
i =0
=
(Note: log bn = log a n ⋅ log b a )
log a b
17
Suppose f (n) = Θ n ( logb a
).
b)
(n /=
 logb a   n logb a   n logb a 
=
Then f (n / b ) Θ
i

i
Θ
=  i logb a  Θ i ,
  b   a 

and thus a i f ( n / bi ) = Θ ( nlogb a ). Then, we have

logb n −1
T (n ) = ∑ a i f ( n / bi ) + n logb a (from the previous slide)
i =0

 logb n −1 log a 
= Θ
 i =0
∑
n= b +n log a
b 

Θ= ( )
nlogb a log n Θ ( f (n) log n )
 

18
When recurrences involve roots

• Solve T (n) = 
( )
 2T n + log n if n > 2

 c otherwise
• Suffices to consider only powers of 2. Let n = 2m.
• Define a new function =
S (m) T= (2m ) T (n).
• The above recurrence translates to
 2 S ( m / 2 ) + m if m > 1
S ( m) = 
c otherwise
• By Master Theorem, S (m) = Θ(m log m).
• So, T (n) = Θ(log n log log n)

19
Strassen's Algorithm for Matrix Multiplication
• Problem: Compute
= C AB, given n × n matrices A and B.
• The straightforward method requires Θ(n3 ) time,
n
using the formula cij = ∑ aik bkj .
k =1

• Toward the end of 1960s, Strassen showed how to multiply

matrices in O(n log 7 ) = O(n 2.81 ) time.
=
• For n 100, n 2.81 ≈ 416,869, and n3 = 1, 000, 000.
• The time complexity was reduced to O(n 2.521813 ) in 1979,
to O(n 2.521801 ) in 1980, and to O(n 2.376 ) in 1986.
• In the following discussion, n is assumed to be a power of 2.
20
• Write
 A11 A12   B11 B12   C11 C12 
A =  , B =  , C  
 21
A A22   21
B B22   21
C C 22 

where each Aij , Bij , Cij is a n / 2 × n / 2 matrix.

 C11 C12   A11 A12   B11 B12 

• Then C  =
A22  B  .
 21 C22   A21  21 B22 

• If we compute
= Cij Ai1 B1 j + Ai 2 B2 j , the running time T ( n )
=
will satisfy the recurrence T ( n ) 8T ( n / 2) + Θ( n 2 )
• T ( n ) will be Θ( n 3 ), not better than the straightforward one.
• Good for parallel processing. What's the running time
using Θ(n 3 ) processors?
21
• Strassen showed

 C11 C12   M2 + M3 M1 + M 2 + M 5 + M 6 
=
C 
 21 C22   M 1 + M 2 + M 4 − M 7 M 1 + M 2 + M 4 + M 5 

where M 1 = ( A21 + A22 − A11 ) × ( B22 − B12 + B11 )

M 2 = A11 × B11
M 3 = A12 × B21
M 4 =−
( A11 A21 ) × ( B22 − B12 )
M5 =
( A21 + A22 ) × ( B12 − B11 )
M6 = ( A12 − A21 + A11 − A22 ) × B22
=
M7 A22 × ( B11 + B22 − B12 − B21 )

•=
T ( n ) 7T ( n / 2) + Θ( n 2 ) ⇒ T ( n ) = Θ( n log 7 )
22
The Closest Pair Problem
• Problem Statement: Given a set of n points in the plane,
A {( xi , yi ) : 1 ≤ i ≤ n} , find two points in A whose
distance is smallest among all pairs.

• Straightforward method: Θ( n 2 ).
• Divide and conquer: O ( n log n ).

23
The Divide-and-Conquer Approach
1. Partition A into two sets: A= B ∪ C.
2. Find a closest pair ( p1 , q1 ) in B.
3. Find a closest pair (p2 , q2 ) in C.
4. Let δ = min {dist( p1 , q1 ), dist( p2 , q2 )}.
5. Find a closest pair (p3 , q3 ) between B and C with
distance less than δ , if such a pair exists.
6. Return the pair of the three which is closest.
• Question : What would be the running time?
2T ( n / 2) + O ( n ) ⇒ T ( n ) =
• Desired: T ( n ) = O ( n log n ).

24
• Now let's see how to implement each step.

• Trivial: steps 2, 3, 4, 6.

25
Step 1: Partition A into two sets: A= B ∪ C.
• A natural choice is to draw a vertical line to divide
the points into two groups.
• So, sort A[1..n] by x-coordinate. (Do this only once.)
• Then, we can easily partition any set A[i.. j ] by
A[i.. j ]= A[i..m] ∪ A[m + 1.. j ]
= (i + j ) 2  .
where m

26
Step 5: Find a closest pair between B and C with
distance less than δ , if exists.
• We will write a procedure

Closest-Pair-Between-Two-Sets ( A[i.. j ], ptr, δ , ( p3, q3) )

which finds a closest pair between A[i..m ] and A[m + 1.. j ]

with distance less than δ , if exists.
• The running time of this procedure must be no more than
O ( A[i.. j ] ) in order for the final algorithm to be O ( n log n ) .

27
Data Structures
• Let the coordinates of the n points be stored in X [1..n]
and Y [1..n].

• For simplicity, let A[i ] = ( X [i ], Y [i ]).

• For convenience, introduce two dummy points:
A[0] = (−∞, −∞) and A[n + 1] =( ∞, ∞ )

• We will use these two points to indicate "no pair"

or "no pair closer than δ ."

• Introduce an array Link[1..n], initialized to all 0's.

28
Main Program
• Global variable: A[0..n + 1]

• Sort A[1..n] such that X [1] ≤ X [2] ≤ ⋅ ⋅ ⋅ ≤ X [n].

That is, sort the given n points by x-coordinate.

• Call Procedure Closest-Pair with appropriate parameters.

29
Procedure Closest-Pair ( A[i.. j ], ( p, q) ) //Version 1//
{*returns a closest pair ( p, q ) in A[i.. j ]*}
•=If j − i 0 : ( p, q ) ← (0, n + 1);
•=If j − i 1: ( p, q ) ← (i, j );
• If j − i > 1: m ← (i + j ) 2 
Closest-Pair ( A[i..m], ( p1 , q1 ) )
Closest-Pair ( A[m + 1.. j ], ( p2 , q2 ) )
ptr ← mergesort A[i.. j ] by y -coordinate into a linked list
δ ← min {dist( p1 , q1 ), dist( p2 , q2 )}
Closest-Pair-Between-Two-Sets ( A[i.. j ], ptr , δ , ( p3 , q3 ) )
( p, q ) ← closest of the three ( p1 , q1 ), ( p2 , q2 ), ( p3 , q3 )
30
Time Complexity of version 1
• Initial call: Closest-Pair ( A[1..n], ( p, q ) ) .

• Assume Closest-Pair-Between-Two-Sets needs Θ(n) time.

• Let T (n) denote the worst-case running time of

Closest-Pair ( A[1..n], ( p, q ) ) .

=
• Then, T (n) 2T (n / 2) + Θ(n log n).

• So, T (n) = Θ ( n log 2 n ) .

• Not as good as desired.

31
How to reduce the time complexity to O(n log n)?
• Suppose we use Mergesort to sort A[i.. j ] :
ptr ← Sort A[i.. j ] by y -coordinate into a linked list
• Rewrite the procedure as version 2.

• We only have to sort the base cases and perform "merge."

• Here we take a free ride on Closest-Pair for dividing.

• That is, we combine Mergesort with Closest-Pair.

32
Procedure Closest-Pair ( A[i.. j ], ( p, q) ) //Version 2//
•=
If j − i 0 : ( p, q ) ← (0, n + 1);
•=
If j − i 1: ( p, q ) ← (i, j );
• If j − i > 1: m ← (i + j ) 2 
Closest-Pair ( A[i..m], ( p1 , q1 ) )
Closest-Pair ( A[m + 1.. j ], ( p2 , q2 ) )
ptr1 ← Mergesort ( A[i..m]) 

ptr 2 ← Mergesort ( A[m + 1.. j ])  mergesort ( A[i.. j])
ptr ← Merge ( ptr1, ptr 2 ) 

(the rest is the same as in version 1)

33
Procedure Closest-Pair ( A[i.. j ], ( p, q), ptr ) //final version//
{*mergesort A[i.. j ] by y and find a closest pair ( p, q) in A[i.. j ]*}
• =
if j − i 0 : ( p, q ) ← (0, n + 1); ptr ← i
• =
if j − i 1: ( p, q) ← (i, j );
if Y [i ] ≤ Y [ j ] then { ptr ← i; Link[i ] ← j}
else { ptr ← j; Link[ j ] ← i}
• if j − i > 1: m ← (i + j ) 2 
Closest-Pair ( A[i..m], ( p1 , q1 ), ptr1)
Closest-Pair ( A[m + 1.. j ], ( p2 , q2 ), ptr 2 )
ptr ← Merge ( ptr1, ptr 2 )
(the rest is the same as in version 1)
34
Time Complexity of the final version
• Initial call: Closest-Pair ( A[1..n], ( p, q ), pqr ) .

• Assume Closest-Pair-Between-Two-Sets needs Θ(n) time.

• Let T (n) denote the worst-case running time of
Closest-Pair ( A[1..n], ( p, q ), pqr ) .

• Then,
= T (n) 2T (n / 2) + Θ(n).

• So, T (n) = Θ ( n log n ) .

• Now, it remains to write the procedure

Closest-Pair-Between-Two-Sets ( A[i.. j ], ptr , δ , ( p3 , q3 ) )
35
Closest-Pair-Between-Two-Sets
• Input: ( A[i.. j ], ptr , δ )
• Output: a closest pair ( p, q ) between B = A[i..m] and
A[m + 1.. j ] with distance < δ , where m =
C= (i + j ) / 2  .
If there is no such a pair, return the dummy pair (0, n + 1).
• Time complexity desired : O ( A[i.. j ] ) .
• For each point b ∈ B, we will compute dist(b, c) for O(1)
points c ∈ C. Similarly for each point c ∈ C.
• Recall that A[i.. j ] has been sorted by y. We will follow the
sorted linked list and look at each point.

36
Closest-Pair-Between-Two-Sets
• L0 : vertical line passing through the point A[m ].
• L1 and L2 : vertical lines to the left and right of L0 by δ .
• We observe that:
 We only need to consider those points between L1 and L2 .
 For each point k in A[i..m ], we only need to consider the
points in A[m + 1.. j ] that are inside the square of δ × δ .
 There are at most three such points.
 And they are among the most recently visited three points
of A[m + 1.. j ] lying between L0 and L2 .
 Similar argument for each point k in A[m + 1.. j ].
37
 L1 L0 L2

Red points are apart

by at least δ
k

Blue points are apart For k, only need to consider the

by at least δ points in the square less the
right and bottom edges

38
Closest-Pair-Between-Two-Sets(A[i.. j ], ptr , δ , ( p3 , q3 ))
// Find the closest pair between A[i..m] and A[m + 1.. j ]
with dist < δ . If there exists no such a pair, then return
the dummy pair (0, n + 1). //
global X [0..n + 1], Y [0..n + 1], Link[1..n]
( p3 , q3 ) ← (0, n + 1)
b1 , b2 , b3 ← 0 //most recently visited 3 points btwn L0 , L1 //
c1 , c2 , c3 ← n + 1 //such points between L0 , L2 //
m ← (i + j ) 2 
k ← ptr

39
while k ≠ 0 do //follow the linked list until end//
1. if X [k ] − X [m] < δ then // consider only btwn L1 , L2 //
if k ≤ m then //point k is to the left of L0 //
compute d ← min{dist(k , ci ) : 1 ≤ i ≤ 3};
if d < δ then update δ and ( p3 , q3 );
b3 ← b2 ; b2 ← b1; b1 ← k ;
else //point k is to the right of L0 //
compute d ← min{dist(k , bi ) : 1 ≤ i ≤ 3};
if d < δ then update δ and ( p3 , q3 );
c3 ← c2 ; c2 ← c1; c1 ← k ;
2. k ← Link[k ]
40
Convex Hull
• Problem Statement: Given a set of n points in the plane,
say, A = { p1 , p2 , p3 ,  , pn } ,
we want to find the convex hull of A.
• The convex hull of A, denoted by CH ( A), is the smallest
convex polygon that encloses all points of A.
• Observation: segment pi p j is an edge of CH ( A) if all

other points of A are on the same side of pi p j (or on pi p j ).
• Straightforward method: Ω( n 2 ).
• Divide and conquer: O ( n log n ).

41
Divide-and-Conquer for Convex Hull
0. Assume all x -coordinates are different, and no three
points are colinear. (Will be removed later.)
1. Let A be sorted by x -coordinate.
2. If A ≤ 3, solve the problem directly. Otherwise, apply
divide-and-conquer as follows.
3. Break up A into A= B ∪ C.
4. Find the convex hull of B.
5. Find the convex hull of C.
6. Combine the two convex hulls by finding the upper and
lower bridges to connect the two convex hulls.
42
Upper and Lower Bridges
• The upper bridge between CH ( B) and CH (C ) is the
the edge vw, where v ∈ CH ( B) and w ∈ CH (C ), such that

 all other vertices in CH ( B) and CH (C ) are below vw, or
 the two neighbors of v in CH ( B) and the two neighbors

of w in CH (C ) are below vw, or
 the counterclockwise-neighbor of v in CH ( B) and the

clockwise-neighbor of w in CH (C ) are below vw,
if v and w are chosen as in the next slide.
• Lower bridge: similar.

43
Finding the upper bridge
• v ← the rightmost point in CH ( B );
w ← the leftmost point in CH (C ).
• Loop

if counterclockwise-neighbor(v ) lies above line vw then
v ← counterclockwise-neighbor(v )

else if clockwise-neighbor(w) lies above vw then
w ← clockwise neighbor(w)
else exit from the loop
• vw is the upper bridge.

44
Data Structure and Time Complexity
• What data structure will you use to represent
a convex hull?
• Using your data structure, how much time will it take
to find the upper and lower bridges?
• What is the over all running time of the algorithm?

• We assumed:
(1) no two points in A share the same x -coordinate
(2) no three points in A are colinear
• Now let's remove these assumptions.
45
Orientation of three points
• Three points: p1 ( x1 , y1 ) , p2 ( x2 , y2 ) , p3 ( x3 , y3 ) .

• ( p1 , p2 , p3 ) in that order is counterclockwise if

x1 y1 1
x2 y2 1 > 0
x3 y3 1

• Clockwise if the determinant is negative.

• Colinear if the determinant is zero.

46