Greedy Algorithm

Lecturer notes is purely design and developed by

Er. Mohd. Arif Siddique
Lecturer Dept. of Computer Science & IT
RGEC, Meerut

unit5 1
 General form of Greedy Algorithms
Algorithm greedy (C) //C is an input set having n element
{
S←∅ // S is a solution set
while not solution (S) and C ≠ ∅ do
x ← extract best element from C.
if (x is feasible)
Then C ← C - {x} // extract X from input set C
S ← S ∪ {x} // add X into solution
if solution (S) = ∅
then return “no solutions”
}

Time complexity excluding sorting: O(n)

Unit 3 Greedy Algorithms 2

 A Greedy Algorithm
A greedy algorithm always makes the choice that looks best at the
moment.
 In dynamic programming, the optimal solution is described in a recursive
manner, and then is computed ``bottom-up''. Dynamic programming is a
powerful technique, but it often leads to algorithms with higher than
desired running times.
 An alternative design technique, called greedy algorithms. This method
typically leads to simpler and faster algorithms, but it is not as powerful
or as widely applicable as dynamic programming.
 The greedy concept make the choice that looks best at the moment in
this hope that local optimal choices lead to global optimal solution
 Even when greedy algorithms do not produce the optimal solution, they
often provide fast heuristics (non-optimal solution strategies), and are
often used in finding good approximations.

Unit 3 Greedy Algorithms 3

 Problem 1: Activity-Selection Problem
Problem:Given a set S = {1, 2, …, n} of n proposed activities, with a
start time si and a finish time fi for each activity i, select a maximum-
size
set of mutually compatible activities.
Compatible Activities:
Activities i and j are compatible if the internal [si, fi) and [sj, fj) do not
overlap, i.e, i and j are compatible if si≥ fj and sj ≥ fi

Goal: Select a maximum-size set of mutually compatible activities

Unit 3 Greedy Algorithms 4

 Greedy Activity-Selection Algorithm
Sort the input activities by increasing finishing time.
f 1 ≤ f2 ≤ . . . ≤ fn

Algo GREEDY-ACTIVITY-SELECTOR (s[ ], f[ ])

{
n = length [s]
A={i}
j=1
For(i = 2 to n ){
if (si ≥ fj ){
A= AU{i}
j=i
}
}
return set A
}

Time complexity excluding sorting: O(n)

Unit 3 Greedy Algorithms 5

 Example
Given 11 activities A p q r s t u v w x y z
s 1 3 0 5 3 5 6 8 8 2 12
f 4 5 6 7 8 9 10 11 12 13 14

Here all input activities are already sorted by increasing finishing time.

z
y
x
w
v
u
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Unit 3 Greedy Algorithms 6

 Example cont…

z
y
x
w
v
u
t
s
r
q

0
p Initialization A={p}
1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p}

Unit 3 Greedy Algorithms 7

 Example cont…

z
y
x
w
v
u
t
s
r
Activity ‘p’ and ‘q’ are not
q compatible (overlapped) So
p don't add it in to solution
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p}

Unit 3 Greedy Algorithms 8

 Example cont…

z
y
x
w
v
u
t
s
r
Activity ‘p’ and ‘r’ are not
compatible (overlapped) So
q don't add it in to solution
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p}

Unit 3 Greedy Algorithms 9

 Example cont…

z
y
x
w
v
u
t
Activity ‘p’ and ‘s’ are
s compatible, So
r add ‘s’ in to solution
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, }

Unit 3 Greedy Algorithms 10

 Example cont…

z
y
x
w
v
u
Activity ‘s’ and ‘t’ are
t not compatible (overlapped) So
s don't add it in to solution
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, }

Unit 3 Greedy Algorithms 11

 Example cont…

z
y
x
w
v
Activity ‘s’ and ‘u’ are
u not compatible (overlapped) So
t don't add it in to solution
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, }

Unit 3 Greedy Algorithms 12

 Example cont…

z
y
x
w
Activity ‘s’ and ‘v’ are
v not compatible (overlapped) So
u don't add it in to solution
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, }

Unit 3 Greedy Algorithms 13

 Example cont…

z
y
x
Activity ‘s’ and ‘w’ are
w compatible, So add
v ‘w’ in to solution
u
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, w, }

Unit 3 Greedy Algorithms 14

 Example cont…

z
y
Activity ‘s’ and ‘x’ are
x not compatible (overlapped) So
w don't add it in to solution
v
u
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, w, }

Unit 3 Greedy Algorithms 15

 Example cont…

z
Activity ‘w’ and ‘y’ are
y not compatible (overlapped), So
x do not add ‘y’ in to solution
w
v
u
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, w, }

Unit 3 Greedy Algorithms 16

 Example cont…

Activity ‘w’ and ‘z’ are

z compatible, So add ‘z’
y in to solution
x
w
v
u
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, w, z }

Unit 3 Greedy Algorithms 17

 Example cont

Activity ‘w’ and ‘z’ are

z compatible, So add ‘z’
y in to solution
x
w
v
u
t
s
r
q
p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ={p, s, w, z }

Unit 3 Greedy Algorithms 18

 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 19
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 20
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B C
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 21
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B A
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 22
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B E
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 23
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B D
E
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 24
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B E F
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 25
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B E G
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 26
 Activity Selection (Interval Scheduling)
B (1)

C (2)

A (3)

E (4)

D (5)

F (6)

G (7)

H (8) Time
0 1 2 3 4 5 6 7 8 9 10 11

B E H
0 1 2 3 4 5 6 7 8 9 10 11
Unit 3 Greedy Algorithms 27
 Elements of the Greedy Strategy
․ When to apply greedy algorithms?
 Greedy-choice property: A global optimal solution can be
arrived at by making a locally optimal (greedy) choice.
 Dynamic programming needs to check the solutions to
subproblems.
 Optimal substructure: An optimal solution to the problem
contains within its optimal solutions to subproblems.
 E.g., if A is an optimal solution to S, then A' = A - {1} is an optimal
solution to S' = {i ∈ S: si ≥ f1}.
․ Greedy algorithms (heuristics) do not always produce
optimal solutions.
․ Greedy algorithms v.s. dynamic programming (DP)
 Common: optimal substructure
 Difference: greedy-choice property
 DP can be used if greedy solutions are not optimal.

Unit 3 Greedy Algorithms 28

 Problem 2: Fractional Knapsack Problem
There are n items in a store. For i =1,2, . . . , n, item i has weight
wi > 0 and benefit bi > 0. Thief can carry a maximum weight of W
pounds in a knapsack. In this version of a problem the items can
be broken into smaller piece, so the thief may decide to carry only
a fraction of object i.

Classification of Knapsack Problem:

Knapsack Problem

The fractional knapsack problem The 0-1 knapsack problem

(Allow to take fraction of items.) (Each item is either taken or not
taken)
Unit 3 Greedy Algorithms 29
 Fractional Knapsack Problem cont..
Algo Greedy-fractional-knapsack (Item[n], w[ ], b[ ], W)
{
Knap=0
Weight = 0
Benefit=0
for each item i
These line takes θ(n)
v[i] = b[i] / w[i] // value per pound
while (weight <=W) {
i= Extract item of maximum value from list
if(weight + w[i] ≤ W)
{
Knap=Knap U Item[i]
weight = weight + w[i]
benefit= benefit +v[i] These line takes θ(n)
}
else{
Knap=Knap U Item[i]
weight = W
benefit = (W - weight) * v[i] / w[i] // Factoring item, W – weight is needed weight
}
} return x
}
Unit 3 Greedy Algorithms 30
 Fractional Knapsack Problem cont..
Algo Greedy-fractional-knapsack (Item[n], w[ ], b[ ], W)
{
Knap=0
( n)
Weight = 0 : O
Benefit=0 i n g
for each item i o rt
s
v[i] = b[i] / w[i] // value per pound
in g
while (weight <=W) {
lu d
i= Extract item of maximum value from list xc
if(weight + w[i] ≤ W) y e
{ xit
p l e
Knap=Knap U Item[i]
o m
weight = weight + w[i] c
benefit= benefit +v[i] e
} tim
else{ So
Knap=Knap U Item[i]
weight = W
benefit = (W - weight) * v[i] / w[i] // Factoring item, W – weight is needed weight
}
} return Knap
}
Unit 3 Greedy Algorithms 31
 Example
Let S ={a, b, c, d, e, f, g} denote a set of objects with weights
and benefits
as given in the table below. What is an optimal solution to the
fractional
knapsack problem for S assuming that we have a sack that
can Item
hold objects
A B C D E F G
with total
Benefits (Rs.) weight
12 18?
10 8 11 14 7 9
Weight (Kg.) 4 6 5 7 3 1 6

Carrying capacity W = 18 Kg.

Unit 3 Greedy Algorithms 32

 Example cont…

First we must calculate the “value" for the

each items,
which is defined as value = benefits/weight. I
do this and
give them
Item in the
A table
B C below:
D E F G
benefits 12 10 8 11 14 7 9
Weight 4 6 5 7 3 1 6
Value 3 1.67 1.6 1.57 4.67 7 1.5

value = benefit /
weight

Unit 3 Greedy Algorithms 33

 Example cont…
Item A B C D E F G
benefits 12 10 8 11 14 7 9
Weight 4 6 5 7 3 1 6
Value 3 1.67 1.6 1.57 4.67 7 1.5

Sort this table according to decreasing value

Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6
Value 7 4.67 3 1.67 1.6 1.57 1.5

Unit 3 Greedy Algorithms 34

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Initially

Knapsack W=18
weight = 0
benefits =0

Unit 3 Greedy Algorithms 35

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Select maximum valued item ‘F’, Here (weight+w[F])<W

Put whole item ‘F’ into knapsack. Add weight[F] with
weight
and benefit[F] with benefit.

Knapsack F W=18
weight = 1
benefits =7

Unit 3 Greedy Algorithms 36

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Now select next maximum valued item ‘E’, Here (weight+w[E])<W

Put whole item ‘E’ into knapsack. Add weight[E] with
weight and benefit[E] with benefit.

Knapsack F, E W=18
weight = 1+3 =4
benefits =7+14 =21

Unit 3 Greedy Algorithms 37

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Now select next maximum valued item ‘A’, Here

(weight+w[A])<=W Put whole item ‘A’ into knapsack. Add
weight[A] with weight and benefit[A] with benefit.

Knapsack F, E, A W=18
weight = 1+3 =4+4=8
benefits =7+14 =21+12=33

Unit 3 Greedy Algorithms 38

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Now select next maximum valued item ‘B’, Here

(weight+w[B])<=W Put whole item ‘B’ into knapsack. Add
weight[B] with weight and benefit[B] with benefit.

Knapsack F, E, A, B W=18
weight = 1+3 =4+4=8+6=14
benefits =7+14 =21+12=33+10=43

Unit 3 Greedy Algorithms 39

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Now select next maximum valued item ‘C’, Here (weight+w[C])

⊀W Put whole item ‘B’ into knapsack. And calculate
weight and benefit as follows:
needed weight = W – weight = 18 – 14 = 4
Put whole item ‘B’ into knapsack.
Knapsack F, E, A, B, C W=18
weight = W = 18
benefits =7+14 =21+12=33+10=43+(needed weight)*
value[C]
=43+(4*1.6) = 43+6.4 = 49.4

Unit 3 Greedy Algorithms 40

 Example cont…
Item F E A B C D G
benefits 7 14 12 10 8 11 9
Weight 1 3 4 6 5 7 6 W=18
Value 7 4.67 3 1.67 1.6 1.57 1.5

Remaining items D, G could not put into Knapsack (bag) because bag is full
i.e weight = W

Knapsack = F, E, A, B, C

Weight in bag = W = 18
Benefits = Rs.
49.4

Unit 3 Greedy Algorithms 41

 Complexity Analysis

  If the items are already sorted into decreasing

order of vi / wi, then
 the while-loop takes a time in O(n);
Therefore, the total time including the sort is in
O(n log n).

If we keep the items in heap with largest vi/wi at

the root. Then
creating the heap takes O(n) time
while-loop now takes O(log n) time (since heap property
must be restored after the removal of root)

Unit 3 Greedy Algorithms 42

 Introduction
Huffman coding is an algorithm used for lossless data compression
developed by David A. Huffman as a PhD student at MIT in 1952,
and published in A Method for the Construction of Minimum-
Redundancy Codes.

"Huffman Codes" are widely used applications

that involve the compression and transmission of
digital data, such as: fax machines, modems,
computer networks, and high-definition television
(HDTV), etc.

Professor David A. Huffman

(August 9, 1925 - October 7, 1999)
Unit 3 Greedy Algorithms 43
 Motivation

 Reducing the space required to store files on

disk or tape

 Reducing the time to transmit large files.

 Huffman savings are between 20% - 90%

Unit 3 Greedy Algorithms 44

 Basic Idea
Huffman Codes:
Suppose we wish to save a text (ASCII) file on the disk or to
transmit it though a network using an encoding scheme that
minimizes the number of bits required. In fixed-size-encoding
-scheme, without compression, characters are typically encoded by
their ASCII codes with 8 bits per character. We can do better if
we variable-size-encoding. In the variable-size-encoding-scheme
we assign different code of different length according to their
frequencies of occurrences.

Unit 3 Greedy Algorithms 45

 Example:
Suppose you have a file with 100K characters.

For simplicity assume that there are only 6 distinct

characters in the file from a through f, with frequencies as
indicated below.
We represent the file using a unique binary string for each
character.
a b c d e f

Frequency 45 13 12 16 9 5
(in 1000s)
Fixed-length 000 001 010 011 100 101
codeword

Space = (45*3 + 13*3 + 12*3 + 16*3 + 9*3 + 5*3) * 1000

= 300K bits

Unit 3 Greedy Algorithms 46

 Can we do better ??
YES !!
By using variable-length codes instead of fixed-length codes.
Idea : Giving frequent characters short codewords, and infrequent
characters long codewords.
i.e. The length of the encoded character is inversely proportional to
that character's frequency.
a b c d e f Bit Required
Frequency 45 13 12 16 9 5
(in 1000s)

Fixed-length
codeword
000 001 010 011 100 101
300K bits

Variable-length
codeword
0 101 100 111 1101 1100
224K bits

Space = (45*1 + 13*3 + 12*3 + 16*3 + 9*4 + 5*4) * 1000=224K bits

Savings = 25%

Unit 3 Greedy Algorithms 47

 PREFIX CODES

Codes in which no codeword is also a prefix of some

other codeword.
("prefix-free codes" would have been a more appropriate name)

Variable-length 0 101 100 111 1101 1100

codeword

It is very easy to encode and decode using prefix

codes.
No Ambiguity !!

It is possible to show (although we won't do so here)

that the optimal data compression achievable by a
character code can always be achieved with a prefix
code, so there is no loss of generality in restricting
attention to prefix codes.

Unit 3 Greedy Algorithms 48

 Constructing a Huffman code

Huffman invented a greedy algorithm that constructs an optimal

prefix code called a Huffman code. The algorithm builds the tree
T corresponding to the optimal code in a bottom-up manner.
It begins with a set of |C| leaves and performs a sequence of |
C| - 1 "merging" operations to create the final tree.

Greedy Choice?

The two smallest nodes are chosen at each step, and this local decision
results in a globally optimal encoding tree.
In general, greedy algorithms use small-grained, or local
minimal/maximal choices to result in a global minimum/maximum

Unit 3 Greedy Algorithms 49

 Greedy Algorithm for Huffman Tree
Algo Greedy-Huffman(C,f)
{
n ← length[C]
Q ← C:f
for (i ← 1 to n-1)
{
z ← Allocate-Node
x ← left[z] ← Extract-Min(Q)
y ← right[z] ← Extract-Min(Q)
f[z] ← f[x]+f[y]
Insert(Q, z)
}
return Extract-Min(Q)
}

Unit 3 Greedy Algorithms 50

 Constructing a Huffman Coding Tree

14
From following Lines
z ← Allocate-Node
x ← left[z] ← Extract-Min(Q)
f:5 e:9 c:12 b:13 d:16 a:45 y ← right[z] ← Extract-Min(Q)
f[z] ← f[x]+f[y]

c:12 b:13 14 d:16 a:45 From following Lines

Insert(Q, z)

f:5 e:9

Unit 3 Greedy Algorithms 51

 Constructing a Huffman Coding Tree

25 From following Lines

z ← Allocate-Node
x ← left[z] ← Extract-Min(Q)
c:12 b:13 14 d:16 a:45
y ← right[z] ← Extract-Min(Q)
f[z] ← f[x]+f[y]
f:5 e:9

From following Lines

Insert(Q, z)
14 d:16 25 a:45

f:5 e:9 c:12 b:13

Unit 3 Greedy Algorithms 52

 Constructing a Huffman Coding Tree
From following Lines
30
z ← Allocate-Node
x ← left[z] ← Extract-Min(Q)
14 d:16 25 a:45
y ← right[z] ← Extract-Min(Q)
f[z] ← f[x]+f[y]
f:5 e:9 c:12 b:13

25 30 a:45 From following Lines

Insert(Q, z)

c:12 b:13 14 d:16

f:5 e:9

Unit 3 Greedy Algorithms 53

 Constructing a Huffman Coding Tree
From following Lines
a:45 55 z ← Allocate-Node
x ← left[z] ← Extract-Min(Q)
25 30
y ← right[z] ← Extract-Min(Q)
f[z] ← f[x]+f[y]
c:12 b:13 14 d:16
Insert(Q, z)

f:5 e:9

Unit 3 Greedy Algorithms 54

 Constructing a Huffman Coding Tree
From following Lines

100
z ← Allocate-Node
x ← left[z] ← Extract-Min(Q)
a:45 55
y ← right[z] ← Extract-Min(Q)
f[z] ← f[x]+f[y]
25 30
Insert(Q, z)

c:12 b:13 14 d:16

f:5 e:9

Unit 3 Greedy Algorithms 55

 Adding binary code in this tree

0 100 1

a:45 55
0 1 Add ‘0’ with left child, and
‘1’with right child of each
node
25 30
0 0 1
1

c:12 b:13 14 d:16

0 1

f:5 e:9

Unit 3 Greedy Algorithms 56

 Finding binary code from this tree

100 1
0 Char Freq Codeword
a:45 55 a 45 0
0 1
b 13 101
25 30 c 12 100
0 1
0 1 d 16 111
e 9 1101
c:12 b:13 14 d:16
f 5 1100
0 1

f:5 e:9

Unit 3 Greedy Algorithms 57

 Running Time analys
The analysis of the running time of Huffman's algorithm
assumes that Q is implemented as a binary min-heap.

• For a set C of n characters, the initialization of Q in

line 2 can be performed in O(n) time using the
BUILD-MIN-HEAP procedure.

• The for loop in lines 3-8 is executed exactly |n| - 1

times. Each heap operation requires time O(log n).
The loop contributes = (|n| - 1) * O(log n)

= O(nlog n)
Thus, the total running time of HUFFMAN on a set of n
characters = O(n) + O(nlog n)
= O(n log n)

Unit 3 Greedy Algorithms 58

 Thank You

Unit 3 Greedy Algorithms 59