Reference Books

Introduction
Time Complexity and Space Complexity
Order of complexities

Design and Analysis of Algorithm

Sanjib Sadhu1
1 Department

of Computer Science & Engineering

NIT Durgapur
Deemed University

2012-2013

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Outline
1

Reference Books

Introduction
Algorithm

Time Complexity and Space Complexity

The RAM model
Time Complexity of Insertion Sort

Order of complexities
Big-Oh (O) notation
Big-Omega () notation
Theta () notation
Small-Oh (o) notation
Small-Omega () notation

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The slide does not contain everything.

You must attend the classes and try to note down the
important points in your copy.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The slide does not contain everything.

You must attend the classes and try to note down the
important points in your copy.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Books on Algorithm

Title: Algorithms - Design Techniques and Analysis

Author-M. H. Alsuwaiyel

Title: Introduction To Algorithms

Authors-Thomas H Cormen, Charles E Leiserson, Ronald
L Rivest, Clifford Stein

Title: Computer Algorithms

Authors-Ellis Horowitz, Sartaj Sahni, Sanguthevar
Rajasekaran

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Algorithm

Outline
1

Reference Books

Introduction
Algorithm

Time Complexity and Space Complexity

The RAM model
Time Complexity of Insertion Sort

Order of complexities
Big-Oh (O) notation
Big-Omega () notation
Theta () notation
Small-Oh (o) notation
Small-Omega () notation

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Algorithm

What is Algorithm
Algorithm
Definition
An algorithm is a procedure that consists of a finite set of
instructions which, given an input from some set of possible
inputs, enables us to obtain an output if such an output exists or
else obtain nothing at all if there is no output for that particular
input through a systematic execution of the instructions.
The algorithm is deterministic in the sense that the same set of
instructions are executed when the algorithm is initiated on a
particular input more than once. However, we will relax this
condition for randomized algorithm.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Algorithm

What is Algorithm
Algorithm
Definition
An algorithm is a procedure that consists of a finite set of
instructions which, given an input from some set of possible
inputs, enables us to obtain an output if such an output exists or
else obtain nothing at all if there is no output for that particular
input through a systematic execution of the instructions.
The algorithm is deterministic in the sense that the same set of
instructions are executed when the algorithm is initiated on a
particular input more than once. However, we will relax this
condition for randomized algorithm.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Algorithm

What is the difference between an algrithm and a program?

Why we try to redesign algorithm for a problem even
though one such algorithm already exists?

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Algorithm

What is the difference between an algrithm and a program?

Why we try to redesign algorithm for a problem even
though one such algorithm already exists?

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Time Complexity and Space Complexity

Time Complexity T(n)
The running time of an algorithm on a particular input is the
number of primitive operations or steps executed. It is
convenient to define the notion of step so that it is as
machine-independent as possible. The time complexity of an
algorithm is denoted by T (n) and measured in terms of n, the
input.
Space Complexity S(n)
The space used by an algorithm is the number of memory cells
(or words) needed to carry out the computational steps
required to solve an instance of the problem. The space
complexity refers to the work space required by the algorithm.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Time Complexity and Space Complexity

Time Complexity T(n)
The running time of an algorithm on a particular input is the
number of primitive operations or steps executed. It is
convenient to define the notion of step so that it is as
machine-independent as possible. The time complexity of an
algorithm is denoted by T (n) and measured in terms of n, the
input.
Space Complexity S(n)
The space used by an algorithm is the number of memory cells
(or words) needed to carry out the computational steps
required to solve an instance of the problem. The space
complexity refers to the work space required by the algorithm.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Outline
1

Reference Books

Introduction
Algorithm

Time Complexity and Space Complexity

The RAM model
Time Complexity of Insertion Sort

Order of complexities
Big-Oh (O) notation
Big-Omega () notation
Theta () notation
Small-Oh (o) notation
Small-Omega () notation

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

RAM model

How can we study the time complexity of an algorithm if we

donot choose a specific machine to measure it on?
We shall assume a generic one-processor, random-access
machine(RAM) model of computation as our
implementation technology and understand that our
algorithms will be implemented as computer program.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

RAM model

How can we study the time complexity of an algorithm if we

donot choose a specific machine to measure it on?
We shall assume a generic one-processor, random-access
machine(RAM) model of computation as our
implementation technology and understand that our
algorithms will be implemented as computer program.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

RAM model
In the RAM model, instructions are executed one after
another, with no concurrent operations.
The RAM model contains instructions commonly found in
real computers:
Arithmetic(add, subtract, multiply, divide, remainder, floor,
ceiling)
Data movement(load, store, copy)
Control(conditional and unconditional branch, subroutione
call and return)

Each such instruction takes a constant amount of time.

The data types in the RAM model are integer and floating
point.
In RAM model, we do not attempt to model the memory
hierarchy that is common in the contemporary computers.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

RAM model
In the RAM model, instructions are executed one after
another, with no concurrent operations.
The RAM model contains instructions commonly found in
real computers:
Arithmetic(add, subtract, multiply, divide, remainder, floor,
ceiling)
Data movement(load, store, copy)
Control(conditional and unconditional branch, subroutione
call and return)

Each such instruction takes a constant amount of time.

The data types in the RAM model are integer and floating
point.
In RAM model, we do not attempt to model the memory
hierarchy that is common in the contemporary computers.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

RAM model
In the RAM model, instructions are executed one after
another, with no concurrent operations.
The RAM model contains instructions commonly found in
real computers:
Arithmetic(add, subtract, multiply, divide, remainder, floor,
ceiling)
Data movement(load, store, copy)
Control(conditional and unconditional branch, subroutione
call and return)

Each such instruction takes a constant amount of time.

The data types in the RAM model are integer and floating
point.
In RAM model, we do not attempt to model the memory
hierarchy that is common in the contemporary computers.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

RAM model
In the RAM model, instructions are executed one after
another, with no concurrent operations.
The RAM model contains instructions commonly found in
real computers:
Arithmetic(add, subtract, multiply, divide, remainder, floor,
ceiling)
Data movement(load, store, copy)
Control(conditional and unconditional branch, subroutione
call and return)

Each such instruction takes a constant amount of time.

The data types in the RAM model are integer and floating
point.
In RAM model, we do not attempt to model the memory
hierarchy that is common in the contemporary computers.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Worst case, Best case and Average case

The worst case complexity of the algorithm is the function

defined by the maximum number of steps taken on any
instance of size n.
The best case complexity of the algorithm is the function
defined by the minimum number of steps taken on any
instance of size n.
The average-case complexity of the algorithm is the
function defined by an average number of steps taken on
any instance of size n.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Worst case, Best case and Average case

The worst case complexity of the algorithm is the function

defined by the maximum number of steps taken on any
instance of size n.
The best case complexity of the algorithm is the function
defined by the minimum number of steps taken on any
instance of size n.
The average-case complexity of the algorithm is the
function defined by an average number of steps taken on
any instance of size n.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Worst case, Best case and Average case

The worst case complexity of the algorithm is the function

defined by the maximum number of steps taken on any
instance of size n.
The best case complexity of the algorithm is the function
defined by the minimum number of steps taken on any
instance of size n.
The average-case complexity of the algorithm is the
function defined by an average number of steps taken on
any instance of size n.

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Outline
1

Reference Books

Introduction
Algorithm

Time Complexity and Space Complexity

The RAM model
Time Complexity of Insertion Sort

Order of complexities
Big-Oh (O) notation
Big-Omega () notation
Theta () notation
Small-Oh (o) notation
Small-Omega () notation

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort
Assumption: Each execution of the i th line takes time ci , where
ci is a constant. Length of array A, denoted by length[A] is n.
cost
INSERTION-SORT(A)
times
c1
1 for j 2 to length[A]
n
c2
2
do key A[j]
n1
3
/* Insert A[j] into the sorted
0
sequence A[1..j-1]*/
n1
c
4
i j-1
n1
4
P
n
c5
5
while ((i>0) and (A[i]>key))
t
Pnj=2 j
c6
6
do A[i+1] A[i]
(t 1)
Pnj=2 j
c7
7
i i-1
j=2 (tj 1)
c8
8
A[i+1] key
n1
Here, tj is the number of times the while loop test in line 5 is
executed for that value of j.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort
Assumption: Each execution of the i th line takes time ci , where
ci is a constant. Length of array A, denoted by length[A] is n.
cost
times
INSERTION-SORT(A)
c1
1 for j 2 to length[A]
n
c2
2
do key A[j]
n1
3
/* Insert A[j] into the sorted
0
sequence A[1..j-1]*/
n1
c
4
i j-1
n1
4
P
n
c5
5
while ((i>0) and (A[i]>key))
t
Pnj=2 j
c6
6
do A[i+1] A[i]
(t 1)
Pnj=2 j
c7
7
i i-1
j=2 (tj 1)
c8
8
A[i+1] key
n1
Here, tj is the number of times the while loop test in line 5 is
executed for that value of j.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort
Assumption: Each execution of the i th line takes time ci , where
ci is a constant. Length of array A, denoted by length[A] is n.
cost
times
INSERTION-SORT(A)
c1
1 for j 2 to length[A]
n
c2
2
do key A[j]
n1
3
/* Insert A[j] into the sorted
0
sequence A[1..j-1]*/
n1
c
4
i j-1
n1
4
P
n
c5
5
while ((i>0) and (A[i]>key))
t
Pnj=2 j
c6
6
do A[i+1] A[i]
(t 1)
Pnj=2 j
c7
7
i i-1
j=2 (tj 1)
c8
8
A[i+1] key
n1
Here, tj is the number of times the while loop test in line 5 is
executed for that value of j.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort
Assumption: Each execution of the i th line takes time ci , where
ci is a constant. Length of array A, denoted by length[A] is n.
cost
times
INSERTION-SORT(A)
c1
1 for j 2 to length[A]
n
c2
2
do key A[j]
n1
3
/* Insert A[j] into the sorted
0
sequence A[1..j-1]*/
n1
c
4
i j-1
n1
4
P
n
c5
5
while ((i>0) and (A[i]>key))
t
Pnj=2 j
c6
6
do A[i+1] A[i]
(t 1)
Pnj=2 j
c7
7
i i-1
j=2 (tj 1)
c8
8
A[i+1] key
n1
Here, tj is the number of times the while loop test in line 5 is
executed for that value of j.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort

The running time of the algorithm is the sum of the running

times for each statement executed.
A statement that takes ci steps to execute and is executed
n times will contribute ci n to the total running times.
The total running time is given by
P
T (n) = c1 n + c2 (n 1) + c4 (n 1) + c5 nj=2 tj +
P
P
c6 nj=2 (tj 1) + c7 nj=2 (tj 1) + c8 (n 1)

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort

Best case cost occurs when the given numbers are

already ordered
Here, tj = 1, and line 6 and 7 will be executed 0 times
T (n) = c1 n + c2 (n 1) + c4 (n 1) + c5 (n 1) + c8 (n 1)
= ((c1 +c2 +c4 +c5 +c8 )n(c2 +c4 +c5 +c8 )) = K1 nK2
where, K1 = (c1 + c2 + c4 + c5 + c8 ) and
K2 = (c2 + c4 + c5 + c8 )

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort
Worst case cost occurs when the given numbers are in
reverse order


P
P
Here, tj = j. Now, nj=2 tj = nj=2 j = n(n+1)

1
and
2
Pn
Pn
n(n1)
j=2 (tj 1) =
j=2 (j 1) =
2


T (n) = c1 n + c2 (n 1) + c4 (n 1) + c5 n(n+1)

1
2




n(n1)
n(n1)
1 + c7
+ c8 (n 1)
+c6
2

2


= (c5 +c26 +c7 ) n2 + c1 + c2 + c4 + c25 c26 c27 + c8 n

(c2 + c4 + c5 + c8 ) = an2 + bn + c
Average case cost occurs when the input is in random
order. In average, tj = 2j . T (n) will still be in the order of n2 ,
same as the worst case.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

The RAM model

Time Complexity of Insertion Sort

Insertion Sort
Worst case cost occurs when the given numbers are in
reverse order


P
P
Here, tj = j. Now, nj=2 tj = nj=2 j = n(n+1)

1
and
2
Pn
Pn
n(n1)
j=2 (tj 1) =
j=2 (j 1) =
2


T (n) = c1 n + c2 (n 1) + c4 (n 1) + c5 n(n+1)

1
2




n(n1)
n(n1)
1 + c7
+ c8 (n 1)
+c6
2

2


= (c5 +c26 +c7 ) n2 + c1 + c2 + c4 + c25 c26 c27 + c8 n

(c2 + c4 + c5 + c8 ) = an2 + bn + c
Average case cost occurs when the input is in random
order. In average, tj = 2j . T (n) will still be in the order of n2 ,
same as the worst case.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Big-Oh O
The O-notation provides an upper bound on the running time.
f (n), g(n)

Definition
A function f (n) is said to be of the order
O(g (n)) (written as f (n) = O(g(n)), if
there exists two positive constants c and
n0 such that n n0 , f (n) c g (n).
Alternately,
f (n)
f (n)
exists and lim
6= ,
n g(n)
n g(n)

if lim

we will say f (n) = O(g(n)) .

Instructor: Sanjib Sadhu

CSE-603

c g(n)
f (n)

n0

f (n)=O(g(n))
implies that f (n)
grows no faster
than some
constant time g(n).

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Big-Oh O
The O-notation provides an upper bound on the running time.
f (n), g(n)

Definition
A function f (n) is said to be of the order
O(g (n)) (written as f (n) = O(g(n)), if
there exists two positive constants c and
n0 such that n n0 , f (n) c g (n).
Alternately,
f (n)
f (n)
exists and lim
6= ,
n g(n)
n g(n)

if lim

we will say f (n) = O(g(n)) .

Instructor: Sanjib Sadhu

CSE-603

c g(n)
f (n)

n0

f (n)=O(g(n))
implies that f (n)
grows no faster
than some
constant time g(n).

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Big-Oh O
Example
If
f (n) = 2n3 + 5n + 7,
Here, f (n) 14 n3 , n 1.
Therefore,
 
f (n) = O n3
Instead of writing f (n) = 2n3 + 8n2 + 5n + 7, we may write
f (n) = 2n3 + O(n2 ). This is helpful if we are not interested
in the details of the lower order terms.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Big-Omega
provides a lower bound within a constant factor of the running
time.
Definition
A function f (n) is said to be of the order (g(n)) (written as
f (n) = (g(n))), if there exists two positive constants c and n0
such that n n0 , f (n) c g (n).
Alternately,
if lim

f (n)
f (n)
exists and lim
6= 0,
n g(n)
g(n)

we will say f (n) = (g(n)). Informally, this definition says that f

grows at least as fast as some constant times g.
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Big-Omega
f (n), g(n)

Example
f (n)

If
3

f (n) = 2n + 5n + 7,
 
f (n) = n3

c g(n)

n0

f (n)=(g(n))
implies that f (n)
grows at least as fast
as some constant
times g(n).
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Theta
The provides an assymptotic tight bound on the running time.
Definition
A function f (n) is said to be of the order (g (n)) (written as
f (n) = (g(n)) ), if there exists three positive constants c1 , c2
and n0 such that n n0 , c1 g (n) f (n) c2 g (n).
Alternately, if
lim

f (n)
f (n)
exists and lim
= K , where 0 < K <
n
g(n)
g(n)

we will say f (n) = (g(n)) .

f (n) = (g(n)) iff f (n) = O(g(n)) and f (n) = (g(n)).
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Theta

Example
If
f (n) = 2n3 + 5n + 7,
 
f (n) = n3

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Small-Oh o I

Definition
A function f (n) is said to be of the order o (g (n)) (written as
f (n) = o(g(n)) ), if for every constant c > 0 there exists a
positive integer n0 such that n n0 , f (n) < c g (n).
Alternately,
f (n)
f (n)
exists and lim
= 0,
n g(n)
n g(n)

if lim

we will say f (n) = o(g(n)) .

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Small-Oh o II

Example
If
f (n) = 2n3 + 5n + 7,


f (n) = o n3.1
but

 
f (n) 6= o n3

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Small-Omega I

Definition
A function f (n) is said to be of the order (g (n)) (written as
f (n) = (g(n)) ), if for any constant c > 0 there exists a positive
integer n0 such that n n0 , f (n) > c g (n).
Alternately,
g(n)
g(n)
exists and lim
= 0,
n f (n)
n f (n)

if lim

we will say f (n) = (g(n)) .

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Small-Omega II

Example
If
f (n) = 2n3 + 5n + 7,
 
f (n) = n2
but

 
f (n) 6= n3

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Properties of O, , , o, I

Transitivity
f (n) = O(g(n)) and g(n) = O(h(n)), f (n) = O(h(n)).
f (n) = (g(n)) and g(n) = (h(n)), f (n) = (h(n)).
f (n) = (g(n)) and g(n) = (h(n)), f (n) = (h(n)).
f (n) = o(g(n)) and g(n) = o(h(n)), f (n) = o(h(n)).
f (n) = (g(n)) and g(n) = (h(n)), f (n) = (h(n)).

Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Properties of O, , , o, II
Reflexivity
f (n) = O(f (n)).
f (n) = (f (n)).
f (n) = (f (n)).
Symmetry
f (n) = (g(n)) if and only if g(n) = (f (n)).
Transpose Symmetry
f (n) = O(g(n)) if and only if g(n) = (f (n)).
f (n) = o(g(n)) if and only if g(n) = (f (n)).
Instructor: Sanjib Sadhu

CSE-603

Reference Books
Introduction
Time Complexity and Space Complexity
Order of complexities

Properties of O, , , o, III
Theorem
If f1 (n) = O(g1 (n)) and f2 (n) = O(g2 (n)), then
1. f1 (n) + f2 (n) = O(max(g1 (n), g2 (n))) // Rule of Sum
2. f1 (n) f2 (n) = O(g1 (n) g2 (n)) // Rule of Product
Questions
Prove the above theorems
Is f1 (n) f2 (n) = O(g1 (n) g2 (n))?
Is

f1 (n)
f2 (n)

(n)
= O( gg21 (n)
)?

Instructor: Sanjib Sadhu

CSE-603