Chapter 1.

Introduction to Algorithmic Thinking

Knowledge of algorithm and data structure is what differentiates dumb programmer from great
programmer. The programs that you write will become faster, more reliable, and simpler when
you can apply algorithmic thinking well.

1: Motivation
Imagine a list of 100 million sorted numbers (e.g. [1, 3, 55, 100, 125, ...]). One day you want to
check if a number X, say 17717, exists in the list. How would you do it?

1.1) Linear Search - Dumb way (and the most the straightforward)
You can scan the list from start to end. One by one.
If you find X, then you are done.
Otherwise, we reach the end of the list without finding X, which the list does not contain
X.

This is the first question we always ask when we come up with an approach:
Q1) How long will it take: in the worst case? best case? on average?
For the sake of discussion, let's say it takes 1 second to scan a number.
If X is somewhere near the end of the list, then we need roughly 100 million seconds.
This is the worst case.
If X is somewhere near the beginning of the list, then we need much much less than 100
million seconds (maybe just a few hundred seconds). This is the best case.
On average, we can intuitively say that we need 50 million seconds to find X.

The second question we always ask:

Q2) Can we do better?
Most often, the answer is yes! And here we'll briefly discuss it.

1.2) Binary Search - Smart way (and more complicated)

Binary means 2 ways. Either left or right. High or low. 0 or 1.
Notice that the list is sorted. The insight:
Imagine you randomly jump to a number in the list, somewhere in the middle. Say this
value is M.
If M is bigger than X (i.e. M > X), then we know that everything to the right of M will also
be bigger than X. This means, we can focus on the left side of M which can still
potentially contain X.
Otherwise, if M is smaller than X (i.e. M < X), then everything to the left of M will be
smaller than X too. Hence we can focus on the right side of M.

So, binary search simply:

 1. Pick the middle of list.
2. If value is found, stop.
3. Otherwise, check if we should throw away left part or right part.
4. Repeat on the remaining items.
5. If no more items remaining, then value is not found.

Illustration:
https://docs.google.com/presentation/d/1p-agsx6MxPD14XnSbjfcSDYBajRun_AGrhvg2E-oOik/
edit?usp=sharing

Let's answer the same question:

Q1 How long will it take? Worst case? Best case? Average case?
We will just concern ourselves with the worst case scenario of binary search.
On each iteration, we throw away half of the items in the list. So if the list started out with 100
million items, on the next iteration we have a list of 50 million, and then 25 million, and so on.
How many iteration can we have at most? That means, how many times we can split the list into
two parts. Let's count from 100 million, halved on each step:
100000000, 50000000, 25000000, 12500000, 6250000, 3125000, 1562500, 781250, 390625,
195312, 97656, 48828, 24414, 12207, 6103, 3051, 1525, 762, 381, 190, 95, 47, 23, 11, 5, 2, 1,
0.
So there are at most 27 iterations before we run out of items to binary search. Hence this
approach takes at most 27 seconds (assuming we take 1 second to jump). This is 3.7 million
times faster than the linear search approach.

Q2 Can we do better?
In this case, given the scenario, binary search is already one of the faster algorithms we can
use for finding item in a sorted list. Binary search is said to take O(log N) (i.e. logarithmic time),
while linear search is said to take O(N) (i.e. linear time). We will understand exactly what those
means over the course of study.

ashmap that allows use to have even

However, it is possible to have a data structure called h
faster search time of O(1) (constant time!). We will learn about those data structures over the
course of study.

As you can see, different algorithms can result in a drastically different performance
characteristics for the same task!
2: Pseudocode
To discuss algorithm more succinctly without having to be bogged down by the details of a
programming language, we can make use of "pseudocode". A pseudocode is just a way of
writing procedures that resembles programming languages without strict rules on syntax. The
main idea is to help communicate algorithms and data structures with each other. Different
programmer might have different styles, and that is fine.

Let's start with the linear search algorithm. We can write it this way in pseudocode:

let A be a l
ist of integers
v al is a n integer

// Returns the index of A for which the value is val.

linear_search(A, val):
for i from 0 to A.length-1
if A[i] == val
return i
return VALUE_NOT_FOUND

Which translates to the following in C:

int linear_search(int A[], int array_length, int val) {

const int VALUE_NOT_FOUND = -1;
for (int i = 0; i < array_length; ++i) {
if (A[i] == val) {
return i;
}
}
return VALUE_NOT_FOUND;
}

As you can see, writing in C could potentially obscure the main idea behind the algorithm itself.

While binary search could be written this way:

let A be a l
ist of sorted integers
v al is a n integer

// Returns the index of A for which the value is val.

binary_search(A, val):
lo = 0, hi = A.length-1
 while lo <= hi
mid = (lo + hi) / 2
if A[mid] == val
return mid
else if A[mid] < val
// Everything in A[lo . . m id] is also less t
han val.
// Hence we only need t o s earch on A[mid+1 . . hi]
lo = mid + 1
else if A[mid] > val
// Everything in A[mid .. hi] is also larger than val.
// Hence we only need to search on A[lo .. mid-1]
hi = mid - 1
return VALUE_NOT_FOUND

The pseudocode style can change over the course, but as long as we can understand exactly
what is going on, it is fine.

3: Exercise
Let's kickstart by implementing the above two algorithms.
You can use Visual C++ to implement the algorithms, and use the pseudocode above as
guidance.
Binary search has a powerful and simple concept, but is notoriously difficult to get right.
You can experiment and see how far you can implement it.
Over the course of study, we will develop a way of thinking that can help you reason about and
debug your implementation effectively.