Lec 1 Introduction Data Structures Algorithm

Data Structures & Algorithm

•

_{Algorithm: Outline, the essence of a}

computational procedure, step-by-step

instructions

•

_{Program: an implementation of an algorithm in}

some programming language

•

_{Data structure: Organization of data needed to}

History

• _{Name: Persian mathematician Mohammed}

al-Khowarizmi, in Latin became Algorismus

• _{First algorithm: Euclidean Algorithm, greatest}

common divisor, 400-300 B.C.

• ₁₉th century – Charles Babbage, Ada Lovelace.

• ₂₀th century – Alan Turing, Alonzo Church, John

Algorithmic problem

• _{Infinite number of input instances satisfying the specification. For}

example:

• _{A sorted, non-decreasing sequence of natural numbers. The sequence}

is of non-zero, finite length:

• _{1, 20, 908, 909, 100000, 1000000000.} • _3.

Specification

of input

?

Specification

of output as

a function of

Algorithmic Solution

• Algorithm describes actions on the input instance

• Infinitely many correct algorithms for the same algorithmic

problem

Input instance,

adhering to

the

specification

Algorithm

Output

related to

the input as

What is a Good Algorithm?

• _Efciency:

• _{Running time}

• _{Space used}

• _{Efciency as a function of input size:}

• _{Number of data elements (numbers, points)}

Measuring the Running Time

•

How should we measure the running

time of an

algorithm

?

•

Approach 1: Experimental Study

• _{Write a program that implements the}

algorithm

• _{Run the program with data sets of varying}

size and composition.

• _{Use a method like System.currentTimeMillis()}

to get an accurate measure of the actual running time.

t (ms)

Limitation Experimental Studies

• _{Experimental studies have several limitations:}

•

It is necessary to

implement

and test

the algorithm in order to determine its

running time.

•

Experiments can be done only on a

limited set of

inputs

, and may not be

indicative of the running time on

other inputs not included in the

experiment.

•

_{In order to compare two algorithms,}

Beyond Experimental Studies

• _{We will now develop a general methodology for}

analyzing the running time of algorithms. In contrast to the "experimental approach", this methodology:

•

Uses a

high-level description

of the

algorithm instead of testing one of its

implementations.

•

Takes into account

all possible inputs.

•

Allows one to evaluate the efciency

of any algorithm in a way that is

independent from the hardware and

Pseudo-code

• Pseudo-code is a description of an algorithm that is more

structured than usual prose but less formal than a programming language.

• _{Example: finding the maximum element of an array.}

Algorithm arrayMax(A, n):

Input: An array A storing n integers. Output: The maximum element in A.

currentMax  A[0] for i 1 to n -1 do

if currentMax < A[i] then currentMax  A[i] return currentMax

• _{Pseudo-code is our preferred notation for describing}

algorithms.

What is Pseudo-code?

• A mixture of natural language and high-level programming

concepts that describes the main ideas behind a generic implementation of a data structure or algorithm.

Expressions:

• _{use standard mathematical symbols to describe numeric and}

boolean expressions

• _use  _{for assignment (“=” in Java)}

• _{use = for the equality relationship (“==” in Java)}

Method Declarations:

What is Pseudo-code?

• Programming Constructs:

• decision structures:if ... then ... [else ] • while-loops: while ... Do

• _{repeat-loops: repeat ... until …} • for-loop: for ... Do

• _{array indexing: A[i]}

• _{Functions/Method}

calls: object method(args)

Analysis of Algorithms

• Primitive Operations: Low-level computations

independent from the programming language can be identified in pseudocode.

• Examples:

• calling a function/method and returning from a

function/method

• arithmetic operations (e.g. addition) • comparing two numbers

• indexing into an array, etc.

• By inspecting the pseudo-code, we can count the

Sort

Example: Sorting

INPUT

sequence of numbers

a

₁

, a

₂

, a

₃

,….,a

_n

b

1

,b

2

,b

3

,….,b

n

OUTPUT

a permutation of the sequence of numbers

2 5 4 10 7 2 4 5 7 10

Correctness

For any given input the algorithm halts with the output:

• b₁ < b₂ < b₃ < …. < b_n • b₁, b₂, b₃, …., b_n is a

permutation of a₁, a₂, a₃,….,a_n

Correctness

For any given input the algorithm halts with the output:

• b₁ < b₂ < b₃ < …. < b_n • _{how (partially) sorted}

they are

• _algorithm

Running time

Depends on

• _{number of elements (n)} • _{how (partially) sorted}

they are

Insertion Sort

• _{Start “empty handed”} • _{Insert a card in the right}

position of the already sorted

hand

• _{Continue until all cards are} inserted/sorted

Strategy

• _{Start “empty handed”} • _{Insert a card in the right}

position of the already sorted

hand

• _{Continue until all cards are}

inserted/sorted

for j=2 to length(A) do key=A[j]

“insert A[j] into the sorted sequence A[1..j-1]”

i=j-1 sorted sequence A[1..j-1]”

i=j-1

while i>0 and A[i]>key do A[i+1]=A[i]

Analysis of Insertion Sort

for j=2 to length(A)

do key=A[j]

“insert A[j] into the sorted sequence A[1..j-1]”

i=j-1

function of the input size

Best/Worst/Average Case

• Best case: elements already sorted _® t_j=1,

running time = f(n), i.e., linear time.

• _{Worst case: elements are sorted in inverse}

order

® tj=j, running time = f(n2), i.e., quadratic time

• Average case: t_j=j/2, running time = f(n2), i.e.,

quadratic time

Total Time= n(c1+c2+c3+c7)+

(c4+c5+c6)-(c2+c3+c5+c6+c7)

2

n j j t

Best/Worst/Average Case (2)

• For a specific size of input n, investigate running times for

different input instances:

Best/Worst/Average Case (3)

• _{For inputs of all sizes:}

1n 2n 3n 4n 5n 6n

R

un

ni

ng

ti

m

e

best-case

Best/Worst/Average Case (4)

• _{Worst case is usually used:}

• It is an upper-bound and in certain application

domains (e.g., air trafc control, surgery) knowing the worst-case time complexity is of crucial importance

• _{For some algorithms worst case occurs fairly}

often

• _{The average case is often as bad as the}

worst case

• Finding the average case can be very

Asymptotic Notation

• _{Goal: to simplify analysis by getting rid of unneeded}

information (like “rounding” 1,000,001≈1,000,000)

• _{We want to say in a formal way 3n}2 _{≈ n}2

• _{The “Big-Oh” Notation:}

• _{given functions f(n) and g(n), we say that f(n) is O(g(n))}

Asymptotic Notation

• The “big-Oh” O-Notation

• _{asymptotic upper bound}

Example

g

(

n

)

=

n

c g

(

n

)

=

4

n

f

(

n

)

=

2

n +

6

For functions

f(n)

and

g(n)

(to the right)

there are positive

constants

c

and

n

₀

such that:

f(n)≤

c g(n)

for

n

≥

n

₀

Another Example

On the other hand…

n

2

is not O(

n

) because there is

no

c

and

n

₀

such that:

n

2

_≤

cn

for

n

≥

n

₀

(As the graph to the right

illustrates, no matter how large

a

c

is chosen there is an

n

big

enough that

n

2

_>

_cn

₎

Asymptotic Notation (cont.)

•

Note

: Even though it is

correct

to say

_{“7n - 3}

is O(n

3

)”, a better statement is “7n - 3 is O(n)”,

that is, one should make the approximation as

tight as possible (for all values of n).

•

Simple Rule

: Drop lower order terms and

constant factors

Asymptotic Analysis of The Running

Time

•

Use the Big-Oh notation to express the number of

primitive operations executed as a function of the

input size.

•

For example, we say that the

arrayMax

algorithm

runs in

O

(n) time.

•

Comparing the asymptotic running time

-an algorithm that runs in O(n) time is better than one that runs in

O(n2_{) time}

-similarly, O(log n) is better than O(n)

-hierarchy of functions: log n << n << n2 _{<< n}3 _{<< 2}n

•

Caution!

Beware of very large constant factors. An

algorithm running in time 1,000,000 n is still

O

(n) but

might be less efcient on your data set than one

Example of Asymptotic Analysis

An algorithm for computing prefix averages

Algorithm prefixAverages1(X):

Input: An n-element array X of numbers.

Output: An n -element array A of numbers such that A[i] is the average of elements X[0], ... , X[i].

Let A be an array of n numbers. for i 0 to n - 1 do

a  0

for j  0 to i do a  a + X[j] A[i]  a/(i+ 1)

return array A

1 step

i iterations

with

i=0,1,2...n-1

Another Example

•

A better algorithm for computing prefix

averages:

Algorithm prefixAverages2(X):

Input: An n-element array X of numbers.

Output: An n -element array A of numbers such that A[i] is the average of elements X[0], ... , X[i].

Let A be an array of n numbers. s 0

for i  0 to n do s  s + X[i] A[i]  s/(i+ 1) return array A

Asymptotic Notation

(terminology)

•

Special classes of algorithms:

logarithmic: O(log n)

linear: O(n)

quadratic: O(n2₎

polynomial: O(nk), k ≥ 1

exponential: O(an), n > 1

•

“Relatives” of the Big-Oh

• The “big-Omega” W-Notation • _{asymptotic lower bound}

• f(n) = W(g(n)) if there exists

constants c and n₀, s.t. c g(n) £ f(n) for n ³ n₀

• Used to describe best-case running

times or lower bounds of algorithmic problems

• E.g., lower-bound of searching in

an unsorted array is W(n).

• The “big-Theta” Q-Notation • _{asymptoticly tight bound}

• f(n) = Q(g(n)) if there exists

Asymptotic Notation (4)

Asymptotic Notation (5)

• _{Two more asymptotic notations}

• _{"Little-Oh" notation f(n)=o(g(n))}

non-tight analogue of Big-Oh

• For every c, there should exist n₀ , s.t. f(n) £ c g(n)

for n ³ n₀

• _{Used for comparisons of running times. If} f(n)=o(g(n)), it is said that g(n) dominates f(n).

• _{"Little-omega" notation} f(n)=w(g(n))

Asymptotic Notation (6)

• _{Analogy with real numbers}

• _{f(n) = O(g(n)) @ f £ g} • _{f(n) = W(g(n)) @ f ³ g}

• _{f(n) = Q(g(n)) @ f = g} • _{f(n) = o(g(n)) @ f < g} • f(n) = w(g(n)) _{@ f > g}

• _{Abuse of notation: f(n) = O(g(n)) actually means}

Comparison of Running Times

Running

Time in

m

s

Maximum problem size (n)

1 second 1 minute 1 hour

400

n

2500

150000

9000000

20

n

log

n

4096

166666

7826087

2

n

2

707

5477

42426

n

4

31

88

244

Math You Need to Review

Logarithms and Exponents

• _{properties of logarithms:}

log_b(xy) = log_bx + log_by log_b (x/y) = log_bx - log_by log_bxa = alog

bx

Log_ba = log_xa/log_xb

• properties of exponentials:

a(b+c) _{= a}b_ac

abc _{= (a}b₎c

ab/ac = a(b-c)

b = a log ab

A Quick Math Review

• _{Geometric progression}

• given an integer n₀ and a real number 0< a ¹ 1

• _{geometric progressions exhibit exponential growth}

• Arithmetic progression

Summations

• _{The running time of insertion sort is determined}

by a nested loop

• _{Nested loops correspond to summations}

Proof by Induction (2)