CS 170: Efficient Algorithms and Intractable Problems on

Algorithms for Integer Arithmetic

Mon, 01 Jan 0001 00:00:00 +0000

💡 Creating good algorithms requires two main building blocks: correctness and efficiency.

Addition #

We would like to define an algorithm $A_{add}$ such that…

$(\forall x,y)(A_{add}(x,y) \to x+y)$

A naive approach #

One straightforward way to add is to “count by fingers”- that is, keep adding one continually until you reach the end of the number.

Given $n$ digits, this algorithm, in the worst case, must add one $10^n - 1$ times. This is quite inefficient! Let’s see if we can do better.

Divide and Conquer (Master Theorem)

Mon, 01 Jan 0001 00:00:00 +0000

Divide and Conquer: an Intro #

![[/cs170/img/Divide-and-Conquer-Master-Theorem/Untitled.png]]

Sometimes, especially for recursive algorithms, we can write a runtime as a recurrence relation: that is, part of a function’s runtime definition is the function itself with a smaller problem size.

This applies directly to a method called divide and conquer, which goes something like this:

We’re given some input $x$ and are trying to evaluate a function $A(x)$.
We can split $x$ into smaller parts $x_1, x_2, \cdots x_a$.
We can then find the output of each individual part $A(x_1), A(x_2), \cdots A(x_a)$.
Sum up all of these outputs to get the original desired value $A(x)$.

Take this example (Karatsuba Multiplication): $T(n) = 3T(n/2) + cn$

Matrix Multiplication

Mon, 01 Jan 0001 00:00:00 +0000

General Method #

If we were just to multiply two matrixes by hand (let’s say we are trying to find $XY$), then we would need to multiply the $i$th row of $X$ with the $j$th column of $Y$ to get the $i,j$ position of the result $Z$.

![[/cs170/img/Matrix-Multiplication/Untitled.png]]

Naive Algorithm #

A straightforward way of going by this is simply to loop through every possible value of $i$ and $j$:

Sorting

Mon, 01 Jan 0001 00:00:00 +0000

General Introduction #

The problem of sorting: given a list of numbers $a_1, \cdots, a_n$, output them in increasing or decreasing order.

The Idea (Divide and conquer):

Split the list in to two halves.
Recursively sort each half.
Merge the two halves together.

This sounds familiar…

Merge Sort #

def mergesort(lst):
	if lst has 1 element:
		return lst[0]
	s_l = mergesort(first half of lst)
	s_r = mergesort(second half of lst)
	s = merge(s_l, s_r)
	return s

def merge(s_l, s_r):
	result = []
	while s_r is not empty:
		if s_l is empty:
			return result + s_r
		if s_l[0] > s[r]:
			result += s_l[0]
			remove s_l[0]
	return result + s_l

The runtime of the merge operation is $\Theta(|s_l| + |s_r|)$ since we go through each element in each list exactly once.

Median Finding

Mon, 01 Jan 0001 00:00:00 +0000

Problem: Given a set $S=\{a_1, \cdots, a_n\}$, find the median $a \in S$ such that half of the values in $S$ are greater than $a$ and the other half is less than $a$.

The median is less sensitive to outliers than the mean.

Finding the Median #

A naive solution #

One method of finding the median is simply sorting the list, then getting the middle element. This requires $n\log(n)$ comparisons (see [[03 Sorting]] for more information).

Fast Fourier Transform (FFT)

Mon, 01 Jan 0001 00:00:00 +0000

Polynomial Multiplication #

The Problem #

Let’s say we have two input polynomials $A(x) = \sum_{i=0}^d a_ix^i$ and $B(x) = \sum_{i=0}^d b_ix^i$.

We would like to find an output $C(x) = A(x) \cdot B(x)$. $C(x)$ has a degree $2d$.

By definition of polynomial multiplication,

$$ C(x) = \sum_{i=0}^d \sum_{j=0}^d a_i b_j x^i x^j $$

Let’s gather all of the coefficients with the same power together to make this expression more useful:

Graphs

Mon, 01 Jan 0001 00:00:00 +0000

Graph Representations #

A graph is a pair $(V,E)$ where $V$ is the vertex set and $E \subseteq V \times V$ is the edge set. (For an introductory data-structures treatment, see [[cs61b/abstract-data-types/graphs]]; for the discrete-math/combinatorial properties used in proofs, see [[cs70/discrete-math/graphs]].)

There are 2 main ways to represent graphs for an algorithm:

Adjacency Matrix #

$M_G = \begin{pmatrix} m_{ij} \end{pmatrix} \in \{0,1\}^{m \times n}$

If $m_{ij} = 1$, then that means the vertices $i$ and $j$ are connected with an edge.

Greedy Algorithms

Mon, 01 Jan 0001 00:00:00 +0000

What are greedy algorithms? #

Greedy algorithms are a general algorithmic approach:

At every step, choose what to do based on local information without considering the next steps. This works nicely in some cases because of the efficiency of such a concept.

A simple example #

Let’s say you have 6 assignments that are due in 1, 1, 2, 2, 4, and 5 hours respectively. Each one takes 1 hour to complete. What is the maximum number you can complete before the deadline?

Minimum Spanning Trees

Mon, 01 Jan 0001 00:00:00 +0000

The Problem #

Given an undirected graph $G$ with positive edge weights, find a subset $T \subseteq E$ such that:

All vertices are connected.
The sum of edge weights in $T$ is minimized.

For step-by-step worked examples of the two classical algorithms, see [[cs61b/algorithms/minimum-spanning-trees/kruskals-algorithm]] and [[cs61b/algorithms/minimum-spanning-trees/prims-algorithm]].

We can notice that $T$ is actually a tree!

Tree Properties #

An undirected graph $T$ is a tree if:

$T$ is connected.
$T$ has no cycles.
$|E| = |V| - 1$.

Any 2 of these properties implies the third.

Huffman Coding

Mon, 01 Jan 0001 00:00:00 +0000

The Problem: Data Compression #

Given a string of characters ABACCDBB... from an alphabet $\{A,B,C,D \cdots\}$, how may bits do we need in order to encode it?

Of course, there is the naive way (just store the entire string), which requires $O(n)$ bits, but with Huffman encoding, we can do better!

Prefix-Free Codes #

One strategy is a greedy algorithm which assigns more frequently seen characters a shorter length bit representation. For example, if A occurs 40% of the time, B 30%, C 20%, and D 10%, we could do something like: A → 0, B → 10, C → 110, D → 111

Horn Formulas

Mon, 01 Jan 0001 00:00:00 +0000

Horn Formulas #

The Problem: Boolean Expressions #

Suppose we have a boolean expression such as $(w \lor y \lor z) \land (y \implies w) \lor (\lnot u \land \lnot v \land \lnot z)$ and so on.

We’d like to answer the question: are there any values of boolean variables that make this statement true?

In the general case, this problem is NP-hard…

A Special Case #

There are some special cases, though, that can be solved with a greedy algorithm. These are called Horn clauses:

Dynamic Programming

Mon, 01 Jan 0001 00:00:00 +0000

What is Dynamic Programming? #

Dynamic Programming (DP) is a general approach to many types of problems.

Main idea: Solve a big problem by breaking it into smaller subproblems, and solve subproblems in order from small to large.

The Approach:

Define subproblems (that are smaller than the original problem).
Show how we can solve the original problem given the answers to the subproblems.
Define base cases (when to return a simple result).
Choose the order to solve subproblems such that we can make the algorithm efficient.

How is this different from recursion though? #

In DP, we solve smaller subproblems first. In recursion, we don’t reach these subproblems until we try to calculate the larger problem.
In other words, when we reach a larger subproblem, all its subproblems must already have been computed.

Some Examples #

Shortest Paths in DAGs #

The Problem: Given a graph $G(V,E)$ with integer weights (negative weights ok), find the shortest path from $s \in V$ to $v \in V$.

Linear Programming

Mon, 01 Jan 0001 00:00:00 +0000

What is Linear Programming? #

Linear programming is expressing and solving linear optimization, typically in higher dimension problems.

An optimization problem has the following form:

$n$ real variables $x_1, \cdots, x_n \in \mathbb{R}$,
An objective function $f(x_1, \cdots, x_r) \in \mathbb{R}$
$m$ constraints $c_1, \cdots c_m$ such that $c_i(x_1, \cdots, x_n)$ returns either true or false.
A goal to get the maximum of $f(x_1, \cdots, x_n)$ such that all constraints are true.

A linear programming problem is a subset of optimization problems, such that:

Network Flow

Mon, 01 Jan 0001 00:00:00 +0000

What is Flow? #

Suppose we have a directed graph with a source $s$ and sink $v$. Each edge has a positive capacity $c > 0$.

A (feasible) flow is a function that maps edges to real numbers $f: E \to \mathbb{R}$ that satisfies:

Capacity constraints: $\forall e \in E$, $0 \le f(e) \le c(e)$. Each edge can only have a particular capacity (think pipes).
Conservation constraints: $\forall u \in V \setminus \{s, t\}$, $\sum f(w, u) = \sum f(u, v)$. The amount of flow that enters a particular vertex is the same amount that leaves that vertex.

The value of a flow is the amount that leaves the source. This should equal the amount that enters the sink if the flow is feasible.

Duality

Mon, 01 Jan 0001 00:00:00 +0000

Primal and Dual: An Example #

Suppose we have an original, known as a primal, linear program with variables $x_1, \cdots, x_n$ and some constraints on those variables.

As an example, let’s say we are trying to maximize $2x_1 + 4x_2$ such that:

$x_1, x_2 \ge 0$
$x_1 \le 50, x_2 \le 80$
$x_1 + x_2 \le 100$

We can rewrite the objective function as a linear combination of the constraints:

Zero-Sum Games

Mon, 01 Jan 0001 00:00:00 +0000

Zero Sum Games #

In this scheme, there are two players (known as the column player and row player, or analogously, minimizer and maximizer) playing a game with a particular strategy.

![[/cs170/img/Zero-Sum-Games/Untitled.png]]

A strategy generally entails assigning a probability to choosing one particular row and column (either A or B in the above image). In the matrix, a numerical element represents the payoff for making that move.

The total payoff can be calculated as a weighted sum, where the payoff of a particular row or column is multiplied by the probability that it is chosen.

Reductions

Mon, 01 Jan 0001 00:00:00 +0000

Introduction #

The strategy of reduction is to reduce a particular Problem A into another Problem B using a subroutine. We can then use Problem B to solve Problem A.

Good news: If Problem B is efficient, then this provides a fast algorithm for Problem A.

Bad news: If Problem A is hard, Problem B must be hard too.

Step by Step:

Preprocess - turn the input of A into an input for B.
Run Problem B.
Postprocess - convert the output of B into the output of A.

We can use reduction to:

Search Problems, P/NP

Mon, 01 Jan 0001 00:00:00 +0000

Intro to Search Problems #

Definitions #

A binary relation is a subset of pairs of finite bit strings, where $(x,w)$ is a (instance, witness) relationship. Remember that any data structure can be represented as a bit string, since it can be programmed into a computer.

The decision problem decide(R) takes an input instance $x$, and determines if there exists a witness $w$ such that $(x, w) \in R$, i.e. does there exist a solution to the problem?

Randomized Algorithms

Mon, 01 Jan 0001 00:00:00 +0000

Types of Randomized Algorithms #

There are two main classes of random algorithms, Las Vegas algorithms and Monte Carlo algorithms.

Las Vegas algorithms are correct, and probably fast.

For example, randomized quicksort (where we choose a random pivot) is guaranteed to return a correct sort regardless of the pivots chosen, but it’s possible to choose bad pivots and result in a poor runtime.

Monte Carlo algorithms are fast, and probably correct.

Streaming

Mon, 01 Jan 0001 00:00:00 +0000

Streaming Algorithms #

In applications like the internet, we might have vast amounts of data “streaming” by. How do we quickly store and update this data via approximation?

Examples of Streaming Algorithms #

Morris’s Algorithm for Approximate Counting #

Problem: Suppose we were counting total sales. Given an input of $n$ sales with prices $p_1, \cdots, p_n$, output the total $p = \sum_{i=1}^n p_i$.

Solution: Initialize a running counter $X$. With probability $\frac{1}{2^X}$, increment $X$ by $1$. Then, return $\tilde{n} = 2^X - 1$.

Lower Bounds

Mon, 01 Jan 0001 00:00:00 +0000

Typically, when we are considering algorithms we talk about the upper bounds of runtime, i.e. how long a function takes to calculate some result.

Now, let’s try to prove a lower bound that applies to any problem that solves a particular problem.

Sorting #

We know that all comparison sorting algorithms take $\Omega(n \log n)$ time. This is because this is the required number of comparisons needed to find a relation between all numbers in a tree structure.