Probability on

Mon, 01 Jan 0001 00:00:00 +0000

The probability section of this guide will likely never be fully completed, due to the fact that the Prob 140 textbook is such an excellent resource in probability theory. Go read it and do the problems!

Instead of a full write-up, the pages in this section will typically just link to relevant sections from the textbook. Personally, I found everything I needed to do well in CS70 probability (and much more) here, including examples that are very similar to problems you might see on the homework.

Mon, 01 Jan 0001 00:00:00 +0000

Introduction #

If you’re reading this, I think it’s safe to assume you already know how to count… (1, 2, 3, whatever) so what’s the big deal about counting?

When we say counting in this context, we mean counting sequences of decisions. For example, we might want to get the total number of ways to choose toppings on a pizza or something.

There are two main types of problems: those where order matters and those where it doesn’t.

Mon, 01 Jan 0001 00:00:00 +0000

Probability Basics #

http://prob140.org/textbook/content/Chapter_02/00_Calculating_Chances.html

Adding and subtracting probabilities - Multiplying probabilities: random draws without replacement, conditional probabilities

Bayes’ Rule #

Bayes’ Rule is used to re-express conditional probabilities $P(A|B)$.

http://prob140.org/textbook/content/Chapter_02/05_Updating_Probabilities.html#bayes-rule

Random Variables #

Probability Spaces #

Probability spaces describe all of the possible values of a random variable, and how likely each of those outcomes are.

http://prob140.org/textbook/content/Chapter_02/00_Calculating_Chances.html

Equality #

Two variables are equal if $X(\omega) = Y(\omega)$ for all $\omega \in \Omega$, where $\Omega$ is a probability space (all possible values).

Mon, 01 Jan 0001 00:00:00 +0000

A hash function assigns a value to each member in a set. It’s often useful to determine the probability of collisions: where two different items are assigned the same hash value. (For the data-structures use of hashing — hash tables, separate chaining, dynamic resizing — see [[cs61b/abstract-data-types/hashing]].)

http://prob140.org/textbook/content/Chapter_01/03_Collisions_in_Hashing.html

An interesting result is explored by the Birthday Problem (sometimes known as the Birthday Paradox, despite not actually being paradoxical), in which the probability of at least two people sharing the same birthday is much higher than expected.

Mon, 01 Jan 0001 00:00:00 +0000

The expectation of a random variable, $E(X)$, is the average of possible values weighted by their probabilities. Formally, it can be defined in two ways:

Domain definition: $E(X) = \sum_{\omega \in \Omega} X(\omega) P(\omega)$.
Range definition: $E(X) = \sum_x x P(X = x)$.

Expectation has nice properties of linearity: $E(X + Y) = E(X) + E(Y)$ and $E(aX + b) = aE(x) + b$.

http://prob140.org/textbook/content/Chapter_08/01_Definition.html

Mon, 01 Jan 0001 00:00:00 +0000

Also see the Data 102 notes on this topic.

Markov’s Inequality: http://prob140.org/textbook/content/Chapter_18/04_Chi_Squared_Distributions.html

Chebyshev’s Inequality: http://prob140.org/textbook/content/Chapter_18/04_Chi_Squared_Distributions.html

Chernoff Bound: http://prob140.org/textbook/content/Chapter_19/04_Chernoff_Bound.html?highlight=chernoff

Mon, 01 Jan 0001 00:00:00 +0000

(Credit: Huiyi Zhang)

All of the continuous probability distributions are deeply connected. Above is a chart describing some of their relationships.

Below are some links:

Mon, 01 Jan 0001 00:00:00 +0000

Markov Chains are a type of stochastic process (a collection of random variables that evolves over time) that satisfy the Markov property (the future state $n+1$ only depends on the current state $n$, and not any of the past states).

Markov chains are often used to model transitions between discrete states.

http://prob140.org/textbook/content/Chapter_10/00_Markov_Chains.html

See also: [[cs188/Markov Models]] (treats Markov chains as time-series Bayes nets) and [[cs188/Markov Decision Processes]] (extends Markov chains with actions and rewards).

Mon, 01 Jan 0001 00:00:00 +0000

The Beta distribution is a family of continuous distributions on [0,1] with two parameters (commonly known as $\alpha$ and $\beta$, but also $r$ and $s$).

Beta distributions are commonly used in situations where we want to continually update a prior distribution given new information.

The beta (1,1) distribution is identical to the uniform distribution.

http://prob140.org/textbook/content/Chapter_21/00_The_Beta_and_the_Binomial.html

Mon, 01 Jan 0001 00:00:00 +0000

Properties:

Conditional Expectation #

$E(X|Y)$is the conditional expectation of $X$given $Y$

$E(X|Y=y)$is a fixed value, but $E(X|Y)$is a random variable $it is a function of $Y$$
Iterated expectation: $E(E(X|Y)) = E(X)$
Additivity: $E(Y+Z | X) = E(Y|X) + E(Z|X)$
- does not work on the right hand side: $E(Y | X+Z) \ne E(Y|X) + E(Y|Z)$
Linearity: $E(aX + b | Y) = aE(X|Y) + b$
Conditioning on the same variable: $E(g(S)T | S) = g(S)E(T|S)$

Conditional Variance #

If $Var(Y)$is difficult to find directly, we can use the variance decomposition to condition the variance on another variable.