Probability And Distribution

Probability And Random Variables

• Probability space allows us to quantify the idea of probability. But often we do not work directly with probability space. Instead, we work with random variables, which transfer the probability to a more convenient (often numerical) space.

• The sample space $\Omega$ is the set of all possible outcomes of the experiment.

• The event space $\mathcal{A}$ is the space of potential results of the experiment. It is obtained by considering the collection of subsets of $\Omega$, and for discrete probability distributions $\mathcal{A}$ is often the power set of $\Omega$. The event space is a $\sigma$-algebra.

• The probability $P$. With each event $A \in \mathcal{A}$, we associate a number $P(A)$ that measures the degree of belief that the event will occur. $P(A)$ is called the probability of event $A$.

In ML, we often avoid explicitly referring to the probability space, but instead refer to the probability of quantities of interest, which we denote by $\mathcal{T}$. We refer to $\mathcal{T}$ as the target space and refer to elements of $\mathcal{T}$ as states.
We introduce a function $X:\Omega \rightarrow \mathcal{T}$ that takes an element of $\Omega$ (an outcome) and returns a particular quantity of interest $x$, a value in $\mathcal{T}$. e.g., In the case of tossing two coins and counting the number of heads, we can define a random variable $X$ that maps outcomes to the number of heads: $X(hh) = 2$, $\dots$.
For any subset $S \subseteq \mathcal{T}$, we associate a probability $P_X(S) \in [0, 1]$ to the event that the random variable $X$ takes a value in $S$.

Statistics

Difference between probability theory and statistics

Discrete And Continuous Probabilities

Discrete Probabilities

Here the target space $\mathcal{T}$ is discrete. Consider the probability distribution of two (or more than two) random variables. The target space of joint probability is the Cartesian product of the target spaces of each random variable. We define the joint probability as the probability of both values occurring jointly

where $n_{ij}$ is the number of events with state $x_i$ and $y_j$, and $N$ is the total number of events. The joint probability is the probability of the intersection of both events.

In machine learning, we use discrete probability distributions to model categorical variables, i.e., variables that take a finite set of unordered values.

The function to describe the (joint) probability of discrete variable(s) is called probability mass function (pmf).（中文称为 概率质量函数，用于衡量离散随机变量的概率的）

Continuous Probabilities

Definition 6.1 (Probability Density Function, pdf). A function $f: \mathbb{R}^D \rightarrow \mathbb{R}$ is called a probability density function (pdf) if

1. $\forall x \in \mathbb{R}^D$: $f(\mathbf{x}) \ge 0$
2. Its integral exists and

In contrast to discrete random variables, the probability of a continuous random variable $X$ taking a particular value $P(X=x)$ is zero. The following is the definition of cdf.

Definition 6.2 (Cumulative Distribution Function, cdf). A cumulative distribution function (cdf) of a multivariate real-valued random variable $X$ with states $\mathbf{x} \in \mathbb{R}^D$ is given by

where $X = [X_1, \dots, X_D]^\top$, $\mathbf{x} = [x_1, \dots, x_D]^\top$

The cdf can also be expressed as the integral of the probability density function $f(\mathbf{x})$:

where $z_i$ are dummy variables.

Sum Rule / Product Rule / Bayesian Theorem

The Sum Rule

If $p(x, y)$ is a joint distribution, then $p(x)$ and $p(y)$ are the corresponding marginal distributions, and $p(y|x)$ is the conditional distribution of $y$ given $x$.

The sum rule states that,

The sum rule is also known as the marginalization property. In general, when the joint distribution contains more than two variables, the sum rule can be applied to any subset of variables. More concretely, if $\mathbf{x} = \begin{bmatrix} x_1, x_2, \cdots, x_n \end{bmatrix}^\top$, we obtain the marginal

where $\mathbf{x}_{-i}$ means all variables except $x_i$.

The Product Rule

The product rule relates the joint distribution to the conditional distribution:

The product rule can be interpreted as the fact that every joint distribution of two variables can be factorized into two other distributions. The two factors are the marginal distribution of the first random variable $\mathbf{x}$ and the conditional distribution of $\mathbf{y}$ given $\mathbf{x}$.

Bayes’ Theorem

$p(\mathbf{x})$ is the prior, which encapsulates our subjective prior knowledge of the unobserved (latent) variable $\mathbf{x}$ before observing any data. We can choose any prior that makes sense, but it is critical to ensure the pdf/pmf of the prior is non-zero for all plausible $\mathbf{x}$, even if they are rare. The prior is a probability distribution that does not rely on observed data and is independent of other random variables.

The likelihood (different from likelihood function, but related) $p(\mathbf{y}|\mathbf{x})$ describes how $\mathbf{x}$ and $\mathbf{y}$ are related. In the case of discrete probability distributions, it is the probability of the data $\mathbf{y}$ given the latent variable $\mathbf{x}$.

The posterior $p(\mathbf{x}|\mathbf{y})$ is the quantity of interest in Bayesian statistics. It expresses “what we know about $\mathbf{x}$ given $\mathbf{y}$”, which is exactly what we care about.

• Bayes’ theorem allows us to invert the relationship between $\mathbf{x}$ and $\mathbf{y}$ given by the likelihood. Therefore, Bayes’ theorem is sometimes called the probabilistic inverse.

Summary Statistics And Independence

A statistic of a random variable is a deterministic function of that variable.

Summary statistics provide a useful view of how a random variable behaves and characterize the distribution.

Means And (Co)Variances

Also called the population mean and covariance, as it refers to the true statistics for the population.

Definition 6.3 (Expected Value). The expected value of a function $g: \mathbb{R} \rightarrow \mathbb{R}$ of a univariate continuous random variable $X \sim p(x)$ is

Correspondingly, the expected value of a function $g(x)$ of a univariate discrete random variable $X \sim p(x)$ is

where $\mathcal{X}$ is the target space of $X$.

The expected value of a random vector $\mathbf{X} = [X_1, X_2, \dots, X_D]^\top$ is

Definition 6.4 (Mean). The mean is a special case of expected value, obtained by choosing $g$ as the identity function. The mean of a random variable $X$ with states $\mathbf{x} \in \mathbb{R}^D$ is defined as

where

Other terminologies:

• Median: 中位数
• Mode: 众数
  • For discrete distributions, the mode is the value with the highest frequency.
  • For continuous distributions, the mode is the peak of the density $p(x)$. A distribution may have multiple modes, and finding all modes can be computationally challenging in high dimensions.

Properties of expected value

1. Expected value is a linear operator. Given $f(\mathbf{x}) = ag(\mathbf{x}) + bh(\mathbf{x})$ where $a, b \in \mathbb{R}$ and $\mathbf{x} \in \mathbb{R}^D$:

Definition 6.5 (Covariance (Univariate)). The covariance between two univariate random variables $X, Y \in \mathbb{R}$ is

The covariance of a variable with itself is its variance, denoted $\mathbb{V}_X[X] = \operatorname{Cov}[X, X]$. The square root of variance is the standard deviation, denoted $\sigma(X) = \sqrt{\mathbb{V}_X[X]}$.

Definition 6.6 (Covariance (Multivariate)). For multivariate random variables $X$ and $Y$ with states $\mathbf{x} \in \mathbb{R}^D$ and $\mathbf{y} \in \mathbb{R}^E$, the covariance is

Definition 6.7 (Variance (Multivariate)). The variance of $X$ with states $\mathbf{x} \in \mathbb{R}^D$ and mean $\mathbf{\mu} \in \mathbb{R}^D$ is

This $D \times D$ matrix is the covariance matrix. It is symmetric, positive semidefinite, and describes the spread of the data. Its diagonal contains the variances of the marginal distributions.

Definition 6.8 (Correlation). The normalized covariance is the correlation:

The correlation matrix is the covariance matrix of standardized variables $X/\sigma(X)$ and $Y/\sigma(Y)$:

• Covariance (and correlation) indicate how two random variables are related.
• Positive correlation: When $X$ increases, $Y$ tends to increase.
• Negative correlation: When $X$ increases, $Y$ tends to decrease.

Empirical Means And Covariances

Definition 6.9 (Empirical Mean and Covariance). Given data, we estimate the mean (Definition 6.4) as the empirical mean (or sample mean):

where $\mathbf{x}_n \in \mathbb{R}^D$. The empirical covariance is

Note the empirical covariance presented above is a biased estimate. The unbiased one, which sometimes called corrected, has the factor $N-1$ in the denominator instead of $N$. The unbiased empirical covariance is

Three Expressions For The Variance

For a random variable $X$, the variance has three equivalent expressions:

Empirically estimating this requires a two-pass algorithm: one pass to compute $\hat{\mu}$, then another to compute the variance. Rearranging gives the raw-score formula:

This is the mean of the square minus the square of the mean.

A third expression uses pairwise differences:

This equals twice the raw-score expression. Thus, the sum of $N^2$ pairwise squared differences equals twice the sum of $N$ squared deviations from the mean.

Sums And Transformation Of Random Variables

For random vectors $\mathbf{X}, \mathbf{Y} \in \mathbb{R}^D$:

where $\mathbf{C} \in \mathbb{R}^D$ is constant.

For an affine transformation $\mathbf{Y} = \mathbf{A}\mathbf{X} + \mathbf{b}$ with mean $\mathbf{\mu}$ and covariance $\Sigma$:

and

Definition 6.10 (Independence). $X, Y$ are statistically independent iff

Independence means $y$ provides no additional information about $x$. If $X, Y$ are independent:

• $p(\mathbf{y}|\mathbf{x}) = p(\mathbf{y})$
• $p(\mathbf{x}|\mathbf{y}) = p(\mathbf{x})$
• $\mathbb{V}[\mathbf{X}+\mathbf{Y}] = \mathbb{V}[\mathbf{X}] + \mathbb{V}[\mathbf{Y}]$
• $\mathbb{V}[\mathbf{X}-\mathbf{Y}] = \mathbb{V}[\mathbf{X}] + \mathbb{V}[\mathbf{Y}]$
• $\operatorname{Cov}[\mathbf{X}, \mathbf{Y}] = 0$

The converse of the last point does not hold: zero covariance does not imply independence.

In machine learning, problems are often modeled with independent and identically distributed (i.i.d.) random variables.

Definition 6.11 (Conditional Independence). $X$ and $Y$ are conditionally independent given $Z$ iff

for all $z \in \mathcal{Z}$. Using the product rule:

Inner Products Of Random Variables

Random variables can be viewed as vectors in a vector space. Define the inner product:

The length of $X$ is

The angle $\theta$ between $X$ and $Y$ satisfies

which is the correlation (Definition 6.8). Thus, correlation is the cosine of the angle between two random variables in this geometric view.

Gaussian Distribution

a.k.a. Normal distribution.

For a univariate random variable, the Gaussian distribution has a density that is given by

The multivariate Gaussian distribution is characterized by mean vector $\mathbf{\mu}$ and a covariance matrix $\mathbf{\Sigma}$, defined as

where $\mathbf{x} \in \mathbb{R}^D$. We write $X \sim \mathcal{N}(\mu, \mathbf{\Sigma})$. The standard normal distribution is the special case of multivariate normal distribution with zero mean and identity covariance $\mathbf{\mu} = 0$ and $\mathbf{\Sigma} = \mathbf{\mathit{I}}$.

Side note, I want to write the multivariate Gaussian distribution as

A major advantage of Gaussian distribution is that variable transformations are often note needed. Since the Gaussian distribution is fully specified by its mean and covariance, we often can obtain the transformed distribution by applying the transformation to the mean and covariance of the random variable.