From outcomes to distributions

Several variables may be evaluated at one outcome

A duration variable assigns one value to each complete token outcome.

Defining a random variable

Given a probability space (\Omega,\mathcal F,\mathbb{P}) and a measurable space (\mathcal X,\mathcal G), a random variable is a measurable function

X:\Omega\rightarrow\mathcal X.

The domain contains outcomes. The codomain contains possible values.

Defining the duration variable

For each token \omega\in\Omega, define

D(\omega)\equiv\text{duration of token }\omega\text{ in milliseconds}.

Thus D(\omega_1)=112, D(\omega_2)=83, and D(\omega_3)=126.

Value statements denote events

For a measurable set B\subseteq\mathcal X, define the shorthand

\{X\in B\} \equiv X^{-1}(B) =\{\omega\in\Omega:X(\omega)\in B\}.

This set contains outcomes, not bare values.

A threshold event

For the three-outcome illustration, let \Omega\equiv\{\omega_1,\omega_2,\omega_3\}. Then

\{D>100\} \equiv\{\omega\in\Omega:D(\omega)>100\} =\{\omega_1,\omega_3\}.

The notation D>100 is shorthand for an event in \mathcal F.

Measurability makes value probabilities well defined

The function X is measurable when

B\in\mathcal G \quad\Longrightarrow\quad X^{-1}(B)\in\mathcal F,

where \mathcal G is the sigma-algebra on \mathcal X.

The pronoun enumeration from the original notes

Let the case-coded pronoun outcomes be ordered as

\begin{aligned} (&\textit{us},\textit{they},\textit{them},\textit{you}_{[-\mathrm{acc}]}, \textit{he},\textit{I},\textit{it}_{[-\mathrm{acc}]},\\ &\textit{me},\textit{him},\textit{she},\textit{we}, \textit{it}_{[+\mathrm{acc}]},\textit{you}_{[+\mathrm{acc}]},\textit{her}). \end{aligned}

Define V(\omega) as the position of \omega in this ordered list. This definition fixes the correspondence between outcomes, integer codes, and probabilities.

Preimages are events

Under the enumeration V,

V^{-1}(\{1,2,3\}) =\{\textit{us},\textit{they},\textit{them}\}.

More generally,

V^{-1}(B) \equiv\{\omega\in\Omega:V(\omega)\in B\}.

The measurability condition requires this preimage to belong to \mathcal F for every B\in\mathcal G.

Distribution of a random variable

The distribution of X assigns each B\in\mathcal G the probability

\mathbb{P}(X\in B) \equiv \mathbb{P}\!\left(X^{-1}(B)\right).

As a function of B, this assignment is a probability measure on (\mathcal X,\mathcal G). We retain \mathbb{P} for the probability measure on (\Omega,\mathcal F) and introduce no second measure symbol.

Discrete random variables

A random variable X is discrete when some finite or countably infinite set \mathcal S satisfies \mathbb{P}(X\in\mathcal S)=1.

Grammatical person and utterance length are discrete. The first has finite support; the second may have countably infinite support.

Defining a probability mass function

For a discrete random variable X, its probability mass function (PMF) is defined by

p_X:\mathcal X\to[0,1], \qquad p_X(x) \equiv\mathbb{P}(X=x).

The subscript names the random variable; the argument is a possible value.

Its support is

\operatorname{supp}(X) \equiv \{x\in\mathcal X:p_X(x)>0\}.

Requirements for a PMF

Its PMF satisfies

p_X(x)\geq0 \qquad\text{and}\qquad \sum_{x\in\operatorname{supp}(X)}p_X(x)=1.

For any A\subseteq\mathcal X,

\mathbb{P}(X\in A) =\sum_{x\in A\cap\operatorname{supp}(X)}p_X(x).

Adding probability mass

Because \{S=3\} and \{S=4\} are disjoint,

\mathbb{P}(S\geq3)=.17+.08=.25.

Absolutely continuous random variables

A real-valued random variable X is absolutely continuous when there is a function f_X:\mathbb R\to[0,\infty) such that, for every Borel set A\subseteq\mathbb R,

\mathbb{P}(X\in A)=\int_A f_X(x)\,\mathrm{d}x.

The function f_X is a probability density function.

Requirements for a density

The density of X satisfies

f_X(x)\geq0 \qquad\text{and}\qquad \int_{-\infty}^{\infty}f_X(x)\,\mathrm{d}x=1.

Conversely, any measurable nonnegative function with integral one defines an absolutely continuous probability distribution by integration.

A uniform duration model

Suppose

f_D(d)\equiv \begin{cases} 1/100,&50\leq d\leq150,\\ 0,&\text{otherwise}. \end{cases}

Then

\mathbb{P}(80\leq D\leq100)=\int_{80}^{100}\frac{1}{100}\,\mathrm{d}u=.20.

Points have no mass under a density

For an absolutely continuous random variable,

\mathbb{P}(X=x)=\int_x^x f_X(t)\,\mathrm{d}t=0.

Thus f_X(x)>0 does not imply \mathbb{P}(X=x)>0.

Defining a cumulative distribution function

Every real-valued random variable has a cumulative distribution function (CDF), defined by

F_X:\mathbb R\to[0,1], \qquad F_X(x) \equiv\mathbb{P}(X\leq x).

The same definition applies to discrete, continuous, and mixed distributions.

Computing a CDF

For a discrete X,

F_X(x)=\sum_{t\leq x}p_X(t).

For an absolutely continuous X,

F_X(x)=\int_{-\infty}^{x}f_X(t)\,\mathrm{d}t.

These are consequences of the CDF definition.

Expectations average with respect to probability

For a random variable X satisfying \mathbb E[|X|]<\infty, define

\mathbb E[X] \equiv \begin{cases} \sum_x x\,p_X(x),&X\text{ discrete},\\[2mm] \int_{-\infty}^{\infty}x f_X(x)\,\mathrm{d}x,&X\text{ absolutely continuous}. \end{cases}

A weighted average of word lengths

If L takes values 1, 2, and 4 with masses .25, .50, and .25, then

\begin{aligned} \mathbb E[L] &=1(.25)+2(.50)+4(.25)\\ &=2.25. \end{aligned}

An expectation need not be a possible value.

Expectations of functions

When \mathbb E[|g(X)|]<\infty, define

\mathbb E[g(X)] \equiv\sum_x g(x)p_X(x)

in the discrete case, with the analogous integral in the continuous case.

In general, \mathbb E[g(X)]\neq g(\mathbb E[X]).

Deriving linearity for a finite discrete model

For constants a and b,

\begin{aligned} \mathbb E[aX+b] &=\sum_x (ax+b)p_X(x)\\ &=a\sum_x xp_X(x)+b\sum_xp_X(x)\\ &=a\mathbb E[X]+b. \end{aligned}

Linearity for sums

If \mathbb E[|X_i|]<\infty for every i,

\mathbb E\!\left[\sum_{i=1}^{n}a_iX_i\right] =\sum_{i=1}^{n}a_i\mathbb E[X_i].

This equality does not require independence. Dependence can change the distribution and variance of the sum.

A distribution need not have a finite mean

Let

p_L(l)\equiv\frac{6}{\pi^2l^2}, \qquad l=1,2,\ldots

Then

\sum_{l=1}^{\infty}p_L(l) =\frac{6}{\pi^2}\sum_{l=1}^{\infty}\frac1{l^2} =1,

but

\mathbb E[L] =\frac{6}{\pi^2}\sum_{l=1}^{\infty}\frac1l =\infty.

The original continuous counterexample: Cauchy

For X\sim\operatorname{Cauchy}(x_0,\gamma) with \gamma>0,

f_X(x) \equiv \frac{1}{\pi\gamma\left[1+\left(\frac{x-x_0}{\gamma}\right)^2\right]}.

The positive and negative parts of \int_{-\infty}^{\infty}x f_X(x)\,\mathrm dx are both infinite. Thus \mathbb E[X] is undefined; the symmetric principal value zero is not an expectation.

Defining variance

For a random variable with finite second moment, define

\operatorname{Var}(X) \equiv\mathbb E\!\left[(X-\mathbb E[X])^2\right].

Variance is an expected squared deviation from the mean.

Deriving the computational identity

Writing \mu\equiv\mathbb E[X],

\begin{aligned} \operatorname{Var}(X) &=\mathbb E[X^2-2\mu X+\mu^2]\\ &=\mathbb E[X^2]-2\mu\mathbb E[X]+\mu^2\\ &=\mathbb E[X^2]-2\mu^2+\mu^2\\ &=\mathbb E[X^2]-\mu^2\\ &=\mathbb E[X^2]-\mathbb E[X]^2. \end{aligned}

Standard deviation

The standard deviation is defined by

\sigma_X \equiv\sqrt{\operatorname{Var}(X)}.

If X is measured in milliseconds, \operatorname{Var}(X) has units ms^2 and \sigma_X has units ms.

Defining central moments

When it exists, the kth central moment is defined by

\mu_k \equiv\mathbb E\!\left[(X-\mathbb E[X])^k\right].

Thus \mu_1=0 and \mu_2=\operatorname{Var}(X).

Distribution notation

The statement

X\sim\mathcal D(\theta)

means that the distribution of X belongs to family \mathcal D with parameter \theta. The symbol \sim does not mean numerical equality.

The categorical distribution

For labels c_1,\ldots,c_K and \boldsymbol\pi\in[0,1]^K with \sum_{k=1}^K\pi_k=1,

V\sim\operatorname{Categorical}(\boldsymbol\pi)

means

p_V(c_k) \equiv\mathbb{P}(V=c_k) =\pi_k.

The pronoun categorical distribution

The original example uses the ordered pronoun labels from the enumeration V and the probability vector

\boldsymbol\pi \equiv(.03,.09,.03,.12,.07,.28,.07,.05,.02,.02,.07,.08,.05,.02).

The kth entry is the probability of the kth pronoun label. Reordering the labels without applying the same permutation to \boldsymbol\pi changes the distribution.

PMF for the pronoun categorical distribution

The Bernoulli distribution

Define the accusative-case indicator

X(\omega) \equiv \begin{cases} 1,&\omega\in A,\\ 0,&\omega\notin A. \end{cases}

The range is \{0,1\} even though the sample space contains fourteen pronoun outcomes. For 0\leq\pi\leq1,

X\sim\operatorname{Bernoulli}(\pi)

means

p_X(x) \equiv\mathbb{P}(X=x) =\pi^x(1-\pi)^{1-x}, \qquad x\in\{0,1\}.

Bernoulli distribution on pronoun case

Using the original empirical value \pi=.27,

p_X(1)=.27, \qquad p_X(0)=.73.

The parameter is the probability of accusative case because the definition of X assigns accusative outcomes the value 1.

The binomial distribution

Let Y_1,\ldots,Y_n be independent Bernoulli variables with common parameter \pi, and define K\equiv\sum_{i=1}^nY_i. Then

K\sim\operatorname{Binomial}(n,\pi),

with support \{0,\ldots,n\}.

Deriving the binomial PMF

Each sequence with k ones has probability \pi^k(1-\pi)^{n-k}, and there are {n\choose k} such sequences. Thus

p_K(k) \equiv\mathbb{P}(K=k) ={n\choose k}\pi^k(1-\pi)^{n-k}.

Sampling without replacement

Suppose a finite archive contains N clauses, M of which are passive. If n distinct clauses are sampled without replacement and K counts passives, then

p_K(k) \equiv\mathbb{P}(K=k) =\frac{{M\choose k}{N-M\choose n-k}}{{N\choose n}}.

This is the hypergeometric PMF for

\max(0,n-N+M)\leq k\leq\min(n,M).

A geometric series defines a PMF

The original notes begin with

\sum_{k=0}^{\infty}\frac{1}{2^{k+1}}=1.

Thus

p_K(k)\equiv\frac{1}{2^{k+1}}, \qquad k=0,1,2,\ldots

is a PMF with countably infinite support.

The geometric family

Let K count failures before the first success in independent Bernoulli trials with common success probability \pi. Define

K\sim\operatorname{Geometric}(\pi)

by the PMF

p_K(k) \equiv\mathbb{P}(K=k) =(1-\pi)^k\pi, \qquad k=0,1,2,\ldots

This is the convention used by dgeom in R. The total-trial count is T\equiv K+1.

The original geometric parameter sequence: \pi=.5

As \pi decreases, the distribution spreads rightward

As \pi increases, mass concentrates at zero

Every geometric PMF decreases from its first support point

For every k\geq0,

\frac{p_K(k+1)}{p_K(k)} =\frac{(1-\pi)^{k+1}\pi}{(1-\pi)^k\pi} =1-\pi <1.

Thus p_K(k+1)<p_K(k). If word length is L\equiv K+1, every geometric model has its mode at L=1.

The CMUdict word-length data

The original worked example uses every noncomment pronunciation entry in CMU Pronouncing Dictionary 0.7b. For an entry \omega, define

L(\omega)\equiv\text{number of phoneme symbols in }\omega.

Alternate pronunciations occupy separate dictionary entries, as they did in the original computation.

Empirical distribution of word length in phonemes

The geometric family cannot reproduce the observed mode

The empirical PMF has its mode at six phonemes. The fitted geometric PMF must have its mode at one phoneme because it is strictly decreasing.

The negative binomial generalizes the geometric

Let K count failures before the rth success, where r\in\{1,2,\ldots\}. Define

K\sim\operatorname{NegBin}(r,\pi)

by

p_K(k) \equiv\mathbb{P}(K=k) ={k+r-1\choose k}(1-\pi)^k\pi^r, \qquad k=0,1,2,\ldots

When r=1, the binomial coefficient equals one and this PMF is exactly geometric.

The original negative-binomial sequence: r=1, \pi=.5

Increasing r permits an interior mode

With \pi fixed, increasing r increases

\mathbb E[K]=r\frac{1-\pi}{\pi}.

With r=5, smaller \pi moves mass farther right

With r=5, larger \pi concentrates mass near zero

A larger r can offset a large \pi

Fitting the word-length distribution

Set K\equiv L-1 so that the observed support begins at zero. Under the negative-binomial parameterization,

\mathbb E[K]=r\frac{1-\pi}{\pi}, \qquad \operatorname{Var}(K)=r\frac{1-\pi}{\pi^2}.

The CMUdict data are underdispersed relative to this family: the empirical variance of K is smaller than its empirical mean. The likelihood thus increases toward the Poisson boundary r\to\infty.

Empirical, geometric, and negative-binomial-limit CDFs

The negative-binomial generalization removes the geometric family’s boundary-mode restriction. For these data, its maximum-likelihood fit approaches its Poisson limit rather than a finite r.

Poisson counts require a declared exposure

If events occur at rate \lambda>0 per exposure unit, a Poisson model for exposure t>0 specifies

Y_t\sim\operatorname{Poisson}(\lambda t),

with

p_{Y_t}(y) \equiv\mathbb{P}(Y_t=y) =e^{-\lambda t}\frac{(\lambda t)^y}{y!}, \qquad y=0,1,2,\ldots

A Poisson distribution is not a complete process claim

The count model implies

\mathbb E[Y_t]=\operatorname{Var}(Y_t)=\lambda t.

A homogeneous Poisson process additionally specifies N(0)=0 and stationary independent increments with rate \lambda.

The continuous uniform distribution

For a<b,

X\sim\operatorname{Uniform}(a,b)

means

f_X(x)\equiv \begin{cases} 1/(b-a),&a\leq x\leq b,\\ 0,&\text{otherwise}. \end{cases}

Density and probability under \operatorname{Uniform}(0,1)

For X\sim\operatorname{Uniform}(0,1),

\mathbb P(.25<X<.75) =\int_{.25}^{.75}1\,\mathrm dx =.50.

Probability is area under the density

CDF of \operatorname{Uniform}(0,1)

The beta distribution

For \alpha,\beta>0,

X\sim\operatorname{Beta}(\alpha,\beta)

has topological support [0,1] and density

f_X(x)\equiv \begin{cases} \dfrac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)},&0<x<1,\\ 0,&\text{otherwise}. \end{cases}

\operatorname{Beta}(1,1) is uniform on (0,1)

Symmetric beta densities with \alpha=\beta>1

Asymmetric beta densities

Boundary modes in the beta family

Two boundary modes when \alpha,\beta<1

Interpreting beta parameters

For X\sim\operatorname{Beta}(\alpha,\beta),

\mathbb E[X]=\frac{\alpha}{\alpha+\beta}.

Holding this ratio fixed while increasing \alpha+\beta concentrates the distribution around the same mean.

The normal distribution

For \sigma>0,

X\sim\mathcal N(\mu,\sigma^2)

means

f_X(x)\equiv\frac{1}{\sigma\sqrt{2\pi}} \exp\!\left[-\frac{(x-\mu)^2}{2\sigma^2}\right].

PDF of the standard normal distribution

The mean controls location; the standard deviation controls scale

Standardizing a normal variable

Define

Z\equiv\frac{X-\mu}{\sigma}.

If X\sim\mathcal N(\mu,\sigma^2), then Z\sim\mathcal N(0,1). Standardization changes units, not relative position within this family.

Standard normal CDF

A beta CDF can have the same sigmoidal form

The chi-squared distribution

For \nu\in\{1,2,\ldots\}, let Z_1,\ldots,Z_\nu be independent standard normal variables and define

Q\equiv\sum_{i=1}^{\nu}Z_i^2.

Then Q\sim\chi^2_\nu, with \mathbb E[Q]=\nu and \operatorname{Var}(Q)=2\nu.

Student’s t distribution

Let Z\sim\mathcal N(0,1) and U\sim\chi^2_\nu be independent. Define

T\equiv\frac{Z}{\sqrt{U/\nu}}.

Then T\sim t_\nu. Because U/\nu can be close to zero, extreme values of T are more probable than extreme values under a standard normal distribution.

Inverse transform sampling

For inverse transform sampling, define the generalized inverse of a CDF F by

F^{-1}(u) \equiv\inf\{x\in\mathbb R:F(x)\geq u\}, \qquad 0<u<1.

If U\sim\operatorname{Uniform}(0,1), define X\equiv F^{-1}(U).

Verifying the target distribution

The generalized inverse satisfies F^{-1}(U)\leq x exactly when U\leq F(x). Thus

\begin{aligned} \mathbb{P}(X\leq x) &=\mathbb{P}(F^{-1}(U)\leq x)\\ &=\mathbb{P}(U\leq F(x))\\ &=F(x). \end{aligned}

Discrete inverse transform sampling

For ordered values x_1,\ldots,x_K, define

c_k\equiv\sum_{j=1}^kp_X(x_j).

Draw U\sim\operatorname{Uniform}(0,1) and return the first x_k for which U\leq c_k.

Random seeds reproduce computations

A random seed fixes the initial state of a pseudorandom number generator. Reusing the seed reproduces the same sequence of draws.

It does not validate the distribution, independence assumptions, or sampling representation.

Several variables use one probability space

Let X:\Omega\rightarrow\mathcal X and Y:\Omega\rightarrow\mathcal Y be random variables on the same probability space.

The pair (X(\omega),Y(\omega)) records two values from the same outcome \omega.

Defining a joint PMF

For discrete X and Y, define

p_{X,Y}(x,y) \equiv\mathbb{P}(X=x,Y=y).

The comma denotes the joint event \{X=x\}\cap\{Y=y\}.

A joint linguistic distribution

Let P code whether a dialogue turn contains a politeness marker and R whether it ends in a final pitch rise.

Reading a joint cell

The upper-left cell states

p_{P,R}(1,1) =.30.

It assigns probability to the paired values for one dialogue turn.

Marginalization is a partition argument

For fixed p, the events \{P=p,R=r\} partition \{P=p\} as r ranges over the support of R:

\{P=p\} =\biguplus_{r\in\operatorname{supp}(R)} \{P=p,R=r\}.

Countable additivity gives

\mathbb{P}(P=p) =\sum_r\mathbb{P}(P=p,R=r).

Applying the PMF definitions to both sides yields

p_P(p) =\sum_r p_{P,R}(p,r).

Computing a marginal PMF

For turns with a politeness marker,

\begin{aligned} p_P(1) &=p_{P,R}(1,1)+p_{P,R}(1,0)\\ &=.30+.20\\ &=.50. \end{aligned}

We sum out pitch rise while holding politeness fixed.

What marginalization removes

The marginal PMFs p_P and p_R retain the row and column totals. They do not retain how probability is paired within the four cells.

Different joint PMFs can have the same marginals.

Marginal density

If X and Y have joint density f_{X,Y}, define the marginal density of X by

f_X(x) \equiv\int_{-\infty}^{\infty}f_{X,Y}(x,y)\,\mathrm{d}y.

The integral removes the Y coordinate.

Defining a conditional PMF

For any y with p_Y(y)>0, define

p_{X\mid Y}(x\mid y) \equiv\mathbb{P}(X=x\mid Y=y) =\frac{p_{X,Y}(x,y)}{p_Y(y)}.

Deriving the chain rule for random variables

Multiply the conditional-PMF definition by p_Y(y):

p_{X,Y}(x,y) =p_{X\mid Y}(x\mid y)p_Y(y).

This is the two-variable chain rule.

Defining independence through conditional invariance

For every y with p_Y(y)>0, independence requires

p_{X\mid Y}(x\mid y)=p_X(x)

for every x. Learning Y=y does not change the distribution of X.

Deriving the independence factorization

Substitute conditional invariance into the chain rule:

\begin{aligned} p_{X,Y}(x,y) &=p_{X\mid Y}(x\mid y)p_Y(y)\\ &=p_X(x)p_Y(y). \end{aligned}

The product form is the equivalent unrestricted criterion for X\perp\!\!\!\perp Y.

Defining conditional independence

For values with p_{Y,Z}(y,z)>0, X and Y are conditionally independent given Z when

p_{X\mid Y,Z}(x\mid y,z)=p_{X\mid Z}(x\mid z).

We write X\perp\!\!\!\perp Y\mid Z.

Conditional independence factorization

For every z with p_Z(z)>0, the conditional chain rule gives the equivalent criterion

X\perp\!\!\!\perp Y\mid Z \quad\Longleftrightarrow\quad p_{X,Y\mid Z}(x,y\mid z) =p_{X\mid Z}(x\mid z)p_{Y\mid Z}(y\mid z).

The conditioning variable remains in both factors.

Defining conditional expectation

For discrete X and any y with p_Y(y)>0, define

\mathbb E[X\mid Y=y] \equiv\sum_x x\,p_{X\mid Y}(x\mid y).

This is the mean of the conditional distribution at Y=y.

Deriving the law of total expectation

For integrable discrete variables,

\begin{aligned} \mathbb E[X] &=\sum_xx\sum_yp_{X,Y}(x,y)\\ &=\sum_{y:p_Y(y)>0}\sum_xx\,p_{X,Y}(x,y)\\ &=\sum_{y:p_Y(y)>0} \left[\sum_xx\,p_{X\mid Y}(x\mid y)\right]p_Y(y)\\ &=\sum_{y:p_Y(y)>0}\mathbb E[X\mid Y=y]p_Y(y). \end{aligned}

Defining covariance

For variables with finite second moments, define

\operatorname{Cov}(X,Y) \equiv\mathbb E\!\left[(X-\mathbb E[X])(Y-\mathbb E[Y])\right].

Its units are the product of the units of X and Y.

Deriving the covariance identity

Expanding the product and applying linearity gives

\operatorname{Cov}(X,Y) =\mathbb E[XY]-\mathbb E[X]\mathbb E[Y].

This identity does not define covariance; it follows from the preceding definition.

Correlation

When both standard deviations are positive, define

\rho_{XY} \equiv \frac{\operatorname{Cov}(X,Y)} {\sigma_X\sigma_Y}.

Correlation is unitless and satisfies -1\leq\rho_{XY}\leq1.

Correlation has a limited interpretation

Correlation summarizes linear association under the joint distribution. It does not determine the complete dependence structure and does not identify a causal direction.