The categorical distribution

The earlier probability mass function assigned probability to the separate values of a discrete variable. We now specify a distribution family for one draw from a finite set. Suppose we sample one dependency from an annotated corpus and record its grammatical relation.

A categorical distribution assigns probability to one draw from such a finite set of alternatives.

Let R be the relation variable with support

\operatorname{supp}(R) \equiv\{\textsc{subject},\textsc{object},\textsc{oblique}\}.

Consider this constructed PMF.

relation r	p_R(r)
subject	.50
object	.30
oblique	.20

The table describes the relation label for one sampled dependency under a declared corpus and annotation process.

Collecting the probabilities in a vector

List the support values

c_1=\textsc{subject}, \qquad c_2=\textsc{object}, \qquad c_3=\textsc{oblique}.

Collect their probabilities in the same order:

\boldsymbol{\pi}=(.50,.30,.20).

We write

R\sim\operatorname{Categorical}(\boldsymbol{\pi})

and read the symbol \sim as “is distributed as.” The PMF is

p_R(c_k)\equiv\mathbb{P}(R=c_k)=\pi_k, \qquad k\in\{1,2,3\}.

Thus p_R(c_2)=.30 means that the object relation receives probability .30.

The subscripts are bookkeeping indices. They connect each category to the corresponding entry of \boldsymbol{\pi}.

Checking the parameter vector

Every categorical probability must be nonnegative:

\pi_k\geq0.

The complete vector must sum to one:

\sum_{k=1}^{3}\pi_k =.50+.30+.20 =1.

Once two entries are fixed, the third is determined. If subject and object receive .50 and .30, then oblique must receive

1-.50-.30=.20.

For K categories, the sum constraint leaves K-1 entries free to vary.

Reordering labels does not reorder the linguistic system

We could instead declare

c_1=\textsc{oblique}, \qquad c_2=\textsc{subject}, \qquad c_3=\textsc{object}.

The probability vector would then need to be

\boldsymbol{\pi}=(.20,.50,.30).

The distribution has not changed because every label retains its probability. The index order does not claim that one grammatical relation is linguistically greater than another.

In software, factor level order may determine the position of a probability in an array without imposing an ordinal interpretation on the labels.

Representing and sampling the PMF in base R

Code

relation_probability <- c(
  subject = .50,
  object = .30,
  oblique = .20
)

sum(relation_probability)

set.seed(31)
sample(
  names(relation_probability),
  size = 8,
  replace = TRUE,
  prob = relation_probability
)

The first call checks the probability constraint. The sample call produces eight independent draws from the declared categorical PMF. A different simulated sample may have proportions that differ from the parameter vector, especially when only eight outcomes are drawn.

What the distribution does not yet represent

One categorical distribution uses one fixed probability vector. It does not yet represent differences across speakers, genres, or syntactic environments. Those differences would require separate conditional probability vectors or a model that makes \boldsymbol{\pi} depend on predictors.

It also describes one label per draw. A sentence containing several dependencies produces several categorical observations only after the unit of analysis has been defined accordingly.

Keeping labels and probabilities in the same order

The order of the category labels must match the order of the probabilities. If the labels are reordered while \boldsymbol{\pi} is left unchanged, the values become attached to the wrong linguistic outcomes.

Label the entries and check one value in words. Here the statement “object receives probability .30” should remain true after any software reordering.

Check your understanding

A discourse annotation has four labels with probabilities .10, .25, .40, and an unknown final value. Compute the final value.
Reorder the support above as object, oblique, subject. Write the matching probability vector.
Why does a categorical index not make the relation labels ordinal?
What additional structure is needed if relation probabilities differ by clause type?

The categorical family permits any finite number of labels. The next page studies the special case with two outcomes coded as zero and one.

--- title: "The categorical distribution" --- The earlier [probability mass function](probability-mass-functions.qmd) assigned probability to the separate values of a discrete variable. We now specify a distribution family for one draw from a finite set. Suppose we sample one dependency from an annotated corpus and record its grammatical relation. A [**categorical distribution**](https://mc-stan.org/docs/functions-reference/categorical-distribution.html) assigns probability to one draw from such a finite set of alternatives. Let $R$ be the relation variable with support $$ \operatorname{supp}(R) \equiv\{\textsc{subject},\textsc{object},\textsc{oblique}\}. $$ Consider this constructed PMF. | relation $r$ | $p_R(r)$ | |---|---:| | subject | $.50$ | | object | $.30$ | | oblique | $.20$ | The table describes the relation label for one sampled dependency under a declared corpus and annotation process. ## Collecting the probabilities in a vector List the support values $$ c_1=\textsc{subject}, \qquad c_2=\textsc{object}, \qquad c_3=\textsc{oblique}. $$ Collect their probabilities in the same order: $$ \boldsymbol{\pi}=(.50,.30,.20). $$ We write $$ R\sim\operatorname{Categorical}(\boldsymbol{\pi}) $$ and read the symbol $\sim$ as “is distributed as.” The PMF is $$ p_R(c_k)\equiv\mathbb{P}(R=c_k)=\pi_k, \qquad k\in\{1,2,3\}. $$ Thus $p_R(c_2)=.30$ means that the object relation receives probability $.30$. The subscripts are bookkeeping indices. They connect each category to the corresponding entry of $\boldsymbol{\pi}$. ## Checking the parameter vector Every categorical probability must be nonnegative: $$ \pi_k\geq0. $$ The complete vector must sum to one: $$ \sum_{k=1}^{3}\pi_k =.50+.30+.20 =1. $$ Once two entries are fixed, the third is determined. If subject and object receive $.50$ and $.30$, then oblique must receive $$ 1-.50-.30=.20. $$ For $K$ categories, the sum constraint leaves $K-1$ entries free to vary. ## Reordering labels does not reorder the linguistic system We could instead declare $$ c_1=\textsc{oblique}, \qquad c_2=\textsc{subject}, \qquad c_3=\textsc{object}. $$ The probability vector would then need to be $$ \boldsymbol{\pi}=(.20,.50,.30). $$ The distribution has not changed because every label retains its probability. The index order does not claim that one grammatical relation is linguistically greater than another. In software, factor level order may determine the position of a probability in an array without imposing an ordinal interpretation on the labels. ## Representing and sampling the PMF in base R ```{r} #| eval: false relation_probability <- c( subject = .50, object = .30, oblique = .20 ) sum(relation_probability) set.seed(31) sample( names(relation_probability), size = 8, replace = TRUE, prob = relation_probability ) ``` The first call checks the probability constraint. The sample call produces eight independent draws from the declared categorical PMF. A different simulated sample may have proportions that differ from the parameter vector, especially when only eight outcomes are drawn. ## What the distribution does not yet represent One categorical distribution uses one fixed probability vector. It does not yet represent differences across speakers, genres, or syntactic environments. Those differences would require separate conditional probability vectors or a model that makes $\boldsymbol{\pi}$ depend on predictors. It also describes one label per draw. A sentence containing several dependencies produces several categorical observations only after the unit of analysis has been defined accordingly. ## Keeping labels and probabilities in the same order The order of the category labels must match the order of the probabilities. If the labels are reordered while $\boldsymbol{\pi}$ is left unchanged, the values become attached to the wrong linguistic outcomes. Label the entries and check one value in words. Here the statement “object receives probability $.30$” should remain true after any software reordering. ## Check your understanding 1. A discourse annotation has four labels with probabilities $.10$, $.25$, $.40$, and an unknown final value. Compute the final value. 2. Reorder the support above as object, oblique, subject. Write the matching probability vector. 3. Why does a categorical index not make the relation labels ordinal? 4. What additional structure is needed if relation probabilities differ by clause type? The categorical family permits any finite number of labels. The [next page](bernoulli-distribution.qmd) studies the special case with two outcomes coded as zero and one.