Suppose that we want to assign probabilities to two distinctions in \Omega_4: whether a pronoun is plural and whether it is accusative. What is the smallest sigma-algebra that lets us ask both questions? It must contain both events, their complements, and every union required by closure. Finding exactly this collection is the generation problem.
Recall from event spaces that a sigma-algebra is a collection of measurable events. If P_4 is the plural event and A_4 is the accusative event, then
\sigma(P_4,A_4)
denotes the smallest sigma-algebra containing both P_4 and A_4. The collection \{P_4,A_4\} is a generating set, and \sigma(P_4,A_4) is the sigma-algebra generated by that set.
Basically, we begin with the distinctions we care about and add only what closure forces us to add. More specifically, any sigma-algebra containing P_4 and A_4 must also contain their complements, intersections, and countable unions. The smallest one contains exactly the events forced by these closure requirements.
For the twelve-form pronoun example, the collection
\mathcal F_{\mathrm{person}}\cup\mathcal F_{\mathrm{case}}
is not a sigma-algebra. The third-person event T_{12} belongs to \mathcal F_{\mathrm{person}}, and the accusative event A_{12} belongs to \mathcal F_{\mathrm{case}}, but their intersection
T_{12}\cap A_{12} =\{\textit{them},\textit{it},\textit{her},\textit{him}\}
does not belong to the union. Thus the union is not closed under intersection.
The combined event space must instead be defined by
\mathcal F_{\mathrm{person\text{-}case}} \equiv\sigma(T_{12},A_{12},N_{12}).
This is the smallest sigma-algebra containing the person and case generators.
Replace each syncretic surface form with two case-coded outcomes:
\begin{aligned} \Omega_{14}\equiv\{&\textit{I},\textit{me}, \textit{you}_{[-\mathrm{acc}]},\textit{you}_{[+\mathrm{acc}]}, \textit{they},\textit{them},\\ &\textit{it}_{[-\mathrm{acc}]},\textit{it}_{[+\mathrm{acc}]}, \textit{she},\textit{her},\textit{he},\textit{him}, \textit{we},\textit{us}\}. \end{aligned}
In this refined space, the accusative event is
A_{14}\equiv\{\textit{me},\textit{you}_{[+\mathrm{acc}]},\textit{them}, \textit{her},\textit{him},\textit{it}_{[+\mathrm{acc}]},\textit{us}\}.
Define the third-person event on this space by
T_{14}\equiv\{\textit{they},\textit{them}, \textit{it}_{[-\mathrm{acc}]},\textit{it}_{[+\mathrm{acc}]}, \textit{she},\textit{her},\textit{he},\textit{him}\}.
The nonaccusative event is A_{14}^c. The four nonempty atoms generated by T_{14} and A_{14} are
T_{14}\cap A_{14}, \quad T_{14}\cap A_{14}^c, \quad T_{14}^c\cap A_{14}, \quad T_{14}^c\cap A_{14}^c.
Every event in \sigma(T_{14},A_{14}) is a union of these four atoms, so the generated sigma-algebra contains 2^4=16 events.
Use the four outcome sample space
\Omega_4 \equiv\{\textit{he},\textit{him},\textit{they},\textit{them}\}.
Define the generators
P_4\equiv\{\textit{they},\textit{them}\}
and
A_4\equiv\{\textit{him},\textit{them}\}.
The two distinctions cross. A form may be plural or singular, and it may be accusative or nominative.
For a finite space, we can solve the generation problem directly. For each generator, choose either the event or its complement, then intersect the choices. The nonempty intersections are
\begin{aligned} P_4\cap A_4 &= \{\textit{them}\},\\ P_4\cap A_4^c &= \{\textit{they}\},\\ P_4^c\cap A_4 &= \{\textit{him}\},\\ P_4^c\cap A_4^c &= \{\textit{he}\}. \end{aligned}
These four sets are the atoms produced by the generators. They do not overlap, and every outcome belongs to exactly one of them.
Every event constructed from P_4, A_4, and the set operations must treat all outcomes within an atom alike. No operation available from the generators can separate two outcomes that have identical membership in every generator.
Now pause at the consequence of the calculation. In this small example, each atom contains one outcome. Every subset of \Omega_4 can thus be written as a union of atoms. It follows that
\sigma(P_4,A_4)=2^{\Omega_4}.
Since there are four atoms, there are 2^4=16 unions of atoms, including the empty union \varnothing and the union of all four atoms \Omega_4.
The finite set operations can be reproduced directly.
omega <- c("he", "him", "they", "them")
plural <- c("they", "them")
accusative <- c("him", "them")
plural_acc <- intersect(plural, accusative)
plural_nom <- intersect(plural, setdiff(omega, accusative))
singular_acc <- intersect(setdiff(omega, plural), accusative)
singular_nom <- intersect(setdiff(omega, plural),
setdiff(omega, accusative))
list(plural_acc, plural_nom, singular_acc, singular_nom)R returns the four singleton atoms listed above.
Now generate a sigma-algebra from P_4 alone. Its two atoms are
P_4=\{\textit{they},\textit{them}\}
and
P_4^c=\{\textit{he},\textit{him}\}.
Every event in \sigma(P_4) is a union of these two atoms. The generated sigma-algebra is thus
\sigma(P_4)=\{\varnothing,P_4,P_4^c,\Omega_4\}.
This space cannot distinguish they from them. Both outcomes have the same membership in the only generator. Adding A_4 splits each number atom by case, producing the four smaller atoms above.
With P_4 as the only generator, case is not measurable. Adding A_4 makes both number and case measurable.
If two outcomes have identical membership in every generator, no event in the generated sigma-algebra can separate them. For instance, suppose a model uses only P_4 to generate its sigma-algebra but later asks whether plural pronouns are nominative or accusative. The outcomes they and them occupy the same atom, so the model cannot represent that case distinction.
Even a corpus containing one million tokens would not put a case event in \sigma(P_4). We must add a case generator before assigning a probability to that event.
For the finite spaces used here, the procedure is mechanical.
The resulting collection is the smallest sigma-algebra containing the generators.
The exact upshot is that generators encode the distinctions the model can make, while atoms show the finest outcome groups those distinctions can separate. The next page keeps the measurable events fixed and assigns numerical probabilities to them.