The Bernoulli distribution

The categorical distribution permits any finite number of unordered labels. Its binary case can be coded with zero and one. Suppose an elicited wh dependency either contains a resumptive pronoun or contains a gap.

X(\omega)\equiv \begin{cases} 1 & \text{if the response contains a resumptive pronoun},\\ 0 & \text{if the response contains a gap}. \end{cases}

The Bernoulli distribution represents one draw with two possible values, 0 and 1.

Its PMF is denoted by p_X, where

p_X(x)\equiv\mathbb{P}(X=x).

Let

\pi\equiv p_X(1).

Since the two masses must sum to one,

p_X(0)=1-\pi.

We write

X\sim\operatorname{Bernoulli}(\pi).

The single parameter \pi is the probability of the outcome coded as 1. The coding statement must thus be part of the model description.

Writing one PMF for both values

The two row PMF can be written as

p_X(x)=\pi^x(1-\pi)^{1-x}, \qquad x\in\{0,1\}.

Substitute x=1:

\begin{aligned} p_X(1) &=\pi^1(1-\pi)^0\\ &=\pi. \end{aligned}

Substitute x=0:

\begin{aligned} p_X(0) &=\pi^0(1-\pi)^1\\ &=1-\pi. \end{aligned}

The exponents select the appropriate factor. Any nonzero number raised to the zeroth power is one, so the unused factor drops out.

Calculating a concrete Bernoulli model

Suppose resumptive pronouns occur with probability .35 in the represented elicitation context. Then

\pi=.35

and

1-\pi=.65.

The complete PMF is

x	linguistic outcome	p_X(x)
0	gap	.65
1	resumptive pronoun	.35

The table concerns one response. It does not say that exactly 35 of every 100 finite samples must contain a resumptive pronoun.

Deriving the mean and variance

The expected value is

\begin{aligned} \mathbb{E}[X] &=0(1-\pi)+1(\pi)\\ &=\pi. \end{aligned}

For a zero and one variable, the mean is the probability of 1.

The variance is

\begin{aligned} \operatorname{Var}(X) &=(0-\pi)^2(1-\pi)+(1-\pi)^2\pi\\ &=\pi(1-\pi). \end{aligned}

With \pi=.35, the variance is

.35(.65)=.2275.

The variance approaches zero as \pi approaches either endpoint because the outcome becomes increasingly predictable.

Checking the PMF in base R

Code

pi_resumptive <- .35
bernoulli_mass <- c(
  gap = 1 - pi_resumptive,
  resumptive = pi_resumptive
)

sum(bernoulli_mass)
sum(c(0, 1) * bernoulli_mass)
pi_resumptive * (1 - pi_resumptive)

The results are 1, .35, and .2275.

Reversing the coding

We could define 1 to mean gap and 0 to mean resumptive pronoun. The new Bernoulli parameter would be .65. The underlying probabilities of the two linguistic outcomes would not change.

Coefficient signs in later binary regression models depend on this coding. Reversing the event marked by 1 reverses which probability the model describes.

Checking which event is coded as 1

The interpretation of \pi depends on which event is coded as 1. The number .35 is not a free standing “Bernoulli probability.” It is the probability of the declared target event.

Write the coding definition immediately before the parameter statement. Then \pi=p_X(1) can be translated back into the linguistic event without guessing.

Check your understanding

Let 1 mark a passive clause. If 18 percent of clauses are passive, write both PMF values.
Reverse the coding in the first question. What is the new value of \pi?
Derive \mathbb{E}[X]=\pi from the two support values.
Why is Bernoulli variance small when \pi is close to zero or one?

The Bernoulli family represents one binary draw. The next page represents the number of target outcomes across a fixed number of comparable draws.

--- title: "The Bernoulli distribution" --- The [categorical distribution](bernoulli-and-categorical-distributions.qmd) permits any finite number of unordered labels. Its binary case can be coded with zero and one. Suppose an elicited wh dependency either contains a resumptive pronoun or contains a gap. $$ X(\omega)\equiv \begin{cases} 1 & \text{if the response contains a resumptive pronoun},\\ 0 & \text{if the response contains a gap}. \end{cases} $$ The [**Bernoulli distribution**](https://bruno.nicenboim.me/bayescogsci/ch-intro.html) represents one draw with two possible values, 0 and 1. Its PMF is denoted by $p_X$, where $$ p_X(x)\equiv\mathbb{P}(X=x). $$ Let $$ \pi\equiv p_X(1). $$ Since the two masses must sum to one, $$ p_X(0)=1-\pi. $$ We write $$ X\sim\operatorname{Bernoulli}(\pi). $$ The single parameter $\pi$ is the probability of the outcome coded as 1. The coding statement must thus be part of the model description. ## Writing one PMF for both values The two row PMF can be written as $$ p_X(x)=\pi^x(1-\pi)^{1-x}, \qquad x\in\{0,1\}. $$ Substitute $x=1$: $$ \begin{aligned} p_X(1) &=\pi^1(1-\pi)^0\\ &=\pi. \end{aligned} $$ Substitute $x=0$: $$ \begin{aligned} p_X(0) &=\pi^0(1-\pi)^1\\ &=1-\pi. \end{aligned} $$ The exponents select the appropriate factor. Any nonzero number raised to the zeroth power is one, so the unused factor drops out. ## Calculating a concrete Bernoulli model Suppose resumptive pronouns occur with probability $.35$ in the represented elicitation context. Then $$ \pi=.35 $$ and $$ 1-\pi=.65. $$ The complete PMF is | $x$ | linguistic outcome | $p_X(x)$ | |---:|---|---:| | 0 | gap | $.65$ | | 1 | resumptive pronoun | $.35$ | The table concerns one response. It does not say that exactly 35 of every 100 finite samples must contain a resumptive pronoun. ## Deriving the mean and variance The expected value is $$ \begin{aligned} \mathbb{E}[X] &=0(1-\pi)+1(\pi)\\ &=\pi. \end{aligned} $$ For a zero and one variable, the mean is the probability of 1. The variance is $$ \begin{aligned} \operatorname{Var}(X) &=(0-\pi)^2(1-\pi)+(1-\pi)^2\pi\\ &=\pi(1-\pi). \end{aligned} $$ With $\pi=.35$, the variance is $$ .35(.65)=.2275. $$ The variance approaches zero as $\pi$ approaches either endpoint because the outcome becomes increasingly predictable. ## Checking the PMF in base R ```{r} #| eval: false pi_resumptive <- .35 bernoulli_mass <- c( gap = 1 - pi_resumptive, resumptive = pi_resumptive ) sum(bernoulli_mass) sum(c(0, 1) * bernoulli_mass) pi_resumptive * (1 - pi_resumptive) ``` The results are 1, `.35`, and `.2275`. ## Reversing the coding We could define 1 to mean gap and 0 to mean resumptive pronoun. The new Bernoulli parameter would be $.65$. The underlying probabilities of the two linguistic outcomes would not change. Coefficient signs in later binary regression models depend on this coding. Reversing the event marked by 1 reverses which probability the model describes. ## Checking which event is coded as 1 The interpretation of $\pi$ depends on which event is coded as 1. The number $.35$ is not a free standing “Bernoulli probability.” It is the probability of the declared target event. Write the coding definition immediately before the parameter statement. Then $\pi=p_X(1)$ can be translated back into the linguistic event without guessing. ## Check your understanding 1. Let 1 mark a passive clause. If 18 percent of clauses are passive, write both PMF values. 2. Reverse the coding in the first question. What is the new value of $\pi$? 3. Derive $\mathbb{E}[X]=\pi$ from the two support values. 4. Why is Bernoulli variance small when $\pi$ is close to zero or one? The Bernoulli family represents one binary draw. The [next page](binomial-distribution.qmd) represents the number of target outcomes across a fixed number of comparable draws.