Covariance

The variance page averaged a squared deviation for one variable. Covariance instead averages the product of paired deviations from a joint distribution. Suppose a constructed probability model places equal mass on four word tokens with contextual surprisal X and reading time Y.

token outcome	surprisal X	reading time Y	probability
\omega_1	1 bit	200 ms	.25
\omega_2	1 bit	250 ms	.25
\omega_3	3 bits	350 ms	.25
\omega_4	3 bits	400 ms	.25

Covariance asks whether the two variables depart from their means in the same direction. In this example, do tokens with above-average surprisal also tend to have above-average reading times?

The covariance is the expected product of paired deviations:

\operatorname{Cov}(X,Y) \equiv\mathbb{E}\!\left[ (X-\mathbb{E}[X])(Y-\mathbb{E}[Y]) \right].

Computing the two means

The surprisal mean is

\mathbb{E}[X] =.25(1+1+3+3) =2\text{ bits}.

The reading time mean is

\mathbb{E}[Y] =.25(200+250+350+400) =300\text{ ms}.

These means center each variable before their paired deviations are multiplied.

Multiplying paired deviations

outcome	X-2	Y-300	product
\omega_1	-1	-100	100
\omega_2	-1	-50	50
\omega_3	1	50	50
\omega_4	1	100	100

Every product is positive because both deviations have the same sign within each paired outcome.

Weight and add:

\begin{aligned} \operatorname{Cov}(X,Y) &=.25(100+50+50+100)\\ &=75. \end{aligned}

The covariance is 75 bit milliseconds.

Reading the sign

A positive covariance means same signed paired deviations tend to dominate. High surprisal values tend to be paired with reading times above their mean, while low surprisal values tend to be paired with reading times below their mean.

A negative covariance means opposite signed products tend to dominate. A covariance of zero means that the average paired product is zero. It does not rule out every form of dependence.

Checking the computation in base R

Code

surprisal <- c(1, 1, 3, 3)
reading_time <- c(200, 250, 350, 400)
mass <- rep(.25, 4)

mean_x <- sum(surprisal * mass)
mean_y <- sum(reading_time * mass)
cov_xy <- sum((surprisal - mean_x) *
              (reading_time - mean_y) * mass)

c(mean_x, mean_y, cov_xy)

The returned values are 2, 300, and 75. This is the covariance of the specified probability distribution, so the calculation divides by total probability rather than using a finite sample correction.

Pairing is essential

Reverse the reading time order while keeping the surprisal order fixed. The two separate distributions remain unchanged, but high surprisal is now paired with low reading time. The covariance becomes negative.

Covariance thus belongs to the joint distribution. Neither marginal distribution identifies the pairing.

Keeping the units visible

Surprisal is measured in bits and reading time in milliseconds, so covariance is measured in bit milliseconds. Converting milliseconds to seconds divides the numerical covariance by 1,000.

This unit dependence makes raw covariance difficult to compare across pairs measured on different scales. Correlation standardizes the quantity.

Zero covariance does not imply independence

Covariance summarizes linear co movement through paired products. A curved or otherwise balanced dependence can have covariance zero, so zero covariance does not in general imply independence.

Independence implies zero covariance when the relevant expectations exist, but the reverse does not generally hold.

Check your understanding

Recalculate the covariance after reversing the reading time vector.
Why can the marginal distributions stay fixed while covariance changes?
What units would the covariance have if surprisal were in bits and latency in seconds?
What kind of relationship can remain when covariance is zero?

Covariance retains measurement units. The next page divides by the two standard deviations to obtain a unitless linear association.

--- title: "Covariance" --- The [variance page](variance-and-covariance.qmd) averaged a squared deviation for one variable. Covariance instead averages the product of paired deviations from a [joint distribution](joint-probability.qmd). Suppose a constructed probability model places equal mass on four word tokens with contextual surprisal $X$ and reading time $Y$. | token outcome | surprisal $X$ | reading time $Y$ | probability | |---|---:|---:|---:| | $\omega_1$ | 1 bit | 200 ms | $.25$ | | $\omega_2$ | 1 bit | 250 ms | $.25$ | | $\omega_3$ | 3 bits | 350 ms | $.25$ | | $\omega_4$ | 3 bits | 400 ms | $.25$ | Covariance asks whether the two variables depart from their means in the same direction. In this example, do tokens with above-average surprisal also tend to have above-average reading times? The [**covariance**](https://bruno.nicenboim.me/bayescogsci/ch-intro.html) is the expected product of paired deviations: $$ \operatorname{Cov}(X,Y) \equiv\mathbb{E}\!\left[ (X-\mathbb{E}[X])(Y-\mathbb{E}[Y]) \right]. $$ ## Computing the two means The surprisal mean is $$ \mathbb{E}[X] =.25(1+1+3+3) =2\text{ bits}. $$ The reading time mean is $$ \mathbb{E}[Y] =.25(200+250+350+400) =300\text{ ms}. $$ These means center each variable before their paired deviations are multiplied. ## Multiplying paired deviations | outcome | $X-2$ | $Y-300$ | product | |---|---:|---:|---:| | $\omega_1$ | $-1$ | $-100$ | $100$ | | $\omega_2$ | $-1$ | $-50$ | $50$ | | $\omega_3$ | $1$ | $50$ | $50$ | | $\omega_4$ | $1$ | $100$ | $100$ | Every product is positive because both deviations have the same sign within each paired outcome. Weight and add: $$ \begin{aligned} \operatorname{Cov}(X,Y) &=.25(100+50+50+100)\\ &=75. \end{aligned} $$ The covariance is 75 bit milliseconds. ## Reading the sign A positive covariance means same signed paired deviations tend to dominate. High surprisal values tend to be paired with reading times above their mean, while low surprisal values tend to be paired with reading times below their mean. A negative covariance means opposite signed products tend to dominate. A covariance of zero means that the average paired product is zero. It does not rule out every form of dependence. ## Checking the computation in base R ```{r} #| eval: false surprisal <- c(1, 1, 3, 3) reading_time <- c(200, 250, 350, 400) mass <- rep(.25, 4) mean_x <- sum(surprisal * mass) mean_y <- sum(reading_time * mass) cov_xy <- sum((surprisal - mean_x) * (reading_time - mean_y) * mass) c(mean_x, mean_y, cov_xy) ``` The returned values are 2, 300, and 75. This is the covariance of the specified probability distribution, so the calculation divides by total probability rather than using a finite sample correction. ## Pairing is essential Reverse the reading time order while keeping the surprisal order fixed. The two separate distributions remain unchanged, but high surprisal is now paired with low reading time. The covariance becomes negative. Covariance thus belongs to the joint distribution. Neither marginal distribution identifies the pairing. ## Keeping the units visible Surprisal is measured in bits and reading time in milliseconds, so covariance is measured in bit milliseconds. Converting milliseconds to seconds divides the numerical covariance by 1,000. This unit dependence makes raw covariance difficult to compare across pairs measured on different scales. [Correlation](correlation.qmd) standardizes the quantity. ## Zero covariance does not imply independence Covariance summarizes linear co movement through paired products. A curved or otherwise balanced dependence can have covariance zero, so zero covariance does not in general imply independence. Independence implies zero covariance when the relevant expectations exist, but the reverse does not generally hold. ## Check your understanding 1. Recalculate the covariance after reversing the reading time vector. 2. Why can the marginal distributions stay fixed while covariance changes? 3. What units would the covariance have if surprisal were in bits and latency in seconds? 4. What kind of relationship can remain when covariance is zero? Covariance retains measurement units. The [next page](correlation.qmd) divides by the two standard deviations to obtain a unitless linear association.