Correlation

The preceding page showed that covariance depends on both association and measurement scale. Correlation divides covariance by the standard deviations of the two variables. Suppose X records whether a dative recipient is pronominal and Y records whether the double object construction is used:

X=1\text{ for a pronominal recipient}

and

Y=1\text{ for a double object construction}.

Consider this constructed joint PMF.

	Y=0	Y=1
X=0	.30	.20
X=1	.10	.40

The table suggests a positive linear association, but covariance depends on the numerical scales. Correlation standardizes covariance by the two standard deviations:

\rho_{XY} \equiv\frac{\operatorname{Cov}(X,Y)} {\sigma_X\sigma_Y}.

The quantity is defined when both standard deviations are positive and finite.

Computing the means and covariance

The target probabilities are

\mathbb{E}[X]=p_X(1)=.50

and

\mathbb{E}[Y]=p_Y(1)=.60.

Since XY=1 only in the lower right cell,

\mathbb{E}[XY]=.40.

An equivalent covariance formula is

\operatorname{Cov}(X,Y) =\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y].

Thus

\operatorname{Cov}(X,Y) =.40-.50(.60) =.10.

Computing the two standard deviations

For a zero and one variable with target probability \pi, the variance is \pi(1-\pi). Thus

\operatorname{Var}(X)=.50(.50)=.25

and

\operatorname{Var}(Y)=.60(.40)=.24.

The standard deviations are

\sigma_X=.50

and

\sigma_Y=\sqrt{.24}\approx.490.

The correlation is

\begin{aligned} \rho_{XY} &=\frac{.10}{.50\sqrt{.24}}\\ &\approx.408. \end{aligned}

The positive value reflects the concentration of mass in the two matching cells, especially X=1,Y=1.

Verifying the calculation in base R

Code

x <- c(0, 0, 1, 1)
y <- c(0, 1, 0, 1)
mass <- c(.30, .20, .10, .40)

mean_x <- sum(x * mass)
mean_y <- sum(y * mass)
cov_xy <- sum((x - mean_x) * (y - mean_y) * mass)
sd_x <- sqrt(sum((x - mean_x)^2 * mass))
sd_y <- sqrt(sum((y - mean_y)^2 * mass))

cov_xy / (sd_x * sd_y)

The result is approximately .408.

Reading the scale

Correlation lies between -1 and 1.

value	linear pattern
1	perfect increasing linear relation
0	no linear association
-1	perfect decreasing linear relation

Intermediate values summarize direction and degree of linear association. A value of zero does not rule out a nonlinear relationship.

Examining why positive unit changes cancel

If X^*=a+bX with b>0, then the covariance with Y is multiplied by b and the standard deviation of X is also multiplied by b. The factor cancels in the correlation.

Adding a constant changes the mean but not the centered deviations. Thus converting a measurement by a positive linear unit transformation does not change correlation.

Reversing a binary coding uses a negative slope and flips the sign. The interpretation must remain tied to the outcome coded as 1.

Correlation does not identify a causal direction

A correlation is not a directional causal effect. Correlation is symmetric:

\rho_{XY}=\rho_{YX}.

It does not select a response, adjust for other variables, or identify what would happen under an intervention. Those claims need a design and model that represent direction.

Check your understanding

Recompute the covariance if the lower right cell changes to .30 and the lower left changes to .20 while the margins are allowed to change.
Why does adding 100 to every value of a continuous X leave correlation unchanged?
What happens to the sign if the coding of Y is reversed?
Give one nonlinear relationship that could have correlation near zero.

This completes the random variables and distributions module. The next module begins the distinction between population quantities, samples, and estimators.

--- title: "Correlation" --- The [preceding page](covariance.qmd) showed that covariance depends on both association and measurement scale. Correlation divides covariance by the [standard deviations](standard-deviation.qmd) of the two variables. Suppose $X$ records whether a dative recipient is pronominal and $Y$ records whether the double object construction is used: $$ X=1\text{ for a pronominal recipient} $$ and $$ Y=1\text{ for a double object construction}. $$ Consider this constructed joint PMF. | | $Y=0$ | $Y=1$ | |---|---:|---:| | $X=0$ | $.30$ | $.20$ | | $X=1$ | $.10$ | $.40$ | The table suggests a positive linear association, but covariance depends on the numerical scales. [**Correlation**](https://bruno.nicenboim.me/bayescogsci/ch-intro.html) standardizes covariance by the two standard deviations: $$ \rho_{XY} \equiv\frac{\operatorname{Cov}(X,Y)} {\sigma_X\sigma_Y}. $$ The quantity is defined when both standard deviations are positive and finite. ## Computing the means and covariance The target probabilities are $$ \mathbb{E}[X]=p_X(1)=.50 $$ and $$ \mathbb{E}[Y]=p_Y(1)=.60. $$ Since $XY=1$ only in the lower right cell, $$ \mathbb{E}[XY]=.40. $$ An equivalent covariance formula is $$ \operatorname{Cov}(X,Y) =\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]. $$ Thus $$ \operatorname{Cov}(X,Y) =.40-.50(.60) =.10. $$ ## Computing the two standard deviations For a zero and one variable with target probability $\pi$, the variance is $\pi(1-\pi)$. Thus $$ \operatorname{Var}(X)=.50(.50)=.25 $$ and $$ \operatorname{Var}(Y)=.60(.40)=.24. $$ The standard deviations are $$ \sigma_X=.50 $$ and $$ \sigma_Y=\sqrt{.24}\approx.490. $$ The correlation is $$ \begin{aligned} \rho_{XY} &=\frac{.10}{.50\sqrt{.24}}\\ &\approx.408. \end{aligned} $$ The positive value reflects the concentration of mass in the two matching cells, especially $X=1,Y=1$. ## Verifying the calculation in base R ```{r} #| eval: false x <- c(0, 0, 1, 1) y <- c(0, 1, 0, 1) mass <- c(.30, .20, .10, .40) mean_x <- sum(x * mass) mean_y <- sum(y * mass) cov_xy <- sum((x - mean_x) * (y - mean_y) * mass) sd_x <- sqrt(sum((x - mean_x)^2 * mass)) sd_y <- sqrt(sum((y - mean_y)^2 * mass)) cov_xy / (sd_x * sd_y) ``` The result is approximately `.408`. ## Reading the scale Correlation lies between $-1$ and $1$. | value | linear pattern | |---:|---| | $1$ | perfect increasing linear relation | | $0$ | no linear association | | $-1$ | perfect decreasing linear relation | Intermediate values summarize direction and degree of linear association. A value of zero does not rule out a nonlinear relationship. ## Examining why positive unit changes cancel If $X^*=a+bX$ with $b>0$, then the covariance with $Y$ is multiplied by $b$ and the standard deviation of $X$ is also multiplied by $b$. The factor cancels in the correlation. Adding a constant changes the mean but not the centered deviations. Thus converting a measurement by a positive linear unit transformation does not change correlation. Reversing a binary coding uses a negative slope and flips the sign. The interpretation must remain tied to the outcome coded as 1. ## Correlation does not identify a causal direction A correlation is not a directional causal effect. Correlation is symmetric: $$ \rho_{XY}=\rho_{YX}. $$ It does not select a response, adjust for other variables, or identify what would happen under an intervention. Those claims need a design and model that represent direction. ## Check your understanding 1. Recompute the covariance if the lower right cell changes to $.30$ and the lower left changes to $.20$ while the margins are allowed to change. 2. Why does adding 100 to every value of a continuous $X$ leave correlation unchanged? 3. What happens to the sign if the coding of $Y$ is reversed? 4. Give one nonlinear relationship that could have correlation near zero. This completes the random variables and distributions module. The [next module](../statistical-inference/index.qmd) begins the distinction between population quantities, samples, and estimators.