Learning from a sample

What can one realized sample tell us?

What can observed speakers, tokens, items, or measurements tell us beyond the realized sample?

Six hundred rows can contain only 30 speakers

A corpus contains 600 relative-clause tokens from 30 speakers. A relative clause modifies a nominal expression, as in the book that Kim read.

A file proportion is not a population proportion

The file proportion, the average speaker proportion, and a population proportion are different quantities.

The realized sample could have been different

One dataset is only one possible result of a sampling process.

Inference connects a sample to a target through a model

First name the population quantity. Then define the estimation rule and study that rule under a stated sampling model.

Thirty speakers supply repeated vowel tokens

Thirty Rochester speakers each produce twenty vowel tokens.

The table has 600 rows and 30 sampled speakers.

Module 1: observational units and rows

Speech-community claims require a target population

Buchstaller and Khattab ask how a sample of speakers can support a claim about a speech community.

Buchstaller and Khattab 2013

Target population and realized sample

A target population contains the units about which we want to make a claim.

A sample contains the units observed through a particular selection process.

The estimand names the population target

Let $P$ denote the population distribution. An estimand

$$\tau\equiv T(P)$$

is the population quantity selected by the linguistic question, such as a population mean or contrast.

Module 1: estimands and observations

Parameter, estimator, and estimate

A parameter, such as $\mu$, is a fixed numerical feature of the modeled population.

An estimator is a random variable such as

$$\widehat T\equiv g(X_1,\ldots,X_n).$$

Its realized value $\widehat t=g(x_1,\ldots,x_n)$ is an estimate of $\tau=T(P)$.

Module 2: random variables map outcomes to values

Repeated rows do not create new sampled speakers

Six hundred tokens can sharpen summaries for 30 observed speakers. They do not create 600 independently sampled speakers.

Notes: populations and samples

Four clause tokens form one realized sample

Let $X_i=1$ when clause token $i$ contains an overtly expressed subject and $X_i=0$ otherwise. The realized values are

$$(x_1,x_2,x_3,x_4)=(1,0,1,1)$$

IID combines a shared distribution with independence

Random variables are independent and identically distributed (IID) when (i) they have the same distribution and (ii) their joint probability factors into marginal probabilities.

This claim concerns the sampling units, not merely distinct rows.

Bernoulli notation for the working model

$$X_i\overset{\mathrm{IID}}{\sim}\operatorname{Bernoulli}(\pi),\qquad \mathbb{P}(X_1=x_1,\ldots,X_n=x_n\mid\pi) =\prod_{i=1}^n p_X(x_i\mid\pi).$$

Recall the Bernoulli distribution, probability factorization, and independence.

The realized sequence has a model probability

For the observed sequence,

$$\mathbb{P}(X_1=1,X_2=0,X_3=1,X_4=1\mid\pi)=\pi^3(1-\pi).$$

Distinct rows may still share a sampling source

Distinct rows need not be independent. Clauses can share speakers, documents, genre, and discourse context.

Notes: independent sampling

The teaching file stores a heuristic accusative label

The Universal Dependencies English Web Treebank teaching extract contains 12,280 personal-pronoun tokens. An older preprocessing rule labels case-distinct forms by surface form; for syncretic you and it, it uses the basic dependency relation as a fallback.

This heuristic accusative label is not the UD morphological feature Case=Acc.

UD English EWT

The file proportion is 0.271

Heuristically accusative tokens: $3{,}327$

Other tokens: $8{,}953$

Define the descriptive file proportion

$$r_{\mathrm{file}}\equiv\frac{3327}{12280}\approx.271.$$

The estimand depends on the sampling claim

Without a population sampling model, $.271$ describes the heuristic labels in this extract. It does not estimate the proportion of current UD Case=Acc annotations, explain case licensing, or justify independence across documents and repeated forms.

Notes: the pronoun-case example · UD English Case

Which parameter value best supports the observed sequence?

Which heuristic-label probability makes the observed token sequence most probable under the working model?

Likelihood treats the data as fixed

The likelihood function evaluates the observed values under each parameter value:

$$L(\pi;\mathbf x_{\mathrm{obs}}) \equiv p_{\mathbf X}(\mathbf x_{\mathrm{obs}}\mid\pi) =\pi^{3327}(1-\pi)^{8953}.$$

Likelihood is not a probability distribution over parameters

$p_{\mathbf X}(\mathbf x\mid\pi)$ is a PMF over possible samples for fixed $\pi$.

$L(\pi;\mathbf x_{\mathrm{obs}})$ compares parameter values for fixed observations.

Module 2: PMFs · Module 2: likelihood and model fit

Log likelihood turns products into sums

The log likelihood turns a product into a sum.

$$\ell(\pi;\mathbf x_{\mathrm{obs}}) =k\log\pi+(n-k)\log(1-\pi).$$

The maximum-likelihood estimator is a rule

The maximum-likelihood estimator is the rule

$$\frac{\partial\ell}{\partial\pi} =\frac{k}{\pi}-\frac{n-k}{1-\pi}=0 \quad\Longrightarrow\quad \widehat\Pi_{\mathrm{MLE}}\equiv\frac{K}{n}.$$

For this sample, the realized maximum-likelihood estimate is $\widehat\pi_{\mathrm{MLE}}=3327/12280\approx.271$.

Optimization does not establish model adequacy

Maximum likelihood selects a parameter value under the declared model. It does not establish that the model is adequate.

Notes: maximum likelihood

The response is absolute linear token-index distance

The English Web Treebank teaching extract contains 2,594 selected wh-form tokens and their annotated basic-dependency heads. For token $i$, let

$$D_i\equiv|\operatorname{index}(\textit{wh}_i)-\operatorname{index}(\text{head}_i)|.$$

Thus $D_i=1$ for adjacent tokens; zero is excluded because a dependency cannot connect a token to itself.

The stored variable is not a filler-gap distance

This response is not a filler-gap distance, and the stored name gap_position denotes the annotated head token rather than a silent gap.

Silveira et al. 2014

The Poisson log likelihood

Under the working model $D_i\overset{\mathrm{IID}}{\sim}\operatorname{Poisson}(\lambda)$,

$$\ell(\lambda;\mathbf d)=\left(\sum_{i=1}^n d_i\right)\log\lambda-n\lambda-\sum_{i=1}^n\log(d_i!).$$

Recall the Poisson distribution.

The MLE equals the sample mean

Setting the derivative to zero gives

$$\frac{\partial\ell}{\partial\lambda} =\frac{\sum_i d_i}{\lambda}-n=0 \quad\Longrightarrow\quad \widehat\Lambda_{\mathrm{MLE}}=\overline D, \qquad \widehat\lambda_{\mathrm{MLE}}=\bar d\approx3.007.$$

The response codomain exposes a model mismatch

The distance data exclude zero, and the Poisson mean variance equality may fail.

Estimation and adequacy are separate questions.

Notes: Poisson maximum likelihood · UD dependency relations

Lexical-decision accuracy supplies a binary response

In a lexical-decision task, a participant judges whether a presented letter string is a word. Let $X_i=1$ for a correct response and $X_i=0$ for an error.

For this illustration, participant-item encounters are independent sampling units and population accuracy is $\pi=.75$.

Lexical-decision response measures

Imagine repeating the sampling process

Each possible sample $\mathbf X=(X_1,\ldots,X_n)$ produces a possibly different sample-proportion estimator

$$\widehat\Pi\equiv\overline X.$$

The sampling distribution belongs to the estimator

The sampling distribution is the probability distribution of an estimator across possible samples.

Parameter, estimator, and estimate are different objects

The parameter $\pi$ is fixed. The estimator $\widehat\Pi$ varies across samples. The estimate $\widehat\pi=\bar x$ is the value for one realized sample.

Larger independent samples concentrate estimates

Larger independent samples concentrate the sampling distribution around the same parameter.

Sampling distributions contain possible estimates

A sampling distribution contains possible estimates, not possible population parameters.

Real lexical-decision experiments usually repeat participants and items. Then the independent-unit assumption must reflect both sources of clustering.

Notes: properties of estimators

How much would the estimator vary across samples?

How much would the estimator’s value vary across repeated samples under the model?

Standard error is sampling-distribution spread

The standard error is the standard deviation of an estimator’s sampling distribution.

Standard error of an IID sample mean

For an IID sample mean,

$$\operatorname{Var}(\overline X) =\frac{1}{n^2}\sum_{i=1}^n\sigma^2 =\frac{\sigma^2}{n}, \qquad \operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}.$$

Four times the sample size halves the standard error

Doubling $n$ does not halve the standard error. Multiplying $n$ by four does.

The denominator must count independent units

Replacing the true independent unit with the row count makes the standard error too small.

Notes: standard errors

Bias averages signed error across samples

The bias of an estimator is

$$\operatorname{Bias}(\widehat\Theta)=\mathbb E[\widehat\Theta]-\theta.$$

Module 2: expected values

Bias is not one sample’s realized error

An unbiased estimator may miss the parameter in one realized sample.

Realized error is not estimator bias.

Unbiased estimators may have different precision

Unbiasedness alone does not rank estimators. Their sampling variation may differ.

Notes: estimator bias

Mean squared error averages squared estimation error

Mean squared error averages squared estimation error.

$$\operatorname{MSE}(\widehat\Theta)=\mathbb E[(\widehat\Theta-\theta)^2]$$

MSE separates variance from squared bias

$$\operatorname{MSE}(\widehat\Theta) =\operatorname{Var}(\widehat\Theta) +\operatorname{Bias}(\widehat\Theta)^2.$$

Small bias can accompany lower MSE

A slightly biased estimator can have lower MSE if its sampling variance is sufficiently smaller.

Bias and variance answer different questions

Bias and variance answer different questions. MSE evaluates their combined squared error.

Notes: mean squared error

Identify the target, estimator, and estimate

Suppose a perception study compares mean prosodic-prominence ratings in conditions $A$ and $B$.

The population contrast $\Delta$ is the estimand. The difference between sample means, $\widehat{\Delta}$, is the estimator.

$$\Delta=\mu_A-\mu_B, \qquad \widehat{\delta}=.42, \qquad \widehat{\operatorname{SE}}(\widehat{\Delta})=.15.$$

Prosodic prominence may depend on several cues; it is not another name for duration or $F_0$.

Estimate and standard error answer different questions

The estimate locates the observed contrast. The standard error describes how much the estimator would vary across samples under the model.

A confidence interval combines them through a procedure with declared repeated-sampling behavior.

Begin with the studentized pivot

Suppose

$$\frac{\widehat{\Delta}-\Delta} {\widehat{\operatorname{SE}}(\widehat{\Delta})} \approx N(0,1).$$

Thus the central $.95$ event is

$$-1.96\leq \frac{\widehat{\Delta}-\Delta} {\widehat{\operatorname{SE}}(\widehat{\Delta})} \leq1.96.$$

Algebra places the parameter between random endpoints

Multiplying by the positive standard error and rearranging gives

$$\widehat{\Delta}-1.96\widehat{\operatorname{SE}}(\widehat{\Delta}) \leq\Delta\leq \widehat{\Delta}+1.96\widehat{\operatorname{SE}}(\widehat{\Delta}).$$

The observed sample gives one interval

$$.42\mathbin{\pm}1.96(.15) =.42\mathbin{\pm}.294 =[.126,.714].$$

After sampling, these endpoints are fixed.

The check has a limited interpretation

This simulation uses a known standard error and an exactly normal sampling distribution. It illustrates coverage; it does not verify the prominence model’s assumptions.

A realized confidence interval is not a posterior interval

After observing the data, the parameter either lies in $[.126,.714]$ or it does not. The model does not license a $.95$ probability for that fixed event.

The interval also does not contain 95% of participant responses. Response variation and estimator uncertainty are different quantities.

Wald intervals can fail near probability boundaries

A Wald interval can extend outside zero and one and can have poor coverage for small samples or probabilities near a boundary.

The Wald interval centers a normal approximation

For $k=18$ successes in $n=20$ trials,

$$\widehat\pi\pm1.96\sqrt{\frac{\widehat\pi(1-\widehat\pi)}{n}} =.90\pm1.96(.0671) =[.7685,1.0315].$$

Clopper-Pearson inverts binomial tails

Write $\mathbb{P}_\pi$ for probability under $K\sim\operatorname{Binomial}(20,\pi)$.

$$\mathbb{P}_{\pi_L}(K\ge18)=.025, \qquad \mathbb{P}_{\pi_U}(K\le18)=.025.$$

The resulting interval is $[.6830,.9877]$.

Exact calculation does not make the sampling model exact

“Exact” refers to the finite sample binomial calculation. It does not establish independence, a shared probability, or representative sampling.

Notes: exact binomial intervals

Fundamental frequency is an acoustic measurement

Twelve speakers supply median fundamental-frequency ($F_0$) measurements. $F_0$ measures the rate of vocal-fold vibration in hertz and serves as an acoustic correlate of perceived pitch.

The target is the population median.

The bootstrap treats observed units as stand-ins

The nonparametric bootstrap treats the empirical distribution of the observed independent units as a stand-in for the population, then resamples those units with replacement.

Percentile intervals use bootstrap quantiles

A percentile bootstrap confidence interval uses quantiles of the bootstrap estimates:

$$C^*_{.95} =\left[ \widehat t^{*}_{(.025)}, \widehat t^{*}_{(.975)} \right].$$

Resampling must preserve within-speaker clustering

If tokens repeat within speakers, resample speakers and retain each selected speaker’s tokens together.

The resampling unit must match the population claim

The resampling unit must match the independent unit in the population claim.

Notes: bootstrap confidence intervals · Praat: frequency and $F_0$

Null and alternative hypotheses partition parameter claims

A null hypothesis states a parameter claim. An alternative hypothesis states the competing parameter values.

A test statistic measures sample disagreement with the null

The random statistic

$$T\equiv\frac{\widehat\Theta-\theta_0} {\widehat{\operatorname{SE}}(\widehat\Theta)}$$

varies across possible samples. The observed sample produces $t_{\mathrm{obs}}$.

The null distribution supplies the reference scale

The null distribution is the reference distribution of the test statistic under the null model.

A p value is a null-model tail probability

For a one-sided upper-tail test,

$$p\text{-value}=\mathbb{P}_{H_0}\!\left( \left\{\omega\in\Omega\mid T(\omega)\geq t_{\mathrm{obs}} \right\} \right).$$

A two-sided test needs a declared extremeness rule. A p value is neither an effect size nor the probability that the null hypothesis is true.

Notes: null hypotheses and test statistics

One speaker supplies one acoustic contrast

Let $D_s$ be one predeclared acoustic contrast from speaker $s$. The speakers, rather than the underlying token rows, are the independent sampling units.

The target is the population mean contrast $\mu_D$.

The one-sample t-test applies the t procedure to these speaker contrasts.

Studentization uses an estimated standard error

$$T=\frac{\overline D-\mu_D}{S_D/\sqrt n}, \qquad T\sim t_{n-1}.$$

One-sample calculation

For $n=10$, $\bar d=18$ ms, and $s_D=20$ ms,

$$\widehat{\operatorname{SE}}(\overline D)=\frac{20}{\sqrt{10}}=6.3249,$$

$$18\pm t_{.975,9}(6.3249) =[3.69,32.31]\text{ ms}.$$

The zero-reference test

$$t_{\mathrm{obs}}=\frac{18}{6.3249}=2.846, \qquad p=2\mathbb{P}(T\ge2.846),\quad T\sim t_9.$$

The two-sided p value is $.0192$.

Token count is not speaker sample size

Many tokens from one speaker do not provide many independent speaker contrasts.

Notes: one-sample t inference

Two vowel measurements share a speaker

Every speaker produces one token in each of two vowel categories. A vowel category is a linguistic category; its acoustic measurements are numerical responses.

The measurements share vocal tract anatomy and recording conditions.

A paired design creates one within-pair response

A paired design forms one within-pair difference for each independent pair.

The difference keeps the speaker identity

$$D_s=Y_{sA}-Y_{sB}.$$

Shared additive speaker shifts cancel inside $D_s$.

Covariance determines the precision gained by pairing

Covariance determines the precision gained by pairing.

Positive within-pair covariance reduces the variance of the difference.

Module 2: covariance

Independent sorting destroys the pairs

Sorting the two response vectors independently destroys the pairing and can create invalid differences.

Notes: paired observations

The estimand is a mean within-speaker difference

Estimate the population mean of the speaker differences, $\mu_D$.

A paired t test is a one-sample test on differences

The paired t-test is a one-sample t-test applied to $D_1,\ldots,D_S$.

$$T\equiv\frac{\overline D-\mu_{D,0}}{S_D/\sqrt S}, \qquad T\sim t_{S-1}.$$

$F_1$ is a formant-frequency estimate

Speaker differences use the Hillenbrand et al. first-formant ($F_1$) frequency estimates. $F_1$ is an estimated center frequency of the first vocal-tract resonance, measured in hertz; it is not a phoneme or vowel-category label.

Hillenbrand et al. 1995

Four paired differences

For $(-69,-58,-82,-37)$ Hz,

$$\bar d=-61.5,\qquad s_D=19.05,\qquad \widehat{\operatorname{SE}}(\overline D)=\frac{19.05}{2}=9.53.$$

$$-61.5\pm t_{.975,3}(9.53) =[-91.82,-31.18]\text{ Hz}.$$

The interval targets a mean contrast

The interval describes the mean within speaker vowel contrast, not the distribution of individual vowel measurements.

Notes: paired-mean inference

What changes when the groups contain different speakers?

How does uncertainty change when the two condition means come from different speakers?

Welch’s test allows different group variances

Welch’s two-sample t-test allows the two groups to have different variances.

Independent groups contribute separate variance terms

$$\widehat{\operatorname{SE}}(\bar X_1-\bar X_2) =\sqrt{\frac{S_1^2}{n_1}+\frac{S_2^2}{n_2}}.$$

For $(\bar y_A,s_A,n_A)=(510,60,20)$ and $(\bar y_B,s_B,n_B)=(550,90,15)$,

$$\widehat\delta=-40\text{ Hz},\qquad \widehat{\operatorname{SE}}=\sqrt{180+540}=26.833\text{ Hz}.$$

Welch degrees of freedom

$$\nu\approx \frac{(s_A^2/n_A+s_B^2/n_B)^2} {(s_A^2/n_A)^2/(n_A-1)+(s_B^2/n_B)^2/(n_B-1)} =23.005.$$

The 95% interval is $[-95.51,15.51]$ Hz.

Equal variance needs substantive justification

Do not use an equal variance assumption merely because a software default or familiar formula includes it.

The procedure must match the design

Paired and independent designs use different sources of variation. The test must match the design.

Notes: comparing independent means

Syllable count and plural realization form a two-by-two table

This teaching sample classifies noun lexemes by singular-lemma syllable count and by plural realization class: productive -(e)s versus another realization.

The observed two-by-two table

The question is whether the two classifications are independent.

Fisher’s test conditions on the observed margins

Fisher’s exact test constructs a conditional null distribution over tables with the observed margins fixed.

One free cell determines the whole table

With the margins fixed, the upper-left count $A$ determines the other three cells. Under conditional independence,

$$\mathbb{P}_{H_0}(A=18) =\frac{{44\choose18}{16\choose12}}{{60\choose30}} \approx.0158.$$

The two-sided p value also includes other tables at least as improbable under the declared ordering rule.

The observed-table probability is not the p value

The two-sided p value is approximately $.0391$.

The observed table probability is only one term in that sum.

The odds ratio describes the sample association

The odds of productive -(e)s are $18/12$ for monosyllabic lexemes and $26/4$ for polysyllabic lexemes. Their sample odds ratio is

$$\operatorname{OR}_{\mathrm{obs}} =\frac{18/12}{26/4}\approx.231.$$

Association does not identify the linguistic mechanism

Association between syllable count and plural realization class does not identify a causal or linguistic mechanism.

Notes: Fisher’s exact test · R: fisher.test

Grammatical role and referring-expression form

A corpus classifies 300 referring-expression tokens by grammatical role and expression form: personal pronoun or lexical noun phrase.

The chi-squared test compares observed and expected counts

The chi-squared test of independence compares observed cell counts with counts expected under independence.

One expected count under independence

For cell $r,c$,

$$E_{rc}=\frac{(\text{row total})(\text{column total})}{\text{grand total}}.$$

Thus the subject-pronoun cell has $O_{11}=120$ but $E_{11}=150(200)/300=100$.

Pearson’s statistic adds standardized cell discrepancies

$$\chi^2=\sum_{r,c}\frac{(O_{rc}-E_{rc})^2}{E_{rc}}, \qquad df=(r-1)(c-1).$$

For this table, $\chi^2=24$ with $df=1$.

Signed differences locate the association

The total statistic records aggregate discrepancy across the whole table. Signed cell differences show which forms occur more or less often than the independence model expects.

Large counts do not repair clustered observations

Small expected counts weaken the approximation. Repeated speakers or documents also violate the independent row assumption.

Notes: chi-squared tests

Plural predication can support distinct readings

A listener classifies an ambiguous plural-subject sentence. Under a collective reading, the predicate holds of the group; under a distributive reading, it holds of each relevant member.

Let $Y_i=1$ for a collective response and $Y_i=0$ for a distributive response.

Experimental evidence on collective and distributive readings

The prior represents parameter uncertainty before these data

A prior distribution assigns probability to parameter values before the present data are observed.

Bayes’ rule reweights prior density

Here $p$ denotes the response PMF and $f$ denotes the parameter density:

$$f_{\Theta\mid\mathbf Y}(\theta\mid\mathbf y) =\frac{p_{\mathbf Y\mid\Theta}(\mathbf y\mid\theta)f_\Theta(\theta)} {p_{\mathbf Y}(\mathbf y)}.$$

Recall Bayes’ rule.

Module 2: PMFs · Module 2: PDFs

The posterior represents uncertainty after conditioning

The posterior distribution represents parameter uncertainty after conditioning on the observed data.

A posterior is a distribution; a likelihood is not

The posterior is a probability distribution over the parameter. The likelihood alone is not.

Notes: posterior distributions

Begin with draws from the declared target

Suppose the analysis produces $S=4{,}000$ draws

$$\Delta^{(1)},\ldots,\Delta^{(4000)}.$$

Their empirical distribution approximates $f_{\Delta\mid\mathbf Y}(\delta\mid\mathbf y)$ only when the computational procedure is valid.

A summary must match the linguistic claim

A posterior summary is a numerical description calculated from the posterior distribution or its draws.

State whether the claim concerns direction, magnitude, or a predicted response; then select the corresponding summary.

Five draws give two centers

For draws $-.09,-.07,-.06,-.05,.02$,

$$\widehat{\mathbb E}[\Delta\mid\mathbf y] =\frac{-.09-.07-.06-.05+.02}{5}=-.05, \qquad \operatorname{median}(\Delta\mid\mathbf y)=-.06.$$

The positive tail pulls the mean upward.

A loss function justifies a point summary

A loss function assigns a cost to estimation error.

• Posterior mean: minimizes posterior expected squared-error loss.
• Posterior median: minimizes posterior expected absolute-error loss.

Neither is automatically the Bayesian estimate.

A credible interval carries conditional probability

If the $.025$ and $.975$ posterior quantiles are

$$[-.08,-.02],$$

then $.95$ of the fitted posterior probability for $\Delta$ lies between them, conditional on the likelihood, prior, and observed data.

Credible and confidence intervals license different claims

Finite-draw endpoints have Monte Carlo error. And posterior probability does not establish that the model or prior represents the linguistic sampling process.

Numerically similar confidence and credible intervals can thus support different interpretations.

Directional probability is not magnitude

Let $\Delta:\Omega\to\mathbb R$ map each outcome to a response-probability difference. For a directional claim, define

$$B_\Delta\equiv\{\omega\in\Omega\mid\Delta(\omega)<0\}.$$

Let $\mathbf 1_{B_\Delta}:\Omega\to\{0,1\}$ equal 1 on $B_\Delta$ and 0 otherwise. Then estimate

$$\mathbb{P}(B_\Delta\mid\mathbf y) \approx\frac{1}{4000}\sum_{s=1}^{4000} \mathbf 1_{B_\Delta}(\omega^{(s)}) =\frac{3800}{4000}=.95.$$

This value is neither a p value nor a measure of association size.

Transform each draw before summarizing

For odds, compute $\Theta^{(s)}/(1-\Theta^{(s)})$ before taking summaries. In general,

$$\mathbb E\!\left[\frac{\Theta}{1-\Theta}\middle|\mathbf y\right] \ne \frac{\mathbb E[\Theta\mid\mathbf y]}{1-\mathbb E[\Theta\mid\mathbf y]}.$$

Preserve joint uncertainty in a contrast

For each paired draw, compute

$$\Delta^{(s)}=\Theta_A^{(s)}-\Theta_B^{(s)}.$$

Separate marginal intervals discard which values occurred together and do not determine the contrast distribution.

Match the summary to the claim

A parameter summary, predicted-response contrast, and directional probability answer different questions. Predeclare diagnostic profiles or show a complete small grid.

Which probability statement does $[-.08,-.02]$ support, and why must a multi-parameter contrast be computed within each draw?

Notes: posterior summaries

Coronal-stop deletion is a coded variant choice

A phonological study codes whether word-final /t/ or /d/ is absent in a predeclared environment. It observes 14 deletion labels and 6 realized-stop labels.

The shared-rate model below suppresses known conditioning by segmental context, morphological class, and speaker.

Coronal-stop deletion and morphological conditioning

Conjugacy preserves the prior’s distribution family

A conjugate prior yields a posterior in the same distribution family after multiplication by the likelihood.

Beta prior and binomial observation model

$$\Pi\sim\operatorname{Beta}(\alpha,\beta),\qquad Y\mid\Pi=\pi\sim\operatorname{Binomial}(n,\pi).$$

Counts update the two Beta shape parameters

$$\Pi\mid\mathbf D=\mathbf d \sim\operatorname{Beta}(\alpha+d,\beta+r).$$

Pseudocounts are an algebraic analogy

The update resembles adding prior successes and failures. This pseudocount analogy explains the algebra; the shape parameters are not literal unobserved tokens.

Notes: conjugate priors · Module 2: Beta distributions · Module 2: binomial distributions

What data does the prior model imply?

What response counts does the complete prior model expect for ten ambiguous sentences?

The prior predictive distribution averages over the prior

The prior predictive distribution averages the observation model over the prior.

$$p_{\widetilde Y}(\widetilde y) =\int p_{\widetilde Y\mid\Theta}(\widetilde y\mid\theta)f_\Theta(\theta)\,\mathrm d\theta.$$

One parameter draw generates one simulated dataset

Draw one parameter value for a complete simulated dataset, then generate the responses conditional on that value.

Inspect prior implications on the response scale

Inspect predictions on the response scale. Ask whether the implied counts are plausible before using the present responses.

Trial-level parameter redraws imply another model

Drawing a new response probability for every trial represents a different sampling structure from drawing one for the complete ten trial dataset.

Notes: prior predictive distributions

The posterior predictive distribution averages over the posterior

The posterior predictive distribution averages the observation model over posterior uncertainty.

$$p_{\widetilde Y}(\widetilde y\mid\mathbf y) =\int p_{\widetilde Y\mid\Theta}(\widetilde y\mid\theta) f_{\Theta\mid\mathbf Y}(\theta\mid\mathbf y)\,\mathrm d\theta.$$

Posterior prediction proceeds draw by draw

Draw $\theta^{(s)}$ from the posterior, then draw the future value $\widetilde y^{(s)}$ from the observation model conditional on $\theta^{(s)}$.

Predictive spread has two sources

Predictive spread contains parameter uncertainty and future response variation.

These are distinct sources of variation.

Prediction and model checking ask different questions

Prediction asks what future observations may look like. Model checking compares replicated observations with the observed data.

Notes: predictive distributions

Closure duration is a measured time interval

A stop’s closure duration is the time interval during which oral airflow is blocked. Four independent speakers supply standardized closure-duration contrasts; each contrast is dimensionless after standardization.

Closure duration of stop consonants

The normalizing constant turns a kernel into a density

The normalizing constant is

$$Z=\int f_{\mathbf Y\mid\Theta}(\mathbf y\mid\theta)f_\Theta(\theta)\,\mathrm d\theta.$$

Relative posterior density does not require the normalizer

Define the posterior kernel

$$\kappa(\theta;\mathbf y) =f_{\mathbf Y\mid\Theta}(\mathbf y\mid\theta)f_\Theta(\theta).$$

Ratios of $\kappa$ do not require $Z$.

Computational difficulty is not posterior uncertainty

Computational difficulty does not imply broad posterior uncertainty. A concentrated posterior can still have an intractable normalizer.

How can draws approximate a posterior expectation?

How can we calculate posterior expectations without evaluating every possible parameter value?

Notes: beyond conjugacy

Posterior probability as an integral

Let $\Pi:\Omega\to[0,1]$ and suppose

$$\Pi\mid\mathbf x\sim\operatorname{Beta}(5,3), \qquad B_\Pi\equiv\{\omega\in\Omega\mid\Pi(\omega)>.60\}.$$

The target is $I\equiv\mathbb P(B_\Pi\mid\mathbf x)$.

Rewriting probability as an expectation

Define $\mathbf 1_{B_\Pi}:\Omega\to\{0,1\}$ by

$$\mathbf 1_{B_\Pi}(\omega)= \begin{cases} 1 & \omega\in B_\Pi,\\ 0 & \omega\notin B_\Pi. \end{cases}$$

If $g(\pi)=1$ when $\pi>.60$ and $0$ otherwise, then

$$\mathbb E[g(\Pi)\mid\mathbf x] =\mathbb P(B_\Pi\mid\mathbf x).$$

Replacing the expectation with an average

Draw independently from the target:

$$\pi^{(1)},\ldots,\pi^{(S)} \overset{\mathrm{iid}}{\sim}\operatorname{Beta}(5,3).$$

$$\widehat I_S =\frac1S\sum_{s=1}^Sg(\pi^{(s)}).$$

Generalizing the calculation

For a continuous density $f$ and integrable $g$,

$$I=\mathbb E_f[g(\Theta)] =\int g(\theta)f(\theta)\,\mathrm d\theta, \qquad \widehat I_S=\frac1S\sum_{s=1}^Sg(\theta^{(s)}).$$

For a discrete target, replace the integral with a sum.

Quantifying Monte Carlo error

Monte Carlo error is variation caused by using a finite number of simulation draws.

For an indicator estimator,

$$\widehat{\operatorname{MCSE}}(\widehat I_S) =\sqrt{\frac{\widehat I_S(1-\widehat I_S)}{S}}.$$

Multiplying $S$ by four tends to divide MCSE by two.

Keeping two uncertainties separate

More simulation draws reduce Monte Carlo error. They do not reduce posterior uncertainty.

More draws are not more linguistic evidence.

Direct target draws may be unavailable

Direct draws from the target distribution may be unavailable.

A proposal distribution supplies available draws

Draw from a proposal distribution $Q$ with density $q$ and reweight the draws toward the target density $f$.

Importance weights correct proposal frequency

The importance weight is

$$w(\theta)=\frac{f(\theta)}{q(\theta)}.$$

Reweighted proposal averages target the expectation

$$\mathbb E_f[h(\Theta)]=\mathbb E_Q[w(\Theta)h(\Theta)]$$

when the target is normalized and the proposal covers its support.

A few large weights can dominate

A few very large weights can dominate the estimate. Unweighted draw frequency comes from $Q$; weighted averages approximate the target with density $f$.

Notes: importance sampling

The current state screens off the earlier path

A Markov chain is a sequence of random states. Given the current state, the distribution of the next state does not depend on the earlier path.

The Markov property in notation

For a permitted (measurable) set $A$ of possible next-state values,

$$\mathbb{P}(X_{t+1}\in A\mid X_t=x_t,\ldots,X_0=x_0) =\mathbb{P}(X_{t+1}\in A\mid X_t=x_t).$$

Module 2: conditional independence

A stationary distribution is unchanged by one transition

A distribution is stationary when one transition leaves it unchanged.

Irreducibility permits movement across relevant states

Irreducibility means that the chain can eventually reach every region that the target distribution assigns positive probability.

Aperiodicity prevents a fixed return cycle

Aperiodicity rules out deterministic cycling through states at a fixed period.

These conditions concern long-run access. They do not guarantee efficient exploration in a finite run.

Notes: Markov chains

Define the target before the chain

For a standardized articulation-rate contrast $\Theta$,

$$I=\mathbb E[g(\Theta)\mid\mathbf y] =\int g(\theta)f_{\Theta\mid\mathbf Y}(\theta\mid\mathbf y)\,\mathrm d\theta.$$

The likelihood and prior define this target.

MCMC constructs dependent states

Markov chain Monte Carlo (MCMC) constructs a Markov chain whose stationary distribution is the target.

$$\theta^{(1)},\theta^{(2)},\ldots,\theta^{(S)}$$

are intended to have long-run density $f_{\Theta\mid\mathbf Y}(\theta\mid\mathbf y)$.

Ergodic averages tolerate dependence

Under conditions that make the chain ergodic,

$$\widehat I_S=\frac{1}{S}\sum_{s=1}^{S}g(\theta^{(s)})\longrightarrow I.$$

Dependence does not invalidate the average. The transition must preserve the target, and the chain must reach its relevant regions.

Initialization is a diagnostic question

The first 1,000 states are omitted because initialization was deliberately extreme. No observable iteration makes a finite chain exactly stationary.

Multiple chains from dispersed plausible values should move toward and repeatedly explore the same region.

Burn-in and warmup are not synonyms

Burn-in omits an initialization period under a fixed transition rule.

Warmup in an adaptive sampler also tunes quantities such as step size and mass matrix; its changing transition states are not posterior draws.

Posterior spread and chain behavior answer different questions

A wide posterior may reflect limited linguistic information. A slowly moving chain reflects computational inefficiency.

More iterations can reduce Monte Carlo error without adding linguistic evidence or narrowing the posterior.

Metropolis-Hastings corrects proposal moves

The Metropolis-Hastings algorithm corrects proposal moves with an acceptance probability.

The normalizing constant cancels

For a symmetric proposal and standard normal target, a move from $.2$ to $1$ has

$$r=\frac{\exp(-1^2/2)}{\exp(-.2^2/2)}=.619.$$

The shared normalizing constant cancels.

Proposal scale controls movement and rejection

A proposal that is too narrow moves slowly. A proposal that is too wide is rejected often.

Notes: Metropolis-Hastings

Random-walk proposals struggle in correlated regions

A local proposal may bounce slowly through a narrow, correlated high-density region.

The gradient supplies local slope. An auxiliary momentum variable supplies direction, allowing a proposal to travel before the accept-or-reject step.

HMC follows a numerical trajectory

Hamiltonian Monte Carlo (HMC) augments the parameter with momentum and follows a numerically approximated Hamiltonian trajectory.

Potential and kinetic energy divide two roles

Potential energy is the negative log target kernel. Kinetic energy uses auxiliary momentum $\rho$ and a positive-definite mass matrix $M$:

$$U(q)=-\log\kappa(q;y), \qquad K(\rho)=\tfrac12\rho^\top M^{-1}\rho.$$

The Hamiltonian is total energy

The Hamiltonian is the sum of potential and kinetic energy.

The gradient updates momentum using the local log-density slope.

Leapfrog updates position and momentum

For step size $\epsilon$,

$$\rho_{t+1/2}=\rho_t-\frac{\epsilon}{2}\nabla U(q_t), \quad q_{t+1}=q_t+\epsilon M^{-1}\rho_{t+1/2},$$

$$\rho_{t+1}=\rho_{t+1/2}-\frac{\epsilon}{2}\nabla U(q_{t+1}).$$

One leapfrog step

For $U(q)=q^2/2$, $(q_0,\rho_0)=(1,.5)$, and $\epsilon=.1$,

$$\rho_{1/2}=.45,\qquad q_1=1.045,\qquad \rho_1=.39775.$$

A Metropolis correction accounts for numerical error.

Gradients support distant, informed proposals

HMC uses local gradient information to make long proposals with high acceptance probability.

Notes: Hamiltonian Monte Carlo · Stan Reference Manual: HMC

The mass matrix maps momentum to velocity

The mass matrix determines the kinetic energy and maps momentum into parameter velocity.

Kinetic energy under a mass matrix

$$K(\rho)=\frac{1}{2}\rho^\top M^{-1}\rho, \qquad \text{velocity}=M^{-1}\rho.$$

A suitable $M$ adjusts movement to posterior scales and correlations.

Warmup adapts computation, not the posterior target

Stan estimates the mass matrix from warmup draws as an inverse covariance approximation. The learned geometry changes computation, not the model’s posterior target.

Notes: mass matrices and posterior geometry

NUTS chooses an HMC trajectory length

The No-U-Turn Sampler (NUTS) builds an HMC trajectory and stops extending it when the path begins to reverse.

A turning condition stops trajectory growth

Grow a balanced trajectory tree, check the turning condition, and select one valid state from the path.

Warmup adapts other HMC controls

NUTS chooses trajectory length. Warmup separately adapts step size and the mass matrix.

NUTS must select from a valid constructed path

Stopping at the first rejected leapfrog point would bias the trajectory. NUTS selects from a valid constructed path.

Notes: the No-U-Turn Sampler · Stan Reference Manual: HMC and NUTS

Stan expresses a joint probability model

Stan is a probabilistic programming language that uses gradient-based MCMC for continuous parameter models.

The model block must match the factorization

The model block should match the declared joint factorization.

Parameter constraints imply transformations and Jacobian adjustments handled by Stan.

Successful sampling does not establish linguistic adequacy

A program can compile and sample from the model that was coded while the model remains inappropriate for the linguistic data.

Notes: implementing a model in Stan · Stan Reference Manual

Trace plots show draws in iteration order

A trace plot displays saved parameter draws against iteration for each chain.

Stable, overlapping movement is a minimum expectation

Look for stable location, overlap across chains, repeated crossing, and continued movement.

Drift and separated chains warn of incomplete exploration

Drift indicates nonstationarity. Separated chains indicate incomplete exploration or distinct regions.

Visual overlap does not prove convergence

Overlapping traces do not prove convergence. They provide one diagnostic view that must agree with numerical summaries.

Notes: checking Markov chains

Independent chains should occupy the same region

If chains explore the same target, variation within each chain should resemble variation between chains.

The $\widehat{R}$ statistic, specifically the rank-normalized split $\widehat{R}$, formalizes this comparison after splitting chains and rank-normalizing their draws.

The basic variance comparison

$$W=\frac1M\sum_{m=1}^M s_m^2, \qquad B=N\operatorname{Var}(\bar\beta_1,\ldots,\bar\beta_M),$$

$$\widehat V=\frac{N-1}{N}W+\frac{B}{N}, \qquad \widehat R=\sqrt{\frac{\widehat V}{W}}.$$

Values near one indicate cross-chain agreement

$\widehat R$ near one indicates that the chains have similar location and scale for the monitored parameter.

Agreement inside one region can still mislead

Chains can agree inside the same incomplete region. $\widehat R$ is not a certificate that the posterior was fully explored.

Inspect every scientifically relevant parameter

Inspect the largest $\widehat R$ across all reported and scientifically relevant parameters, not one convenient coefficient.

Notes: cross-chain agreement · Stan: convergence diagnostics

Autocorrelation compares draws separated by a lag

Autocorrelation is correlation between draws separated by a fixed lag within one stationary chain.

The autocorrelation function scans across lags

Let $\operatorname{Corr}(U,V)$ denote correlation between random variables $U$ and $V$ under the stationary distribution. Then

$$\rho_t=\operatorname{Corr}(\Theta_s,\Theta_{s+t}),$$

For $X_s=.85X_{s-1}+\varepsilon_s$, $\rho_t=.85^t$.

Correct targets can still be sampled inefficiently

Autocorrelation can reduce Monte Carlo efficiency even when the chain has the correct stationary distribution.

Thinning usually discards useful information

Thinning discards draws and usually does not improve estimation for a fixed computational run.

Notes: autocorrelation

Effective sample size translates dependence into precision

Effective sample size is the number of independent draws that would provide comparable Monte Carlo precision.

Serial dependence changes effective sample size

For a scalar posterior quantity $H_s\equiv h(\Theta_s)$, define its lag-$t$ autocorrelation as $\rho_t\equiv\operatorname{Corr}(H_s,H_{s+t})$. Then, for one stationary chain,

$$N_{\mathrm{eff}}(h)=\frac{N}{1+2\sum_{t=1}^{\infty}\rho_t}.$$

Positive autocorrelation usually lowers ESS; negative autocorrelation can raise it above $N$.

A geometric ESS calculation

If $N=3000$ and $\rho_t=.5^t$, then

$$1+2\sum_{t=1}^{\infty}.5^t=3, \qquad N_{\mathrm{eff}}=\frac{3000}{3}=1000.$$

Bulk and tail ESS target different summaries

Bulk ESS and tail ESS are summary-specific refinements. Bulk ESS concerns central summaries; tail ESS concerns quantiles and tail probabilities.

ESS counts simulation information, not linguistic units

Effective sample size counts information in simulation draws. It does not count speakers, items, or experimental observations.

Notes: effective sample size · Stan diagnostics guidance

A divergence marks excessive integration error

A divergent transition occurs when numerical integration error becomes too large along an HMC trajectory.

Curved posterior geometry can create a funnel

A positive scale parameter and latent deviations can create a narrow, highly curved posterior region.

This shape is called a funnel.

Locate divergent draws before changing the model

Locate divergent draws in parameter space. Then reconsider parameterization, scale, prior information, and model geometry.

Discarding divergent draws does not repair exploration

Discarding divergent draws changes the empirical distribution and does not repair the sampler’s failure to explore the target.

Notes: divergent transitions · Stan: divergent transitions

Pairs plots show two-parameter projections

A pairs plot displays pairwise projections of posterior draws.

Funnels, ridges, and separated regions reveal geometry

Inspect funnels, narrow ridges, strong correlations, multimodal separation, and the locations of divergent draws.

Geometric association is not causal direction

A geometric association between parameters does not show that one parameter causes the other.

Divergence locations connect warnings to geometry

Pairs plots connect a sampler warning with the posterior region where computation is difficult.

Notes: parameter pairs plots

Generated quantities preserve draw-level dependence

The generated quantities block computes derived values or simulations after each saved parameter draw.

Deterministic contrasts and stochastic replications

Compute a deterministic contrast within each draw to preserve joint uncertainty.

Simulate a replicated response when the quantity is stochastic.

Post-sampling calculations do not change the target

The block does not change the posterior target because its calculations occur after parameter sampling.

Replication must preserve the fitted indexing structure

Preserve participant, item, and observation indices when simulated quantities must retain the fitted dependence structure.

Notes: generated quantities · Stan Reference Manual: program blocks

Posterior predictive checks compare same-design replications

A posterior predictive check compares observed data with replicated data generated under posterior draws.

Generate one replicated dataset per posterior draw

For each posterior draw, generate the replicated dataset $\mathbf Y^{\mathrm{rep}}$ and retain its realization $\mathbf y^{\mathrm{rep},(s)}$ using the observed design and indexing.

Recall the posterior predictive distribution.

A discrepancy function targets a model implication

A discrepancy function reduces observed and replicated data to a feature the model should reproduce.

A variance check

Observed low-frequency decision-time standard deviation: $210$ ms

Replicated 95% interval: $[135,184]$ ms

Only $3/1000$ replications reach $210$ ms.

Linguistically relevant discrepancies

Compare observed and replicated variance, the proportion at a response-scale endpoint, group means, or the frequency of very long word types.

A check can miss features it never examines

Checking only a statistic that the fitted model must reproduce closely can miss failures elsewhere in the distribution.

Mismatch diagnoses a feature, not a unique cause

A mismatch identifies a feature the fitted model fails to reproduce. Agreement does not establish that the model is true.

Notes: posterior predictive checks · Stan User’s Guide: predictive checks

What can one realized sample tell us?

What can observed speakers, tokens, items, or measurements tell us beyond the realized sample?

Which layer failed?

Every result must identify its population target, sampling model, procedure, computational check, and fitted-model check.

Targets and procedures · Posterior distributions · Posterior predictive checks