Turning linguistic records into data

Suppose we have recorded a speaker producing the word heed. We could analyze the entire waveform, the word token, the vowel token, or a sequence of acoustic measurements taken during the vowel. Each analysis extracts a different set of observations from the same recording.

Before counting the observations, we must decide what one observation represents. The entity represented by one observation is the unit of analysis.

The stored record

A record is a stored artifact, such as a recording, corpus document, response log, or annotation file.

One record may contain many possible observations. A ten-minute interview is one audio file, but it may contain hundreds of word tokens and dozens of vowel tokens. Conversely, several records may contribute information about one observation. If we summarize a speaker across three recording sessions, the speaker-level observation draws on three files.

File count thus tells us how the material is stored. It does not tell us how many observations the analysis contains.

Decide what one row means

Consider a reading-time experiment. Each participant reads several sentences, and each sentence contains several regions. The response log might look like this:

participant	sentence	region	reading time
p01	s17	1	318
p01	s17	2	441
p01	s18	1	287

If one observation is one participant’s reading of one region in one sentence, then a row is identified by three pieces of information:

\langle\text{participant},\text{sentence},\text{region}\rangle.

We can check each candidate key against the table. p01 does not identify a row because that participant appears repeatedly. The pair \langle\texttt{p01},\texttt{s17}\rangle also appears more than once because the sentence contains several regions. The full triple identifies exactly one row.

The smallest collection of columns that uniquely identifies an observation is its observation key.

A quick key check

If two rows share the proposed observation key, the key is incomplete or the same unit was measured more than once. Decide which description fits the design before fitting a model.

Watch what aggregation changes

If we first sum the regional reading times within each participant and sentence, the new table has one row per participant and sentence, and the region column is no longer part of the key. If we then average across participants, the table has one row per sentence, and the participant column is no longer part of the key.

The three tables retain different sources of variation:

one row represents	variation still visible
participant, sentence, and region	participants, sentences, and regions
participant and sentence	participants and sentences
sentence mean	sentences

Once we replace 100 participant responses with one sentence mean, the resulting table cannot show how participants differ on that sentence. That information was removed before the statistical model saw the data.

Summing or averaging observations is not automatically a mistake. It is appropriate only when the analysis does not need the variation discarded by that operation.

Tokens and types

Linguists often switch between tokens and types. A token is an occurrence in a particular context. A type groups occurrences that the analysis treats as equivalent.

Consider ten occurrences of runs. They are ten orthographic word tokens. They belong to one word-form type, runs. A lemmatized analysis may group them with occurrences of run, ran, and running under the lemma run.

The grouping follows from the research question. A study of inflection must distinguish run from runs, whereas a study of lemma frequency may deliberately collapse them.

The appropriate unit depends on the question:

question	useful unit
Does discourse context affect the choice of a dative construction?	one construction token
Do verbs differ in their preference for the dative construction?	one verb type, perhaps summarized across tokens
Does one speaker change construction use over an interview?	one speaker by time segment observation

The first analysis preserves contextual differences among tokens. The second compares lexical types after collapsing across their tokens. Neither representation is preferable in general because they answer different questions.

A unique row is not an independent row

A rectangular table does not claim that its rows are independent. In the reading-time table, participant p01 occurs in several rows and sentence s17 occurs in several rows. Those repetitions remain present even if every row has a unique observation key.

There are two separate facts about these rows:

a unique key tells us that two rows are not the same observation;
repeated identifiers tell us that two observations share a participant, item, speaker, word, language, or another source of variation.

To model differences among participants or sentences later, we must retain both identifiers now.

Measurement, analysis, and generalization

To connect a dataset to a claim, answer three questions.

Where was the response measured? A judgment might be recorded on every participant and sentence trial.
What does one response in the analysis represent? The analysis might retain each trial or average judgments by sentence.
What should the claim generalize to? The target might concern the tested sentences, new sentences instantiating the same constructions, the tested participants, or a broader population of speakers.

These answers need not be identical. A trial-level analysis may support generalization to new participants and new items when the design contains replication across both. But one thousand tokens from one speaker still provide observations from only one speaker. They tell us nothing about variation across speakers.

A pre-analysis specification

Before plotting or modeling, specify the unit represented by an observation:

One observation represents ______.

Then ask:

Which columns uniquely identify that observation?
Which speakers, items, words, languages, documents, or sites repeat?
Which variation has already been removed by preprocessing or averaging?
Does the dataset contain replication at the level of the intended claim?

For a corpus of dative tokens, a useful answer might be:

One observation represents one attested dative construction token. The observation key combines the document and token position. Speakers, verbs, and documents repeat across observations. The analysis retains both lexical and discourse-context variation.

Test the distinction

A corpus contains 50,000 word tokens from 20 interviews with five speakers. Give an observation key for a token analysis. Then describe a speaker-level table and state which variation the aggregation removes.
An acceptability experiment presents 40 sentences to each of 100 participants. What does one row represent if the table contains one judgment per row? Which identifiers must be retained?
A typological database contains two inventories for the same language from different sources. Is language a sufficient observation key? What additional identifier is needed?

--- title: "Turning linguistic records into data" --- Suppose we have recorded a speaker producing the word *heed*. We could analyze the entire waveform, the word token, the vowel token, or a sequence of acoustic measurements taken during the vowel. Each analysis extracts a different set of observations from the same recording. Before counting the observations, we must decide *what one observation represents*. The entity represented by one observation is the [**unit of analysis**](https://bookdown.org/pkaldunn/Book/UnitsObsAnalysis.html). ## The stored record A [**record**](https://openstax.org/books/introductory-statistics-2e/pages/1-1-definitions-of-statistics-probability-and-key-terms) is a stored artifact, such as a recording, corpus document, response log, or annotation file. One record may contain many possible observations. A ten-minute interview is one audio file, but it may contain hundreds of word tokens and dozens of vowel tokens. Conversely, several records may contribute information about one observation. If we summarize a speaker across three recording sessions, the speaker-level observation draws on three files. File count thus tells us how the material is stored. It does not tell us how many observations the analysis contains. ## Decide what one row means Consider a reading-time experiment. Each participant reads several sentences, and each sentence contains several regions. The response log might look like this: | participant | sentence | region | reading time | |---|---|---:|---:| | p01 | s17 | 1 | 318 | | p01 | s17 | 2 | 441 | | p01 | s18 | 1 | 287 | If one observation is one participant's reading of one region in one sentence, then a row is identified by three pieces of information: $$ \langle\text{participant},\text{sentence},\text{region}\rangle. $$ We can check each candidate key against the table. `p01` does not identify a row because that participant appears repeatedly. The pair $\langle\texttt{p01},\texttt{s17}\rangle$ also appears more than once because the sentence contains several regions. The full triple identifies exactly one row. The smallest collection of columns that uniquely identifies an observation is its [**observation key**](https://datacarpentry.github.io/spreadsheet-ecology-lesson/01-format-data.html). ::: {.callout-note title="A quick key check"} If two rows share the proposed observation key, the key is incomplete or the same unit was measured more than once. Decide which description fits the design before fitting a model. ::: ## Watch what aggregation changes If we first sum the regional reading times within each participant and sentence, the new table has one row per participant and sentence, and the region column is no longer part of the key. If we then average across participants, the table has one row per sentence, and the participant column is no longer part of the key. The three tables retain different sources of variation: | one row represents | variation still visible | |---|---| | participant, sentence, and region | participants, sentences, and regions | | participant and sentence | participants and sentences | | sentence mean | sentences | Once we replace 100 participant responses with one sentence mean, the resulting table cannot show how participants differ on that sentence. That information was removed before the statistical model saw the data. Summing or averaging observations is not automatically a mistake. It is appropriate only when the analysis does not need the variation discarded by that operation. ## Tokens and types Linguists often switch between [**tokens**](https://web.stanford.edu/~jurafsky/slp3/old_aug25/ed3book.pdf) and [**types**](https://web.stanford.edu/~jurafsky/slp3/old_aug25/ed3book.pdf). A token is an occurrence in a particular context. A type groups occurrences that the analysis treats as equivalent. Consider ten occurrences of *runs*. They are ten orthographic word tokens. They belong to one word-form type, *runs*. A lemmatized analysis may group them with occurrences of *run*, *ran*, and *running* under the lemma *run*. The grouping follows from the research question. A study of inflection must distinguish *run* from *runs*, whereas a study of lemma frequency may deliberately collapse them. The appropriate unit depends on the question: | question | useful unit | |---|---| | Does discourse context affect the choice of a dative construction? | one construction token | | Do verbs differ in their preference for the dative construction? | one verb type, perhaps summarized across tokens | | Does one speaker change construction use over an interview? | one speaker by time segment observation | The first analysis preserves contextual differences among tokens. The second compares lexical types after collapsing across their tokens. Neither representation is preferable in general because they answer different questions. ## A unique row is not an independent row A rectangular table does not claim that its rows are independent. In the reading-time table, participant `p01` occurs in several rows and sentence `s17` occurs in several rows. Those repetitions remain present even if every row has a unique observation key. There are two separate facts about these rows: 1. a unique key tells us that two rows are not the same observation; 2. repeated identifiers tell us that two observations share a participant, item, speaker, word, language, or another source of variation. To model differences among participants or sentences later, we must retain both identifiers now. ## Measurement, analysis, and generalization To connect a dataset to a claim, answer three questions. 1. *Where was the response measured?* A judgment might be recorded on every participant and sentence trial. 2. *What does one response in the analysis represent?* The analysis might retain each trial or average judgments by sentence. 3. *What should the claim generalize to?* The target might concern the tested sentences, new sentences instantiating the same constructions, the tested participants, or a broader population of speakers. These answers need not be identical. A trial-level analysis may support generalization to new participants and new items when the design contains replication across both. But one thousand tokens from one speaker still provide observations from only one speaker. They tell us nothing about variation across speakers. ## A pre-analysis specification Before plotting or modeling, specify the unit represented by an observation: > One observation represents ______. Then ask: 1. Which columns uniquely identify that observation? 2. Which speakers, items, words, languages, documents, or sites repeat? 3. Which variation has already been removed by preprocessing or averaging? 4. Does the dataset contain replication at the level of the intended claim? For a corpus of dative tokens, a useful answer might be: > One observation represents one attested dative construction token. The observation key combines the document and token position. Speakers, verbs, and documents repeat across observations. The analysis retains both lexical and discourse-context variation. ## Test the distinction 1. A corpus contains 50,000 word tokens from 20 interviews with five speakers. Give an observation key for a token analysis. Then describe a speaker-level table and state which variation the aggregation removes. 2. An acceptability experiment presents 40 sentences to each of 100 participants. What does one row represent if the table contains one judgment per row? Which identifiers must be retained? 3. A typological database contains two inventories for the same language from different sources. Is `language` a sufficient observation key? What additional identifier is needed?