Working with CELEX morphological data

Reading

Baayen et al. (1995) on the design and contents of CELEX.

CELEX contains lexical data for English, Dutch, and German, including morphological structure labels (Baayen et al. 1995). What do we need from those labels? For each English lemma, we want two objects: a sequence of morphemes for segmentation and a tree of labeled constituents for grammar estimation.

Data availability

CELEX (LDC96L14) requires a Linguistic Data Consortium license. Place an extracted copy in morphological-patterns/celex/ to run the code on this page. The page is not executed as part of the public site build.

Loading the entries

The file eml.cd contains backslash-separated records. In that file, field 1 contains the headword and field 21 contains its StrucLab analysis. The loader checks the field count rather than assigning an eight-column schema to a 25-field record.

import sys
from dataclasses import asdict

import pandas as pd

sys.path.insert(0, "_code")
from celex import load_celex_entries, parse_morph_tree

entries = load_celex_entries("celex/english/eml/eml.cd")
celex = pd.DataFrame(asdict(entry) for entry in entries)

print(f"Entries with morphological structures: {len(celex)}")
celex.head()

Parsing a structure label

A StrucLab value places a category in square brackets after each bracketed constituent. Consider the following normalized analysis of unhappiness:

(((un)[Prefix] (happy)[Adjective])[Adjective] (ness)[Suffix])[Noun]

The inner constituent combines un and happy as an adjective; the outer constituent combines that adjective with ness as a noun. The category labels correspond to CFG variables.

analysis = (
    "(((un)[Prefix] (happy)[Adjective])[Adjective] "
    "(ness)[Suffix])[Noun]"
)
tree = parse_morph_tree(analysis)

tree.morphemes

The same tree supplies a set of rules.

for rule in sorted(tree.rules, key=str):
    print(rule)

The lexical rules introduce un, happy, and ness. The nonlexical rules record how their categories compose. Counting these rules across the database will give us the supervised estimates used by the probabilistic grammar.

Question

Why retain the full tree if tree.morphemes already gives us the segmentation?

Answer

The segmentation records only the terminal sequence. It does not say whether un first combines with happy or whether ness does. The internal nodes determine which CFG rules occurred and thus which attachment structures the grammar can learn.

CELEX uses underlying morpheme spellings in some analyses, so concatenating the leaves need not reproduce the surface headword exactly. We will handle that mismatch with string alignment when constructing character-level boundary labels.

References

Baayen, R. Harald, Richard Piepenbrock, and Leon Gulikers. 1995. The CELEX Lexical Database (CD-ROM). Linguistic Data Consortium, University of Pennsylvania.

--- title: Working with CELEX morphological data bibliography: ../references.bib jupyter: python3 execute: eval: false --- ::: {.callout-tip} ## Reading @baayen_celex_1995 on the design and contents of CELEX. ::: CELEX contains lexical data for English, Dutch, and German, including morphological structure labels [@baayen_celex_1995]. What do we need from those labels? For each English lemma, we want two objects: a sequence of morphemes for segmentation and a tree of labeled constituents for grammar estimation. ::: {.callout-warning} ## Data availability CELEX (LDC96L14) requires a Linguistic Data Consortium license. Place an extracted copy in `morphological-patterns/celex/` to run the code on this page. The page is not executed as part of the public site build. ::: ## Loading the entries The file `eml.cd` contains backslash-separated records. In that file, field 1 contains the headword and field 21 contains its `StrucLab` analysis. The loader checks the field count rather than assigning an eight-column schema to a 25-field record. ```{python} import sys from dataclasses import asdict import pandas as pd sys.path.insert(0, "_code") from celex import load_celex_entries, parse_morph_tree entries = load_celex_entries("celex/english/eml/eml.cd") celex = pd.DataFrame(asdict(entry) for entry in entries) print(f"Entries with morphological structures: {len(celex)}") celex.head() ``` ## Parsing a structure label A `StrucLab` value places a category in square brackets after each bracketed constituent. Consider the following normalized analysis of *unhappiness*: ``` (((un)[Prefix] (happy)[Adjective])[Adjective] (ness)[Suffix])[Noun] ``` The inner constituent combines *un* and *happy* as an adjective; the outer constituent combines that adjective with *ness* as a noun. The category labels correspond to CFG variables. ```{python} analysis = ( "(((un)[Prefix] (happy)[Adjective])[Adjective] " "(ness)[Suffix])[Noun]" ) tree = parse_morph_tree(analysis) tree.morphemes ``` The same tree supplies a set of rules. ```{python} for rule in sorted(tree.rules, key=str): print(rule) ``` The lexical rules introduce *un*, *happy*, and *ness*. The nonlexical rules record how their categories compose. Counting these rules across the database will give us the supervised estimates used by the probabilistic grammar. ::: {.callout-caution} ## Question Why retain the full tree if `tree.morphemes` already gives us the segmentation? ::: ::: {.callout-tip collapse=true} ## Answer The segmentation records only the terminal sequence. It does not say whether *un* first combines with *happy* or whether *ness* does. The internal nodes determine which CFG rules occurred and thus which attachment structures the grammar can learn. ::: CELEX uses underlying morpheme spellings in some analyses, so concatenating the leaves need not reproduce the surface headword exactly. We will handle that mismatch with string alignment when constructing character-level boundary labels.