Loading a Lexicon

One use case for regular expressions in the context of computational linguistics is querying a lexicon. To look at this use case, we will use the lexicon of word forms found in the CMU Pronouncing Dictionary.

from pathlib import Path
from urllib.request import urlretrieve

cmudict_path = Path("cmudict-0.7b")

if not cmudict_path.exists():
    urlretrieve(
        "https://raw.githubusercontent.com/Alexir/CMUdict/master/cmudict-0.7b",
        cmudict_path,
    )

with cmudict_path.open(encoding="ISO-8859-1") as f:
    for i, line in enumerate(f):
        if i >= 200:
            break
        print(line, end="")

with cmudict_path.open(encoding="ISO-8859-1") as f:
    cmudict_rows: list[list[str]] = [
        line.split() for line in f if not line.startswith(";;;")
    ]
    cmudict: dict[str, list[str]] = {
        row[0].lower(): row[1:] for row in cmudict_rows
    }
    
cmudict["abstraction"]

This dictionary uses what’s known as the ARPABET and represents stress using numeric indicators: 0 for no stress, 1 for primary stress, and 2 for secondary stress. The ARPABET maps to more recognizable IPA representations in the following way.

arpabet_to_phoneme: dict[str, str] = {'AA': 'ɑ',
                      'AE': 'æ', 
                      'AH': 'ə', 
                      'AO': 'ɔ', 
                      'AW': 'aʊ', 
                      'AY': 'aɪ', 
                      'B': 'b', 
                      'CH': 'tʃ', 
                      'D': 'd', 
                      'DH': 'ð', 
                      'EH': 'ɛ',
                      'ER': 'ɝ', 
                      'EY': 'eɪ', 
                      'F': 'f', 
                      'G': 'g', 
                      'HH': 'h', 
                      'IH': 'ɪ', 
                      'IY': 'i', 
                      'JH': 'dʒ', 
                      'K': 'k', 
                      'L': 'l', 
                      'M': 'm', 
                      'N': 'n',
                      'NG': 'ŋ', 
                      'OW': 'oʊ', 
                      'OY': 'ɔɪ', 
                      'P': 'p', 
                      'R': 'ɹ', 
                      'S': 's', 
                      'SH': 'ʃ', 
                      'T': 't', 
                      'TH': 'θ', 
                      'UH': 'ʊ', 
                      'UW': 'u', 
                      'V': 'v',
                      'W': 'w', 
                      'Y': 'j', 
                      'Z': 'z', 
                      'ZH': 'ʒ'}

We’ll use this mapping to construct the IPA representation from the ARPABET representation.

import re

def arpabet_phone_to_ipa(phone: str) -> str:
    """Convert one stress-marked ARPABET phone to IPA."""
    if phone == "AH0":
        return "ə"
    return arpabet_to_phoneme[re.sub(r"\d", "", phone)]


entries: dict[str, list[str]] = {
    w: [arpabet_phone_to_ipa(phoneme) for phoneme in transcription]
        for w, transcription in cmudict.items()
        if len(set(w)) > 1
        if len(transcription) > 1
        if w[0].isalpha()
}

For instance, the IPA representation for the word abstraction can be accessed in the following way.

entries["abstraction"]

with Path("cmudict-ipa").open("w", encoding="utf-8") as f:
    for w, ipa in entries.items():
        ipa = " ".join(ipa)
        f.write(f"{w},{ipa}\n")