One of the major uses for regular expressions is for extracting substrings from a string. This can be done with groups. For instance, suppose I want all of stems that have the morpheme with the form /ʃən/.
with open("cmudict-ipa", encoding="utf-8") as f:
entry_rows: list[list[str]] = [
line.strip().split(",", maxsplit=1) for line in f
]
entries: dict[str, list[str]] = {
word: ipa.split() for word, ipa in entry_rows
}import re
regex_ʃən = '(.+)ʃən'
n_matches = 0
for w, ipa in entries.items():
if re.fullmatch(regex_ʃən, "".join(ipa)):
if n_matches < 30:
n_matches += 1
print("".join(ipa), re.findall(regex_ʃən, "".join(ipa)), f"({w})")
else:
print("...")
breakThis works to some extent, but notice that it will capture cases where /ʃən/ is not a morpheme. For instance, the word passion will get matched. It will also return the wrong stem when the morpheme is realized as /eɪʃən/, such as accreditation.
re.findall(regex_ʃən, "".join(entries["passion"])), re.findall(regex_ʃən, "".join(entries["accreditation"]))To handle the second, we might look for /eɪʃən/ and /ʃən/. We can use the quantifier ? to say that /eɪ/ is optional. Because it is a digraph, we need to surround it with parentheses.
regex_ʃən = '(.+)(eɪ)?ʃən'
n_matches = 0
for w, ipa in entries.items():
if re.fullmatch(regex_ʃən, "".join(ipa)):
if n_matches < 30:
n_matches += 1
print("".join(ipa), re.findall(regex_ʃən, "".join(ipa)), f"({w})")
else:
print("...")
breakThe problem is that this makes Python think we want to capture it. So what we need is a non-capturing group, which we get by putting ?: after the open parenthesis.
regex_ʃən = '(.+)(?:eɪ)?ʃən'
n_matches = 0
for w, ipa in entries.items():
if re.fullmatch(regex_ʃən, "".join(ipa)):
if n_matches < 30:
n_matches += 1
print("".join(ipa), re.findall(regex_ʃən, "".join(ipa)), f"({w})")
else:
print("...")
breakIt still seems to be capturing /eɪ/ in accreditation. What gives? The reason this is happening is that quantifiers like + are greedy by default. That means they will match as much as they can. And because /eɪ/ is optional, (.+) can match it.
To make sure it doesn’t match it if it doesn’t need to, we can make the quantifier non-greedy by appending a ?.
regex_ʃən = '(.+?)(?:eɪ)?ʃən'
n_matches = 0
for w, ipa in entries.items():
if re.fullmatch(regex_ʃən, "".join(ipa)):
if n_matches < 30:
n_matches += 1
print("".join(ipa), re.findall(regex_ʃən, "".join(ipa)), f"({w})")
else:
print("...")
breakOkay. So how do we deal with cases where /ʃən/ is not a morpheme? One thing we can do is to look for stems that show up without /ʃən/. This will exclude passion, since /pæ/ is not a word.
regex_ʃən = '(.+?)(?:eɪ)?ʃən'
n_matches = 0
seen = set()
for w1, ipa1 in entries.items():
possible_morpheme = re.findall(regex_ʃən, "".join(ipa1))
if possible_morpheme:
for w2, ipa2 in entries.items():
if re.fullmatch(possible_morpheme[0], "".join(ipa2)):
if n_matches < 20 and "".join(ipa2) not in seen:
n_matches += 1
seen |= {"".join(ipa2)}
print("".join(ipa2), f"({w2})", "+", "ʃən", "=", "".join(ipa1), f"({w1})")
else:
break
if n_matches >= 20:
print("...")
breakAn issue here is that /ʃən/ doesn’t simply get appended to a stem. There is an additional phonological process that deletes a portion of that stem–e.g. /æbstɹækt/ + /ʃən/ is /æbstɹækʃən/, not /æbstɹæktʃən/. So we need to consider cases where a final consonant–usually a t–was deleted. But we need to make sure we do so only when the morpheme wasn’t realized as /eɪʃən/, so we need to go back to capturing it.
regex_ʃən = '(.+?)(eɪ)?ʃən'
n_matches = 0
seen = set()
for w1, ipa1 in entries.items():
possible_morpheme = re.findall(regex_ʃən, "".join(ipa1))
if possible_morpheme:
for w2, ipa2 in entries.items():
if possible_morpheme[0][1]:
regex_stem = possible_morpheme[0][0]
else:
regex_stem = possible_morpheme[0][0] + "t"
if re.fullmatch(regex_stem, "".join(ipa2)):
if n_matches < 20 and "".join(ipa2) not in seen:
n_matches += 1
seen |= {"".join(ipa2)}
print("".join(ipa2), f"({w2})", "+", "ʃən", "=", "".join(ipa1), f"({w1})")
else:
break
if n_matches >= 20:
print("...")
breakThere’s still some wonky stuff in here–e.g. ancient coming from ain’t and ashen coming from at–but we’re getting closer. We can’t really deal with cases like ashen coming from at, but we can deal with ancient coming from ain’t. The issue is that re.findall searches for matching substrings, whereas re.fullmatch requires the entire string to match. If we want the suffix pattern to extend to the end of the string, we have to say so explicitly with $, which means “end of string”.1
regex_ʃən = '(.+?)(eɪ)?ʃən$'
n_matches = 0
seen = set()
for w1, ipa1 in entries.items():
possible_morpheme = re.findall(regex_ʃən, "".join(ipa1))
if possible_morpheme:
for w2, ipa2 in entries.items():
if possible_morpheme[0][1]:
regex_stem = possible_morpheme[0][0]
else:
regex_stem = possible_morpheme[0][0] + "t"
if re.fullmatch(regex_stem, "".join(ipa2)):
if n_matches < 20 and "".join(ipa2) not in seen:
n_matches += 1
seen |= {"".join(ipa2)}
print("".join(ipa2), f"({w2})", "+", "ʃən", "=", "".join(ipa1), f"({w1})")
else:
break
if n_matches >= 20:
print("...")
breakTo get much better than this, we’d need to start matching on the orthographic representation as well–e.g. matching the ion at the end of the orthographic representation of the word, thus filtering things like /æt/ + /ʃən/ = /æʃən/. One thing we’ll still miss are cases like adoration, where there is a vowel quality change (which is consequently why we get adder + ion = adoration currently). To handle those cases, we would need to account for the conditions under which vowel quality changes, which we could do in principle using regular expressions but which I won’t do here.
The dual of $ is ^, which asserts the beginning of a string. Inside a character class, a ^ immediately after [ instead marks negation.↩︎