diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 99568344..2f9464c1 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -438,6 +438,14 @@ Excluded (MAIDEN_MARKERS, per nameparser/config/maiden_markers.py): - 2026-08 #319 — the wholly-suffix predicate was lifted into the vocabulary layer so the comma decision and the honorific peel's segment test cannot drift apart. - 2026-08-23 #296/#325 — the listing form's post-comma part is read for what it is. Three commits of the approved #291/#296 bundle (spec 2026-07-30, plan 2026-08-01; see the comma-suffix arc below), in the plan's order. (1) A comma followed only by titles fixed no family boundary, so the pre-comma name keeps its positional read: `John Smith, Mr.` reads given 'John', family 'Smith', title 'Mr.' where 2.0 had merged the family ('Bob Jones, author' moves with it at every baseline). "Nothing but titles" is title-shaped AND not suffix vocabulary — the plan's helper said only the first, and would have read `Smith, PhD Jr.` as the title run 'PhD Jr.' and `Smith, Mr. Jr.` as 'Mr. Jr.' (both master's suffix 'Jr.'; the gate and a measurement against master caught them). (2) The TITLES/suffix overlap audit and the ordering, one commit because each alone is wrong (the plan's F1): reading the credential run first would make `Smith, Dr.` a suffix while 'dr' was SUFFIX_WORDS residue, and the audit alone leaves `Smith, Jr.` a title by the period-abbreviation inference. The audit table's dispositions ship as approved — 'do', 'jr', 'junior', 'phd', 'se' out of TITLES; 'dr' and 'sra' out of the suffix sets; 'ms' and 'sa' into the ambiguous set — with ONE deviation on evidence the table did not have: 'md' keeps dual membership, because bare 'Md' before a name is the Bengali and South Asian Muslim abbreviation of Muhammad (#343/#345's corpus rows `Md Abdul Karim`, `Md. Abdul Karim`; the drop read the first as given 'Md', middle 'Abdul') while 'MD' after the name is the degree — position decides, as for 'sr'. The ordering is wider than the plan's lone piece: a post-comma part that is nothing but suffix pieces is the credential run, whole, which is what #325 needed (`Smith, Ph. D. Jr.` had put the split credential in the given name once a second suffix made it no longer lone — a 1.4.0 regression; v1 read suffix 'Ph. D.', title 'Jr.'). Vocabulary decides which words qualify and position breaks the tie for the genuine duals: `Smith, Sr.` is Senior where `Sr. Garcia` is Señor, `Smith, Esq.` is the postnominal where H2 still reads `Esq. Smith` as a title (H2 carries that as Accepted). `Smith, Dr. Jr.` keeps the 2.0 deviation's other case — a title, then the last piece as the suffix (v1 read first 'Jr.'; 2.0 through master suffix 'Jr.') — and `Smith, John Jr.` its given name. Consequences the audit carries, all measured and pinned: a leading 'PhD' or 'Jr' is a name word; `Do Quang Minh` reads given 'Do' with the particle-or-given fork, and `Dr. Do Van Johnson, MD` given 'Do', family 'Van Johnson' (the #367 rule for that name is replaced; TITLES ∩ ambiguous is {freiherr, st} now, which keeps the chained emitter reachable); a trailing bare 'Dr' is a name word, joining every other title-only word there (#316 is the open question); `田中さん, PhD` reads suffix 'さん, PhD'; the #100 prefix-join tests use 'lt', a genuine dual, as their token. #291 (multi-word credentials) is the bundle's fourth commit and its own PR. - 2026-08-23 (review of PR #428, four reviewers) — the repair's condition is "no name word", not "all titles": `John Smith, Mr. Jr.` (a title and a postnominal) had kept the merge the rationale argues against, and reads given 'John', family 'Smith', suffix 'Jr.' now. Its guard counts NAME pieces: the positional read peels a trailing suffix first, so `Smith Jr., Mr.` — two pieces, one name — read positionally lost its family (the code review); it keeps family 'Smith', suffix 'Jr.' at every baseline. The positional read records its ORDER, and post_rules' leading-piece scan keys on "assign records no order after a family comma" rather than on the structure, so P1's family-first fold reaches the pre-comma name as it does without the comma — `de Mesnil Juan, Dr.` under a family-first order had read family 'de' (the test review). The credential run does NOT step over a delimiter core the policy names (`Smith, RN - CRNA` reads given 'RN' under `extra_suffix_delimiters` as without it): the code review asked for it, a draft did it, and tests/test_suffixes.py's "the delimiter must not affect parsing at all" outside the suffix-comma form is v1 parity pinned on purpose — C1 carries the limit as Accepted. A second deviation from the audit table: `ms` and `sa` do NOT join the ambiguous set — the gate is position-blind and the collision is not; gated, `John Smith, MS` lost its suffix-comma route and read title 'MS', and `Smith, Ms.` passed the gate on its one period and read as a credential anyway. Both are genuine duals, and position decides: `Smith, Ms.` reads suffix 'Ms.' (title at every baseline; C1 carries it as Accepted, with `Smith, Ms. Jane` keeping the title) and `John Smith, MS` keeps suffix 'MS'. And the gate itself now means what S2 says: an ambiguous acronym counts when written with ITS periods, one after each letter (`M.A.`, `J.D.`), not on any period — `Smith, Ed.` had become a credential on a name's trailing period, and `Jack Ma.` had read suffix 'Ma.' since 2.0 (S2 carries the boundary; 1.4.0 read family 'Ma.'). Two limits recorded rather than fixed: `田中さん, Dr.` keeps the honorific glued, joining master's `田中さん, Mr.` — the honorific peel runs in script_segment on the suffix-comma and no-comma structures, before group or assign can say the comma fixed nothing (C1 Accepted); and group's chain emitter still reports no PARTICLE_OR_GIVEN on the family-comma path, so `Freiherr von Berg Ma, Mr.` reads as its no-comma twin but without the report — group runs before assign decides which read applies, and plumbing a deferred report through the state is more than the shape is worth today. +- 2026-08-23 #429 — one comma segment is one suffix entry, and the two stages disagreed about which segments those are. group asked by segment INDEX (`tail_start`: 1 after a suffix comma, 2 after a family comma), which is right wherever assign reads the segment as suffixes for the same structural reason — but not after a family comma, where segment 1 is a name slot assign re-reads by CONTENT with `_segment_holds_no_name` and routes piece by piece (#296/#325). So the roles were right and the render was not: `Smith, MD PhD` gave suffix 'MD, PhD' with a comma the writer never typed, and `Smith, CBE MC`, `Smith, BSc MBA` and `Smith, PhD Jr.` the same, while the full-name `John Smith, MD PhD` has rendered 'MD PhD' since 1.4.0. The predicate moved down into group beside the piece predicates it already reads — the same direction `_leading_titles` moved in #424, and the only direction available, since assign imports group. + SCOPE, stated as the condition actually reads rather than as the first draft described it: any family comma whose following segment holds no name word, NOT only a one-word family. `John Smith, Jr. III` renders suffix 'Jr. III' (1.4.0's reading), and a title-led segment is in too — `Smith, Dr. MD PhD` renders 'MD PhD'. The differential cannot witness the widening: of 1069 corpus names, 303 carry a structure comma, the predicate is true for 36, only 9 of those have two or more pieces in the segment, and none of the 9 combines a multi-word pre-comma name with two suffix pieces. 0 unexplained is therefore evidence about the corpus, not about the class — the case table carries it instead. + THE REGRESSION THE REVIEW FOUND, and the reason this entry is long: the first draft let ANY piece open an entry, as the tail block always had. That is safe on a tail segment, where assign routes every piece to SUFFIX — which is what `tail` means — and wrong off it, where a title piece routes to TITLE. Two failures, one cause. The `joined` tag is role-BLIND and the facade heals it for every role, so a title opening an entry collapsed `title_list` from ['Rev.', 'Dr.'] to ['Rev. Dr.'] on `Smith, Rev. Dr.`; and where the pre-comma name left a suffix, the tag glued the next suffix BACKWARD across the writer's own comma — `Smith Jr., Mr. Jr.` rendered suffix 'Jr. Jr.', the exact inverse of the bug being fixed. Neither is visible to the differential, which compares strings, nor to the case table as it stood, which asserts the title STRING (space-joined either way). The rule that replaces it separates two joins that had been one: WITHIN a piece the tag renders a merged piece as one unit, whatever role it holds; BETWEEN pieces it continues an entry, and only a piece that renders into the same run may do that. The entry is sticky across a piece that is not in it, so an interleaved title does not split the run it sits in (`Smith, MD Dr. PhD` renders 'MD PhD'); a delimiter core still closes it, which is the one thing that separates two entries. + Scoped deliberately, and the scoping is the other decision: `tail` still gates delimiter-core dropping, which is v1 `expand_suffix_delimiter` parity (#206, commit 021823e — three code comments say #191, which is the German/Dutch vocabulary PR, and are corrected here) and a TAIL rule. The core set is threaded to the block as `seg_cores` so the widened condition cannot drop cores where `tail` says not to. Worth knowing before anyone tries to pin that: for every realistic delimiter the two spellings are indistinguishable by construction, because the core must first pass the very predicate that gates the block — the only reachable class is a policy whose delimiter core is itself suffix or title vocabulary (`Policy(extra_suffix_delimiters={" jr "})`, `Smith, MD jr`), which the case table does not enshrine. + Cost, measured rather than optimised away: `_segment_holds_no_name` now runs TWICE per family-comma parse, once here and once in assign, on the same pieces — verified pure recomputation, not two questions (408 of 506 comma-bearing test literals reach both sites, 0 disagreements, and off-tail `seg_cores` is empty so `kept` never filters, which is why assign sees exactly the pieces group tested). Removing the second call is worth 0.6–1.4 µs, 1.2–2.2% of a family-comma parse and 0% of every other parse (NO_COMMA and SUFFIX_COMMA make no call at all). Caching it needs a derived ParseState field the two sites would otherwise not share, so the number is recorded instead. The order is cheapest-first per 7009fcf: two bool loads and an int compare gate the call. + R1 gained the separator sentence with this change. That the invariant it states — one comma segment is one suffix entry — was written down in no rule is a large part of why two stages disagreed about it for years: C1 governs routing, R1 governed word ORDER and never mentioned separation, and the only statement of the rule was a code comment. + Left alone, measured and pre-existing: the NO-COMMA path comma-joins a space-separated run too (`John Smith MD PhD` renders 'MD, PhD' at 1.4.0, 2.0.0, 2.1.0 and master alike). The fix requires FAMILY_COMMA and does not reach it; that path has no writer-drawn boundary and is a separate question, recorded here and NOT yet filed. It is load-bearing for one thing: `str()` of a fixed parse is a no-comma string, so re-parsing it re-inserts the comma — round-tripping is not stable on these shapes, and an earlier draft of the release bullet claimed it was. + The name `Smith, MD PhD` entered the corpora as this fix's C1 example — a FIRING one, carrying no `· boundary` annotation, which marks the non-firing example a rule must have — so its ledger entries arrive with it. A fourth written for the 1.4.0 ledger EXPLAINED NOTHING and was dropped: `fix(comma-family) lone post-comma piece routes to suffix/title, not first` already claims the name there, its fields covering all three that move. That is #373's dormant-rule diagnosis doing its job, and #426 is the precedent for dropping rather than keeping a shadowed rule. ### T1 — separators, not joiners diff --git a/docs/design/rules.md b/docs/design/rules.md index 6b05d9bb..a1ab7bb9 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -623,7 +623,8 @@ C1. Rationale: a credential run after the comma means the name is in "Smith, PhD" → suffix="PhD" "Smith, Jr." → suffix="Jr." "Smith, Sr." → suffix="Sr." - "Smith, Ph. D. Jr." → suffix="Ph. D., Jr." + "Smith, Ph. D. Jr." → suffix="Ph. D. Jr." + "Smith, MD PhD" → suffix="MD PhD" "Smith, Dr." → title="Dr." "Smith, Dr. Jr." → suffix="Jr." "John Smith, Mr." → given="John" @@ -648,7 +649,7 @@ C1. Rationale: a credential run after the comma means the name is in form alone, and that limitation is kept as parity: "Smith, RN - CRNA" reads given "RN" under the policy as without it. "John Smith, LEED AP" → family="Smith" deviates: #291 (today: family="John Smith") - history: decisions.md#C1 · interacts: H2, P6 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py + history: decisions.md#C1 · interacts: H2, P6 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py C2. Rationale: text beyond the recognized comma parts should be taken in without silent guessing. @@ -874,10 +875,17 @@ R1. Rationale: a field is a way of reading the parse, not a stored words — O3's fold and, since #379, P6's attached tussenvoegsel — which render before the rest of the family wherever they stood in the string. + Words are separated as the writer separated them. The suffix view + is the one place this is visible, because it is the only field + that can hold parts the writer comma-separated: a run of + post-nominals written with spaces renders with spaces, and one + written with commas keeps them. "Dr. Juan Q. Xavier de la Vega III" → family="de la Vega" "Hassan, Mohamad Ahmad Ali" middle_as_family → family="Ahmad Ali Hassan" "Hassan, Mohamad Ahmad Ali" → family="Hassan" · boundary - implemented: nameparser/_types.py + "Smith, MD PhD" → suffix="MD PhD" + "John Smith, MD, Bart" → suffix="MD, Bart" + history: decisions.md#C1 · implemented: nameparser/_types.py R2. Rationale: callers need the surname with and without its particles — sorting wants "Vega", display wants "de la Vega". diff --git a/docs/release_log.rst b/docs/release_log.rst index e515800a..ddc6f3b6 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -47,7 +47,8 @@ Release Log - Fix pure postnominals being claimed as titles: ``jr``, ``junior``, ``phd``, ``do`` and ``se`` have left the default ``titles`` vocabulary, and ``dr``/``sra`` have left the suffix vocabulary they never belonged in. v1's word lists recorded where the v1 parser needed a word to be in order to do its job, not where the word can actually occur -- ``"Smith, PhD"`` returned title ``PhD`` because the title peel legitimately got there first. Each of the nineteen words in the ``titles``/suffix overlap was reviewed and given its true positional class; twelve are genuine duals and keep both memberships, with position deciding (``"Lt. Smith"`` is a title, ``"Smith, LT"`` a postnominal; ``"Ms. Smith"`` is the honorific, ``"John Smith, MS"`` the degree -- and so, after a family comma, is ``"Smith, Ms."``, the slot being postnominal). ``md`` is one of them: bare ``Md`` before a name is the Bengali and South Asian abbreviation of Muhammad (``"Md Abdul Karim"``), ``MD`` after it the degree. The cost is in leading position, where a dropped word now reads as a name: ``"PhD Smith"`` gives given ``PhD`` where 1.x gave title ``PhD``. That is the correct reading of a string nobody writes, and it is what makes ``"Do Nguyen"`` parse as the Vietnamese name it is -- and ``"Dr. Do Van Johnson"`` read given ``Do``. ``dr`` and ``sra`` are the only two words that lose *suffix* membership, so they also stop being recognized in trailing position: ``"John Smith Dr."`` now gives family ``Dr.`` rather than suffix ``Dr.``. That is the same reading every other title-only word already gets there -- ``"John Smith Prof."`` and ``"John Smith Mr."`` have always parsed this way, and the suffix entry was the only thing making ``dr`` an exception. Routing a trailing title word to ``title``, which the comma path already does and the no-comma path does not, is a separate open question (#316). An ambiguous credential acronym (``ma``, ``ed``, ``jd``, ``do``) now counts as a suffix only when written with its periods, one after each letter -- ``"John Smith M.A."`` -- and no longer on any period: ``"Jack Ma."`` keeps family ``Ma.``, as 1.4.0 read it, where 2.0 read a suffix - - Fix a credential run after a one-word family comma reading as a title or a given name: ``"Smith, Jr."`` and ``"Smith, PhD"`` now give suffix ``Jr.``/``PhD`` where they gave title, and ``"Smith, Ph. D. Jr."`` gives suffix ``Ph. D., Jr.`` where the split credential landed in the given name (a regression from 1.4.0, which read suffix ``Ph. D.``). The position right after a family comma is postnominal position, so a run that is nothing but suffix words is read as the credential run it is before the title peel's whole-segment exception can claim it. Vocabulary still decides which words qualify -- ``"Smith, Dr."`` keeps title ``Dr.``, because ``dr`` is not postnominal vocabulary at all -- and the leading readings are untouched (``"Sr. Garcia"`` is still title ``Sr.`` while ``"Smith, Sr."`` is now suffix ``Sr.``); a name word in the run makes it the given-and-suffix reading it always had (``"Smith, John Jr."``) (closes #296, closes #325) + - Fix a credential run after a one-word family comma reading as a title or a given name: ``"Smith, Jr."`` and ``"Smith, PhD"`` now give suffix ``Jr.``/``PhD`` where they gave title, and ``"Smith, Ph. D. Jr."`` gives suffix ``Ph. D. Jr.`` where the split credential landed in the given name (a regression from 1.4.0, which read suffix ``Ph. D.``). The position right after a family comma is postnominal position, so a run that is nothing but suffix words is read as the credential run it is before the title peel's whole-segment exception can claim it. Vocabulary still decides which words qualify -- ``"Smith, Dr."`` keeps title ``Dr.``, because ``dr`` is not postnominal vocabulary at all -- and the leading readings are untouched (``"Sr. Garcia"`` is still title ``Sr.`` while ``"Smith, Sr."`` is now suffix ``Sr.``); a name word in the run makes it the given-and-suffix reading it always had (``"Smith, John Jr."``) (closes #296, closes #325) + - Fix a space-separated credential run after a family comma rendering with a comma the name never had: ``"Smith, MD PhD"`` gives suffix ``MD PhD`` where it gave ``MD, PhD``, and ``"Smith, CBE MC"``, ``"Smith, BSc MBA"``, ``"Smith, PhD Jr."`` and ``"Smith, Dr. MD PhD"`` the same. The roles are unchanged; only the rendered string carried the extra comma. One comma segment is one suffix entry, and grouping decided which segments those were by their position while assignment decided the same segment by its content, so the two disagreed about the segment a family comma leaves after it. The reading follows assignment now, which is where the segment's roles were already decided. This reaches any family comma whose following segment holds no name word, not only a one-word family: ``"John Smith, Jr. III"`` gives suffix ``Jr. III``, which is also what 1.4.0 gave. The full-name comma form was never affected -- ``"John Smith, MD PhD"`` has given suffix ``MD PhD`` since 1.4.0 -- and this brings the rest into line with it. A run written with commas keeps them (``"Smith, MD, PhD"`` is still suffix ``MD, PhD``), a comma between two entries survives (``"Smith Jr., Mr. Jr."`` is still suffix ``Jr., Jr.``), and a name word in the segment still makes it the given-and-suffix reading (``"Smith, John Jr."``). A name written without a comma is unaffected and still renders its run comma-joined, so re-parsing ``str()`` output does not reproduce the run (closes #429) - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 2459633a..3ab1d89c 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -42,8 +42,8 @@ effective_script, is_suffix_lenient, resolve_script_set, ) from nameparser._pipeline._group import ( - _is_leading_title, _is_suffix_piece, _leading_titles, _peel_trailing, - _peel_walk, + _is_suffix_piece, _leading_titles, _peel_trailing, _peel_walk, + _segment_holds_no_name, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -256,32 +256,6 @@ def _assign_main(seg_idx: int, state: ParseState, return order -def _segment_holds_no_name(pieces: tuple[tuple[int, ...], ...], - ptags: tuple[frozenset[str], ...], - tokens: list[WorkToken]) -> bool: - """The segment is titles and suffixes only ('John Smith, Dr.', - 'John Smith, Mr. Jr.') -- nothing in it is a name word. - - The FAMILY_COMMA rule "segment 0 is wholly the family name" rests on - the writer having said where the family name ends. A comma followed - by no name word said no such thing -- 'John Smith, Dr.' is 'Dr. John - Smith' with the honorific moved, and 'John Smith, Mr. Jr.' the same - with the postnominal along -- so the pre-comma name keeps its - positional read instead of being merged. Uses the same - _is_leading_title predicate the peel does, period-abbreviation - inference included, so the two cannot disagree about what a title - is; a suffix piece counts as what it is, so a mixed run like - 'Smith, Dr. Jr.' is a title and a postnominal, each read where it - stands, and never a title run 'Dr. Jr.'. An empty segment - ('Doe,, Jr.') holds no title to read by. - """ - if not pieces: - return False - return all(_is_suffix_piece(pieces[k], ptags[k], tokens) - or _is_leading_title(pieces[k], ptags[k], tokens) - for k in range(len(pieces))) - - def assign(state: ParseState) -> ParseState: tokens = list(state.tokens) ambiguities = list(state.ambiguities) diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 1a91c481..738fd0a6 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -23,7 +23,8 @@ piece-level walk that assign applies and that P5's reserve, P2's chain and M2's walk read (#425, #424), and assign's leading-title test (_is_leading_title, _leading_titles), which the chain's scan -shares since #424. +shares since #424, and its no-name-segment test +(_segment_holds_no_name), which the one-entry join reads since #429. """ from __future__ import annotations @@ -144,6 +145,44 @@ def _is_suffix_piece(piece: Sequence[int], ptags: Set[str], return "vocab:suffix" in tags and "initial" not in tags +def _segment_holds_no_name(pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], + tokens: Sequence[WorkToken]) -> bool: + """The segment is titles and suffixes only ('John Smith, Dr.', + 'John Smith, Mr. Jr.') -- nothing in it is a name word. + + The FAMILY_COMMA rule "segment 0 is wholly the family name" rests on + the writer having said where the family name ends. A comma followed + by no name word said no such thing -- 'John Smith, Dr.' is 'Dr. John + Smith' with the honorific moved, and 'John Smith, Mr. Jr.' the same + with the postnominal along -- so the pre-comma name keeps its + positional read instead of being merged. Uses the same + _is_leading_title predicate the peel does, period-abbreviation + inference included, so the two cannot disagree about what a title + is; a suffix piece counts as what it is, so a mixed run like + 'Smith, Dr. Jr.' is a title and a postnominal, each read where it + stands, and never a title run 'Dr. Jr.'. An empty segment + ('Doe,, Jr.') holds no title to read by. + + TWO callers, asking it for different reasons, and the difference + matters. assign uses it to decide whether the comma fixed the family + name (above). group's one-entry join (#429) uses it to decide + whether the segment is a credential run at all. + + True does NOT mean "every piece is a suffix" -- the title tolerance + is the whole point, and a true segment can still hold pieces assign + routes to TITLE, so a caller rendering the segment as one unit must + ask _is_suffix_piece per piece as well. What assuming otherwise cost + is recorded at the one-entry join in group(), the caller that made + the assumption. + """ + if not pieces: + return False + return all(_is_suffix_piece(pieces[k], ptags[k], tokens) + or _is_leading_title(pieces[k], ptags[k], tokens) + for k in range(len(pieces))) + + class Peel(NamedTuple): """What assign's trailing peel made of a walk. `names` is a count of positions in the caller's `rest`: rest[:names] are the name @@ -818,7 +857,7 @@ def group(state: ParseState) -> ParseState: # v1 parity: additional_parts_count=1 applies only to FAMILY_COMMA # parts; the SUFFIX_COMMA pre-comma segment gets 0. additional = 1 if state.structure is Structure.FAMILY_COMMA else 0 - # v1 expand_suffix_delimiter parity (#191): tail segments (wholly + # v1 expand_suffix_delimiter parity (#206): tail segments (wholly # consumed as suffixes by assign) drop delimiter-core tokens, the # same structural mechanism as the maiden marker (taken out in # _group_segment, recorded in `dropped` just below) @@ -836,10 +875,11 @@ def group(state: ParseState) -> ParseState: # suppresses it there: the family name is already fixed, so # there is no fork left to report. tail = tail_start is not None and seg_idx >= tail_start + seg_cores = cores if tail else frozenset() pieces, ptags, taken = _group_segment( seg, additional, tokens, bound_join, None if family_comma else ambiguities, - cores if tail else frozenset(), + seg_cores, state.lexicon.given_name_titles) # the marker is dropped and the maiden name's tokens become # MAIDEN (#274); which pieces those are was settled in @@ -851,7 +891,31 @@ def group(state: ParseState) -> ParseState: for i in piece: tokens[i] = dataclasses.replace( tokens[i], role=Role.MAIDEN) - if tail: + # rules.md#C1: "a part that is nothing but suffix words is the + # credential run and reads as suffixes, whole" -- WHOLE is this + # block's half of the rule, the routing being assign's. + # One comma segment is one suffix entry. `tail` answers that by + # INDEX, which is right wherever assign reads the segment as + # suffixes for the same structural reason -- but not after a + # ONE-WORD family comma, where segment 1 is a name slot that + # assign re-reads by CONTENT: a segment of nothing but + # credentials is the credential run, whole (#296/#325). group + # asking the index while assign asked the content is what made + # 'Smith, MD PhD' render 'MD, PhD' with a comma the writer + # never typed, where the full-name 'John Smith, MD PhD' has + # rendered 'MD PhD' since 1.4.0 (#429). Ask assign's own + # predicate, over the pieces group just built. + # `family_comma` is redundant by invariant and kept for + # locality: segment() emits at most one segment for NO_COMMA, so + # seg_idx == 1 already implies a comma, and under SUFFIX_COMMA + # tail_start is 1, so `tail` short-circuits before this. Nothing + # can pin it -- dropping it is an equivalent mutant over the + # corpora and 65,725 generated inputs -- so it is documented + # rather than tested. + one_entry = tail or ( + family_comma and seg_idx == 1 + and _segment_holds_no_name(pieces, ptags, tokens)) + if one_entry: # v1 renders each tail COMMA SEGMENT as one suffix entry # ('Smith, V MD' -> suffix 'V MD'); a delimiter core inside # a segment separates entries and is dropped, but a segment @@ -859,23 +923,50 @@ def group(state: ParseState) -> ParseState: # within a part, never erases a lone part). Continuation # tokens within an entry take the stable "joined" tag so # the suffix view space-joins them (the fix_phd mechanism). + # Core dropping stays keyed on `tail` via seg_cores: the + # #206 parity is a TAIL rule, and the one-entry join is the + # only half that follows assign's content read. entry_open = False kept: list[int] = [] for k in range(len(pieces)): is_core = (len(pieces[k]) == 1 - and tokens[pieces[k][0]].text in cores + and tokens[pieces[k][0]].text in seg_cores and len(pieces) > 1) if is_core: dropped.extend(pieces[k]) entry_open = False continue kept.append(k) + # Two different joins, and conflating them is what a + # widened condition gets wrong. WITHIN a piece (pos > 0) + # the tag renders a merged piece as one unit; the branch + # is written role-blind because the merge is (the ph-d + # pair reaches GIVEN as one element), though no + # multi-token TITLE piece witnesses it -- none turned up + # in 38,892 generated family-comma inputs. BETWEEN pieces it continues an ENTRY, + # and only pieces that render into the same run may do + # that. On a tail segment every kept piece does -- that + # is what `tail` means -- but off it assign routes piece + # by piece (_assign.py), so a title piece is not part of + # the suffix entry. Letting one continue the entry tags + # a token the SUFFIX view never joins and the TITLE view + # does: 'Smith, Rev. Dr.' collapsed title_list to + # ['Rev. Dr.'], and after a pre-comma suffix the tag + # glued across the writer's own comma ('Smith Jr., Mr. + # Jr.' rendered suffix 'Jr. Jr.') -- the inverse of the + # bug this block exists to fix. + in_entry = tail or _is_suffix_piece( + pieces[k], ptags[k], tokens) for pos, i in enumerate(pieces[k]): - if entry_open or pos > 0: + if pos > 0 or (in_entry and entry_open): tokens[i] = dataclasses.replace( tokens[i], tags=tokens[i].tags | {"joined"}) - # piece-level state: the NEXT piece continues this entry - entry_open = True + # Sticky across a piece that is not in the entry, so an + # interleaved title does not split the run it sits in: + # 'Smith, MD Dr. PhD' renders suffix 'MD PhD', not + # 'MD, PhD'. A delimiter core still closes the entry + # (above) -- that is the one thing that separates two. + entry_open = entry_open or in_entry if len(kept) != len(pieces): pieces = [pieces[k] for k in kept] ptags = [ptags[k] for k in kept] diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index aadcfe7c..98c1d7d2 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -265,7 +265,7 @@ def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, return False predicate = (is_suffix_lenient if policy.lenient_comma_suffixes else is_suffix_strict) - # v1 expand_suffix_delimiter parity (#191): a configured delimiter + # v1 expand_suffix_delimiter parity (#206): a configured delimiter # is TRANSPARENT in the all-suffix tests -- v1 split the part string # on the delimiter before checking, so the delimiter never counted cores = delimiter_cores(policy.extra_suffix_delimiters) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 210204e5..447da311 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -388,6 +388,17 @@ def __post_init__(self) -> None: notes="v1 made the lone post-comma strict-suffix piece the " "given; 2.0 routes it to suffix (same family as the " "'Smith, Dr.' row)"), + Case("family_comma_suffix_run_renders_unjoined", "Smith, MD PhD", + {"family": "Smith", "suffix": "MD PhD"}, + classification="fix(comma-family)", + notes="a space-separated post-nominal run after a family " + "comma RENDERED with a comma the input never had " + "('MD, PhD'), because the one-entry join was asked by " + "segment index and this segment is not a tail one. The " + "full-name 'John Smith, MD PhD' has rendered 'MD PhD' " + "since 1.4.0, and #429 brought this form into line with " + "it. The roles were already right after #428; this was " + "its remaining half"), Case("period_joined_ambiguous_chunk", "John Doe, Msc.Ed.", {"given": "John", "family": "Doe", "suffix": "Msc.Ed."}, notes="chunk-level suffix membership is v1's is_suffix: bare " @@ -571,7 +582,7 @@ def __post_init__(self) -> None: Case("suffix_delimiter_tail_segment", "Doe, John, RN - CRNA", {"given": "John", "family": "Doe", "suffix": "RN, CRNA"}, policy=_SD, - notes="v1 suffix_delimiter parity (#191): the delimiter token " + notes="v1 suffix_delimiter parity (#206): the delimiter token " "is dropped from consumed tail segments (pinned live " "2026-07-16)"), Case("suffix_delimiter_detection", "Doe, John RN - CRNA", @@ -1583,24 +1594,79 @@ def __post_init__(self) -> None: "suffix, so the inference does not run"), # -- #325: the whole credential run, not the lone piece Case("family_comma_split_credential_run", "Smith, Ph. D. Jr.", - {"family": "Smith", "suffix": "Ph. D., Jr."}, - classification="fix(#325)", + {"family": "Smith", "suffix": "Ph. D. Jr."}, + classification="fix(#325) + fix(#429)", notes="one word before the comma, the space-split 'Ph. D.' " "and a suffix after it: the lone-piece route did not " "apply and the merged credential fell through to the " "given name (a 1.4.0 regression -- v1 read suffix 'Ph. " "D.', title 'Jr.'). A run that is nothing but suffix " - "pieces is the credential run C1 describes, whole"), + "pieces is the credential run C1 describes, whole -- " + "and #429 made it render whole too, where #325 shipped " + "it as 'Ph. D., Jr.' with a comma the writer never " + "typed. The full-name 'John Smith, Ph. D. Jr.' rendered " + "it unjoined at 2.0.0 and after -- 1.4.0 rendered " + "'Ph. D., Jr.' there too -- so this row now agrees with " + "the form it has agreed with since 2.0"), Case("family_comma_credential_run_then_numeral", "Smith, Ph. D. III", - {"family": "Smith", "suffix": "Ph. D., III"}, - classification="fix(#325)"), + {"family": "Smith", "suffix": "Ph. D. III"}, + classification="fix(#325) + fix(#429)", + notes="the numeral does not end the run; the render lost its " + "inserted comma with the rest (#429)"), Case("family_comma_two_credentials", "Smith, PhD Jr.", - {"family": "Smith", "suffix": "PhD, Jr."}, - classification="fix(#325)", + {"family": "Smith", "suffix": "PhD Jr."}, + classification="fix(#325) + fix(#429)", notes="'PhD' led the run as a title until the audit; the audit " "alone would have made it the given name, which is why " "the ordering shipped in the same commit. The run is " - "suffixes"), + "suffixes, and since #429 renders as one entry rather " + "than 'PhD, Jr.'"), + Case("family_comma_title_led_credential_run", "Smith, Dr. MD PhD", + {"title": "Dr.", "family": "Smith", "suffix": "MD PhD"}, + classification="fix(#429)", + notes="the title-led run: assign routes Dr. to TITLE and the " + "two credentials to SUFFIX, so the ENTRY is the " + "credential run, not the whole segment. 384 inputs of " + "this shape moved with #429 and none was pinned until " + "the review said so"), + Case("family_comma_title_between_credentials", "Smith, MD Dr. PhD", + {"title": "Dr.", "family": "Smith", "suffix": "MD PhD"}, + classification="fix(#429)", + notes="the entry is sticky across a piece that is not in it: " + "an interleaved title must not split the run it sits " + "in, or the render inserts the very comma #429 removes"), + Case("family_comma_title_led_run_keeps_the_written_comma", + "Smith Jr., Mr. Jr.", + {"title": "Mr.", "family": "Smith", "suffix": "Jr., Jr."}, + notes="THE #429 REGRESSION GUARD, and unchanged since 1.4.0. " + "The pre-comma name leaves a suffix, and the segment " + "after the comma is title-led. #429's first draft let " + "the title piece OPEN the entry, so the following Jr. " + "was tagged as a continuation and the view joined it " + "backward across the writer's own comma -- suffix " + "'Jr. Jr.', the exact inverse of the bug #429 fixes. " + "Only a piece that renders into the same run may " + "continue an entry"), + Case("family_comma_written_commas_are_kept", "Smith, MD, PhD", + {"family": "Smith", "suffix": "MD, PhD"}, + notes="the negative control for #429, and the distinction the " + "whole change rests on: the parser renders the run as " + "the writer spaced it, and never stops emitting a comma " + "the writer typed. Parity at every baseline"), + Case("family_comma_run_matches_the_full_name_form", "John Smith, MD PhD", + {"given": "John", "family": "Smith", "suffix": "MD PhD"}, + notes="the full-name twin of family_comma_suffix_run_renders_" + "unjoined, and the reference #429 brought the one-word " + "form into line with. Unchanged since 1.4.0, so this row " + "fails if a future change fixes one form by breaking the " + "other"), + Case("family_comma_segment_zero_is_not_the_run", "MD PhD Jr., John", + {"given": "John", "family": "MD", "suffix": "PhD, Jr."}, + notes="segment 0 is the family segment even when it is wholly " + "credential-shaped, so the one-entry join is asked of " + "segment 1 alone. Dropping that conjunct left the whole " + "suite green while this shape's suffix silently became " + "'PhD Jr.' (the mutation matrix found it)"), Case("family_comma_run_with_a_name_is_not_a_run", "Smith, John Jr.", {"given": "John", "family": "Smith", "suffix": "Jr."}, notes="the non-flip: a name word in the run makes it the " diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 0b692383..67d9298d 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -269,7 +269,7 @@ def test_initials_do_not_count_as_rootnames_for_conjunction_carveout() -> None: def test_extra_suffix_delimiter_splits_tail_entries() -> None: - # v1 expand_suffix_delimiter parity (#191): a configured delimiter + # v1 expand_suffix_delimiter parity (#206): a configured delimiter # is transparent in a FAMILY_COMMA tail segment -- it separates # suffix ENTRIES (each rendered independently) and is itself # dropped, rather than becoming a suffix token or fusing the @@ -436,7 +436,7 @@ def test_the_connective_carveout_counts_the_surviving_name() -> None: def test_a_delimiter_core_in_a_suffix_tail_is_not_maiden_text() -> None: - """A tail segment drops its delimiter cores (#191) and the marker + """A tail segment drops its delimiter cores (#206) and the marker takes what is left, in that order -- the order group() had before the marker pass moved ahead of the joins. A core is not a word the marker can take: it is structure, like the marker itself.""" diff --git a/tests/v2/test_facade.py b/tests/v2/test_facade.py index fc1dbbf6..784b804b 100644 --- a/tests/v2/test_facade.py +++ b/tests/v2/test_facade.py @@ -176,6 +176,49 @@ def test_suffix_list_heals_joined_continuations() -> None: # v1 fix_phd assert n.suffix_list == ["Ph. D."] # ONE element, v1 parity +def test_the_joined_tag_never_reaches_a_title( # #429 regression guard +) -> None: + """The "joined" tag is role-BLIND and _list_for heals it for every + role, so a tag placed for the suffix view is read by the title view + too. + + #429 widened the one-entry join off the tail path, where assign + routes every piece to SUFFIX, onto the family-comma path, where it + does not. Its first draft let any piece open an entry; a title piece + doing so tagged the title behind it, and title_list collapsed. + + Asserted on the LIST views because the string views cannot see it -- + titles render space-joined either way, which is why the case table + and the differential (both string-only) stayed green while this was + broken. + """ + n = HumanName("Smith, Rev. Dr.") + assert n.title == "Rev. Dr." # unchanged, and why it hid + assert n.title_list == ["Rev.", "Dr."] # TWO elements + + # The other half: with a suffix already standing from the pre-comma + # segment, a title opening the entry glued the next suffix backward + # across the writer's own comma. + m = HumanName("Smith Jr., Mr. Jr.") + assert m.suffix == "Jr., Jr." + assert m.suffix_list == ["Jr.", "Jr."] + + # The control: where the pieces really are one entry, they DO heal. + k = HumanName("Smith, MD PhD") + assert k.suffix_list == ["MD PhD"] # ONE element + + # And the pin for the TAG CONDITION itself, which the three above + # miss: they all pin the sticky entry_open update, and the whole + # suite stays green with `in_entry and` dropped from the tag test. + # This needs a suffix piece FIRST -- opening the entry legitimately + # -- and then TWO titles, so the second title is a continuation of + # an entry it does not belong to. + j = HumanName("Smith, MD Rev. Dr.") + assert j.suffix == "MD" + assert j.title == "Rev. Dr." # identical either way + assert j.title_list == ["Rev.", "Dr."] # the mutant gives ['Rev. Dr.'] + + def test_str_uses_string_format_with_v1_cleanup() -> None: n = HumanName("Dr. Juan de la Vega III") assert str(n) == "Dr. Juan de la Vega III" diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index f92232ff..ecd2bfa2 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1248,7 +1248,7 @@ def _claim(rule: dict) -> _Claim: "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('family', 'middle'), "973617235cda"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(274, ('given', 'suffix', 'title'), "dc19a4822c61"), + _Claim(275, ('given', 'suffix', 'title'), "58f0d640d1d9"), "fix(comma-family) a comma followed only by titles keeps the given/family split": _Claim(2, ('family', 'given'), "5bd9c6d96c38"), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": @@ -1268,9 +1268,9 @@ def _claim(rule: dict) -> _Claim: "fix(#367) an inferred title no longer displaces a leading particle either": _Claim(1, ('family', 'given', 'middle'), "d8ee9cd5da5f"), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(274, ('family', 'given'), "dc19a4822c61"), + _Claim(275, ('family', 'given'), "58f0d640d1d9"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1068, ('family', 'given', 'suffix'), "c0770ee57f2a"), + _Claim(1069, ('family', 'given', 'suffix'), "b1ccb5c7532c"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1343,6 +1343,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "e62caedec864"), }, "expected_since_2.0.0.toml": { + "fix(#429) a wholly-credential segment after a one-word family renders as one entry": + _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('family', 'middle'), "973617235cda"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": @@ -1439,6 +1441,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'maiden'), "2150936a8c55"), }, "expected_since_2.1.0.toml": { + "fix(#429) a wholly-credential segment after a one-word family renders as one entry": + _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('family', 'middle'), "973617235cda"), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index dc26fc19..05009f10 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -135,6 +135,7 @@ "Smith, Esq." "Smith, John" "Smith, Jr." +"Smith, MD PhD" "Smith, Major. John" "Smith, Ms." "Smith, Ms. Jane" diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 49e9b3c3..fed2117a 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -833,3 +833,28 @@ issue = "fix(#418) accepted: a suffix word inside the maiden name ends it, conne # expected_since_2.1.0.toml copy carries the fuller account. name_regex = "(?i)^jane\\s+n[eé]e\\s+jr\\s+y\\s+jones$" fields = ["middle", "family", "maiden"] + +[[change]] +issue = "fix(#429) a wholly-credential segment after a one-word family renders as one entry" +# 'Smith, MD PhD': one word before the comma and nothing but +# credentials after it. The baseline read the leading 'MD' as a title +# and left suffix 'PhD'; the segment is the credential run rules.md#C1 +# describes and now lands whole in `suffix`. +# +# The roles moved in #325; what #429 adds is the RENDER. #325 shipped +# them as 'MD, PhD' -- a comma the writer never typed -- because group +# decided "one comma segment is one suffix entry" by segment INDEX +# while assign decided the same segment by CONTENT. group now asks +# assign's own predicate, so the run renders as written and this path +# agrees with the full-name 'John Smith, MD PhD', which has rendered +# 'MD PhD' since 1.4.0. +# +# The name entered the corpus as the rules.md#C1 example that fix +# added, so this entry arrives with it rather than explaining a +# name that was already there. `family` and `given` are deliberately +# OUT of the field list: neither moves on this rule, so a regression +# in either stays loud. Literal-anchored -- the class has no other +# corpus name, and a harvested one should arrive UNEXPLAINED and be +# read once rather than absorbed here. +name_regex = "(?i)^smith,\\s*md\\s+phd$" +fields = ["title", "suffix"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index bd5d5c65..125d38f9 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -770,3 +770,28 @@ issue = "fix(#418) accepted: a suffix word inside the maiden name ends it, conne # once rather than absorbed here. name_regex = "(?i)^jane\\s+n[eé]e\\s+jr\\s+y\\s+jones$" fields = ["middle", "family", "maiden"] + +[[change]] +issue = "fix(#429) a wholly-credential segment after a one-word family renders as one entry" +# 'Smith, MD PhD': one word before the comma and nothing but +# credentials after it. The baseline read the leading 'MD' as a title +# and left suffix 'PhD'; the segment is the credential run rules.md#C1 +# describes and now lands whole in `suffix`. +# +# The roles moved in #325; what #429 adds is the RENDER. #325 shipped +# them as 'MD, PhD' -- a comma the writer never typed -- because group +# decided "one comma segment is one suffix entry" by segment INDEX +# while assign decided the same segment by CONTENT. group now asks +# assign's own predicate, so the run renders as written and this path +# agrees with the full-name 'John Smith, MD PhD', which has rendered +# 'MD PhD' since 1.4.0. +# +# The name entered the corpus as the rules.md#C1 example that fix +# added, so this entry arrives with it rather than explaining a +# name that was already there. `family` and `given` are deliberately +# OUT of the field list: neither moves on this rule, so a regression +# in either stays loud. Literal-anchored -- the class has no other +# corpus name, and a harvested one should arrive UNEXPLAINED and be +# read once rather than absorbed here. +name_regex = "(?i)^smith,\\s*md\\s+phd$" +fields = ["title", "suffix"]