Regulatory tokenizer

nlpgraph ships a zero-dependency word tokenizer + sentence splitter (nlpgraph/tokenizer) tuned for legal and compliance text. It keeps regulatory constructs whole instead of shattering them on punctuation — which is exactly where general-purpose tokenizers fail on reg-tech documents.

import { tokenize } from 'nlpgraph/tokenizer';

tokenize('See Article 33(1) of Regulation (EU) 2016/679.');
// [{ text: 'See', kind: 'word' },
//  { text: 'Article 33(1)', kind: 'reference' },
//  { text: 'of', kind: 'word' },
//  { text: 'Regulation (EU) 2016/679', kind: 'citation' },
//  { text: '.', kind: 'punct' }]

Token kinds

Beyond word and punct, the tokenizer recognizes atomic identifiers common in regulations:

kind	examples
citation	Regulation (EU) 2016/679, 15 U.S.C. 78j, Pub. L. 116-283
reference	Article 33(1), Section 4.2.1, § 500.03, Appendix B
control-id	AC-2, NIST SP 800-53, T1059, A.12.4
currency	€50,000, USD 1.2 million
date	25 May 2018, 2016-04-27
enum	(a), (iii), 1)
url, email, number	as expected

How it works

At each position the scanner tries an ordered list of anchored recognizers (most specific first), else a word (with clitic splitting and abbreviation-dot handling), else a single punctuation character. It also strips leading list-item markers so broken paragraphs and bulleted regulatory text tokenize cleanly.

The tokenizer is evaluated on a 551-case corpus mined from 52 real regulations using a span-preservation metric (a span is “preserved” if it survives whole inside a single token). It is the first milestone of the project and has no runtime dependencies — pure string operations.