Accuracy & evaluation
We report our own numbers, measured by our committed eval harness — not the model author’s.
Parser (UAS / LAS)
Section titled “Parser (UAS / LAS)”| Test set | UAS | LAS |
|---|---|---|
| UD English-EWT (500 sentences) | 94.9 | 93.1 |
| Hand-annotated compliance gold | 99.2 | 97.7 |
- UAS (unlabeled attachment score) = % of tokens with the correct head.
- LAS (labeled attachment score) = % with the correct head and relation label.
The parser is fed gold word segmentation and scored against gold dependencies. Throughput is
~38 sentences/s single-thread on CPU (fp16), fully deterministic. All numbers come from a committed
eval harness (eval/) so anyone can reproduce them.
The compliance gold set is hand-annotated in the UD scheme over NIST/MAS/CIS-style sentences (ISO text is deliberately excluded — it is copyrighted). The point of a separate compliance set is that reg-tech prose (modal obligations, passives with agents, nested references) differs from newswire.
Tokenizer
Section titled “Tokenizer”Evaluated on a 551-case corpus mined from 52 real regulations using a span-preservation metric: a span counts as preserved if it survives whole inside a single token (not fractured across boundaries). We track overall preservation plus a focused atomic-identifier subtotal (citation/reference/control-id/date/currency/enum/number/url/email).
Reproduce
Section titled “Reproduce”npm run eval # parser: compliance gold + UD English-EWT (needs the model)npm run eval:tokenizer # tokenizer span-preservation on the 551-case corpus (no model)