ShinrAIShinrAI Benchmarks
Every number on this page comes from a published evaluation report with strict character-span matching — including the unflattering ones.
Results on Real Medical Letters
The Golden ePA corpus: real German doctor letters — discharge summaries, findings, lab reports. Small by design, real by design, frozen so generations stay comparable.
Precision / recall / F1, strict span matching. The static rule engine is our own tuned regex + dictionary stack — the technology every classic PII tool is built on. 4 documents, 21 gold entities: a qualitative head-to-head on the hardest real corpus, not a statistical study. Larger human-annotated gold sets per language are in progress.
Results by Version and Language
Pick a model generation and a language. Bars show span F1 on surface-disjoint evaluation suites — entities the model never saw in training.
Improvement per Model Generation
Frozen v1.1 scored on the v1.2 six-language suites, against v1.2 itself — identical data, identical scoring.
The previous generation lands between 33.6 and 49.3 macro F1 on the six-language suites; v1.2 lands between 89.2 and 94.4. Single hardest category jump across generations: Japanese STREET, F1 6.6 → 88.6 (v1.0 → v1.1).
Attribute Accuracy: Origin, Gender, Name Part
Replacement quality depends on these attributes. Accuracy on the frozen v1-suite (v1.1) and the six v1.2 language suites:
Model Size vs. Accuracy
The compact builds are what most deployments run. Here is exactly what they cost in accuracy.
| Evaluation suite | fp32 · 1.23 GB | INT8 | Δ INT8 | INT4 |
|---|
Macro F1 over the four entity heads. Worst INT8 cost anywhere: 0.33 points. ONNX export parity vs PyTorch: maximum logit delta 3.5 × 10⁻⁵ against a 10⁻³ gate.
Measured Latency
Measured sustained-load numbers on a laptop-class CPU (ONNX fp32, single engine instance) — not lab-best-case.
| Payload | p50 single stream | p95 single stream | Requests/s (c=1) |
|---|
| By hardware — p50 per request | Laptop CPU — measured (fp32) | Mac mini M4 — estimated | GPU service (P40) — measured | Modern GPU (L40S/A100) — estimated | Production end-to-end — measured |
|---|---|---|---|---|---|
| Short (~200 chars) | 26.2 ms | ~28 ms | 12.2 ms | ~6 ms | 17.0 ms |
| Paragraph (~400 tokens) | 101.1 ms | ~105 ms | 28.3 ms | ~13 ms | 37.6 ms |
| Long (1–2k tokens) | 588.4 ms | ~620 ms | 106.7 ms | ~50 ms | 125.3 ms |
Bold = measured, ~ = estimated. GPU cells: round-trips against the two-replica P40 service (2016 hardware); paragraph inference alone is 9.4 ms — 4.3× CPU. The Modern GPU column scales the same cells to L40S/A100 class. Production adds screening, gazetteers and the hop (≈11 ms). Zero 5xx in every cell.
Deterministic paths: static screening <1 ms per stage · dictionary lookups <0.1 ms · replacement draw ~0.4 ms · language detection <5 ms. Production paragraph p95: 41.6 ms — well under the 60 ms bar. Oversized texts degrade to the static path, never to an error.
Comparison with Existing Tools
Third-party figures are vendor-reported or self-reported on their own corpora — solid tools, different tests, so read the numbers as orientation. The capability columns are the direct comparison: this is what the multi-dimensional approach adds.
| Tool | Approach | Reported recall* | Rarity tiers | Cultural origin | Gender-safe replacement | Lossless restore |
|---|---|---|---|---|---|---|
| ShinrAI (hybrid static + neural) | Regex + gazetteers + purpose-trained encoder | measured: F1 89.2–94.4 (v1.2 suites) | ✓ | ✓ 50 classes | ✓ invariant | ✓ client-held map |
| Open RoBERTa-class PII models | RoBERTa/DeBERTa fine-tunes (public) | ~85–95% self-reported, own synthetic suites* | — | — | — | — |
| Microsoft Presidio | Regex + ML hybrid | ~70–80% common entities* | — | — | — | — |
| AWS Comprehend PII | BERT-based | ~85% EN names/locations* | — | — | — | — |
| Google Cloud DLP | Regex + dictionaries | ~60–75% names* | — | — | — | — |
*Third-party estimates, different corpora — not directly comparable to our suite numbers. A public head-to-head on open NER benchmarks (MultiNERD, WikiANN, UNER, CoNLL-2003, AI4Privacy) is specified and will be published with the open-weight release.
Methodology and Limitations
- Scoring: seqeval-strict character spans. A hit counts only when boundaries and type both match exactly.
- Suites are surface-disjoint: evaluation entities never occur in training data. Suites are LLM-generated; the Golden ePA corpus is real documents.
- Known limitation: tier classification (common/uncommon/rare etc.) sits at 49.6–68.5% on the v1.2 languages against an internal 85% gate — v1.3 targets it explicitly.
- Known regression: v1.2 regressed Japanese (macro 60.8 on the v1.1 suite). Production routes Japanese to v1.1 automatically; v1.3 retrains Japanese at the full recipe.
- Latency setup: M-series laptop CPU, sustained 15–60 s cells, any server error fails the run. Server-class Xeon extrapolation: roughly 2–3× slower per thread.
- Release gates rise per generation: v1.2 shipped against 85–90-point category gates with precision ≥ 90; v1.3 and v1.4 are pinned at F1 ≥ 95 per category and track. No core metric may drop more than 1 point versus the previous release.