ShinrAIShinrAI Benchmarks
Every number on this page comes from a published evaluation report with strict character-span matching — including the unflattering ones.
Results on Real Medical Letters
The Golden ePA corpus: real German doctor letters — discharge summaries, findings, lab reports. Small by design, real by design, frozen so generations stay comparable.
Precision / recall / F1, strict span matching. The static rule engine is our own tuned regex + dictionary stack — the technology every classic PII tool is built on. 4 documents, 21 gold entities: a qualitative head-to-head on the hardest real corpus, not a statistical study. Larger human-annotated gold sets per language are in progress.
Results by Version and Language
Pick a model generation and a language. Bars show span F1 on surface-disjoint evaluation suites — entities the model never saw in training.
Improvement per Model Generation
Frozen v1.1 scored on the v1.2 six-language suites, against v1.2 itself — identical data, identical scoring.
The previous generation lands between 33.6 and 49.3 macro F1 on the six-language suites; v1.2 lands between 89.2 and 94.4. Single hardest category jump across generations: Japanese STREET, F1 6.6 → 88.6 (v1.0 → v1.1).
Attribute Accuracy: Origin, Gender, Name Part
Replacement quality depends on these attributes. Accuracy on the frozen v1-suite (v1.1) and the six v1.2 language suites:
Model Size vs. Accuracy
The compact builds are what most deployments run. Here is exactly what they cost in accuracy.
| Evaluation suite | fp32 · 1.23 GB | INT8 | Δ INT8 | INT4 |
|---|
Macro F1 over the four entity heads. Worst INT8 cost anywhere: 0.33 points. ONNX export parity vs PyTorch: maximum logit delta 3.5 × 10⁻⁵ against a 10⁻³ gate.
Measured Latency
Measured sustained-load numbers on a laptop-class CPU (ONNX INT8, single engine instance) — not lab-best-case.
| Payload | p50 single stream | p95 single stream | Requests/s (c=1) |
|---|
Deterministic paths are faster still: static screening <1 ms per stage · dictionary lookups <0.1 ms · replacement draw ~0.4 ms · language detection <5 ms. GPU serving budget: ≤25 ms p95 per paragraph. Oversized texts degrade to the static path — never to an error.
Comparison with Existing Tools
Third-party figures are vendor-reported or self-reported on their own corpora — solid tools, different tests, so read the numbers as orientation. The capability columns are the direct comparison: this is what the multi-dimensional approach adds.
| Tool | Approach | Reported recall* | Rarity tiers | Cultural origin | Gender-safe replacement | Lossless restore |
|---|---|---|---|---|---|---|
| ShinrAI (hybrid static + neural) | Regex + gazetteers + purpose-trained encoder | measured: F1 89.2–94.4 (v1.2 suites) | ✓ | ✓ 50 classes | ✓ invariant | ✓ client-held map |
| Open RoBERTa-class PII models | RoBERTa/DeBERTa fine-tunes (public) | ~85–95% self-reported, own synthetic suites* | — | — | — | — |
| Microsoft Presidio | Regex + ML hybrid | ~70–80% common entities* | — | — | — | — |
| AWS Comprehend PII | BERT-based | ~85% EN names/locations* | — | — | — | — |
| Google Cloud DLP | Regex + dictionaries | ~60–75% names* | — | — | — | — |
*Third-party estimates, different corpora — not directly comparable to our suite numbers. A public head-to-head on open NER benchmarks (MultiNERD, WikiANN, UNER, CoNLL-2003, AI4Privacy) is specified and will be published with the open-weight release.
Methodology and Limitations
- Scoring: seqeval-strict character spans. A hit counts only when boundaries and type both match exactly.
- Suites are surface-disjoint: evaluation entities never occur in training data. Suites are LLM-generated; the Golden ePA corpus is real documents.
- Known limitation: tier classification (common/uncommon/rare etc.) sits at 49.6–68.5% on the v1.2 languages against an internal 85% gate — v1.3 targets it explicitly.
- Known regression: v1.2 regressed Japanese (macro 60.8 on the v1.1 suite). Production routes Japanese to v1.1 automatically; v1.3 retrains Japanese at the full recipe.
- Latency setup: M-series laptop CPU, sustained 15–60 s cells, any server error fails the run. Server-class Xeon extrapolation: roughly 2–3× slower per thread.
- Release gates rise per generation: v1.2 shipped against 85–90-point category gates with precision ≥ 90; v1.3 and v1.4 are pinned at F1 ≥ 95 per category and track. No core metric may drop more than 1 point versus the previous release.