ShinrAIShinrAI Benchmarks
Every number on this page comes from a published evaluation report with strict character-span matching — including the unflattering ones.
Current model evidence
ShinrAI 1.3 is available across 15 locales, including Hebrew beta. ShinrAI 1.4 is in development, focusing on longer documents, paperwork and Hebrew. No release date announced.
Median strict-span F1 · ShinrAI 1.3
Innovius evaluation: 200 business and clinical texts per locale, 15 locales, Hebrew beta included. Model-only scores; compare identical datasets and scoring methods.
Historical results: ShinrAI 1.0–1.2
Archived release measurements. The medical-letter comparison covers four documents and 21 entities, and compares ShinrAI 1.0/1.1 with our own static baseline. It is not a current competitor benchmark. Hardware and quality figures below belong to the historical releases and setups shown.
ShinrAI 1.0–1.1: medical-letter evaluation
The Golden ePA corpus: real German doctor letters — discharge summaries, findings, lab reports. Small by design, real by design, frozen so generations stay comparable.
Precision / recall / F1, strict span matching. The static rule engine is our own tuned regex + dictionary stack — the technology every classic PII tool is built on. 4 documents, 21 gold entities: a qualitative head-to-head on the hardest real corpus, not a statistical study. Larger human-annotated gold sets per language are in progress.
Results by Version and Language
Pick a model generation and a language. Bars show span F1 on surface-disjoint evaluation suites — entities the model never saw in training.
Improvement per Model Generation
Frozen v1.1 scored on the v1.2 six-language suites, against v1.2 itself — identical data, identical scoring.
The previous generation lands between 33.6 and 49.3 macro F1 on the six-language suites; v1.2 lands between 89.2 and 94.4. Single hardest category jump across generations: Japanese STREET, F1 6.6 → 88.6 (v1.0 → v1.1).
Attribute Accuracy: Origin, Gender, Name Part
Replacement quality depends on these attributes. Accuracy on the frozen v1-suite (v1.1) and the six v1.2 language suites:
Model Size vs. Accuracy
The compact builds are what most deployments run. Here is exactly what they cost in accuracy.
| Evaluation suite | fp32 · 1.23 GB | INT8 | Δ INT8 | INT4 |
|---|
Macro F1 over the four entity heads. Worst INT8 cost anywhere: 0.33 points. ONNX export parity vs PyTorch: maximum logit delta 3.5 × 10⁻⁵ against a 10⁻³ gate.
Measured Latency
Measured sustained-load numbers on a laptop-class CPU (ONNX fp32, single engine instance) — not lab-best-case.
| Payload | p50 single stream | p95 single stream | Requests/s (c=1) |
|---|
| By hardware — p50 per request | Laptop CPU — measured (fp32) | Mac mini M4 — estimated | GPU service (P40) — measured | Modern GPU (L40S/A100) — estimated | Production end-to-end — measured |
|---|---|---|---|---|---|
| Short (~200 chars) | 26.2 ms | ~28 ms | 12.2 ms | ~6 ms | 17.0 ms |
| Paragraph (~400 tokens) | 101.1 ms | ~105 ms | 28.3 ms | ~13 ms | 37.6 ms |
| Long (1–2k tokens) | 588.4 ms | ~620 ms | 106.7 ms | ~50 ms | 125.3 ms |
Bold = measured, ~ = estimated. GPU cells: round-trips against the two-replica P40 service (2016 hardware); paragraph inference alone is 9.4 ms — 4.3× CPU. The Modern GPU column scales the same cells to L40S/A100 class. Production adds screening, gazetteers and the hop (≈11 ms). Zero 5xx in every cell.
Deterministic paths: static screening <1 ms per stage · dictionary lookups <0.1 ms · replacement draw ~0.4 ms · language detection <5 ms. Production paragraph p95: 41.6 ms — well under the 60 ms bar. Oversized texts degrade to the static path, never to an error.
Methodology and Limitations
- Scoring: seqeval-strict character spans. A hit counts only when boundaries and type both match exactly.
- Suites are surface-disjoint: evaluation entities never occur in training data. Suites are LLM-generated; the Golden ePA corpus is real documents.
- Archived release measurements. The medical-letter comparison covers four documents and 21 entities, and compares ShinrAI 1.0/1.1 with our own static baseline. It is not a current competitor benchmark. Hardware and quality figures below belong to the historical releases and setups shown.
- Latency setup: M-series laptop CPU, sustained 15–60 s cells, any server error fails the run. Server-class Xeon extrapolation: roughly 2–3× slower per thread.