scnsoft-pii-encoder-pruned-6l: int8 ONNX, tokenizer, config and model card (training seed 1234)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,213 @@
+---
+license: other
+license_name: proprietary-sciencesoft
+license_link: https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-pruned-6l/blob/main/README.md#licence
+language: [en, de, fr, it, es, ru, ar]
+pipeline_tag: token-classification
+tags: [pii, ner, privacy, dlp, onnx, xlm-roberta, pruning, distillation]
+library_name: onnxruntime
+base_model: ScienceSoft/scnsoft-pii-encoder
+base_model_relation: finetune
+model-index:
+- name: scnsoft-pii-encoder-pruned-6l
+  results:
+  - task:
+      type: token-classification
+      name: PII detection (format-less entities)
+    dataset:
+      type: ScienceSoft/piibench
+      name: PII Bench (piibench)
+    metrics:
+    - type: f1
+      value: 0.6887
+      name: F1, character-level, label-agnostic, averaged over 7 datasets (production engine, before post-filters)
+    - type: precision
+      value: 0.5730
+      name: Precision
+    - type: recall
+      value: 0.9064
+      name: Recall
+---
+
+# ScienceSoft PII encoder — pruned to 6 layers
+
+[`ScienceSoft/scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder)
+with its top six transformer layers removed and the remaining six retrained. The full model is
+used as teacher. It detects personal data **without a fixed format** (names of people and
+organisations, locations, addresses and dates) in seven languages, and it runs **1.8× faster**
+than the full model on CPU.
+
+**It is less accurate than the full model.** Its recall matches the full model's, but its
+precision does not: on real text it marks more words that are not personal data. Before training,
+a bar was set for releasing a faster model:
+
+- average PII Bench F1 of at least 0.72;
+- no language more than 0.08 below the full model;
+- at least 2× the full model's speed.
+
+**This model meets none of the three.** It reaches 0.689 average F1 and 1.8× the speed. It is
+published with those results stated, so the trade between speed and accuracy can be judged on the
+numbers.
+
+Architecture: `xlm-roberta-base` with 6 layers (the full model's layers 0–5), 768 wide, same
+tokenizer, labels and classifier head as the full model. Exported to ONNX and quantised to int8;
+the file is **235 MB** (most of it the 250,002-token embedding table), and inference runs on CPU.
+
+## Labels
+
+`PERSON` `ORGANIZATION` `LOCATION` `ADDRESS` `DATE_TIME` (BIO tagging, 11 classes), the same set
+and order as the full model. Identifiers with a fixed format (email addresses, telephone,
+payment-card, IBAN and passport numbers) are not labels. See *touch rate* below for how often this
+model marks them anyway.
+
+## Speed
+
+Measured on 300 PII Bench rows:
+
+- int8 ONNX Runtime 1.19.2 on CPU, batch size 1;
+- all three models interleaved row by row;
+- an Apple M4 Max with 16 cores, a shared workstation with a 1-minute load average of 18–19.
+
+Absolute milliseconds depend on the host; the ratio is the informative part.
+
+| model | layers × width | int8 size | median ms per row | 95th percentile | speed-up |
+|---|---|---|---|---|---|
+| `scnsoft-pii-encoder` | 12 × 768 | 278 MB | 22.2 | 36.0 | 1.0× |
+| **this model** | 6 × 768 | 235 MB | **12.2** | **20.7** | **1.8×** |
+| [`scnsoft-pii-encoder-small`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-small) | 6 × 384 | 107 MB | 5.7 | 8.5 | 3.9× |
+
+Halving the depth roughly halves the compute per token. The width stays at 768, which is why the
+6 × 384 model is faster still.
+
+## Evaluation on PII Bench
+
+[PII Bench](https://huggingface.co/datasets/ScienceSoft/piibench) (piibench) scores detection
+of the five entity types above in text serialised the way outbound traffic carries it (JSON,
+XML, SQL, CSV, key=value, log lines, Markdown tables, OCR output) as well as prose.
+
+- It comprises 3,500 rows, 500 from each of seven third-party datasets.
+- Precision, recall and F1 are computed over characters and are label-agnostic.
+- Scores are micro-averaged within each dataset and then averaged across the seven.
+- Leaderboard: [ScienceSoft/piibench-leaderboard](https://huggingface.co/spaces/ScienceSoft/piibench-leaderboard).
+
+**How these numbers were produced.** All models below are scored the same way: by the production
+Rust inference engine, before its post-filters. That is the same path as the leaderboard's
+`scnsoft-pii-encoder-engine-raw` row, which this scoring reproduces exactly (0.7725). The
+intervals are 95 % percentile bootstrap intervals over rows, resampled within each dataset
+(1,000 resamples).
+
+**Conflict of interest.** PII Bench is built and published by ScienceSoft, the developer of
+this model. Four of its seven datasets (TAB, FactRuEval, MEDDOCAN, IAHLT) informed changes to the
+training corpus this model shares with the full model.
+
+| | this model | `scnsoft-pii-encoder` | bar set before training |
+|---|---|---|---|
+| F1 | **0.689** [0.681, 0.697] | 0.773 [0.765, 0.780] | ≥ 0.72 |
+| precision | 0.573 | 0.685 | |
+| recall | 0.906 | 0.908 | |
+| touch rate (lower is better) | 0.160 | 0.020 | |
+
+| language | this model | `scnsoft-pii-encoder` | bar (full model − 0.08) | |
+|---|---|---|---|---|
+| ar | 0.6012 | 0.6812 | 0.6012 | at the bar |
+| de | 0.5378 | 0.6573 | 0.5773 | below |
+| en | 0.7317 | 0.8040 | 0.7240 | above |
+| es | 0.7736 | 0.8437 | 0.7637 | above |
+| ru | 0.6776 | 0.7543 | 0.6743 | above |
+
+**Where the accuracy is lost.**
+
+- Recall is the full model's (0.906 against 0.908). The loss is in **precision** (0.573 against
+  0.685), across every dataset (−0.05 to −0.12 F1) and every free-text format.
+- On an in-distribution synthetic evaluation, drawn from the training generator, the model scores
+  F1 0.982, close to the full model's 0.993. It fires too often on real text, which its training
+  data never showed it.
+- **Touch rate**, the share of characters belonging to format-fixed identifiers that a model marks
+  anyway, is 0.160 here, against 0.020 for the full model.
+
+### Variability across training seeds
+
+Two training runs of the same recipe, differing only in the random seed:
+
+| training seed | F1 | 95 % CI | precision | recall |
+|---|---|---|---|---|
+| 1234 — **this model** (`main`) | 0.6887 | [0.6809, 0.6966] | 0.573 | 0.906 |
+| 4321 (branch `seed-4321`) | 0.6870 | [0.6792, 0.6952] | 0.580 | 0.890 |
+
+Seed 1234 is published on `main` by convention, not selected by score.
+
+## Training
+
+- **Initialisation:** the full model's embeddings, transformer layers 0–5 and classifier head.
+  Keeping the lower layers follows the finding that dropping the top layers of a pretrained
+  transformer holds up best after retraining (Sajjad et al., 2020, *On the Effect of Dropping
+  Layers of Pre-trained Transformer Models*).
+- **Corpus:** the same synthetic corpus as `scnsoft-pii-encoder` (v23), rebuilt from that run's
+  own generator snapshot and configuration and verified by count: 155,141 training and 5,300
+  evaluation rows.
+- **Distillation from `scnsoft-pii-encoder`:** 0.5 × cross-entropy on the gold labels plus
+  0.5 × T² × KL(teacher ‖ student) at T = 2, over non-padding tokens.
+- **Schedule:** two epochs, 31 minutes on one GPU.
+- **Stack:** `transformers==4.57.1`, `tokenizers==0.22.1`, `torch==2.9.0a0`, `faker==37.12.0`.
+- **int8 against fp32:** +0.0036 F1 on the synthetic evaluation, 95 % CI [+0.0020, +0.0052]. The
+  int8 file scored slightly higher here.
+
+## Limitations
+
+- **Below the release bar** on average F1, on German, and on speed (see above). Where accuracy
+  matters, use [`scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder).
+  Where speed matters most, [`scnsoft-pii-encoder-small`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-small)
+  is 2× faster again, at lower accuracy.
+- **Lower precision on real text** than the full model, and more format-fixed identifiers marked.
+- **Trained only on synthetic data**, like the full model. Its other limitations apply too: Arabic
+  is the weakest language; Arabizi is a reconstruction; `DATE_TIME` is kept by the product only
+  next to a birth-date cue.
+
+## Use
+
+Required files: `config.json`, `tokenizer.json`, `onnx/model_quantized.onnx`.
+
+```python
+import json, onnxruntime as ort
+from transformers import AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("ScienceSoft/scnsoft-pii-encoder-pruned-6l")
+cfg = json.load(open("config.json"))
+id2label = {int(k): v for k, v in cfg["id2label"].items()}
+sess = ort.InferenceSession("onnx/model_quantized.onnx",
+                            providers=["CPUExecutionProvider"])
+
+enc = tok(text, return_offsets_mapping=True, return_tensors="np",
+          truncation=True, max_length=256)
+offsets = enc.pop("offset_mapping")[0]
+logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
+# argmax -> id2label -> merge BIO spans using `offsets` for character positions
+```
+
+Encode with the special tokens `<s>` and `</s>`, as the model was trained;
+`AutoTokenizer.__call__` adds them by default.
+
+## Provenance
+
+The training corpus is synthetic text produced by ScienceSoft's own generator; no production
+traffic and no scraped text is used for training. Entity values come from project-written lists
+and from Faker (person, company and city names), libpostal (address vocabulary) and GeoNames
+(Arabic-script city names).
+
+## Licence
+
+Proprietary to ScienceSoft. The weights are published so that the results on this page can be
+evaluated and reproduced; no licence to use them in production, to modify them or to redistribute
+them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).
+
+**Basis for these terms.**
+
+- **Derived from** ScienceSoft's own `scnsoft-pii-encoder`, which is derived from
+  [`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base) (MIT licence).
+- **Training corpus** includes entity values from Faker (MIT), libpostal (MIT) and GeoNames
+  (CC BY 4.0).
+
+All of these licences permit a derivative work to be distributed under proprietary terms,
+provided their notices are retained and GeoNames is attributed. None is share-alike, and none
+restricts commercial use. The notices and the attribution are in
+[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,55 @@
+# Third-party notices
+
+This model is proprietary to ScienceSoft. It is derived from, or was trained with material
+drawn from, the third-party works below. Each is used under its own licence, all of which
+permit distribution of derivative works under different terms provided the notices below are
+retained and the attributions given. None is share-alike and none restricts commercial use.
+
+## Base model — XLM-RoBERTa
+
+`FacebookAI/xlm-roberta-base` (https://huggingface.co/FacebookAI/xlm-roberta-base), including
+its tokenizer. Licensed under the MIT License:
+
+```
+MIT License
+
+Copyright (c) Facebook, Inc. and its affiliates.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+```
+
+## Training-data sources
+
+The training corpus is synthetic text produced by ScienceSoft's own generator. Entity values
+inserted into that text were drawn in part from the following sources.
+
+**Faker** (https://github.com/joke2k/faker), version 37.12.0 — person, company and city name
+pools. MIT License, Copyright (c) 2012 Daniele Faraglia; the MIT permission notice above
+applies with that copyright line.
+
+**libpostal** (https://github.com/openvenues/libpostal), `resources/dictionaries/` — address
+sub-building vocabulary. MIT License, Copyright (c) 2015 openvenues; the MIT permission notice
+above applies with that copyright line.
+
+**GeoNames** (https://www.geonames.org/) — Arabic-script city names from the per-country
+exports (https://download.geonames.org/export/dump/). Licensed under Creative Commons
+Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/). Changes made:
+names were filtered to populated places of at least 5,000 inhabitants and to Arabic-script
+text, then inserted as entity values into generated training sentences; no GeoNames record is
+reproduced in this model repository.

config.json

@@ -0,0 +1,53 @@
+{
+  "architectures": [
+    "XLMRobertaForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "O",
+    "1": "B-PERSON",
+    "2": "I-PERSON",
+    "3": "B-ORGANIZATION",
+    "4": "I-ORGANIZATION",
+    "5": "B-LOCATION",
+    "6": "I-LOCATION",
+    "7": "B-ADDRESS",
+    "8": "I-ADDRESS",
+    "9": "B-DATE_TIME",
+    "10": "I-DATE_TIME"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "B-ADDRESS": 7,
+    "B-DATE_TIME": 9,
+    "B-LOCATION": 5,
+    "B-ORGANIZATION": 3,
+    "B-PERSON": 1,
+    "I-ADDRESS": 8,
+    "I-DATE_TIME": 10,
+    "I-LOCATION": 6,
+    "I-ORGANIZATION": 4,
+    "I-PERSON": 2,
+    "O": 0
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.1",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

onnx/model_quantized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20883bcc40b21b7c30287bd000b18b4a62bd585becdfa2aade38f7f1bb9234c3
+size 235332276

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a514807cffabd8abaf028cfaffe7ff0c4f60b97ea2db80c41f14172ae6b018ca
+size 17082987

tokenizer_config.json

@@ -0,0 +1,62 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 256,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}