Banking77 bank: 87.0% with bge-small-en-v1.5 (probe head, 4000 iters, class-balanced)

README.md

@@ -0,0 +1,96 @@
+---
+license: cc-by-4.0
+library_name: ruvector
+tags:
+  - text-classification
+  - intent-classification
+  - ruvector
+  - typesafe
+datasets:
+  - PolyAI/banking77
+---
+
+# ruvector-typesafe — Banking77 bank
+
+A trained example bank for
+[`@ruvector/typesafe`](https://www.npmjs.com/package/@ruvector/typesafe):
+typed decisions over text, locally, with no API bill and no network in the
+decision path. 77 fine-grained banking intents — the standard hard test for intent classification.
+
+```sh
+npm install @ruvector/typesafe
+curl -LO https://huggingface.co/ruvnet/ruvector-typesafe-banking77/resolve/main/bank.json
+curl -LO https://huggingface.co/ruvnet/ruvector-typesafe-banking77/resolve/main/questions.json
+
+echo "my card was declined at an atm" \
+  | npx typesafe decide --questions questions.json --bank bank.json --embedder onnx \
+      --engine-options '{"probeIterations":4000,"probeClassBalanced":true,"head":"probe"}'
+```
+
+**The `--engine-options` are not optional.** A bank stores examples and their
+frozen splits — never hyperparameters — and the head is refit from it on load.
+Omit them and you refit with the library defaults (400 iterations, `head: auto`),
+which is a different and materially worse model than the one measured below.
+These are the exact options this bank was trained under.
+
+## What this artifact is
+
+**Labelled examples with frozen split assignments — not weights.** The heads
+(nearest-prototype, or a multinomial probe once a class has enough examples) and
+the temperature calibration are refit from the bank when the engine loads it.
+
+Two consequences worth knowing:
+
+- The bank is **encoder-independent** — it holds text and content-hashed split
+  tags, nothing encoder-derived. For this dataset that is measured, not assumed: both bundled encoders exported byte-identical banks.
+  Only the accuracy below is encoder-specific.
+- **The first decision after loading is slow.** `importBankJson` just admits the
+  examples; the head is fitted lazily on the first `decide`, and at
+  `probeIterations: 4000` over 9,993 examples and
+  77 classes that fit is **minutes**, not milliseconds.
+  Every later call is the steady-state latency in the table. Import once, warm
+  it with one throwaway decision, and keep the engine alive — do not load a bank
+  per request.
+
+A reloaded bank reproduces the trained engine's answers exactly — that
+round-trip is asserted by `test/bank-roundtrip.test.mjs`, not assumed.
+
+## Accuracy
+
+Held-out `test` split, 3,077 utterances,
+77 classes.
+
+| encoder | accuracy | p50 latency | p95 latency |
+|---|---|---|---|
+| `all-MiniLM-L6-v2` | 76.7% | 15 ms | 16 ms |
+| `bge-small-en-v1.5` | 87.0% | 6 ms | 10 ms |
+
+Trained on 9,993 labelled examples
+(`splitsHash: f51a48af2895c08d…`).
+
+## Training
+
+```sh
+node scripts/typesafe-banks/build-bank.mjs --dataset banking77 --encoder all-MiniLM-L6-v2
+```
+
+The probe head is full-batch gradient descent with a **fixed iteration count**,
+and that count is the thing to tune when you add data: the default 400
+iterations fits ~1k examples well and underfits ~10k badly. Raise it through
+`EngineOptions`:
+
+```ts
+createTypesafe({ embedder: …, engine: { probeIterations: 4000 } })
+```
+
+## Limitations
+
+- English only; both bundled encoders are English sentence encoders.
+- The label set is closed. New intents need new examples and a refit.
+- Accuracy is reported on this dataset's own test split — it is not a claim
+  about your traffic.
+
+## Credit
+
+Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders* (2020), arXiv:2003.04807. Dataset licence: CC-BY-4.0; this bank redistributes
+the utterance text under that licence. The `@ruvector/typesafe` code is MIT.

eval-all-MiniLM-L6-v2.json

@@ -0,0 +1,46 @@
+{
+  "engineOptions": {
+    "probeIterations": 4000,
+    "probeClassBalanced": true
+  },
+  "dataset": "banking77",
+  "encoder": "all-MiniLM-L6-v2",
+  "questionId": "intent",
+  "counts": {
+    "train": 9993,
+    "test": 3077,
+    "labels": 77
+  },
+  "splitsHash": "f51a48af2895c08ddafcb661530bd07b9ebad6b2d73f04358965a788acc72017",
+  "train": {
+    "examples": 9993,
+    "seconds": 328.446,
+    "report": {
+      "question": "intent",
+      "accepted": 9993,
+      "rejected": 0,
+      "head": "linear-probe",
+      "calibrated": true
+    }
+  },
+  "eval": {
+    "inScope": {
+      "n": 3077,
+      "accuracy": 0.7673058173545662,
+      "meanConfidence": 0.7062555000480987,
+      "meanAbstain": 1.0337861248008484e-8,
+      "p50Ms": 15,
+      "p95Ms": 16
+    },
+    "oos": null
+  },
+  "bank": {
+    "bytes": 2392682,
+    "sha256": "661afa0340ecde8ea9efeac69d130f4a33d36383502d3f2cd0244d3240ae4e75"
+  },
+  "binding": {
+    "backend": "native"
+  },
+  "node": "v22.23.2",
+  "finishedAt": "2026-09-22T20:31:18.802Z"
+}

eval-bge-small-en-v1.5.json

@@ -0,0 +1,46 @@
+{
+  "engineOptions": {
+    "probeIterations": 4000,
+    "probeClassBalanced": true
+  },
+  "dataset": "banking77",
+  "encoder": "bge-small-en-v1.5",
+  "questionId": "intent",
+  "counts": {
+    "train": 9993,
+    "test": 3077,
+    "labels": 77
+  },
+  "splitsHash": "f51a48af2895c08ddafcb661530bd07b9ebad6b2d73f04358965a788acc72017",
+  "train": {
+    "examples": 9993,
+    "seconds": 428.759,
+    "report": {
+      "question": "intent",
+      "accepted": 9993,
+      "rejected": 0,
+      "head": "linear-probe",
+      "calibrated": true
+    }
+  },
+  "eval": {
+    "inScope": {
+      "n": 3077,
+      "accuracy": 0.8696782580435489,
+      "meanConfidence": 0.8615509237211584,
+      "meanAbstain": 6.934685016282097e-7,
+      "p50Ms": 6,
+      "p95Ms": 10
+    },
+    "oos": null
+  },
+  "bank": {
+    "bytes": 2392682,
+    "sha256": "661afa0340ecde8ea9efeac69d130f4a33d36383502d3f2cd0244d3240ae4e75"
+  },
+  "binding": {
+    "backend": "native"
+  },
+  "node": "v22.23.2",
+  "finishedAt": "2026-09-22T20:31:44.920Z"
+}

questions.json

@@ -0,0 +1,85 @@
+{
+  "intent": {
+    "type": "choice",
+    "instructions": "Which banking intent does this message express",
+    "criteria": {
+      "card_arrival": "card arrival",
+      "card_linking": "card linking",
+      "exchange_rate": "exchange rate",
+      "card_payment_wrong_exchange_rate": "card payment wrong exchange rate",
+      "extra_charge_on_statement": "extra charge on statement",
+      "pending_cash_withdrawal": "pending cash withdrawal",
+      "fiat_currency_support": "fiat currency support",
+      "card_delivery_estimate": "card delivery estimate",
+      "automatic_top_up": "automatic top up",
+      "card_not_working": "card not working",
+      "exchange_via_app": "exchange via app",
+      "lost_or_stolen_card": "lost or stolen card",
+      "age_limit": "age limit",
+      "pin_blocked": "pin blocked",
+      "contactless_not_working": "contactless not working",
+      "top_up_by_bank_transfer_charge": "top up by bank transfer charge",
+      "pending_top_up": "pending top up",
+      "cancel_transfer": "cancel transfer",
+      "top_up_limits": "top up limits",
+      "wrong_amount_of_cash_received": "wrong amount of cash received",
+      "card_payment_fee_charged": "card payment fee charged",
+      "transfer_not_received_by_recipient": "transfer not received by recipient",
+      "supported_cards_and_currencies": "supported cards and currencies",
+      "getting_virtual_card": "getting virtual card",
+      "card_acceptance": "card acceptance",
+      "top_up_reverted": "top up reverted",
+      "balance_not_updated_after_cheque_or_cash_deposit": "balance not updated after cheque or cash deposit",
+      "card_payment_not_recognised": "card payment not recognised",
+      "edit_personal_details": "edit personal details",
+      "why_verify_identity": "why verify identity",
+      "unable_to_verify_identity": "unable to verify identity",
+      "get_physical_card": "get physical card",
+      "visa_or_mastercard": "visa or mastercard",
+      "topping_up_by_card": "topping up by card",
+      "disposable_card_limits": "disposable card limits",
+      "compromised_card": "compromised card",
+      "atm_support": "atm support",
+      "direct_debit_payment_not_recognised": "direct debit payment not recognised",
+      "passcode_forgotten": "passcode forgotten",
+      "declined_cash_withdrawal": "declined cash withdrawal",
+      "pending_card_payment": "pending card payment",
+      "lost_or_stolen_phone": "lost or stolen phone",
+      "request_refund": "request refund",
+      "declined_transfer": "declined transfer",
+      "Refund_not_showing_up": "Refund not showing up",
+      "declined_card_payment": "declined card payment",
+      "pending_transfer": "pending transfer",
+      "terminate_account": "terminate account",
+      "card_swallowed": "card swallowed",
+      "transaction_charged_twice": "transaction charged twice",
+      "verify_source_of_funds": "verify source of funds",
+      "transfer_timing": "transfer timing",
+      "reverted_card_payment?": "reverted card payment?",
+      "change_pin": "change pin",
+      "beneficiary_not_allowed": "beneficiary not allowed",
+      "transfer_fee_charged": "transfer fee charged",
+      "receiving_money": "receiving money",
+      "failed_transfer": "failed transfer",
+      "transfer_into_account": "transfer into account",
+      "verify_top_up": "verify top up",
+      "getting_spare_card": "getting spare card",
+      "top_up_by_cash_or_cheque": "top up by cash or cheque",
+      "order_physical_card": "order physical card",
+      "virtual_card_not_working": "virtual card not working",
+      "wrong_exchange_rate_for_cash_withdrawal": "wrong exchange rate for cash withdrawal",
+      "get_disposable_virtual_card": "get disposable virtual card",
+      "top_up_failed": "top up failed",
+      "balance_not_updated_after_bank_transfer": "balance not updated after bank transfer",
+      "cash_withdrawal_not_recognised": "cash withdrawal not recognised",
+      "exchange_charge": "exchange charge",
+      "top_up_by_card_charge": "top up by card charge",
+      "activate_my_card": "activate my card",
+      "cash_withdrawal_charge": "cash withdrawal charge",
+      "card_about_to_expire": "card about to expire",
+      "apple_pay_or_google_pay": "apple pay or google pay",
+      "verify_my_identity": "verify my identity",
+      "country_support": "country support"
+    }
+  }
+}