Glint-Research/Glintclassify

Glint-Classify-1M

A 984,209-parameter decision model. It does not write text. It reads an unstructured state — a string, a JSON object, a list, or an image up to 4096px — together with a map of typed questions, and returns one structured, calibrated answer per question, for every question, in a single forward pass.

This is a local, 1M-parameter re-creation of the idea behind TypeSafe AI's "System One" models (Jev). Jev is text-only; image input is this release's addition.

parameters	984,209
vocabulary	4096 (byte-level BPE, the Glint tokenizer, pad 0)
text encoder	3 shared bidirectional blocks, 128 wide, 4 heads, QK-normalised attention
image encoder	3 stride-2 convs, GroupNorm; any size ≤4096px → global 32px view + up to 4×4 tiles
training	105 minutes cumulative on one RTX 5090 across three chained passes, 169,529 steps, loss 0.384
held-out accuracy	72.2% against 26.0% chance, ECE 0.090
latency	3.6 ms median text-only, 3.3 ms with a 32px image (RTX 5090)
model file	3.9 MB

Read the Failure modes section before deploying this. Its aggregate numbers are respectable for the size and its behaviour on genuinely novel prompts is not.

Quick start

python infer.py --info
python infer.py --ask request.json
python infer.py --ask '{"state": {"tool": "shell", "command": "rm -rf /srv"}, "questions": {"destructive": {"type": "noul", "instructions": "Is this command destructive or irreversible?"}}}'
python infer.py --ask request.json --image photo.jpg
python infer.py --serve --port 8321          # POST /v1/systemone, GET /health
curl -X POST localhost:8321/v1/systemone -d @request.json

Python:

from infer import SystemOne

system = SystemOne()
answers = system.ask(
    {"ticket": "I was charged twice for order A-104. Please refund the duplicate."},
    {
        "refund": {"type": "noul", "instructions": "Does `ticket` request a refund?"},
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this ticket?",
            "criteria": {
                "returns": "exchanges, refunds and damaged items",
                "billing": "charges, invoices and duplicate payments",
                "shipping": "delivery delays and lost packages",
            },
        },
        "frustration": {
            "type": "score",
            "instructions": "How frustrated is the customer?",
            "criteria": ["calm and neutral", "concerned but civil", "very angry"],
        },
    },
)

infer.py needs only torch and tokenizers (Pillow only to read image files). It carries its own copy of the architecture; nothing else has to be cloned or installed. Every answer is JSON-serialisable and can be checked before acting.

--ask is deterministic argmax: the same request always returns the same answer. --serve samples choices from the model's own distribution and breaks decision loops: when an unchanged state would repeat the same top choice, later calls progressively flatten the sampling distribution so the agent acts its way out instead of repeating one answer forever. Reported probabilities are always the model's own, so thresholds keep their meaning; /v1/systemone echoes which mode was used in its explore field.

The three primitives

primitive	request	response
noul	{"type": "noul", "instructions": "...", "criteria": {"true": "...", "false": "..."}}	{"type": "noul", "noul": p_yes}
choice	{"type": "choice", "instructions": "...", "criteria": {"name": "description", ...}}	{"type": "choice", "choice", "confidence", "probabilities"}
score	{"type": "score", "instructions": "...", "criteria": ["low", ..., "high"]}	{"type": "score", "score", "confidence", "probabilities", "legend"}

confidence is 1 − normalised entropy: a flat distribution is 0, a one-hot is 1. score is the probability-weighted mean level and can fall between levels. Choice takes 2..255 options, score 2..10 levels. Reference state elements in the instructions with backticks — `order.status`, `ticket.messages[0].text`.

All questions share one forward pass. Adding questions to a request costs almost nothing: 5 questions and 1 question both answer in about 4 ms.

Acting on the answer

The model is calibrated, so the probabilities are usable as thresholds. Suggested pattern, adapted from TypeSafe's own guidance:

• confidence > 0.85 — act automatically.
• 0.5 .. 0.85 — proceed with verification, or ask the user.
• < 0.5 — abstain; route to a human or a slower model.

Measured ECE on the held-out mixture is 0.090, and the procedural_ood suite (93.1% accuracy, novel vocabulary) sits at 0.015, so where the model is right it tends to know it. Where it is wrong it is often confidently wrong — treat image predictions and novel phrasings with more caution than the confidence suggests.

Accuracy

Every number below was measured on this machine against the shipped weights with python train.py --systemone --eval --size=1m, and metrics.json beside this README is that report verbatim. Held-out rows were never trained on, and the temperature was fitted on even rows while every number below is measured on odd rows.

	measured	chance
all held-out sources (unweighted mean)	72.2%	26.0%
noul (yes/no)	78.7%	—
choice	73.9%	—
score	64.9%	—

source	accuracy	chance		source	accuracy	chance
sms_spam	98.4	50.0		qnli	67.6	50.0
enron_spam	98.0	50.0		boolq	67.2	50.0
mnist	97.2	10.0		snli	60.2	33.3
procedural_ood	93.1	34.4		stl10	56.4	10.0
clinc intents	88.0	8.3		cifar10	54.4	10.0
dbpedia	87.6	8.3		rte	52.2	50.0
fashion_mnist	86.8	10.0		cifar100	52.0	8.3
ag_news	84.8	25.0		yelp stars	49.6	20.0
massive intents	81.2	8.3		yahoo topics	49.2	10.0
sst2	74.0	50.0		tweet_sentiment	48.0	33.3
emotion	70.8	16.7

What this says plainly: the model learned the format of typed decisions and the lexical cues that separate its sources — spam, topic, intent, digit and garment recognition, and the procedural robot/mail/web/extraction/guardrail domains it was generated from. Natural-language inference is still the weak spot. RTE at 52.2% against 50% chance is chance-level, and SNLI at 60.2% against 33.3% is above chance but half of it is the three-way guess: those datasets ask for entailment between two sentences, which needs knowledge a million parameters do not hold. The distillation and polarity data moved SNLI up from 36.6%, which is why it is no longer reported as no-signal, but neither score is something to rely on.

Held-out probes (probes.json, 15 hand-written states, 25 questions) score 16/25. They are written to be unlike anything in training, and the failures are systematic rather than random.

Failure modes (its what you would expect)

Dont use it in prod, use it for fun, its not perfect.

• Images are small-picture recognition. 32px CIFAR-10 classes land at 54.4% (chance 10%), while MNIST (97.2%) and Fashion-MNIST (86.8%) are strong. A gallery image of a cat was answered deer (0.39) with cat second at 0.24. A 4096px image is summarised as 17 tiles of 32px, so fine detail, text in images, and small objects are out of reach.

How it was trained

• Cross-entropy over the option logits — a proper scoring rule, which is also what makes the probabilities calibratable — then one temperature fitted on the calibration half and shipped in the checkpoint.
• 35% procedural (robot, web, mail, fact-passage, JSON extraction, shell/SQL guardrail, request routing), 45% public text (15 datasets recast as typed decisions), 20% public images (CIFAR-10/100, Fashion-MNIST, MNIST, STL-10), randomly rescaled so every tile grid sees gradient.
• Three chained passes on one RTX 5090: 15 minutes, then two resumes of 45 minutes each (105 cumulative, 169,529 steps). validate_minutes caps any single pass at 45 minutes; --resume continues an existing checkpoint, so a longer budget is always an explicit choice rather than a default.

Limits

• State text is truncated at 192 tokens, instructions at 40, each option at 24.
• No generation, no chat: ask typed questions, act on the structured answers.
• It knows the sources in the accuracy table and their shape, not the world.
• Everything here is a measurement from this machine, not a leaderboard.

Reproducing

make systemone-prep                                    # cache the 20 public sources
make systemone                                         # 15-minute run + held-out report
python scripts/build_systemone_release.py --trained-minutes 105   # rebuild this directory, with a parity check

The build script loads this release's own infer.py, runs the whole probe suite through both the release and the training code, and refuses to build if any probability differs by more than 1e-4. The shipped files agree exactly (max |Δp| = 0.00e+00).