OLMo-2-1B-Instruct — LiteRT-LM (blockwise int4)

allenai/OLMo-2-0425-1B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime.

OLMo-2 is AllenAI's fully-open model family — open weights, open data, open training code, Apache-2.0. This 1B variant is small enough to run on a phone, and is converted with the official upstream litert-torch with no fork patches.

Model description

File	OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm (0.93 GB)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding int8
Compute	integer
Context (KV cache)	4096
Base model	allenai/OLMo-2-0425-1B-Instruct

The bundle carries the tokenizer and OLMo-2's native Tülu prompt template (<|user|> / <|assistant|>, stop token <|endoftext|>), so no separate tokenizer files are needed.

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the model and answers the prompt:

litert-lm run --from-huggingface-repo litert-community/OLMo-2-1B-Instruct \
  OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm \
  --prompt "Explain on-device AI in one short sentence."

3. Keep it for repeated use — import once, then refer to it by name:

litert-lm import --from-huggingface-repo litert-community/OLMo-2-1B-Instruct \
  OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm olmo-2-1b-instruct

litert-lm run olmo-2-1b-instruct     # interactive chat
litert-lm serve                      # local OpenAI-compatible API server

The same bundle runs on macOS, Linux and Windows.

4. Run it on Android

Google AI Edge Gallery 1.0.16+ imports .litertlm models directly from Hugging Face — tap + (bottom-right) inside the app and pick this repository; no computer or adb needed. To sideload a local file instead:

adb push OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm /sdcard/Download/

then tap + and select the file. At 0.93 GB it fits comfortably on an 8 GB phone. To embed the model in your own Android app, use the LiteRT-LM Kotlin API (com.google.ai.edge.litertlm:litertlm-android).

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine:

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	347 tok/s	33.3 tok/s	0.87 s
Apple M4 Max (macOS)	GPU (Metal)	3223 tok/s	150.1 tok/s	0.10 s

Reproducibility: the GPU figures repeat to within about 1% across invocations; the CPU figures are noisier — six invocations of this model spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU row as accurate to roughly ±7%.

On device — iPhone 17 Pro (iOS 27.0), GPU/Metal backend, single cold run with no warm-up turn, prompt "Explain on-device AI in one short sentence.", max 512 tokens:

Device	Backend	Prefill	Decode	TTFT	Load	Peak footprint
iPhone 17 Pro	GPU (Metal)	27.6 tok/s	24.4 tok/s	0.95 s	5.2 s	1330 MB

The iPhone prefill figure is measured on a short chat prompt, so it is dominated by fixed per-turn overhead rather than prefill throughput — it is not comparable to the 256-token desktop prefill column. The model also fits and runs on a Pixel 8a (8 GB) via Gallery; no timing was taken there, so none is quoted.

Accuracy note

GSM8K, n=100, greedy, 0-shot chain-of-thought, identical prompt and answer extraction for every row. The bf16 reference runs in PyTorch on CPU; the quantized rows run as .litertlm through the LiteRT-LM runtime.

Configuration	GSM8K	Size
bf16 (reference)	72.0%	—
int4, blockwise-32 + OCTAV — this model	63.0%	0.93 GB
int8 (dynamic_wi8_afp32)	65.0%	1.50 GB

63% is a coherent, non-degenerate score for a 1B: answers terminate cleanly at <|endoftext|> and do not degenerate. At this size 4-bit quantization costs about 9 points against bf16 — a 1B has less redundancy to absorb int4 rounding than a 3B+, where the same recipe lands at parity. int8 recovers only about 2 points for 60% more size (and the 2-point gap is within n=100 noise), so int4 is shipped as the size/quality trade-off for on-device use.

Conversion

Converted with the official upstream litert-torch export_hf (clean git worktree at upstream/main, dev-fork patches excluded). Olmo2ForCausalLM rides the stock converter with no custom code: QK-norm and OLMo-2's reordered post-norm lower to generic ops. The int4 recipe is blockwise (block 32) + OCTAV with the embedding at int8. At 0.93 GB the whole bundle is a single sub-2 GiB section, so it memory-maps on iOS without needing an externalized embedder.

Training data & PII

This is a weights-exact format conversion of allenai/OLMo-2-0425-1B-Instruct; no new training was performed. OLMo-2 is fully open: pretrained on AllenAI's public Dolma-family corpus (web text, code, academic papers, encyclopedic and Q&A data) and post-trained with the public Tülu 3 SFT + preference mix. Because the pretraining corpus is web-derived it may incidentally contain names and other PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for full dataset and safety details.

License and changes

Apache-2.0, inherited from the base model allenai/OLMo-2-0425-1B-Instruct. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and prompt template repackaged unmodified into the .litertlm bundle. This repository is a community conversion and is not affiliated with AllenAI.