Inkling 2 Small for Hyperflare
This model is based on Inkling-2-Small-NVFP4-CSF by Local Inference Lab, Inc., a non-profit organization, available at https://huggingface.co/local-inference-lab/Inkling-2-Small-NVFP4-CSF. Inkling-2-Small-NVFP4-CSF is licensed under the Local Inference Lab License, Version 1.0.
Built on the work of Thinking Machines, Local Inference Lab, NVIDIA, Luke Alonso, Martin Vit, Arthur Cheng, the
Flare, FlashInfer, vLLM and Hugging Face Transformers projects and Ke Bao. See Credits and CREDITS.md.
This repository publishes a signed Hyperflare (Flare) runtime package for Inkling 2 Small by Thinking Machines, prepared for tensor-parallel serving on four GPUs. The package revision holds the prepared weights, the compiled kernels, the captured execution graphs, the signed package metadata and the Flare executable that serves it. Download the revision for the hardware and start the executable: nothing is compiled or tuned on the machine that runs it. Transformers, vLLM, SGLang and llama.cpp cannot load this format.
The model serves text and images, takes up to 8 requests at once and up to 1,048,576 tokens per request,
and answers on an OpenAI-compatible API under the model name Inkling-2-Small.
The package of 2026-11-18
The revision gb202 holds a new package since 2026-11-18. It replaces the text-only package of 2026-11-17.
- The vision tower of Inkling 2 Small is prepared: a request can carry up to 4 images (see Serve).
- The language model is unchanged. Its weights are byte for byte those of the package of 2026-11-17, and the perplexity on the test passage is the same, 6.2105.
- The vision tower, its working memory for the largest image and four image slots take 2.0 GiB of each card that the text-only package gives to the key and value cache. Four 96 GB cards hold 1,769,472 tokens of key and value cache beside the weights (the package of 2026-11-17 held 2,236,416).
- The included Flare executable is new: it adds the image path (see The executable).
- 18 of the 210 package files changed: the metadata primary, the four
weights/tp4/rank-*/misc.safetensorsfiles, which now also hold the vision tower, and the compiled kernels, captured graphs and initial state of each card. The other 192 files are byte for byte those of the package of 2026-11-17. - Flare's package cache lists this package as
Inkling-2-Small-NVFP4-CSF-Static-Flare. The model name on the API is unchanged. - The text-only package of 2026-11-17, with the larger key and value cache, and its model card remain in the
repository's history: package at commit
ca92afae02b1e466756ab223c9adf3a496fbdfc6(pass it tohf download --revision), card at commitcef114aedb5f681b1660723d640cb31300dffcdf. The package of 2026-11-14 is at commit085a16677047bc931f85fe2d62dcd1eaf2267e56.
Results
Measured on 2026-11-18 on four RTX PRO 6000 Blackwell cards (96 GB each), with the model's draft head proposing
7 tokens a round. The measurements ran on the package before it was signed (signing replaces only the metadata
primary model.safetensors), served by the Flare executable included in the revision.
| Result | |
|---|---|
| Decode, 1 request | 179.3 tokens/s |
| Decode, 2 requests at once (total) | 286.6 tokens/s |
| Decode, 4 requests at once (total) | 471.9 tokens/s |
| Decode, 8 requests at once (total) | 615.6 tokens/s |
| Decode, 1 request with 32K tokens of context | 177.1 tokens/s |
| Decode, 1 request with 128K tokens of context | 152.8 tokens/s |
| Prompt processing, 8K-token prompt | 11,199 tokens/s |
| Prompt processing, 64K-token prompt | 10,487 tokens/s |
| Prompt processing, 128K-token prompt | 9,153 tokens/s |
| Image questions: 8 one at a time, 2 in sequence, 4 at once, and 1 text question in the same run | 15 of 15 right |
| One request with a 3,200 x 2,000-pixel image (4,033 prompt tokens), start to answer | 4.5 seconds |
| Key and value cache beside the weights | 1,769,472 tokens |
| Start, weights from NVMe to the four cards | 60.5 seconds |
| Perplexity on a 4,095-token English passage | 6.21 |
The gb202 setup has the prepared settings and the commands. Validation describes each test.
About Hyperflare
Hyperflare, also called Flare (NVIDIA/hyperflare-inference), is a local inference server developed by Local
Inference Lab and NVIDIA for NVIDIA GPUs of the SM120 and SM121 generation, in workstations and compact systems
such as DGX Spark. It is an open-source research prototype, built for long conversations, coding agents and
repeated tool use, and meant to make the memory, bandwidth and compute of such a machine go further while
generation stays responsive. Its authors describe it as a specialized option for the models and machines it
covers, not a replacement for production inference engines such as vLLM.
The idea is to do the expensive work before serving starts:
- Prepare, then serve. A preparation stage written in Python on PyTorch, FlashInfer and the b12x kernels loads the model, tunes its kernels on the cards it will run on by racing the variants of each kernel and keeping the fastest, compiles the forward passes and captures them as CUDA graphs. A runtime written in Rust then serves those graphs without Python.
- Recipes. Each model has a recipe that owns the model's computation. The runtime owns scheduling, cache storage and request handling, and stays small and shared between models.
- Planned memory and state. Buffers and capacities are fixed ahead of time. A conversation's state, the key and value cache together with any recurrent state of the model, is treated as one thing, so that it can be reused, branched and resumed correctly.
- Tensor parallelism. On several cards every layer is split across the cards, and the cards exchange partial results directly with each other, here over PCIe.
This repository holds the result of that preparation for Inkling 2 Small on four cards.
The weights
The package is prepared from
local-inference-lab/Inkling-2-Small-NVFP4-CSF
at revision 614db8960cbce9e7d6ff04e886f4c2815972007e, Local Inference Lab's checkpoint of Inkling 2 Small for
tensor-parallel serving. That checkpoint holds the routed experts of all 48 decoder layers in NVFP4 with
calibrated static activation scales, stores the NVFP4 block scales with CSF, a lossless encoding, and takes its
weights from thinkingmachines/Inkling-2-Small-MXFP8
at revision 086a047e8b7ff2ac902dd52355328ddb66b16914. With static scales no value depends on a token's own
activations at run time, so each card runs its share of an expert without asking the other cards.
Inkling 2 Small is a mixture-of-experts model by Thinking Machines with 48 layers, hidden width 4,608, 384 routed experts of which 8 run for each token, 200,058 vocabulary entries, a context window of up to 1,048,576 tokens, a draft module for speculative decoding and a vision tower for image input.
What the package changes in that checkpoint:
- The attention input and output projections of the decoder layers are stored as MXFP8, converted from the checkpoint's BF16 values during preparation. Perplexity on the test passage is 6.2105 with them and 6.2112 with the BF16 projections.
- The 15 MXFP8 feed-forward matrices of the draft module are stored as BF16. Preparation proves for every value that the BF16 number equals the MXFP8 one exactly before it accepts the file.
- The vision tower keeps the checkpoint's BF16 values, with a full copy on each card. It has 23 transformer layers of width 1,024, reads 20 x 20-pixel patches and merges each 2 x 2 group of patches into one token for the language model.
- The weights are laid out as one share per card in Flare's package format, with the compiled kernels, the captured execution graphs and the signed metadata beside them.
The routed experts keep the checkpoint's NVFP4 values, block scales and static activation scales. Nothing was retrained, recalibrated or quantized again apart from item 1.
The revision is a self-contained package for tensor parallelism over four cards: a model.safetensors metadata
primary, companion files under weights/ and deployments/, and the publisher profile. Keep the primary and
every companion together under their original names.
Pick a platform
| Revision | Hardware | Package | Platform setup |
|---|---|---|---|
gb202 |
Linux x86-64; 4x RTX PRO 6000 Blackwell 96 GB (SM120) on one host | 347 GB | gb202 setup |
The main revision holds this model card, the license files, the notice, the credits and the publisher profile.
Download the revision for the hardware; its README covers download, launch, Docker and the serving settings.
Flare fundamentals
The executable
Use the Flare executable included in the revision, runner/flare-linux-x86_64. Inkling 2 is not part of a
released Flare build at the time of writing, so released Flare images and installations cannot serve this package.
The included executable was built on 2026-11-17 with the image path, which the executable of the earlier
packages lacks. It reads this package, verifies its signatures and served every measurement on this card. The
revision also holds a small Dockerfile that packages the same executable; it compiles nothing.
Download and verify
Download the revision with the Hugging Face hf CLI, always naming the revision. At the first start Flare checks
the package signature against the publisher profile in the revision, shows the publisher and the model license,
and asks for acceptance. The signature covers the metadata primary and every companion, so a missing or renamed
file fails the check. A trust store kept between starts remembers the acceptance.
Serve
The package needs four GPUs. Rank zero serves the OpenAI-compatible API at /v1, with health at /health;
clients use the model name Inkling-2-Small. Reasoning is returned apart from the answer, in
reasoning_content.
The prepared package has these capabilities:
- 8 concurrent requests and a 1,048,576-token per-request limit, subject to available key and value cache memory. The requests share one cache of 1,769,472 tokens on four dedicated 96 GB cards.
- 4,096-token prefill chunks and drafts of up to 7 tokens a round with adaptive verification.
- Text and images. Images go in
image_urlparts of a chat message as base64data:image/png,data:image/jpegordata:image/webpURLs, up to 32 MiB each and up to 4 per request. An image is not resized: Flare pads it on the right and bottom to whole cells of 40 x 40 pixels, and each cell becomes one token. One image may have up to 8,192 cells (a 12-megapixel photo of 4,032 x 3,024 pixels has 7,676), and the images of one request up to 16,384 together. Four requests with images are prepared at a time; a fifth waits for a free image slot. - Sampling defaults of temperature 1.0, top-p 1.0, top-k disabled, min-p 0 and neutral penalties. Thinking
Machines gives no sampling recommendation for Inkling 2 Small; the defaults follow Inkling Small. Request values
override them:
temperaturefrom 0.01 to 2 (a request for 0 is served at 0.01),top_p,top_k(1 gives greedy decoding),min_p,repetition_penalty,presence_penalty,frequency_penaltyandseed.
Concurrency, prefill chunk size and the draft layout are properties of the prepared package. Changing them requires a different package.
Validation
All tests of this section ran on 2026-11-18 on the package of that date, on four cards, with drafts of 7 tokens, except the comparison of the vision tower, which ran on one card before the package was prepared.
- Speed. llm-inference-bench v0.4.29 at default sampling: decode at 1, 2, 4 and 8 requests at once, decode with 32K and 128K tokens of context, and prompt processing at 8K, 64K and 128K tokens. Tokens per second vary with draft acceptance and with the generated text. Of the offered draft tokens, 46 % were accepted in the decode runs with no prior context and 50 % in the runs with 32K and 128K tokens of context.
- Images. Eight test images made for this check, one question each, answered at greedy decoding: a word and a number to read from a sign, three red circles and two blue squares to count, the tallest bar of a four-bar chart to name, the color of an image's left half, and single words to read from four images, the largest 3,200 x 2,000 pixels. All 8 were right one request at a time; 2 of 2 were right when the same words came with two different images, one request after the other; and 4 of 4 were right with four image requests at once. A text-only arithmetic question in the same run was right as well.
- Vision tower. The tower's computation was compared stage by stage with Hugging Face Transformers' Inkling 2 vision code on the same input at each of the six prepared image sizes, from 64 to 32,768 patches. The largest relative L2 difference of any stage was 0.0035.
- Perplexity. 6.21 on a 4,095-token English passage (mean negative log-likelihood 1.8262 nats), the same 6.2105 as the text-only package of 2026-11-17. A build with the checkpoint's BF16 attention projections scores 6.2112.
- Long requests. The longest request measured on this package holds 131,072 tokens. A build of the same recipe on 2026-11-16, prepared for 16 requests, answered a question about a marker placed in a prompt of 1,036,894 tokens.
- Package. Flare verified every stored byte of the package when it wrote it, and the package's signatures when it loaded it.
No divergence from a BF16 reference (KL divergence) was measured, no image benchmark was run, and beyond these tests no other accuracy evaluation was run on this package.
Results of the package of 2026-11-17
The text-only package of 2026-11-17 has the same language model weights. These were measured on it and were not
repeated on the package of 2026-11-18. The card at commit cef114aedb5f681b1660723d640cb31300dffcdf describes
them in full.
- Decode 197.8, 293.2, 468.2 and 645.4 tokens/s at 1, 2, 4 and 8 requests at once, and 172.0 and 161.3 tokens/s with 32K and 128K tokens of context; prompt processing 11,218, 10,554 and 9,497 tokens/s at 8K, 64K and 128K tokens; 50 % and 47 % of the offered draft tokens accepted.
- Coding task, llm-inference-bench's coding-peak test with a prompt that asks for a complete browser game, 8,192
generated tokens, 1 request, median and best of 5 runs: 256.1 and 266.5 tokens/s; 268.8 and 301.0 at
temperature 0; 273.6 and 314.3 with thinking at its lowest level (
reasoning_effortset tononethroughchat_template_kwargs); 318.4 and 337.1 with thinking at its lowest level at temperature 0. - Two test profiles of llm-inference-bench, five runs each with five requests in flight: LAVD 5 of 5 exact, Estonia 5 of 5 pass.
Results of the package of 2026-11-14
These were measured on the package of 2026-11-14, prepared from the publisher's checkpoint, and were not repeated
on the later packages. The card at commit 31af0a35e9d40b0083ad6e8bb765d7ad8837817c describes them in full.
- LAVD 10 of 10 exact, Estonia 10 of 10 pass, hotel lights 10 of 10 exact.
- MMLU-Pro, llm-inference-bench's pinned 1,000-question subset at temperature 0: 86.0 % as the benchmark scores it (860 of 1,000); 84.9 % counting only answers the model stated; 88.1 % on the 964 questions answered within the 131,072-token output limit.
- KLCBench, a Kentucky appellate-brief benchmark out of 100: 82.6 through Kentucky Local Counsel and 66.4 for the model alone, with no fabricated citation in either.
- Perplexity 6.19 on the same passage.
Credits
- Base model: Inkling 2 Small by Thinking Machines (
thinkingmachines/Inkling-2-Small; weight sourcethinkingmachines/Inkling-2-Small-MXFP8, revision086a047e8b7ff2ac902dd52355328ddb66b16914; Apache-2.0). The weights, the vision tower, the draft module (mtp.safetensors), the tokenizer and the chat template are theirs. - Checkpoint: Inkling-2-Small-NVFP4-CSF by Local Inference Lab, Inc.
(
local-inference-lab/Inkling-2-Small-NVFP4-CSF, revision614db8960cbce9e7d6ff04e886f4c2815972007e; Local Inference Lab License 1.0): the NVFP4 experts of all 48 layers, the calibrated static activation scales, the CSF-coded block scales and the MXFP8 feed-forward weights of the draft module. - Scale compression: the CSF scale encoding of Local Inference Lab (Martin Vit and Luke Alonso).
- Serving: Hyperflare, also called Flare (
NVIDIA/hyperflare-inference, Apache-2.0), developed by Local Inference Lab and NVIDIA with the Flare community contributors, among them Luke Alonso, Martin Vit, Jason Cook, Eugene Rakhmatulin and yatesdr. The Inkling 2 recipe builds on Arthur Cheng's Inkling-Small recipe and Triton operators (pull request 191 of that repository). - Image input: the image path of the Inkling 2 recipe and of the Flare server (the padded-grid image
preparation, the class-token image layout and the preparation of the vision tower) is ported from Luke Alonso's
commit
8628b7c57333ccc4aa481f021256dfcab033a0c9of 2026-11-14 in pull request 312 ofNVIDIA/hyperflare-inference(Apache-2.0). The recipe's vision tower classes are copied from Hugging Face Transformers 5.23.0 (Apache-2.0). - Kernels: b12x by Luke Alonso, with Martin Vit, Jason Cook and other contributors (Apache-2.0), in Local Inference Lab's fork of FlashInfer; FlashInfer by NVIDIA and the FlashInfer community (Apache-2.0).
- Model arithmetic: checked against two open-source servers, the vLLM project's Inkling 2 model code (Apache-2.0) and Ke Bao's SGLang pull request 50186 (Apache-2.0). The draft head is driven as the vLLM project's DSpark and DFlash speculator code drives it. Both were read; no code was copied from either.
- Formats and tools: NVFP4 and Model Optimizer (NVIDIA), with which Local Inference Lab packed the experts; MXFP8 from the Open Compute Project's Microscaling Formats specification.
- Speed and test measurement: llm-inference-bench v0.4.29 by Local Inference Lab.
- Accuracy benchmarks of the earlier package: MMLU-Pro by TIGER-Lab ("MMLU-Pro: A More Robust and Challenging
Multi-Task Language Understanding Benchmark", Yubo Wang and others, arXiv:2406.01574; dataset
TIGER-Lab/MMLU-Pro), run through the pinned 1,000-question subset that llm-inference-bench ships; KLCBench by Brandon M. Music (brandonmmusic-max/KLCBench). - Hyperflare port and package: Brandon M. Music.
- Release layout: the split into a model card on
mainand a signed package with its executable on the hardware revision follows Local Inference Lab's Flare releases on Hugging Face. The Dockerfile inrunner/is the runtime stage of Flare's own Dockerfile.
License
The package is licensed under the Local Inference Lab License, Version 1.0 (LICENSE; SPDX
LicenseRef-LIL-1.0), the license of the checkpoint it is prepared from. That license reproduces the terms of
the Apache License, Version 2.0, and adds conditions that restrict them. Two of them matter to most readers: the
checkpoint and substantially similar copies of it, this package among them, may not be uploaded or mirrored
elsewhere, and a page that describes a project built on it begins with the attribution notice that opens this
card. LICENSE holds the terms, and this paragraph does not replace them.
Brandon M. Music publishes this package here with the permission of Local Inference Lab, Inc. under Section 2.4 of that license. The permission covers this repository only.
Inkling 2 Small is by Thinking Machines and is licensed under the Apache License, Version 2.0
(LICENSES/Apache-2.0.txt). Its model card also links an
Acceptable Use Policy. NOTICE names the works, where
they come from and what was changed. The Flare executable in runner/ is licensed under the Apache License,
Version 2.0; its license and notices are beside it.
Model tree for brandonmusic/Inkling-2-Small-NVFP4-CSF-Flare
Base model
thinkingmachines/Inkling-2-Small-MXFP8