Inkling 2 Small for Hyperflare

This model is based on Inkling-2-Small-NVFP4-CSF by Local Inference Lab, Inc., a non-profit organization, available at https://huggingface.co/local-inference-lab/Inkling-2-Small-NVFP4-CSF. Inkling-2-Small-NVFP4-CSF is licensed under the Local Inference Lab License, Version 1.0.

Built on the work of Thinking Machines, Local Inference Lab, NVIDIA, Luke Alonso, Martin Vit, Arthur Cheng, the Flare, FlashInfer, vLLM and Hugging Face Transformers projects and Ke Bao. See Credits and CREDITS.md.

This repository publishes a signed Hyperflare (Flare) runtime package for Inkling 2 Small by Thinking Machines, prepared for tensor-parallel serving on four GPUs. The package revision holds the prepared weights, the compiled kernels, the captured execution graphs, the signed package metadata and the Flare executable that serves it. Download the revision for the hardware and start the executable: nothing is compiled or tuned on the machine that runs it. Transformers, vLLM, SGLang and llama.cpp cannot load this format.

The model serves text and images, takes up to 8 requests at once and up to 1,048,576 tokens per request, and answers on an OpenAI-compatible API under the model name Inkling-2-Small.

The package of 2026-11-18

The revision gb202 holds a new package since 2026-11-18. It replaces the text-only package of 2026-11-17.

  • The vision tower of Inkling 2 Small is prepared: a request can carry up to 4 images (see Serve).
  • The language model is unchanged. Its weights are byte for byte those of the package of 2026-11-17, and the perplexity on the test passage is the same, 6.2105.
  • The vision tower, its working memory for the largest image and four image slots take 2.0 GiB of each card that the text-only package gives to the key and value cache. Four 96 GB cards hold 1,769,472 tokens of key and value cache beside the weights (the package of 2026-11-17 held 2,236,416).
  • The included Flare executable is new: it adds the image path (see The executable).
  • 18 of the 210 package files changed: the metadata primary, the four weights/tp4/rank-*/misc.safetensors files, which now also hold the vision tower, and the compiled kernels, captured graphs and initial state of each card. The other 192 files are byte for byte those of the package of 2026-11-17.
  • Flare's package cache lists this package as Inkling-2-Small-NVFP4-CSF-Static-Flare. The model name on the API is unchanged.
  • The text-only package of 2026-11-17, with the larger key and value cache, and its model card remain in the repository's history: package at commit ca92afae02b1e466756ab223c9adf3a496fbdfc6 (pass it to hf download --revision), card at commit cef114aedb5f681b1660723d640cb31300dffcdf. The package of 2026-11-14 is at commit 085a16677047bc931f85fe2d62dcd1eaf2267e56.

Results

Measured on 2026-11-18 on four RTX PRO 6000 Blackwell cards (96 GB each), with the model's draft head proposing 7 tokens a round. The measurements ran on the package before it was signed (signing replaces only the metadata primary model.safetensors), served by the Flare executable included in the revision.

Result
Decode, 1 request 179.3 tokens/s
Decode, 2 requests at once (total) 286.6 tokens/s
Decode, 4 requests at once (total) 471.9 tokens/s
Decode, 8 requests at once (total) 615.6 tokens/s
Decode, 1 request with 32K tokens of context 177.1 tokens/s
Decode, 1 request with 128K tokens of context 152.8 tokens/s
Prompt processing, 8K-token prompt 11,199 tokens/s
Prompt processing, 64K-token prompt 10,487 tokens/s
Prompt processing, 128K-token prompt 9,153 tokens/s
Image questions: 8 one at a time, 2 in sequence, 4 at once, and 1 text question in the same run 15 of 15 right
One request with a 3,200 x 2,000-pixel image (4,033 prompt tokens), start to answer 4.5 seconds
Key and value cache beside the weights 1,769,472 tokens
Start, weights from NVMe to the four cards 60.5 seconds
Perplexity on a 4,095-token English passage 6.21

The gb202 setup has the prepared settings and the commands. Validation describes each test.

About Hyperflare

Hyperflare, also called Flare (NVIDIA/hyperflare-inference), is a local inference server developed by Local Inference Lab and NVIDIA for NVIDIA GPUs of the SM120 and SM121 generation, in workstations and compact systems such as DGX Spark. It is an open-source research prototype, built for long conversations, coding agents and repeated tool use, and meant to make the memory, bandwidth and compute of such a machine go further while generation stays responsive. Its authors describe it as a specialized option for the models and machines it covers, not a replacement for production inference engines such as vLLM.

The idea is to do the expensive work before serving starts:

  • Prepare, then serve. A preparation stage written in Python on PyTorch, FlashInfer and the b12x kernels loads the model, tunes its kernels on the cards it will run on by racing the variants of each kernel and keeping the fastest, compiles the forward passes and captures them as CUDA graphs. A runtime written in Rust then serves those graphs without Python.
  • Recipes. Each model has a recipe that owns the model's computation. The runtime owns scheduling, cache storage and request handling, and stays small and shared between models.
  • Planned memory and state. Buffers and capacities are fixed ahead of time. A conversation's state, the key and value cache together with any recurrent state of the model, is treated as one thing, so that it can be reused, branched and resumed correctly.
  • Tensor parallelism. On several cards every layer is split across the cards, and the cards exchange partial results directly with each other, here over PCIe.

This repository holds the result of that preparation for Inkling 2 Small on four cards.

The weights

The package is prepared from local-inference-lab/Inkling-2-Small-NVFP4-CSF at revision 614db8960cbce9e7d6ff04e886f4c2815972007e, Local Inference Lab's checkpoint of Inkling 2 Small for tensor-parallel serving. That checkpoint holds the routed experts of all 48 decoder layers in NVFP4 with calibrated static activation scales, stores the NVFP4 block scales with CSF, a lossless encoding, and takes its weights from thinkingmachines/Inkling-2-Small-MXFP8 at revision 086a047e8b7ff2ac902dd52355328ddb66b16914. With static scales no value depends on a token's own activations at run time, so each card runs its share of an expert without asking the other cards.

Inkling 2 Small is a mixture-of-experts model by Thinking Machines with 48 layers, hidden width 4,608, 384 routed experts of which 8 run for each token, 200,058 vocabulary entries, a context window of up to 1,048,576 tokens, a draft module for speculative decoding and a vision tower for image input.

What the package changes in that checkpoint:

  1. The attention input and output projections of the decoder layers are stored as MXFP8, converted from the checkpoint's BF16 values during preparation. Perplexity on the test passage is 6.2105 with them and 6.2112 with the BF16 projections.
  2. The 15 MXFP8 feed-forward matrices of the draft module are stored as BF16. Preparation proves for every value that the BF16 number equals the MXFP8 one exactly before it accepts the file.
  3. The vision tower keeps the checkpoint's BF16 values, with a full copy on each card. It has 23 transformer layers of width 1,024, reads 20 x 20-pixel patches and merges each 2 x 2 group of patches into one token for the language model.
  4. The weights are laid out as one share per card in Flare's package format, with the compiled kernels, the captured execution graphs and the signed metadata beside them.

The routed experts keep the checkpoint's NVFP4 values, block scales and static activation scales. Nothing was retrained, recalibrated or quantized again apart from item 1.

The revision is a self-contained package for tensor parallelism over four cards: a model.safetensors metadata primary, companion files under weights/ and deployments/, and the publisher profile. Keep the primary and every companion together under their original names.

Pick a platform

Revision Hardware Package Platform setup
gb202 Linux x86-64; 4x RTX PRO 6000 Blackwell 96 GB (SM120) on one host 347 GB gb202 setup

The main revision holds this model card, the license files, the notice, the credits and the publisher profile. Download the revision for the hardware; its README covers download, launch, Docker and the serving settings.

Flare fundamentals

The executable

Use the Flare executable included in the revision, runner/flare-linux-x86_64. Inkling 2 is not part of a released Flare build at the time of writing, so released Flare images and installations cannot serve this package. The included executable was built on 2026-11-17 with the image path, which the executable of the earlier packages lacks. It reads this package, verifies its signatures and served every measurement on this card. The revision also holds a small Dockerfile that packages the same executable; it compiles nothing.

Download and verify

Download the revision with the Hugging Face hf CLI, always naming the revision. At the first start Flare checks the package signature against the publisher profile in the revision, shows the publisher and the model license, and asks for acceptance. The signature covers the metadata primary and every companion, so a missing or renamed file fails the check. A trust store kept between starts remembers the acceptance.

Serve

The package needs four GPUs. Rank zero serves the OpenAI-compatible API at /v1, with health at /health; clients use the model name Inkling-2-Small. Reasoning is returned apart from the answer, in reasoning_content.

The prepared package has these capabilities:

  • 8 concurrent requests and a 1,048,576-token per-request limit, subject to available key and value cache memory. The requests share one cache of 1,769,472 tokens on four dedicated 96 GB cards.
  • 4,096-token prefill chunks and drafts of up to 7 tokens a round with adaptive verification.
  • Text and images. Images go in image_url parts of a chat message as base64 data:image/png, data:image/jpeg or data:image/webp URLs, up to 32 MiB each and up to 4 per request. An image is not resized: Flare pads it on the right and bottom to whole cells of 40 x 40 pixels, and each cell becomes one token. One image may have up to 8,192 cells (a 12-megapixel photo of 4,032 x 3,024 pixels has 7,676), and the images of one request up to 16,384 together. Four requests with images are prepared at a time; a fifth waits for a free image slot.
  • Sampling defaults of temperature 1.0, top-p 1.0, top-k disabled, min-p 0 and neutral penalties. Thinking Machines gives no sampling recommendation for Inkling 2 Small; the defaults follow Inkling Small. Request values override them: temperature from 0.01 to 2 (a request for 0 is served at 0.01), top_p, top_k (1 gives greedy decoding), min_p, repetition_penalty, presence_penalty, frequency_penalty and seed.

Concurrency, prefill chunk size and the draft layout are properties of the prepared package. Changing them requires a different package.

Validation

All tests of this section ran on 2026-11-18 on the package of that date, on four cards, with drafts of 7 tokens, except the comparison of the vision tower, which ran on one card before the package was prepared.

  • Speed. llm-inference-bench v0.4.29 at default sampling: decode at 1, 2, 4 and 8 requests at once, decode with 32K and 128K tokens of context, and prompt processing at 8K, 64K and 128K tokens. Tokens per second vary with draft acceptance and with the generated text. Of the offered draft tokens, 46 % were accepted in the decode runs with no prior context and 50 % in the runs with 32K and 128K tokens of context.
  • Images. Eight test images made for this check, one question each, answered at greedy decoding: a word and a number to read from a sign, three red circles and two blue squares to count, the tallest bar of a four-bar chart to name, the color of an image's left half, and single words to read from four images, the largest 3,200 x 2,000 pixels. All 8 were right one request at a time; 2 of 2 were right when the same words came with two different images, one request after the other; and 4 of 4 were right with four image requests at once. A text-only arithmetic question in the same run was right as well.
  • Vision tower. The tower's computation was compared stage by stage with Hugging Face Transformers' Inkling 2 vision code on the same input at each of the six prepared image sizes, from 64 to 32,768 patches. The largest relative L2 difference of any stage was 0.0035.
  • Perplexity. 6.21 on a 4,095-token English passage (mean negative log-likelihood 1.8262 nats), the same 6.2105 as the text-only package of 2026-11-17. A build with the checkpoint's BF16 attention projections scores 6.2112.
  • Long requests. The longest request measured on this package holds 131,072 tokens. A build of the same recipe on 2026-11-16, prepared for 16 requests, answered a question about a marker placed in a prompt of 1,036,894 tokens.
  • Package. Flare verified every stored byte of the package when it wrote it, and the package's signatures when it loaded it.

No divergence from a BF16 reference (KL divergence) was measured, no image benchmark was run, and beyond these tests no other accuracy evaluation was run on this package.

Results of the package of 2026-11-17

The text-only package of 2026-11-17 has the same language model weights. These were measured on it and were not repeated on the package of 2026-11-18. The card at commit cef114aedb5f681b1660723d640cb31300dffcdf describes them in full.

  • Decode 197.8, 293.2, 468.2 and 645.4 tokens/s at 1, 2, 4 and 8 requests at once, and 172.0 and 161.3 tokens/s with 32K and 128K tokens of context; prompt processing 11,218, 10,554 and 9,497 tokens/s at 8K, 64K and 128K tokens; 50 % and 47 % of the offered draft tokens accepted.
  • Coding task, llm-inference-bench's coding-peak test with a prompt that asks for a complete browser game, 8,192 generated tokens, 1 request, median and best of 5 runs: 256.1 and 266.5 tokens/s; 268.8 and 301.0 at temperature 0; 273.6 and 314.3 with thinking at its lowest level (reasoning_effort set to none through chat_template_kwargs); 318.4 and 337.1 with thinking at its lowest level at temperature 0.
  • Two test profiles of llm-inference-bench, five runs each with five requests in flight: LAVD 5 of 5 exact, Estonia 5 of 5 pass.

Results of the package of 2026-11-14

These were measured on the package of 2026-11-14, prepared from the publisher's checkpoint, and were not repeated on the later packages. The card at commit 31af0a35e9d40b0083ad6e8bb765d7ad8837817c describes them in full.

  • LAVD 10 of 10 exact, Estonia 10 of 10 pass, hotel lights 10 of 10 exact.
  • MMLU-Pro, llm-inference-bench's pinned 1,000-question subset at temperature 0: 86.0 % as the benchmark scores it (860 of 1,000); 84.9 % counting only answers the model stated; 88.1 % on the 964 questions answered within the 131,072-token output limit.
  • KLCBench, a Kentucky appellate-brief benchmark out of 100: 82.6 through Kentucky Local Counsel and 66.4 for the model alone, with no fabricated citation in either.
  • Perplexity 6.19 on the same passage.

Credits

  • Base model: Inkling 2 Small by Thinking Machines (thinkingmachines/Inkling-2-Small; weight source thinkingmachines/Inkling-2-Small-MXFP8, revision 086a047e8b7ff2ac902dd52355328ddb66b16914; Apache-2.0). The weights, the vision tower, the draft module (mtp.safetensors), the tokenizer and the chat template are theirs.
  • Checkpoint: Inkling-2-Small-NVFP4-CSF by Local Inference Lab, Inc. (local-inference-lab/Inkling-2-Small-NVFP4-CSF, revision 614db8960cbce9e7d6ff04e886f4c2815972007e; Local Inference Lab License 1.0): the NVFP4 experts of all 48 layers, the calibrated static activation scales, the CSF-coded block scales and the MXFP8 feed-forward weights of the draft module.
  • Scale compression: the CSF scale encoding of Local Inference Lab (Martin Vit and Luke Alonso).
  • Serving: Hyperflare, also called Flare (NVIDIA/hyperflare-inference, Apache-2.0), developed by Local Inference Lab and NVIDIA with the Flare community contributors, among them Luke Alonso, Martin Vit, Jason Cook, Eugene Rakhmatulin and yatesdr. The Inkling 2 recipe builds on Arthur Cheng's Inkling-Small recipe and Triton operators (pull request 191 of that repository).
  • Image input: the image path of the Inkling 2 recipe and of the Flare server (the padded-grid image preparation, the class-token image layout and the preparation of the vision tower) is ported from Luke Alonso's commit 8628b7c57333ccc4aa481f021256dfcab033a0c9 of 2026-11-14 in pull request 312 of NVIDIA/hyperflare-inference (Apache-2.0). The recipe's vision tower classes are copied from Hugging Face Transformers 5.23.0 (Apache-2.0).
  • Kernels: b12x by Luke Alonso, with Martin Vit, Jason Cook and other contributors (Apache-2.0), in Local Inference Lab's fork of FlashInfer; FlashInfer by NVIDIA and the FlashInfer community (Apache-2.0).
  • Model arithmetic: checked against two open-source servers, the vLLM project's Inkling 2 model code (Apache-2.0) and Ke Bao's SGLang pull request 50186 (Apache-2.0). The draft head is driven as the vLLM project's DSpark and DFlash speculator code drives it. Both were read; no code was copied from either.
  • Formats and tools: NVFP4 and Model Optimizer (NVIDIA), with which Local Inference Lab packed the experts; MXFP8 from the Open Compute Project's Microscaling Formats specification.
  • Speed and test measurement: llm-inference-bench v0.4.29 by Local Inference Lab.
  • Accuracy benchmarks of the earlier package: MMLU-Pro by TIGER-Lab ("MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark", Yubo Wang and others, arXiv:2406.01574; dataset TIGER-Lab/MMLU-Pro), run through the pinned 1,000-question subset that llm-inference-bench ships; KLCBench by Brandon M. Music (brandonmmusic-max/KLCBench).
  • Hyperflare port and package: Brandon M. Music.
  • Release layout: the split into a model card on main and a signed package with its executable on the hardware revision follows Local Inference Lab's Flare releases on Hugging Face. The Dockerfile in runner/ is the runtime stage of Flare's own Dockerfile.

License

The package is licensed under the Local Inference Lab License, Version 1.0 (LICENSE; SPDX LicenseRef-LIL-1.0), the license of the checkpoint it is prepared from. That license reproduces the terms of the Apache License, Version 2.0, and adds conditions that restrict them. Two of them matter to most readers: the checkpoint and substantially similar copies of it, this package among them, may not be uploaded or mirrored elsewhere, and a page that describes a project built on it begins with the attribution notice that opens this card. LICENSE holds the terms, and this paragraph does not replace them.

Brandon M. Music publishes this package here with the permission of Local Inference Lab, Inc. under Section 2.4 of that license. The permission covers this repository only.

Inkling 2 Small is by Thinking Machines and is licensed under the Apache License, Version 2.0 (LICENSES/Apache-2.0.txt). Its model card also links an Acceptable Use Policy. NOTICE names the works, where they come from and what was changed. The Flare executable in runner/ is licensed under the Apache License, Version 2.0; its license and notices are beside it.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for brandonmusic/Inkling-2-Small-NVFP4-CSF-Flare

Paper for brandonmusic/Inkling-2-Small-NVFP4-CSF-Flare