Inkling-2-Mini in 8.9 GB (2.72 bpp)

thinkingmachines/Inkling-2-Mini (26.4B physical / 5.4B active 3x-looped MoE, 18.89 GB native text weights) compressed to 8.87 GB (2.72 bits per text parameter) by the LookAtMyQuant compiler and Tier-3 SQYFR proximal QAT across all 24 MoE layers, packed for the Apple Silicon MLX runtime.

Evaluated untruncated at 1,536 tokens, it retains 99.2% 3-task mean (345 / 349 = 98.9% pooled) of the BF16 original (192 / 200 = 96.0% GSM8K, 149 / 164 = 90.9% strict HumanEval, 4 / 4 Vision) while cutting KL 2.34x on English/chat/code (0.0208 vs 0.0486) and 29.6% across all 13 languages (0.0571 vs 0.0811) below Unsloth's larger 9.14 GB UD-Q2_K_XL GGUF.

Results (max_new_tokens = 1,536)

All models evaluated on the same BF16 teacher, openai/gsm8k (test[:200], greedy, reasoning_effort="none", 1,536 max new tokens), executable openai/openai_humaneval (all 164 problems, strict <|content_text|> execution), 4 Vision VQA tests (with the untouched 637 MB BF16 vision tower; this repository contains the packed 8.87 GB text model), and 16+13-language TextMix windows:

Model Text GB Text bpp free_KL (en/chat/code) all13_KL GSM8K (200) HumanEval Strict (164) Vision (4) 3-Task Mean vs Teacher
Original BF16 Teacher 18.89 5.80 0.0000 0.0000 193 / 200 (96.5%) 152 / 164 (92.7%) 4 / 4 100.0%
LookAtMyQuant 12.8 GB 12.76 3.92 0.0070 0.0268 193 / 200 (96.5%) 152 / 164 (92.7%) 4 / 4 100.0%
LookAtMyQuant 11.2 GB 11.18 3.43 0.0115 0.0400 190 / 200 (95.0%) 151 / 164 (92.1%) 4 / 4 99.3% (98.9% pooled)
This file (8.9 GB) 8.87 2.72 0.0208 0.0571 192 / 200 (96.0%) 149 / 164 (90.9%) 4 / 4 99.2% (98.9% pooled)
LookAtMyQuant 5.4 GB 5.38 1.65 0.0864 0.2773 174 / 200 (87.0%) 117 / 164 (71.3%) 4 / 4 89.0% (84.5% pooled)
LookAtMyQuant 4.7 GB 4.72 1.45 0.1162 0.3724 188 / 200 (94.0%) 113 / 164 (68.9%) 4 / 4 90.6% (87.4% pooled)
LookAtMyQuant 3.9 GB 3.85 1.18 0.2161 0.6037 179 / 200 (89.5%) 97 / 164 (59.1%) 4 / 4 85.5% (80.2% pooled)

Run it on Apple Silicon

Install the LookAtMyQuant MLX runtime:

pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git

Generate text directly from Hugging Face (on Apple M5 Max, 63.5 tok/s serial and 106.0 tok/s median lossless drafted decode / 125.8 tok/s reasoning / 123.8 tok/s math in 9.41 GB active MLX RAM when paired with the 179 MB 4-bit DSpark drafters):

lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB \
    -p "What is 17 times 23? Show the steps." --stats

lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB \
    --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
    -p "Write a Python function that merges overlapping intervals." --stats

lamq chat shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB \
    --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter

Python API:

from runtime_mlx import Runtime

rt = Runtime.load(
    "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB",
    drafter="shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter",
)
for chunk in rt.stream("Solve 3x + 4y = 31 and 5x - 2y = 17.", max_tokens=256):
    print(chunk.text, end="", flush=True)

How it was made

  1. Empirical looped-MoE knapsack: Measured per-layer and per-expert-bucket ([16, 16, 32, 64] usage-sorted) KL sensitivity across all 48 logical visits (L6โ€“17 looped 3x) with pass-indexed sigmoid fixed routing, allocating 4 KB SRAM hashed block-vector trellis codes (ht2, ht3) and row trellis codes (rtr1, rtr4) with a tr2 floor on the tail lm_head bucket (head.7) to preserve all 13 languages.
  2. Full-MoE proximal QAT (SQYFR): Trained all 24.46B MoE expert weights, 38.8M native steering parameters, and lm_head/attention linear row scales for 2,400 steps (9.83M fresh tokens) against the online BF16 teacher with 48-visit router Bernoulli-KL distillation, 3 full 24-layer Hessian-weighted trellis projection cycles, and a 600-step cosine settle.
Downloads last month
218
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB

Quantized
(171)
this model