Instructions to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Trellis
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with Trellis:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- MLX
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling-2-Mini in 8.9 GB (2.72 bpp)
thinkingmachines/Inkling-2-Mini (26.4B physical / 5.4B active 3x-looped MoE, 18.89 GB native text weights) compressed to 8.87 GB (2.72 bits per text parameter) by the LookAtMyQuant compiler and Tier-3 SQYFR proximal QAT across all 24 MoE layers, packed for the Apple Silicon MLX runtime.
Evaluated untruncated at 1,536 tokens, it retains 99.2% 3-task mean (345 / 349 = 98.9% pooled) of the BF16 original (192 / 200 = 96.0% GSM8K, 149 / 164 = 90.9% strict HumanEval, 4 / 4 Vision) while cutting KL 2.34x on English/chat/code (0.0208 vs 0.0486) and 29.6% across all 13 languages (0.0571 vs 0.0811) below Unsloth's larger 9.14 GB UD-Q2_K_XL GGUF.
Results (max_new_tokens = 1,536)
All models evaluated on the same BF16 teacher, openai/gsm8k (test[:200], greedy, reasoning_effort="none", 1,536 max new tokens), executable openai/openai_humaneval (all 164 problems, strict <|content_text|> execution), 4 Vision VQA tests (with the untouched 637 MB BF16 vision tower; this repository contains the packed 8.87 GB text model), and 16+13-language TextMix windows:
| Model | Text GB | Text bpp |
free_KL (en/chat/code) |
all13_KL |
GSM8K (200) | HumanEval Strict (164) | Vision (4) | 3-Task Mean vs Teacher |
|---|---|---|---|---|---|---|---|---|
| Original BF16 Teacher | 18.89 | 5.80 | 0.0000 | 0.0000 | 193 / 200 (96.5%) | 152 / 164 (92.7%) | 4 / 4 | 100.0% |
| LookAtMyQuant 12.8 GB | 12.76 | 3.92 | 0.0070 | 0.0268 | 193 / 200 (96.5%) | 152 / 164 (92.7%) | 4 / 4 | 100.0% |
| LookAtMyQuant 11.2 GB | 11.18 | 3.43 | 0.0115 | 0.0400 | 190 / 200 (95.0%) | 151 / 164 (92.1%) | 4 / 4 | 99.3% (98.9% pooled) |
| This file (8.9 GB) | 8.87 | 2.72 | 0.0208 | 0.0571 | 192 / 200 (96.0%) | 149 / 164 (90.9%) | 4 / 4 | 99.2% (98.9% pooled) |
| LookAtMyQuant 5.4 GB | 5.38 | 1.65 | 0.0864 | 0.2773 | 174 / 200 (87.0%) | 117 / 164 (71.3%) | 4 / 4 | 89.0% (84.5% pooled) |
| LookAtMyQuant 4.7 GB | 4.72 | 1.45 | 0.1162 | 0.3724 | 188 / 200 (94.0%) | 113 / 164 (68.9%) | 4 / 4 | 90.6% (87.4% pooled) |
| LookAtMyQuant 3.9 GB | 3.85 | 1.18 | 0.2161 | 0.6037 | 179 / 200 (89.5%) | 97 / 164 (59.1%) | 4 / 4 | 85.5% (80.2% pooled) |
Run it on Apple Silicon
Install the LookAtMyQuant MLX runtime:
pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git
Generate text directly from Hugging Face (on Apple M5 Max, 63.5 tok/s serial and 106.0 tok/s median lossless drafted decode / 125.8 tok/s reasoning / 123.8 tok/s math in 9.41 GB active MLX RAM when paired with the 179 MB 4-bit DSpark drafters):
lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB \
-p "What is 17 times 23? Show the steps." --stats
lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB \
--drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
-p "Write a Python function that merges overlapping intervals." --stats
lamq chat shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB \
--drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter
Python API:
from runtime_mlx import Runtime
rt = Runtime.load(
"shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB",
drafter="shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter",
)
for chunk in rt.stream("Solve 3x + 4y = 31 and 5x - 2y = 17.", max_tokens=256):
print(chunk.text, end="", flush=True)
How it was made
- Empirical looped-MoE knapsack: Measured per-layer and per-expert-bucket (
[16, 16, 32, 64]usage-sorted) KL sensitivity across all 48 logical visits (L6โ17looped 3x) with pass-indexed sigmoid fixed routing, allocating 4 KB SRAM hashed block-vector trellis codes (ht2,ht3) and row trellis codes (rtr1,rtr4) with atr2floor on the taillm_headbucket (head.7) to preserve all 13 languages. - Full-MoE proximal QAT (
SQYFR): Trained all24.46BMoE expert weights,38.8Mnative steering parameters, andlm_head/attention linear row scales for 2,400 steps (9.83Mfresh tokens) against the online BF16 teacher with 48-visit router Bernoulli-KL distillation, 3 full 24-layer Hessian-weighted trellis projection cycles, and a 600-step cosine settle.
- Downloads last month
- 218
Quantized
Model tree for shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-8.9GB
Base model
thinkingmachines/Inkling-2-Mini