Instructions to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter --local-dir Inkling-2-Mini-LookAtMyQuant-drafter
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
4-bit DSpark Drafters for Inkling-2-Mini
4-bit MLX quantizations (178.9 MB each, down 3.55x from the 635.8 MB BF16 mtp.safetensors inside thinkingmachines/Inkling-2-Mini) of Inkling-2-Mini's 3-layer DSpark MTP block-8 speculative drafter, for use with the LookAtMyQuant Apple Silicon MLX runtime beside 3.9GB, 4.7GB, 5.4GB, 8.9GB, 11.2GB, or 12.8GB.
The drafter reads 4 target hidden taps (logical visits 5, 23, 41, and 48) and proposes adaptive linear blocks (6 <-> 8, or opt-in 8-node causal trees and N-gram prompt lookup) verified in one batched pass β supporting lossless greedy, exact unbiased temp > 0 speculative sampling (1.38x over serial at temp=0.6, top_p=0.9, top_k=40), and opt-in tight-margin greedy (+31β36%). Live on Apple M5 Max with mtp_4bit.safetensors:
3.9GB(1.18 bpp,SQYFR v38 GB iPhone):75.8 tok/sserial ->113.0 tok/smedian (117.3math,108.5code,3.54 GiBactive /4.57 GiBdecode peak).4.7GB(1.45 bpp,SQYFR v312 GB iPhone + DSpark):69.2 tok/sserial ->129.0 tok/smath (3.25 tok/check,94.7code,4.35 GiBactive /5.35 GiBdecode peak).5.4GB(1.65 bpp,SQYFR v2):1,807 tok/sT=512prefill,75.3 tok/sserial ->124.4 tok/smedian (142.9math,126.5reasoning,5.33 GBactive).8.9GB(2.72 bpp):63.5 tok/sserial ->106.0 tok/smedian (125.8reasoning,123.8math;mtp_4bit_8p9gb_v2.safetensorslifts held-outtpc_bl83.26 -> 3.53).11.2GB(3.43 bpp):2,032 tok/sT=512prefill,62.4 tok/sserial ->103.7 tok/smedian (111.7math).12.8GB(3.92 bpp):2,016 tok/sT=512prefill,60.5 tok/sserial ->112.7 tok/smedian (119.2math1.97x,106.2code).
Run it
pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git
lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-4.7GB \
--drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
-p "What is 17 times 23? Show the steps." --stats
Files
mtp_4bit.safetensors(default): 4-bit (group_size = 64) MLX-quantized stock DSpark weights (norms,fuse_weight,mask_embed, andhidden_outkept in native precision)mtp_4bit_5p4gb_v2.safetensors: 4-bit DSpark drafter distilled onto5.4GBmtp_4bit_8p9gb_v2.safetensors: 4-bit DSpark drafter distilled onto8.9GBconfig.json: model &mtp_configmetadatamanifest.json: drafter format descriptor (inkling2-dspark-4bit-v1)
- Downloads last month
- 460
Quantized
Model tree for shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter
Base model
thinkingmachines/Inkling-2-Mini