add model card

README.md

@@ -0,0 +1,92 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.8-27B
+tags: [mlx, quantized, mixed-precision, 2bit, 4bit, auto-round]
+pipeline_tag: image-text-to-text
+---
+
+# Qwen3.8-27B MLX 4-bit with 100% of FFN at 2-bit
+
+One rung of a five-model ladder built to measure **decode throughput vs. FFN bit-width**
+on Apple Silicon. Accuracy was deliberately not tuned — these exist to answer one question:
+
+> Is MLX's 2-bit `qmv` kernel as efficient as the 4-bit one?
+
+## This model
+
+Base scheme is uniform **4-bit, group_size 64, affine**, matching
+[`mlx-community/Qwen3.8-27B-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit)
+(498 quantized modules, vision tower bf16, MTP dropped). On top of that, **192 of the
+192 FFN tensors are dropped to 2-bit** (still g64/affine). Attention, `lm_head` and
+`embed_tokens` stay at 4-bit.
+
+- Layers with all three FFN projections at 2-bit: **64 (L0-63)**
+- Layers only partially converted: **0**
+
+| | |
+|---|---|
+| 2-bit FFN tensors | 192 / 192 |
+| On disk | 11.78 GB |
+| Weights read per decoded token | 10.13 GB |
+| Bandwidth-bound ceiling vs. baseline | **1.422x** |
+
+**That last number is arithmetic, not a measurement.** It is
+`baseline_bytes / this_model_bytes`, assuming batch-1 decode is purely memory-bandwidth
+bound. `embed_tokens` is excluded from the read figure because decoding gathers a single
+row rather than streaming the matrix. **No tok/s has been measured on any hardware.**
+Real measurements, when they exist, belong below this line.
+
+## The full ladder
+
+| model | 2-bit FFN tensors | disk | ceiling |
+|---|---|---|---|
+| [`test-4bit`](https://huggingface.co/hancheolp/test-4bit) | 0 / 192 | 16.05 GB | 1.000x |
+| [`test-4bit25`](https://huggingface.co/hancheolp/test-4bit25) | 48 / 192 | 14.98 GB | 1.080x |
+| [`test-4bit50`](https://huggingface.co/hancheolp/test-4bit50) | 96 / 192 | 13.92 GB | 1.174x |
+| [`test-4bit75`](https://huggingface.co/hancheolp/test-4bit75) | 144 / 192 | 12.85 GB | 1.286x |
+| [`test-4bit100`](https://huggingface.co/hancheolp/test-4bit100) | 192 / 192 | 11.78 GB | 1.422x |
+
+Even at 100% FFN coverage the ceiling is 1.42x, and quantizing *everything* to 2-bit would
+only reach 1.80x. The g64 metadata (fp16 scale + fp16 bias = 0.5 bpw) does not shrink with
+bit-width, so 4-bit is really 4.5 bpw and 2-bit is 2.5 bpw.
+
+## Which tensors go to 2-bit
+
+Selected by ascending KL sensitivity, using the per-tensor sweep published in
+[`mlx-community/Qwen3.8-27B-OptiQ-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-OptiQ-4bit)
+(`optiq/sensitivity.json`). FFN sensitivity in this model falls monotonically with depth —
+mean KL is 0.01584 for L0-7 and 0.00072 for L56-63, a 22x spread — so the least-sensitive
+tensors all sit near the output. All three FFN projections hold the same parameter count,
+so coverage alone fixes size and speed; the ranking only decides which tensors take the
+damage.
+
+## Build
+
+Quantized with [AutoRound](https://github.com/intel/auto-round) 0.15.0 in plain RTN mode
+(`iters=0`, `disable_opt_rtn=True`, data-free), exported via `--format mlx`, then repaired.
+
+The repair step is not optional. AutoRound's MLX exporter leaves 97 layers unquantized:
+
+- `embed_tokens` — `SUPPORTED_LAYER_TYPES` is `(Linear, Conv2d, Conv1D)`; `nn.Embedding`
+  entries are dropped by the layer-config resolver.
+- `linear_attn.in_proj_a` / `in_proj_b` (96 tensors, shape `[48, 5120]`) —
+  `_is_mlx_quantizable()` requires `out_dim % 64 == 0`. MLX imposes no such rule, and
+  `mlx-community/Qwen3.8-27B-4bit` quantizes all 96.
+
+Those 97 are filled in afterwards with `mx.quantize` at 4-bit/g64, so the packing is
+bit-exact MLX rather than a reimplementation of the affine formula. The exporter also omits
+`"mode": "affine"` and emits ~57 stray `false` entries for vision layers; both are fixed.
+
+## Usage
+
+```python
+from mlx_vlm import load, generate
+model, processor = load("hancheolp/test-4bit100")
+```
+
+## Caveats
+
+- Accuracy is unmeasured. This rung is not recommended for real use.
+- Plain RTN, no calibration. Not representative of AutoRound's tuned modes.
+- MTP is dropped, as in the mlx-community conversion. For speculative decoding see
+  [`mlx-community/Qwen3.8-27B-MTP-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-4bit).