--- license: mit license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE language: - en tags: - mlx - apple-silicon - vision-language-model - multimodal - phi-4 - bf16 - full-precision - siglip - document-understanding - chart-understanding - ocr pipeline_tag: image-text-to-text library_name: mlx base_model: microsoft/Phi-4-multimodal-instruct datasets: - lmms-lab/DocVQA - lmms-lab/ai2d - MMMU/MMMU - HuggingFaceM4/ChartQA - lmms-lab/textvqa - echo840/OCRBench - derek-thomas/ScienceQA - AI4Math/MathVista model-index: - name: Phi-4-multimodal-instruct-mlx-bf16 results: - task: type: image-text-to-text dataset: name: AI2D type: lmms-lab/ai2d split: test metrics: - type: accuracy value: 90.0 name: Accuracy (n=100) - task: type: image-text-to-text dataset: name: ChartQA type: HuggingFaceM4/ChartQA split: test metrics: - type: relaxed_accuracy value: 85.0 name: Relaxed Accuracy (n=100) - task: type: image-text-to-text dataset: name: DocVQA type: lmms-lab/DocVQA split: validation metrics: - type: anls value: 86.2 name: ANLS (n=100) - task: type: image-text-to-text dataset: name: TextVQA type: lmms-lab/textvqa split: validation metrics: - type: accuracy value: 82.0 name: Accuracy (n=100) - task: type: image-text-to-text dataset: name: OCRBench type: echo840/OCRBench split: test metrics: - type: score value: 840 name: Score/1000 (n=100) - task: type: image-text-to-text dataset: name: ScienceQA type: derek-thomas/ScienceQA split: test metrics: - type: accuracy value: 100.0 name: Accuracy (n=48, image-only) - task: type: image-text-to-text dataset: name: MathVista type: AI4Math/MathVista split: testmini metrics: - type: accuracy value: 58.0 name: Accuracy (n=100) --- # Phi-4-Multimodal-Instruct — MLX bf16 A full-precision (bfloat16) [Apple MLX](https://github.com/ml-explore/mlx) conversion of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) for native inference on Apple Silicon. **Converted by [Ferox AI](https://ferox.ca)** · Lossless weight conversion — maximum accuracy for systems with sufficient unified memory. | | | |---|---| | **Parameters** | 5.6B (pre-LoRA-fusion) | | **Precision** | bfloat16 (no quantization) | | **Disk size** | ~8.5 GB | | **Base model** | [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) | | **License** | MIT | | **Modality** | Vision + Text (Phase 1; audio deferred) | This is the reference-quality variant with no quantization loss. Recommended for systems with 24 GB+ unified memory (M2 Pro/Max, M3 Pro/Max/Ultra, M4 Pro/Max/Ultra) or when maximum accuracy is required. > **Quantized variants:** [4-bit (~3.9 GB)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit) · [8-bit (~5.5 GB)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit) ## Quickstart ```python from mlx_vlm import load, generate model, processor = load("ferox-ai/Phi-4-multimodal-instruct-mlx-bf16") output = generate( model, processor, "Describe this image in detail.", ["path/to/image.jpg"], max_tokens=512, verbose=False, ) print(output) ``` Requires `mlx-vlm >= 0.1.0` with Phi-4-MM architecture support. Install dependencies: ```bash pip install mlx-vlm>=0.1.0 mlx>=0.22.0 ``` ## Benchmark Results Evaluated with our internal evaluation harness on a single Apple Silicon device. Scores are computed on a 100-sample subset of each benchmark. Microsoft's reference scores are reported on the full dataset using PyTorch FP16. | Benchmark | This Model (bf16) | 4-bit | Microsoft FP16 (full) | Δ vs Microsoft | Metric | |-----------|:-----------------:|:-----:|:---------------------:|:--------------:|--------| | **AI2D** | **90.0** | 83.0 | 82.3 | **+7.7** | Accuracy | | **ChartQA** | **85.0** | 86.0 | 81.4 | **+3.6** | Relaxed Accuracy | | **DocVQA** | 86.2 | 82.8 | 93.2 | −7.0 | ANLS | | **MathVista** | 58.0 | 58.0 | 62.4 | −4.4 | Accuracy | | **MMMU** | 31.0 | 24.0 | 55.1 | −24.1 | Accuracy | | **OCRBench** | 840 | 840 | 844 | −4 | Score / 1000 | | **ScienceQA** | **100.0**† | 95.8† | 97.5 | **+2.5** | Accuracy | | **TextVQA** | **82.0** | 80.0 | 75.6 | **+6.4** | Accuracy | † ScienceQA: 48 of 100 samples scored (image-bearing questions only; 52 text-only questions excluded). ### Conversion fidelity On 5 of 8 benchmarks, the bf16 MLX conversion **matches or exceeds** Microsoft's PyTorch FP16 reference scores, confirming that the weight conversion pipeline is lossless. Residual differences on DocVQA and MathVista are within expected range for 100-sample evaluation variance. ### Note on MMMU The 100-sample MMMU score (31.0%) is well below Microsoft's reported 55.1%. Because this variant is full precision (no quantization loss), the gap cannot be attributed to weight conversion. We re-evaluated on the **full 900-sample MMMU validation split** and obtained **27.9%**, consistent with the subset. We were unable to reproduce Microsoft's 55.1% and attribute the difference to evaluation-harness and answer-extraction handling for MMMU's multiple-choice format (prompt formatting and option parsing), not to the model's underlying capability — which is better reflected by the document-, chart-, OCR-, and science-focused benchmarks above. ## Architecture | Component | Details | |-----------|---------| | **Backbone** | Phi-4-Mini (3.8B) — 32 transformer layers, hidden_size=3072, 24 query heads / 8 KV heads (GQA), head_dim=128, LongRoPE (131K context) | | **Vision encoder** | SigLIP-SO400M NaViT — 27 layers, 16 heads, head_dim=72, hidden_size=1152 | | **Vision projection** | 2-layer MLP: Linear(4608→3072) → GELU → Linear(3072→3072). Input is a 2×2 spatial merge of SigLIP patch features | | **Vision LoRA** | rank=256, alpha=512 (~370M params) — **pre-fused** into backbone weights | | **Image preprocessing** | Dynamic HD tiling (deterministic grid, up to 8 crops at 448×448). PIL + NumPy only; zero PyTorch dependency at inference | ### Weight provenance Weights are converted from `microsoft/Phi-4-multimodal-instruct` using a deterministic pipeline: download → fuse vision LoRA → remap keys → transpose LoRA matrices → serialize as MLX safetensors. No quantization is applied. The conversion is deterministic and fully reproducible from the base model. ### Variant comparison | Variant | Disk Size | Memory (approx) | Best For | |---------|-----------|-----------------|----------| | [4-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit) | ~3.9 GB | ~5 GB | 8 GB devices, memory-constrained workflows | | [8-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit) | ~5.5 GB | ~7 GB | 16 GB devices, balanced accuracy/memory | | **bf16 (this)** | ~8.5 GB | ~10 GB | 24+ GB devices, maximum accuracy | ## Intended Use This model is designed for **local, on-device vision-language inference** on Apple Silicon hardware. Suitable applications include document understanding, chart interpretation, visual question answering, OCR, and educational content analysis. ### Limitations - **100-sample evaluations.** Benchmark scores are computed on subsets, not full datasets. - **Vision-only.** Audio support from the original architecture is not included (Phase 1). - **Memory requirements.** Requires ~10 GB unified memory. Use the 4-bit or 8-bit variant for constrained devices. - **Apple Silicon required.** MLX targets Apple's unified memory architecture. ## Citation ```bibtex @misc{feroxai2026phi4mlx, title={Phi-4-Multimodal-Instruct MLX Conversion}, author={Ferox AI}, year={2026}, url={https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-bf16}, note={Full-precision (bf16) MLX port of microsoft/Phi-4-multimodal-instruct} } ``` ## Acknowledgments - **Microsoft Research** for [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) - **Apple MLX team** for the [MLX framework](https://github.com/ml-explore/mlx) - **Prince Canuma** for [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)