tugot17/LFM2-8B-A1B-5.2-packed

LFM2-8B-A1B — Transformers v5 packed-MoE format

This is a format conversion of LiquidAI/LFM2-8B-A1B, not a new model. The weights are numerically identical; only the MoE expert tensor layout differs.

Why this exists

Transformers ≥ 5.0 (huggingface/transformers#41580) packs MoE expert weights into a single 3D tensor per projection instead of per-expert Linear weights:

	Original on-disk format	This repo (Transformers v5 packed)
gate/up	…experts.{i}.w1.weight, …experts.{i}.w3.weight ([I, H] each)	…experts.gate_up_proj ([E, 2*I, H])
down	…experts.{i}.w2.weight ([H, I])	…experts.down_proj ([E, H, I])

For LFM2-8B-A1B: E=32, H=2048, I=1792, so gate_up_proj is (32, 3584, 2048) and down_proj is (32, 2048, 1792) — i.e. out-features-major, matching an inference engine's fused w13_weight/w2_weight layout directly (no transpose needed).

This is the layout an in-memory Transformers v5 model exposes in its state_dict, which is exactly what gets pushed over the update_weights_from_tensor / RLHF weight-sync path. An inference-engine weight loader that only understands the per-expert format will silently skip these tensors, leaving experts uninitialized (garbage output). This repo is a minimal reproducer / test fixture for that path (used to validate LFM2-MoE packed-expert loading in SGLang).

How it was produced

from transformers import AutoModelForCausalLM  # transformers >= 5.2
from safetensors.torch import save_file
import torch
m = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM2-8B-A1B", dtype=torch.bfloat16)
sd = {k: v.contiguous().clone() for k, v in m.state_dict().items()}
save_file(sd, "model.safetensors", metadata={"format": "pt"})

Tokenizer and config are copied unchanged from the base model.