Instructions to use philipjohnbasile/hy3-demolition-mlx-lite-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use philipjohnbasile/hy3-demolition-mlx-lite-v1 with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("philipjohnbasile/hy3-demolition-mlx-lite-v1") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use philipjohnbasile/hy3-demolition-mlx-lite-v1 with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "philipjohnbasile/hy3-demolition-mlx-lite-v1"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "philipjohnbasile/hy3-demolition-mlx-lite-v1" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use philipjohnbasile/hy3-demolition-mlx-lite-v1 with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "philipjohnbasile/hy3-demolition-mlx-lite-v1"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "philipjohnbasile/hy3-demolition-mlx-lite-v1" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "philipjohnbasile/hy3-demolition-mlx-lite-v1", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use philipjohnbasile/hy3-demolition-mlx-lite-v1 with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "philipjohnbasile/hy3-demolition-mlx-lite-v1"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default philipjohnbasile/hy3-demolition-mlx-lite-v1
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use philipjohnbasile/hy3-demolition-mlx-lite-v1 with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "philipjohnbasile/hy3-demolition-mlx-lite-v1"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "philipjohnbasile/hy3-demolition-mlx-lite-v1" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Hy3-Demolition-MLX lite-v1
Explore the model guide · All public work
Release at a glance
| This artifact | |
|---|---|
| Purpose | A fused LoRA on Tencent Hy3's 295B MoE, keeping all 192 experts per layer. |
| Runtime | The release uses a pinned mlx-lm fork plus its documented FP32 router patch. The card records a 112.3 GB peak on an M5 Max with 128 GB. |
| Status | Recorded runtime; see the evidence and limits below. |
| Tensor download | 111.15 GB (103.52 GiB) of root .safetensors files, including any root sidecars. This is a file-size total, not peak RAM. |
| Read first | The experimental SSD pager is a different serving path. Reduced cache tiers measured on a large Mac are not separate tests on every smaller Mac. |
A standalone, fused MLX model directory built from Tencent Hy3 (295B MoE, 21B active, 192 experts / top-8, 80 layers) for local Apple Silicon use. A LoRA adapter trained on verifier-filtered agent data (coding, repair, tool-call discipline, strict JSON, security, and domain facets) is fused into the quantized weights. No experts are pruned in the lite variant.
This is a research artifact, shared as-is with receipts: no support or fitness claims, Apache-2.0 like its base model. Recorded measurements and experimental projections are distinguished below; the receipts live in the source repo.
Usage
⚠️ Runtime prerequisite — read first. This is a
hy_v3(Tencent Hy3) model.hy_v3is not in mainlinemlx-lmyet (ml-explore/mlx-lm#1211), so stockmlx-lm, LM Studio, and Ollama cannot load it (you'll getValueError: Model type hy_v3 not supported). Run it with the pinned fork:pip install "mlx-lm @ git+https://github.com/eauchs/mlx-lm@a7cc3054b1ff48c19950513b422a66cfed7baa60"Reapply the FP32-router patch documented in RESTORE.md. The pinned environment is the recorded release setup, not a fresh runtime qualification. A future upstream merge or release still needs loader and generation validation. Upstream PR #1211 remained open when checked on September 10, 2026. — For a model that runs on stock mlx-lm / LM Studio today, use the fast sibling
hy3-family-mini-qwen35b-v1.
MLX only. Requires an Apple Silicon Mac with enough unified memory (measured peak 112.3 GB on an M5 Max 128 GB).
mlx_lm.generate --model <this-directory> --prompt "Write a tiny Python fizzbuzz." --max-tokens 512
mlx_lm.chat --model <this-directory>
mlx_lm.server --model <this-directory> --port 8080 # OpenAI-compatible /v1
No adapter path, agent framework, or custom runtime wrapper is required —
the directory is the whole artifact. For clean, direct answers, serve with
--chat-template-args '{"reasoning_effort":"no_think"}'; use high for hard
agent tasks where step-by-step reasoning helps (the model reasons inside
<think:opensource>…</think:opensource> tags). Suggested settings for
coding/agent work: temperature 0–0.2, top-p 1.0.
⚠️ max_tokens too low fails silently, not loudly. With reasoning left
on (the default), a small token budget can be entirely consumed by the
<think:opensource> trace, leaving zero output with no error — this
looks like a clean empty result, not a truncation. Confirmed on the sibling
model in a real production pipeline (a max_tokens: 1600 cap returned zero
results for every input before the cause was found; 7000–9000 fixed it).
If output looks suspiciously empty, raise max_tokens or set
reasoning_effort: no_think before assuming the task legitimately had
nothing to produce.
⚠️ For strict-JSON/structured output, pin repeat_penalty: 1.0 explicitly
in the request — don't trust a server's UI/global default. A nonzero
repeat penalty punishes exactly the repeated structural tokens JSON needs
("name":, "type":, array delimiters), and can silently corrupt output on
batch/array-heavy extraction tasks. Confirmed in production: a serving UI
had repeat_penalty: 1.1 active globally, invisible to the request itself,
on a strict-JSON extraction workload. A pipeline's full sampling contract
(temperature, top-k, repeat-penalty) belongs in every request body, never
assumed from server state — server defaults and UI settings can change
between calls or restarts without you noticing.
Serving multiple concurrent requests: mlx_lm.server (above) does batch
requests, but was measured unreliable at 8-way concurrency on the reap25
tier (dropped ~25% of connections under load) — solid at ≤4 concurrent
(2.72× aggregate, 4/4 completed). For reliable high-concurrency serving,
oMLX is a viable alternative: loads with one dependency
swap (install this project's pinned mlx-lm fork into its venv instead of
its default mainline pin — RESTORE.md has the exact command), measured
3.49× aggregate at 8/8 concurrent with no drops on reap25, plus active
memory-ceiling enforcement and a restart-surviving KV cache. Not a blanket
"faster" claim — single-stream speed was comparable either way in that
test. These numbers were measured on the smaller reap25 tier (87 GB); this
card's larger 112 GB checkpoint should behave the same qualitatively but
hasn't been separately benchmarked.
Source and recipe
- Base:
ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx(mixed-quant MLX checkpoint of Tencent Hy3), used as an AR-only view (MTP sidecar omitted,num_nextn_predict_layers=0). - LoRA: rank 8 on 8 layers (211M trainable params, 0.072%), 200 iterations, batch 1, lr 1e-5, adamw, gradient checkpointing, masked prompts, max-seq 2048, val loss 1.154 → 0.722.
- Training data: 305/16/16 verifier-gated examples (repair, coding, agentic, security, music, perfumery facets), length-normalized so nothing truncates.
- Fused with a streamed lazy fuse (one shard at a time), keeping the stock Hy3 chat template.
Build scripts, receipts, and the full recipe:
https://github.com/PhilipJohnBasile/hy3-demolition-mlx (tag lite-v1).
Experimental SSD streaming
The source project includes an expert pager that reads the same stored weights on demand. The smaller-cache measurements were made on the reap25 artifact on an M5 Max, not independently on this Lite checkpoint or on physical 16/32/64 GB machines. They do not establish a minimum-RAM or output-equivalence guarantee for this release.
See the streaming experiment
and scripts/41_streaming_load.py for the prototype. The recorded fully
resident path above is the default release recipe.
Verification receipts
- Fresh full baseline (suite+hard+brutal) 46/46 (2026-07-08); receipts in the source repo.
All measured on the fused artifact, 2026-07-07, receipts committed under
eval/receipts/ in the source repo:
- Direct
mlx_lm.generatesmoke: correct output, clean EOS stop. - Agent eval 5/5 through
mlx_lm.server/v1/chat/completions(coding ×2, tool-call JSON, repair, strict JSON schema), outputs verified by an automated verifier ensemble (internal tooling; the receipts themselves are committed in the source repo). - Strict-JSON server smoke: parseable JSON,
finish_reason: stop. - Peak inference memory 112.3 GB; warm decode ~7.4 tok/s (measured,
hy3_mtp_smoke.json). First-token latency is high on cold load.
Limitations
- The model learned self-checking and repair habits; it cannot actually execute compilers, tests, shells, or tools. Verification of its outputs still needs a real harness.
- Quantized MoE base: expect the usual quantization artifacts.
- Trained/evaluated primarily in English on agent/code tasks.
- ~7.4 tok/s warm decode on M5 Max (measured) — a usable local daily driver with prompt caching, not a high-throughput server. Cold load pages ~104 GB from disk, so the first response lags.
Runtime support
Supported: mlx_lm.generate, mlx_lm.chat, mlx_lm.server.
Not supported (today): GGUF, llama.cpp, vLLM, SGLang, CUDA serving.
- LM Studio / Ollama: pending — needs the
hy_v3architecture in mainline mlx-lm (ml-explore/mlx-lm#1211). Today these load only on the pinned fork (eauchs/mlx-lm@a7cc3054b1ff48c19950513b422a66cfed7baa60); future mainline and LM Studio versions require separate validation.
- Downloads last month
- 38
2-bit
Model tree for philipjohnbasile/hy3-demolition-mlx-lite-v1
Base model
tencent/Hy3