Instructions to use ubergarm/DeepSeek-R1T-Chimera-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="ubergarm/DeepSeek-R1T-Chimera-GGUF",
	filename="DeepSeek-R1T-Chimera-IQ4_KS/DeepSeek-R1T-Chimera-IQ4_KS-00001-of-00035.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
llama cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
llama cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Use Docker

docker model run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF

LM Studio
Jan

vLLM

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/DeepSeek-R1T-Chimera-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/DeepSeek-R1T-Chimera-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF

Ollama
How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Ollama:
```
ollama run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF
```

Unsloth Studio

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/DeepSeek-R1T-Chimera-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/DeepSeek-R1T-Chimera-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ubergarm/DeepSeek-R1T-Chimera-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF
```

Lemonade

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/DeepSeek-R1T-Chimera-GGUF

Run and chat with the model

lemonade run user.DeepSeek-R1T-Chimera-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

`ik_llama.cpp` imatrix Quantizations of tngtech/DeepSeek-R1T-Chimera

This quant collection REQUIRES ik_llama.cpp fork to support advanced non-linear SotA quants. Do not download these big files and expect them to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc!

NOTE ik_llama.cpp can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.

These quants provide best in class quality for the given memory footprint.

Big Thanks

Special thanks to u/un_passant for additional hardware access for this special project and patience as it uploaded!

Shout out to Wendell and the Level1Techs crew, the community Forums, YouTube Channel! BIG thanks for providing BIG hardware expertise and access to run these experiments and make these great quants available to the community!!!

Also thanks to all the folks in the quanting and inferencing community here and on r/LocalLLaMA for tips and tricks helping each other run all the fun new models!

Excited to share and learn together. Thanks!

Quant Collection

So far these are my best recipes offering the great quality in good memory footprint breakpoints.

DeepSeek-R1T-Chimera-IQ4_KS

NOTE: This quant may take a long time to upload, hopefully less than a month lol...

338.456 GiB (4.326 BPW)

type f32: 361 tensors - norms etc.
type q6_0: 61 tensors - attn_k_b (not divisible by 256 so can't use iq6_k)
type iq6_k: 551 tensors - balance of attn, token_embd, output, output_norm, shared experts
type iq4_ks: 174 tensors - ffn_(down|gate|up)_exps routed experts

This quant is designed to take advantage of faster iq4_ks CUDA performance and is not pre-repacked allowing multi-GPU users to offload additional layers easily. If you have enough RAM to hold it all you can use -rtr for run-time-repacking of remaining layers on CPU for improved performance, or use the offline repack tool for a custom solution tailored to you exact hardware configuration.

Quantization

👈 Secret Recipe

#!/usr/bin/env bash

custom="
# Token embedding and output tensors
# note token_embd cannot be repacked quant type
token_embd\.weight=iq6_k
output\.weight=iq6_k
output_norm\.weight=iq6_k

# First 3 dense layers (0-3)
blk\.[0-2]\.attn_k_b.*=q6_0
blk\.[0-2]\.attn_.*=iq6_k
blk\.[0-2]\..*=iq6_k

# All attention, norm weights, and bias tensors for MoE layers (3-60)
# Except blk.*.attn_k_b.weight is not divisible by 256 and no iq6_k so go with q6_0
blk\.[3-9]\.attn_k_b.*=q6_0
blk\.[1-5][0-9]\.attn_k_b.*=q6_0
blk\.60\.attn_k_b.*=q6_0

blk\.[3-9]\.attn_.*=iq6_k
blk\.[1-5][0-9]\.attn_.*=iq6_k
blk\.60\.attn_.*=iq6_k

blk\.[3-9]\.ffn_norm\.weight=iq6_k
blk\.[1-5][0-9]\.ffn_norm\.weight=iq6_k
blk\.60\.ffn_norm\.weight=iq6_k

blk\.[3-9]\.exp_probs_b\.bias=iq6_k
blk\.[1-5][0-9]\.exp_probs_b\.bias=iq6_k
blk\.60\.exp_probs_b\.bias=iq6_k

# Shared Experts (3-60)
blk\.[3-9]\.ffn_down_shexp\.weight=iq6_k
blk\.[1-5][0-9]\.ffn_down_shexp\.weight=iq6_k
blk\.60\.ffn_down_shexp\.weight=iq6_k

blk\.[3-9]\.ffn_(gate|up)_shexp\.weight=iq6_k
blk\.[1-5][0-9]\.ffn_(gate|up)_shexp\.weight=iq6_k
blk\.60\.ffn_(gate|up)_shexp\.weight=iq6_k

# The bulk of the model size is below
# Routed Experts (3-60)
# usually ffn_down is made a bit bigger than ffn_(gate|up) but you do you
blk\.[3-9]\.ffn_down_exps\.weight=iq4_ks
blk\.[1-5][0-9]\.ffn_down_exps\.weight=iq4_ks
blk\.60\.ffn_down_exps\.weight=iq4_ks

blk\.[3-9]\.ffn_(gate|up)_exps\.weight=iq4_ks
blk\.[1-5][0-9]\.ffn_(gate|up)_exps\.weight=iq4_ks
blk\.60\.ffn_(gate|up)_exps\.weight=iq4_ks
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

./build/bin/llama-quantize \
    --imatrix /mnt/models/ubergarm/DeepSeek-R1T-Chimera-GGUF/DeepSeek-R1T-Chimera.imatrix \
    --custom-q "$custom" \
    /media/b/data2/models/ubergarm/DeepSeek-R1T-Chimera-GGUF/DeepSeek-R1T-Chimera-256x21B-BF16-00001-of-00030.gguf \
    /media/b/data2/models/ubergarm/DeepSeek-R1T-Chimera-GGUF/DeepSeek-R1T-Chimera-IQ4_KS.gguf \
    IQ4_KS \
    40

imatrix

Based on some discussions on imatrix methodology I chose the tried and true old school methodology using default context length 512. This is one of the first imatrix generated using the updated imatrix calculation fix for MLA so went lower than Q8_0 on attention tensors for this MLA quant (iq6_k) given the discussions there and recent CUDA speed improvements.

👈 Imatrix Methodology

wget https://gist.githubusercontent.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c/raw/571fda718462de863e5a0171078c175420c7649a/calibration_data_v5_rc.txt

numactl -N 0 -m 0 \
./build/bin/llama-imatrix \
    --verbosity 1 \
    -m /media/b/data2/models/ubergarm/DeepSeek-R1T-Chimera-GGUF/DeepSeek-R1T-Chimera-Q8_0.gguf \
    -f calibration_data_v5_rc.txt \
    -o DeepSeek-R1T-Chimera.imatrix \
    --layer-similarity \
    --ctx-size 512 \
    --numa numactl \
    --threads 40

# NOTE: I actually forgot --layer-similarity otherwise would publish that here. Sorry!

References

Downloads last month: 3

GGUF

Model size

672B params

Architecture

deepseek2

Hardware compatibility

Model tree for ubergarm/DeepSeek-R1T-Chimera-GGUF

Base model

tngtech/DeepSeek-R1T-Chimera

Quantized

(3)

this model

ik_llama.cpp imatrix Quantizations of tngtech/DeepSeek-R1T-Chimera