You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model reads computer-support messages for one application (TicketAI) and is published as evidence for a method, not as a general-purpose assistant. Tell us who you are and what you intend to use it for.

Log in or Sign Up to review the conditions and access this model content.

support-slm v1 and v2: GGUF for llama.cpp

The support-slm v1 and v2 LoRA adapters converted for llama.cpp, plus the Q8_0 base model it runs on. TicketAI serves exactly these files: the base with the v1 LoRA (port 8860) and the base with the v2 LoRA as an off-topic judge (port 8861).

File What it is sha256
qwen2.5-0.5b-instruct-Q8_0.gguf Qwen2.5-0.5B-Instruct @ 7ae557604adf, converted with llama.cpp convert_hf_to_gguf.py --outtype q8_0 6294a715454ca6d0…
support-slm-v1-lora-F16.gguf the LoRA adapter, convert_lora_to_gguf.py --outtype f16 9cb2858fae6429fc…
support-slm-v2-lora-F16.gguf the v2 LoRA adapter (off-topic judge), same conversion 9623c80ca2df5156…

Run it

llama-server -m qwen2.5-0.5b-instruct-Q8_0.gguf --lora support-slm-v1-lora-F16.gguf -c 4096 --host 127.0.0.1 --port 8860

Send the chat-formatted prompt with the exact system_prompt.txt and greedy settings (temperature 0, top_k 1, stop at <|im_end|>). TicketAI sends the prompt as token ids made by the base model's own chat template, so the text is tokenised exactly as in training.

v2 runs the same way with --lora support-slm-v2-lora-F16.gguf. See the main card for v1 vs v2.

How v1 compares with the PyTorch adapter

Same 420 held-out messages, same scoring:

PyTorch adapter (fp32) this GGUF pair
Whole record right 0.481 0.555
Issue 0.717 0.756
Requested action 0.906 0.923
Seconds per message (laptop CPU) 6.649 1.689

The two give word-for-word identical output on 305 of 420 held-out messages; where they differ, this pair scored slightly higher here. That is a small-sample effect, not evidence that quantisation improves the model. See the main card for the full results and limits.

License: Apache-2.0, as Qwen2.5-0.5B-Instruct.

Downloads last month
-
GGUF
Model size
0.5B params
Architecture
qwen2
Hardware compatibility
Log In to add your hardware

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Smriti10raj/support-slm-GGUF

Adapter
(1,059)
this model

Space using Smriti10raj/support-slm-GGUF 1