Quark-150M-128K-Zero

Quark-150M-128K-Base trained with reinforcement learning (GRPO) directly, with no supervised fine-tuning stage: a small chat model that can think in a private monologue and call tools.

Research artifact. This model is released for research on small language models: architecture, training dynamics, long context and reinforcement learning from a base model. It is not a product and has not been evaluated for production use. Its only safety training is a small set of refusal tasks during RL, and its safety has not been evaluated. Like any small language model, it makes factual and reasoning errors and can produce biased or harmful text. Do not use it for decisions that affect people, or deploy it without your own evaluation and safeguards.

Parameters 165.3M (140.0M without embeddings)
Layers 16 (12 local, 4 global)
Context 131,072 tokens
Training tokens ~506B pretraining, then RL
Vocabulary 32,772 (embedding table padded to 32,896)
Weights float32 safetensors

What this checkpoint is

This is the RL model. It was trained with GRPO starting from Quark-150M-128K-Base, with no supervised fine-tuning stage ("zero", as in RL-from-base). It answers in the chat format, can think first in a [MONOLOGUE] (it decides when, or you can ask it to with enable_thinking=True), and can call tools.

The Quark-150M family

Model What it is
Quark-150M-32K-Raw Pre-anneal checkpoint: end of the constant learning-rate phase (~472B tokens, trained at up to 4K context)
Quark-150M-32K-Base Base model after the 32K-context anneal (~500B tokens)
Quark-150M-128K-Base Base model extended to 128K context (+10B tokens)
Quark-150M-128K-Zero Reinforcement learning (GRPO) straight from the 128K base, without supervised fine-tuning

Architecture

  • Decoder-only transformer hybrid, 16 layers, width 768, 12 attention heads (head size 64), tied input/output embeddings.
  • Local layers (12): sliding-window attention over the last 256 tokens, with RoPE, QK-norm and a learned attention sink per head. A double-gated short causal convolution (kernel 4) runs in parallel, and the two are blended per channel.
  • Global layers (4, every fourth layer): full causal grouped-query attention (12 query heads, 4 key/value heads) with RoPE and QK-norm.
  • Gated value residuals: every attention after the first layer mixes its values with the first layer's values, through a per-token, per-head gate.
  • Residual stream: depth-scaled residual updates with an RMSNorm before and after every sublayer (sandwich norm). The stream is kept in float32 even when the weights are in bfloat16.
  • MLP: gated MLP with a squared-ReLU gate, hidden size 2048.
  • Small inference state: only the 4 global layers keep a full key/value cache. Local layers keep 255 tokens of keys/values and a 3-token convolution state.
  • Tokenizer: byte-level BPE with 32,768 tokens; every chat tag is a single token.

Training

  • Algorithm: GRPO with group-relative advantages and a token-level loss, on-policy samples at temperature 1, no KL penalty and no reference model. Under 1,300 steps on 8Γ—H100, stopped early for budget while its scores were still improving; the learning rate was annealed over the final 3%.
  • Rewards: about 95% of reward groups were scored by programmatic checks: exact answers, format and instruction rules, unit tests, and file checks inside sandboxes. The rest were scored by LLM judges: chat quality, creative writing, second opinions on safety refusals, and rendering of SVG/HTML answers.
  • Tasks: arithmetic and math word problems, commonsense, world-state tracking, theory of mind, perception, format and instruction following, system prompts, long-context retrieval, identity, values and safety, chat, creative writing, code, and tool use. Tool use covered simulated tools, live web search, and shell/file tasks in real sandboxes, written both in Quark's own tool format and in the formats of common coding-agent harnesses.
  • No supervised fine-tuning: every token the model was trained on was its own sample, apart from short code-written hints (e.g. given solution steps on hard reasoning tasks, or the opening of a tool call).

Results after RL

These come from training rollouts and our in-house task suite, not from held-out benchmarks.

  • Simple terminal tasks in 15 coding-agent harness formats (real commands in real sandboxes, training rollouts): 2.6% solved at the start of harness training, 64.5% over the last 100 steps of the run. Harness formats added late in the run started at 42% to 54% solved.
  • Task suite (15 families, four samples per task at temperature 1, step 50 to the final step): well-formed replies 71% to 96%, replies cut off at the length limit 11.7% to 0.5%, correct answers 12.8% to 36.4%.

Usage

Requires transformers>=5.23 (tested with 5.23.0) and trust_remote_code=True (the modeling code ships with the weights).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "empero-ai/Quark-150M-128K-Zero"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda")

messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
  • Thinking: apply_chat_template(..., enable_thinking=True) opens a [MONOLOGUE] and the model thinks before it answers. Without add_generation_prompt, the model chooses for itself whether to think, call a tool or answer.
  • Stopping: generation stops at [/ASSISTANT], [/TOOL_CALL] or <|endoftext|> (set in generation_config.json).
  • Tools: pass tools=[...], either Hugging Face/OpenAI-style function specs or {"name", "description", "parameters": {arg: {"type": ...}}}. When the model stops at [/TOOL_CALL], run the call, append {"role": "tool_result", "content": result}, and generate again.
  • Sampling: the model was trained on its own samples at temperature 1.0. Greedy decoding and lower temperatures also work.

Chat format

Turns are wrapped in tags, and each tag is a single special token:

<|endoftext|>[SYSTEM]
You are quark, a helpful assistant made by Empero AI
[/SYSTEM]
[USER]
What is 2 + 3?
[/USER]
[MONOLOGUE]
Two plus three is five.
[/MONOLOGUE]
[ASSISTANT]
5
[/ASSISTANT]

[SYSTEM] and [MONOLOGUE] (the model's private thinking) are optional. <|endoftext|> marks the start of a document; the tokenizer and the chat template add it.

The 128K models also have tool-call tokens. Tools are offered in the system turn, and the template writes them for you from tools=[...]. The model calls a tool with [TOOL_CALL] {"name": ..., "arguments": {...}} [/TOOL_CALL]. The caller runs the tool and appends the result as a [TOOL_RESULT] ... [/TOOL_RESULT] turn (role tool_result, or tool), then the model continues.

Sample completions

Generated with the code above in float32, one run per prompt with the settings shown: nothing was re-rolled or selected. Generation stops at <|endoftext|>, [/ASSISTANT] or [/TOOL_CALL], or at the token limit. Special tokens are shown as text; the prompt is everything up to where the model's output begins.

Chat (greedy, up to 200 new tokens)

Prompt:

[USER]
Hi! Who are you?
[/USER]
[ASSISTANT]

Output:

I am quark, and I am here to chat with you.
[/ASSISTANT]

Chat (greedy, up to 200 new tokens)

Prompt:

[USER]
Explain in two sentences why the sky is blue.
[/USER]
[ASSISTANT]

Output:

The sky looks blue because sunlight is made of all colors and when it hits the air, the blue light scatters in all directions more than the others.
[/ASSISTANT]

Chat (greedy, up to 200 new tokens)

Prompt:

[USER]
Write a haiku about autumn.
[/USER]
[ASSISTANT]

Output:

The leaves turn gold and red, falling from the trees like a gentle rain.
[/ASSISTANT]

Chat with thinking (enable_thinking=True) (greedy, up to 200 new tokens)

Prompt:

[USER]
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
[/USER]
[MONOLOGUE]

Output:

Say the ball costs $1.10 more than the bat.
So the bat costs $1.10 + $1.00 = $2.10.
Answer: $2.10.
[/MONOLOGUE]
[ASSISTANT]
$2.10
[/ASSISTANT]

Chat with a web-search tool offered (the model chooses how to continue) (greedy, up to 200 new tokens)

Prompt:

[SYSTEM]
You are quark, a helpful assistant made by Empero AI

[TOOLS]
You have the following tools available:
[
{"name": "web_search", "description": "Search the web.", "parameters": {"query": {"type": "string"}}}
]

To use a tool, write a tool call in this format:
[TOOL_CALL]
{"name": <tool name>, "arguments": {<argument>: <value>, ...}}
[/TOOL_CALL]

Example:
[TOOL_CALL]
{"name": "web_search", "arguments": {"query": "<query>"}}
[/TOOL_CALL]

The result comes back in a [TOOL_RESULT] block. Then call another tool, or answer in [ASSISTANT]. You may think in [MONOLOGUE] before or after a call.
Only call the tools listed above, with exactly their arguments. Never write a [TOOL_RESULT] yourself. If no tool is needed, answer directly.
[/TOOLS]
[/SYSTEM]
[USER]
Who won the most recent Nobel Prize in Literature?
[/USER]

Output:

[TOOL_CALL]
The answer is Literature, Nobel laureate Elena Ferrante.
[/ASSISTANT]

Limitations and intended use

  • A ~165M-parameter model: little world knowledge, frequent factual and reasoning errors, and confident hallucinations.
  • Trained mainly on English. Other languages appear in the pretraining data but are weaker.
  • Little safety training (a small set of refusal tasks during RL) and no safety evaluation. Outputs can be wrong, biased or inappropriate.
  • Released as-is for research. Expect rough edges, and evaluate it on your own task before relying on anything it says.
  • Tool use is unreliable beyond simple tools: live web search did not work well during training, and the model sometimes writes an answer inside a tool call (see the samples).
  • Long-context accuracy fell during RL (from 35% to 25% on our long-context tasks). For long inputs, compare it with Quark-150M-128K-Base on your task.
  • Thinking is not always better: on some tasks (for example, questions about what other people know or believe) forcing a monologue lowers accuracy.

License

The weights and code are released under the Apache License 2.0.


Compute for this work was partly sponsored by lium.io.

Downloads last month
422
Safetensors
Model size
0.2B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for empero-ai/Quark-150M-128K-Zero

Finetuned
(1)
this model
Quantizations
1 model