Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
Abstract
A small hybrid language model using sparse attention and short convolutions achieves faster CPU inference and better benchmark scores than larger conventional models despite training on far less data.
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.
Community
Daedalus-150M is a 160M-parameter language model designed backwards from its deployment target: one user, one token at a time, 4-bit weights, an ordinary CPU. Only 6 of its 18 blocks use full attention; the other 12 use short depthwise convolutions whose recurrent state is two timesteps wide no matter how long the context gets, so two thirds of the network never re-reads a growing KV cache.
Trained from scratch on 59.9B tokens, it scores 47.31 on a five-task benchmark (HellaSwag, ARC-Easy, PIQA, OpenBookQA, WinoGrande) against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-Neo-125M — all trained on 3–6× more data — and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685.
To test the architecture rather than the training recipe, a parameter-matched all-attention twin was trained on identical data, with the winning condition written down before either arm was scored. The hybrid won the pre-registered quality metric by 0.81%, matched the twin on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76× faster at 2048 tokens of context — 2.08× against an external 135M peer. In every measurement the speed advantage is near zero at empty context and grows with depth, which is what the mechanism predicts and what a merely leaner model would not show. A bandwidth-only cost model predicts just 1.17×, so memory volume alone does not explain the gap.
The paper also reports what did not work: an unmitigated 4-bit quality cost after quantisation-aware training failed on its first step, roughly half the convolution channels ending up inert and impossible to prune under the stock runtime's shape checks, and a vocabulary larger than this model size warrants.
Model: https://huggingface.co/Unseen1980/daedalus-150m-instruct
Checkpoints: https://huggingface.co/Unseen1980/daedalus-checkpoints
Code: https://github.com/unseen1980/daedalus
Fixing the target before picking the architecture is the right call — most of these hybrids read like the authors picked a fancy mix and then hunted for a benchmark that flatters it. The cache-bounded convolutions are genuinely clever, but I want numbers I can feel: token/s on a specific CPU, with a real batch size, not a score delta. 47.31 vs 42.20 tells me the model is better at the benchmark, not that it's faster where I'd actually run it. What's the latency at batch 1 on, say, an i5 or an EPYC core? That's the number that decides whether I swap my current setup. Until I see that, the cleverness stays theoretical.
Thanks Aamer for your comment and actually for reading the paper! Fair point. The 47.31 vs 42.20 figure is the quality result, not the speed result. I agree that I should have tied those numbers to an exact host configuration.
For the CPU measurements, I tested on my MacBook Pro with an M3 Pro and 18 GB unified memory. With Q4_0 weights, 8 threads, and 128 generated tokens, Daedalus measured approximately 1,112 / 960 / 739 tok/s at context depths 0 / 512 / 2048, compared with 923 / 664 / 420 tok/s for the parameter-matched dense twin. At a 2048-token context, that is roughly 1.35 ms per decoded token versus 2.38 ms/token. Teh main claim is that on the same machine, with the same runtime, quantisation, thread count, and alternating runs, the hybrid’s advantage grows from 1.20× at empty context to 1.76× at 2048 tokens.
I only had access to the machine used for those measurements, so I could not produce a credible i5/EPYC hardware matrix before release. I agree that a follow up should include a broader CPU matrix!
Thanks again 👍
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decoding (2026)
- Variable Bit-width Quantization: Learning Per-Group Precision for"Bigger-but-Smaller"Language Models (2026)
- Opt.Gear Technical Report (2026)
- Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models (2026)
- Multi-Head Attention Residuals (2026)
- A Controlled Study of Attention-Only Transformers (2026)
- Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2608.20210 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper