complex-kda-ladder

Paper: https://arxiv.org/abs/2609.24797

Code: https://github.com/OpenEuroLLM/ComplexKDA

Authors: Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein

The ComplexKDA scaling ladder: 114 trained models in one repository, one subfolder per cell. Each cell is a complete model directory.

A cell is named <arm>_<rung>_<budget>BT: the arm, the parameter rung, and the token budget it was annealed over -- complex-kda_302M_20BT is ComplexKDA at the 302M rung, annealed over 20 billion tokens. cells.json carries each cell's parameter count and its global batch size.

Arms:

• complex-kda
• complex-kda-hybrid

Every cell here is ComplexKDA and loads through the bundled modeling_complex_kda.py. The baselines the ladder was measured against -- the unsigned KDA gate and the two attention arms -- are not part of this repository.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("openeurollm/complex-kda-ladder")          # shared, at the root
model = AutoModelForCausalLM.from_pretrained(
    "openeurollm/complex-kda-ladder", subfolder="complex-kda-hybrid_1.7B_120BT",
    trust_remote_code=True, dtype="bfloat16")

cells.json lists every cell with its parameter count.

Training data

Every cell is trained on the same stream: the high-quality subset of Nemotron-CC, a 15% downsample holding 319B tokens, encoded with the EleutherAI/gpt-neox-20b tokenizer at context 4096. Each cell's budget is a single epoch of it -- the largest, 120BT, is 38% of the corpus -- so no cell repeats data, and cells at the same rung differ only in how much of the same stream they saw.

These are research checkpoints from a scaling study: base language models with no instruction tuning, no alignment, and no safety filtering beyond the corpus's own.

Tokenizer, and how to start a prompt

The shared gpt-neox-20b tokenizer at the repository root is the one these cells were trained through, with model_max_length set to their trained context. Training packed documents as [text..., <|endoftext|>], so to condition on the start of a document, prefix <|endoftext|>; mid-document continuations need no prefix. On this tokenizer bos_token_id == eos_token_id == 0, so there is only one such token and no BOS/EOS choice to make -- unlike the 1.3B FineWeb-Edu models, whose Llama-2 tokenizer has a distinct <s> that these models' corpora never contain.

For the Triton kernels these models were trained with:

pip install git+https://github.com/OpenEuroLLM/ComplexKDA

Install that from its current main. Every cell here ties its input and output embeddings, and versions before commit 8ebf36f cannot build a tied checkpoint under transformers 5.x: they raise 'list' object has no attribute 'keys' from inside post_init. The bundled code in this repository is unaffected either way, so the portable path works with any version.