SlayerLab/fabryka-english-250m-e01-sft-v1

Fabryka English 250M E01 — experimental SFT v1

An experimental instruction fine-tune of the 249,944,832-parameter Fabryka E01 base. The base has only 50M pretraining targets and remains substantially undertrained. This release preserves the original 32,768-token vocabulary and 2,048-token context. It is not a proven coding improvement or a competitive general coding assistant.

Verified before/after benchmarks

Benchmark	E01 base	This SFT
BananaMind Base Bench 1.1 overall Elo	842	853
BananaMind Base Bench correct	106/350	113/350
Code completion correct	5/50	6/50
Code completion Elo	729	724
Quantitative correct	16/50	9/50
ARC Easy acc_norm	29.42%	29.34%
ARC Challenge acc_norm	23.21%	21.59%
PIQA acc_norm	51.47%	51.85%
HellaSwag acc_norm	24.97%	24.86%

The coding category gained three correct items and lost two. Its Elo decreased because item difficulty and weights differ. These 50 examples do not establish a meaningful coding gain; quantitative and ARC Challenge performance regressed. This is a mixed experimental result. Base Bench measures continuation choice, not executable code correctness or chat quality. Its Elo is not comparable to BananaMind Instruct Bench Elo.

The four standard tasks used lm-eval 0.4.13, Transformers 5.3.0, PyTorch 2.11.0, FP32 on RTX 3090, batch 8, zero-shot, context 2048, no BOS and no chat template. Canonical task hashes and dataset revisions were pinned to the base evaluation. All saved sample IDs, choice scores, accuracies and standard errors were verified.

BananaMind Base used the original official runner and dataset revision d4aade51312889e8580963e1ce960c6eaef1a450, FP32, batch 2 records, no BOS/chat, mean conditional token log-probability. This run used CUDA; E01 used CPU, so small numerical differences are possible. All 350 original records, choices, predictions, weighted scores and fixed-item Elo calculations were verified. No benchmark items are redistributed here. INT and native CORE have not been evaluated for this SFT; the base model's scores must not be attributed to these weights.

Standard verification · BananaMind category breakdown · Paired category comparison

BananaMind Instruct Bench 1.1

The complete 300-task run scored 296 overall Elo, 5/300 passed (1.67%), and 1.14% weighted score. This evaluates this SFT checkpoint, not E01 base.

Category	This SFT passed	This SFT Elo	BananaMind Chat reported Elo
Overall	5/300	296	888
General	4/120	340	720
Multi-turn	0/75	318	911
System prompts	1/60	516	793
Context recall	0/30	537	1194
Code	0/15	667	1337

BananaMind-2-Pro-Preview-Chat reports 98/300 overall and 11/15 code. Those are publisher self-reports; the comparison model was not rerun locally. Category Elo values have a rating prior and different item difficulties, so a nonzero Elo does not imply any passed tasks.

Protocol: unmodified official runner, dataset revision 40494cb4a9224bfd78722968efd2bff440e08186, RTX 3090, BF16, native chat template, greedy generation, repetition penalty 1.1, seed 42, KV cache enabled, per-item generation limits and model context 2048. All 300 item identities, prompts, scoring metadata, pass judgments, weighted scores and Elo aggregates were checked against the pinned source. There were no context overflows; 191 generations hit their limit and 109 ended naturally. Code grading checks syntax and patterns; it does not execute programs.

Access was obtained after SFT training. A post-hoc lexical overlap screen flagged one common-fact item in one training conversation; this is not proof of benchmark copying. Excluding that item gives 297 Elo, 5/299 passed, a non-official sensitivity result. Semantic and short-prompt overlap are not ruled out. Gated task contents and model transcripts are not redistributed.

Aggregate results and verification

Fine-tuning

One epoch, 509 AdamW updates, 8,144 training conversations, 2,499,600 assistant prediction targets. The selected conversations have equal coding-source/general counts; token proportions are not necessarily equal. Source: SmolTalk revision f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc, with self-oss-instruct as the coding source. Whole conversations longer than 1,024 tokens were excluded. A separate 256-conversation validation set was held out. All labels were checked: loss covers assistant content and EOS only. Peak LR 5e-5, 5% warmup, cosine decay to 10%, AdamW betas (0.9, 0.95), weight decay 0.01, gradient clipping 1.0, seed 42, microbatch 2 and effective batch 16 conversations, length bucketing, BF16 autocast with FP32 master weights.

Held-out code loss fell 5.615742 -> 3.039462; general instruction loss fell 4.860617 -> 4.122322. These are validation-loss measurements, not task accuracy.

Preparation removed 24 conversations with lexical overlap against available CORE, MMLU or BananaMind Base references. Exact normalized conversation deduplication was applied; semantic and repository-level split separation are not guaranteed. The gated Instruct benchmark was unavailable during training; the post-hoc screen is described above. Source-tag selection does not guarantee every coding answer is executable Python. Training answers were not independently executed.

Training result · Checkpoint verification

Usage

Requires custom model code (trust_remote_code=True). Use Transformers 5.3.0 and PyTorch 2.11.0 for the evaluated runtime. The saved tokenizer includes the training chat template. Example:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "SlayerLab/fabryka-english-250m-e01-sft-v1"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).to("cuda")
model.eval()
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a short Python function that adds two numbers."}],
    add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Expect repetition, incorrect answers and unreliable code. Review and test generated code. The exported weights exactly match the final training checkpoint (290 stored tensors; tied output embedding checked separately). Model weights SHA256: ddc598302ef545e41db513e35df40935bb0bcf03c7a8947cc99214fefa9b5ca6.