kamal-018/maithili-clm-124m

Maithili Causal Language Model (109.5M)

A decoder-only transformer pretrained from scratch on the Maithili language corpus.

Architecture

Component	Detail
Type	Decoder-only causal transformer
Parameters	~109.5M
Layers	12
Hidden dim	768
Attention heads	12
Head dim	64
Position encoding	RoPE (Rotary Position Embedding)
Normalization	RMSNorm
Feed-forward	SwiGLU
Vocab size	32,000
Max sequence length	1024
Tokenizer	Byte-level BPE (purpose-built for Maithili)

Training

Detail	Value
Training data	AiventraLab/maithili-corpus (Maithili gold corpus)
Total training tokens	~983,040,000
Steps	30000
Effective batch size	32,768 tokens
Precision	fp16 mixed precision
Optimizer	AdamW (betas=0.9/0.95, weight_decay=0.1)
LR schedule	Cosine with linear warmup (500 steps)
Peak LR	3e-4
Gradient checkpointing	Yes
Hardware	1x NVIDIA T4 (16GB)

Evaluation

Metric	Value
Final validation loss	3.1215
Final validation perplexity	22.68

Validation Loss Over Training

[EVAL] step 26000: val_loss=3.1731, val_ppl=23.88 [EVAL] step 27000: val_loss=3.2049, val_ppl=24.65 [EVAL] step 28000: val_loss=3.2059, val_ppl=24.68 [EVAL] step 29000: val_loss=3.2060, val_ppl=24.68 [EVAL] step 30000: val_loss=3.1897, val_ppl=24.28