SinhalaMMLU-SharedTask/team-cognilex

Team Cognilex – Sinhala MMLU Shared Task System

System summary

• Base model: Qwen3.5-4B.
• What we did: We first fine-tuned Qwen3.5-4B with LoRA on synthetic question–answer pairs generated from Sri Lankan Grade 6–11 textbooks to provide Sinhala syllabus context. We merged this adapter into the base model, then trained a new LoRA adapter on examination MCQs. Our final system uses the merged synthetic-QA model with the MCQ adapter and greedy decoding to predict the correct answer number.
• Total inference parameters: 4,571,730,432 (approximately 4.572B).
• Extra data used for training: 100,370 synthetic direct Q&A pairs generated from textbooks available at the Educational Publications Department, and MCQs extracted from Sri Lankan examination papers. The MCQ dataset contained 46,373 records for the first three epochs and 47,025 records for the fourth epoch after expansion and filtering.

This work was carried out for the ICTER Sinhala MMLU Shared Task 2026, conducted by UCSC and IIT, Sri Lanka.

Team information

• Team name: Congnilex
• CodeBench username: lakshan

System architecture

Models and resources in this submission

Repository	What it is
SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-synthetic-qa-instruct	Qwen3.5-4B fine-tuned on synthetic textbook Q&A, with the adapter merged into the model. This model is the base for MCQ adapter training and final inference.
SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-mcq-lora	MCQ LoRA adapter trained on top of the merged synthetic-QA model.

Dataset creation

Synthetic textbook Q&A dataset

Textbook collection and text extraction

We collected textbooks for Grades 6–11 from the Educational Publications Department. We used a Codex agent integrated with Playwright MCP to scrape and download the textbook PDFs.

We converted the PDFs to text using pypdf, with Sinhala and Tamil Unicode/font support for text extraction. We then divided the extracted textbook text into overlapping chunks using:

CHUNK_CHARS = 6000
OVERLAP_CHARS = 300

Synthetic Q&A generation

We sent each textbook chunk to deepseek-v4-flash (284B) through the Embercloud API platform, with a maximum output length of 15,000 tokens. Our prompt asked the model to generate as many factual question–answer pairs as possible from each chunk while excluding image-based questions.

We used an SQLite database to maintain records of the synthetic Q&A generation process.

Statistic	Value
Grades processed	6 (Grades 6–11)
Subject JSON files	94
Total Q&A pairs	100,370

Examination MCQ dataset

Paper collection

In parallel, we collected MCQ data from Sri Lankan online examination-paper platforms using Codex integrated with Playwright MCP servers. We downloaded available papers from 1990–2026, including:

• A/L past papers and Pesuru papers.
• O/L papers.
• Provincial papers.
• Grade 6–9 school and provincial papers.
• General knowledge papers.

The websites used included:

Source	Link
Alapi Edu	Past-paper hub
AlevelApi	alevelapi.com
Department of Examinations	Evaluation reports
PaperBook	paperbook.lk
North Western Provincial Education	Downloads
e-Thaksalawa	e-thaksalawa.moe.gov.lk/lcms
Past Papers Wiki	pastpapers.wiki
GovDoc	G.C.E. O/L model papers
e-Kalvi	Sinhala-medium O/L papers

MCQ extraction and checking

We uploaded the downloaded papers, along with their answers, to ChatGPT using **four ChatGPT Plus accounts** with **GPT-6.1-Sol in medium reasoning mode**. We also used **one Claude Sonnet 5.5 account** and **one Gemini 3.8 Flash account**. We manually checked samples of the extracted MCQs to verify that the questions and answers were mapped correctly.

Dataset filtering and expansion

The initial MCQ dataset contained more than 47,000 records. Before fine-tuning, we removed duplicate questions and questions that exactly matched the hidden set. We include this filtering step for transparency in our experimental process. The resulting dataset contained 46,373 records.

Because easy-level MCQs were underrepresented compared with medium- and hard-level questions, we collected additional easy questions before the fourth training epoch. After removing duplicates and exact matches, the expanded dataset contained 47,025 records.

Dataset stage	Records
Initial collected dataset	More than 47,000
Filtered dataset for the first three MCQ epochs	46,373
Expanded, filtered dataset for the fourth MCQ epoch	47,025

Subject distribution

Humanities (38.8%) and STEM (29.5%) account for the largest shares of the MCQ dataset.

Difficulty distribution

Medium questions comprise 52.9% of the MCQ dataset, followed by hard (32.8%) and easy (14.3%) questions.

Training methodology

Stage 1: Sinhala syllabus adaptation using synthetic Q&A

We fine-tuned Qwen3.5-4B on the synthetic textbook Q&A dataset using LoRA and Unsloth. This stage was designed to provide Sinhala syllabus context to the language model used in our experiments.

Initial training

We divided the dataset into 97.5% training and 2.5% validation. The initial run was configured for two epochs, with a maximum sequence length of 1,024 tokens.

We used early stopping with:

early_stopping_patience = 4
early_stopping_threshold = 0.001

Early stopping occurred at approximately 11,000 steps.

The initial training configuration was:

output_dir = str(checkpoint_dir)
num_train_epochs = 2
per_device_train_batch_size = 1
per_device_eval_batch_size = 1
gradient_accumulation_steps = 16  # Effective batch size of 16 on one GPU.
learning_rate = 2e-5
warmup_ratio = 0.03
lr_scheduler_type = "cosine"
optim = "adamw_8bit"
weight_decay = 0.01
max_grad_norm = 1.0
bf16 = is_bfloat16_supported()
fp16 = not is_bfloat16_supported()
logging_strategy = "steps"
logging_steps = 25
logging_first_step = True
report_to = "tensorboard"
eval_strategy = "steps"
eval_steps = 250
save_strategy = "steps"
save_steps = 250
save_total_limit = 4
load_best_model_at_end = True
metric_for_best_model = "eval_loss"
greater_is_better = False
max_length = 1024
packing = False
completion_only_loss = True
dataset_num_proc = 1
dataset_kwargs = {"add_special_tokens": False}
gradient_checkpointing = True

The LoRA configuration was:

r = 16
lora_alpha = 16
lora_dropout = 0.05
bias = "none"
target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj",
]
use_gradient_checkpointing = "unsloth"

Continued training after early stopping

After early stopping, we continued fine-tuning the adapter on the training set only, without a validation set, up to a target of three total epochs. We reduced the learning rate to 1e-5 to reduce catastrophic forgetting and retained the same LoRA rank and adapter settings.

The continuation settings were:

TARGET_TOTAL_EPOCHS = 3
NEW_LEARNING_RATE = 1e-5
MICRO_BATCH = 1
GRAD_ACCUMULATION = 16
warmup_ratio = 0.01
lr_scheduler_type = "cosine"
optim = "adamw_8bit"
weight_decay = 0.01
max_grad_norm = 1.0

Merging the model and adapter

We merged the synthetic-QA adapter with the model using:

trainer.model.save_pretrained_merged(
    str(merged_dir),
    tokenizer,
    save_method="merged_16bit",
)

We used this merged model as the base for training the MCQ adapter.

Stage 2: MCQ adapter fine-tuning

First three epochs

We initialized a new LoRA adapter on top of the merged synthetic-QA model and fine-tuned it on 46,373 MCQ records for three epochs, using a learning rate of 2e-5 and a context length of 2,048 tokens.

The MCQ LoRA configuration was:

r = 16
lora_alpha = 32
lora_dropout = 0.5
finetune_vision_layers = False
finetune_language_layers = True
finetune_attention_modules = True
finetune_mlp_modules = True
bias = "none"

The training settings included:

num_train_epochs = 3
learning_rate = 2e-5
max_length = 2048
warmup_ratio = 0.05
lr_scheduler_type = "cosine"
optim = "adamw_8bit"
weight_decay = 0.01
max_grad_norm = 1.0

Fourth epoch

After the first three epochs, we fine-tuned the MCQ adapter for one additional epoch on the expanded dataset of 47,025 records. We reduced the learning rate to 1e-5 and retained the other MCQ training settings.

We observed a small loss reduction during the fourth epoch. We used this final adapter together with the merged synthetic-QA model for inference.

Training overview

Stage	Dataset	Training duration	Learning rate	Context length
Initial synthetic-QA adaptation	100,370 Q&A pairs, split 97.5% / 2.5%	Configured for 2 epochs; early stopping at approximately 11,000 steps	2e-5	1,024
Continued synthetic-QA adaptation	Training partition only	Continued up to 3 total epochs	1e-5	1,024
Initial MCQ adapter training	46,373 MCQs	3 epochs	2e-5	2,048
Final MCQ adapter training	47,025 MCQs	1 additional epoch	1e-5	2,048

How to run

For inference, use the merged synthetic-QA model with the MCQ LoRA adapter.

Our model predicts the correct MCQ answer number in the range 1–5. We use greedy decoding with a maximum of 8 output tokens.

• GPU used: NVIDIA A100 80 GB on Google Colab.
• Approximate runtime on the hidden test set: 13–15 minutes.

Parameter count of the system

Component	Parameter count
Base parameters	4,539,265,536 (4.539B)
LoRA parameters	32,464,896 (32.46M)
Total parameters	4,571,730,432 (4.572B)

Hardware and training time

We used Google Colab with an NVIDIA A100 80 GB GPU for training and inference.

Training stage	Time (HH:MM:SS)
Direct Q&A model: first 11,000 steps	18:32:38
Direct Q&A model: continuation up to the third epoch	12:39:29
MCQ adapter: first three epochs	15:14:40
MCQ adapter: fourth epoch	06:34:17
Total training time	53:01:04

Repository structure

• data/
  • All_Grades_direct_QA_dataset.json
  • MCQ_dataset.json
• training/
  • MCQ_preprocessing_and_visualization.ipynb
  • McqAdapterTraining/
    • MCQ_Qwen3_5_4B_ZeroShot_LoRA_Colab_v2.ipynb
    • Qwen3_5_4B_MCQ_v2_Continue_3_to_4epochs_LR1e_5__v2 (1).ipynb
  • SyntheticDirectQATraining/
    • Qwen3_5_4B_Sinhala_FRESH_16bit_LoRA_Training (3).ipynb
    • Qwen3_5_4B_Sinhala_Continue_From_11k_LowLR_3or4_Epochs (2).ipynb
• inference.ipynb
• predictions.jsonl
• architecture.jpeg
• subject-distribution.png
• difficulty-distribution.png
• README.md
• .gitattributes

Achieved results

Evaluation metric	Result
Overall score	68.55
Easy-level score	61.21
Medium-level score	67.96
Hard-level score	76.49

Team members

• Lakshan Cooray
• Vihindi Kothalawala
• Binara Mendis
• Deshan Sumanathilaka

Citation

Please cite the SinhalaMMLU benchmark paper:

@inproceedings{pramodya-etal-2025-sinhalammlu,
  title     = {SinhalaMMLU: A Comprehensive Benchmark
               for Evaluating Multitask Language
               Understanding in Sinhala},
  author    = {Pramodya, Ashmari and Nelki, Nirasha and
               Shalinda, Heshan and Liyanage, Chamila and
               Sakai, Yusuke and Pushpananda, Randil and
               Weerasinghe, Ruvan and Kamigaito, Hidetaka
               and Watanabe, Taro},
  booktitle = {Proceedings of the 2025 Conference on
               Empirical Methods in Natural Language
               Processing},
  year      = {2025},
  pages     = {32943--32961},
  url       = {https://aclanthology.org/2025.emnlp-main.1673/}
}