Pythia-410M Dolly LoRA on Modal
A compact instruction-tuning project that fine-tunes the standard, non-deduplicated
EleutherAI/pythia-410m model on
databricks/databricks-dolly-15k
using 16-bit LoRA and Modal GPU compute.
Project summary
| Item | Value |
|---|---|
| Base model | EleutherAI/pythia-410m |
| Dataset | databricks/databricks-dolly-15k |
| Training method | FP16 LoRA |
| Maximum sequence length | 768 tokens |
| Epochs | 3 |
| Training GPU | Modal A10 |
| Inference GPU | Modal T4 |
| Completed optimizer steps | 2,760 |
| Measured training time | 52 minutes 40 seconds |
| Training throughput | 13.962 samples/second |
| Final adapter path | /data/outputs/pythia-410m-dolly-lora/final-adapter |
The project saves checkpoints, the final PEFT adapter, tokenizer files, and training
metadata to a persistent Modal Volume named pythia-410m-dolly-output.
Results
Observed test generations:
Simple explanation
Prompt
Explain artificial intelligence in simple terms.
Response
Artificial Intelligence (AI) is the term for a new generation of computer
systems that are capable of performing tasks without human assistance. This can
be done by combining artificial neural networks with computational intelligence
algorithms to automate and improve the efficiency of our everyday life. AI
technology has already been used in areas such as robotics, video games,
healthcare, and manufacturing.
Closed factual question
Prompt
What is the capital of France?
Response
Paris.
Creative-writing limitation
Prompt
Write a short poem about a robot exploring the Moon.
Response
I love to explore, it's the best way to meet new people.
The model performs well on simple explanations and basic factual questions, but creative constraint-following remains unreliable at this model size.
Project layout
pythia-410m-dolly/
├── train.py
├── test_model.py
├── README.md
└── .venv/
Persistent files inside the Modal Volume:
/data/
├── cache/
│ └── huggingface/
└── outputs/
└── pythia-410m-dolly-lora/
├── checkpoint-*/
├── final-adapter/
│ ├── adapter_config.json
│ ├── adapter_model.safetensors
│ ├── tokenizer.json
│ ├── tokenizer_config.json
│ └── special_tokens_map.json
├── trainer_state.json
└── training_summary.json
Requirements
- Python 3.11 or newer
- A Modal account
- Modal CLI authentication
- Internet access for the initial package and model downloads
Local environment
mkdir -p ~/pythia-410m-dolly
cd ~/pythia-410m-dolly
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install --upgrade modal
modal setup
Training configuration
The training script uses approximately the following settings:
MODEL_NAME = "EleutherAI/pythia-410m"
DATASET_NAME = "databricks/databricks-dolly-15k"
MAX_LENGTH = 768
NUM_TRAIN_EPOCHS = 3
LEARNING_RATE = 2e-4
PER_DEVICE_TRAIN_BATCH_SIZE = 4
GRADIENT_ACCUMULATION_STEPS = 4
LORA_R = 16
LORA_ALPHA = 32
LORA_DROPOUT = 0.05
The effective training batch size is 16 examples per optimizer step.
The LoRA adapter targets Pythia/GPT-NeoX projection modules:
target_modules = [
"query_key_value",
"dense",
]
Training labels mask the instruction and context tokens with -100, so loss is
calculated only over response tokens.
Run training
Check the script before launching a paid GPU container:
python -m py_compile train.py
Start a detached Modal run:
modal run --detach train.py
Modal's --detach option keeps the remote application running if the local
process exits or disconnects.
Persistent storage
The same Modal Volume cannot be mounted at two different paths in one function. This project mounts it once:
VOLUME_ROOT = "/data"
volumes = {
VOLUME_ROOT: output_volume,
}
Outputs and caches then use separate subdirectories:
OUTPUT_DIR = "/data/outputs/pythia-410m-dolly-lora"
CACHE_DIR = "/data/cache/huggingface"
Test the adapter on Modal
Run the default prompt:
modal run test_model.py
Run a custom prompt:
modal run test_model.py \
--prompt "Explain artificial intelligence in simple terms."
Run deterministic greedy decoding:
modal run test_model.py \
--prompt "What is the capital of France?" \
--temperature 0 \
--max-new-tokens 80
Run sampled decoding:
modal run test_model.py \
--prompt "Write exactly four short lines about a robot on the Moon." \
--temperature 0.45 \
--top-p 0.85 \
--top-k 30 \
--repetition-penalty 1.12 \
--max-new-tokens 100
At inference time, the script loads the original base model and then attaches the saved LoRA weights:
base_model = AutoModelForCausalLM.from_pretrained(
"EleutherAI/pythia-410m",
torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(
base_model,
ADAPTER_DIR,
is_trainable=False,
)
Download the trained adapter
mkdir -p downloaded-output
modal volume get \
pythia-410m-dolly-output \
outputs/pythia-410m-dolly-lora/final-adapter \
downloaded-output/
Inspect downloaded files:
find downloaded-output -type f -printf '%p %k KB\n'
Training metrics
The completed run reported:
train_runtime: 3160.6921 seconds
train_samples_per_second: 13.962
train_steps: 2760
epoch: 3.0
Late-run training losses were mostly around 1.90 to 2.14. Gradient norms
were generally stable, mostly near 0.7 to 1.3, with no obvious sign of
gradient explosion in the supplied logs.
Limitations
- The model is small and may ignore multi-part or creative instructions.
- It can generate inaccurate or outdated factual claims.
- Dolly 15K is English-only and does not make the model reliably multilingual.
- LoRA instruction tuning does not replace the base model's knowledge.
- The model has not undergone dedicated safety alignment.
- Outputs should be reviewed before use in consequential settings.
Dataset and model licensing
- Pythia-410M is published under the Apache 2.0 license.
- Dolly 15K is published under CC BY-SA 3.0.
- Review both upstream licenses before redistributing a merged model, adapter, or derivative dataset.
- Final model license: CC BY-SA 3.0.
This repository should not imply endorsement by EleutherAI, Databricks, Hugging Face, or Modal.
References
Model tree for RexTRO111/InstructPythia-410M-LoRA-v1
Base model
EleutherAI/pythia-410m