TRL documentation
빠른 시작
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v1.17.1).
빠른 시작
TRL은 Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO) 등의 기법을 사용하여 foundation model을 post-training하는 데 사용되는 포괄적인 라이브러리입니다.
빠른 예제
TRL의 가장 인기 있는 트레이너로 즉시 시작해 보세요. 각 예제는 빠른 실험을 위해 컴팩트한 모델을 사용합니다.
Supervised Fine-Tuning
from trl import SFTTrainer
from datasets import load_dataset
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B",
train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()Group Relative Policy Optimization
accuracy_reward에는 math-verify가 필요하며, pip install trl[math_verify]로 설치할 수 있습니다.
from trl import GRPOTrainer
from datasets import load_dataset
from trl.rewards import accuracy_reward
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
reward_funcs=accuracy_reward,
)
trainer.train()Direct Preference Optimization
from trl import DPOTrainer
from datasets import load_dataset
trainer = DPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()보상 모델링
from trl import RewardTrainer
from datasets import load_dataset
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
trainer = RewardTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
train_dataset=dataset,
)
trainer.train()명령줄 인터페이스
코드를 완전히 건너뛰고 터미널에서 직접 학습을 시작하세요:
# SFT: Fine-tune on instructions
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
--dataset_name trl-lib/Capybara
# DPO: Align with preferences
trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized
# Reward: Train a reward model
trl reward --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized다음 단계
📚 자세히 알아보기
- SFT Trainer - SFT 완벽 가이드
- DPO Trainer - 선호도 정렬
- GRPO Trainer - 그룹 상대 정책 최적화
🚀 확장
💡 예시
문제 해결
메모리 부족?
batch size를 줄이고 최적화를 활성화합니다:
SFT
DPO
GRPO
training_args = SFTConfig(
per_device_train_batch_size=1, # Start small
gradient_accumulation_steps=8, # Maintain effective batch size
)training_args = DPOConfig(
per_device_train_batch_size=1, # Start small
gradient_accumulation_steps=8, # Maintain effective batch size
)training_args = GRPOConfig(
per_device_train_batch_size=1, # Start small
gradient_accumulation_steps=8, # Maintain effective batch size
num_generations=4, # Reduce from default 8 (GRPO generates num_generations completions per prompt)
max_completion_length=512, # Tune based on task; longer sequences cost more memory
)Loss가 감소하지 않나요?
학습률을 조정해 보세요:
training_args = SFTConfig(learning_rate=2e-5) # Good starting point추가 도움이 필요하면 GitHub에서 이슈를 생성하세요.
Update on GitHub