# Further pre-train roberta model

**URL:** <https://discuss.huggingface.co/t/further-pre-train-roberta-model/271>\
**Category:** Beginners\
**Created:** [July 14, 2020, 8:28am UTC](https://discuss.huggingface.co/t/further-pre-train-roberta-model/271 "2020-07-14T08:28:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mr-nlp](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@mr-nlp](https://discuss.huggingface.co/u/mr-nlp)\
**Post date:** [July 14, 2020, 8:28am UTC](https://discuss.huggingface.co/t/further-pre-train-roberta-model/271/1 "2020-07-14T08:28:21Z")

</div>

I have gone through this code [train from scratch](https://colab.research.google.com/drive/1Whs8Rn2ZFH78-Cw_rtxUbwYcqAcaU47k) and understood how to pre-train a model from scratch. I have the following doubts in this code

- What does `block_size` in `LineByLineTextDataset` represent?

- If I want to further pretrain Robert-base model (instead of training from scratch) using my own corpus, what are the changes I have to make in the above code besides the following changes

```auto
from transformers import RobertaForMaskedLM, RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
model = RobertaForMaskedLM('roberta-base') 

```

I am aware that I need not to train tokenizer from scratch.  
@thomwolf @julien-c

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [July 14, 2020, 12:12pm UTC](https://discuss.huggingface.co/t/further-pre-train-roberta-model/271/2 "2020-07-14T12:12:56Z")

</div>

Hi @mr-nlp, I think you can use the same `run_language_modelling` script to further pre-train roberta, just provide your own datasets.

`block_size` is used for `max_length`