# DistilBertModel to sequence classification

**URL:** <https://discuss.huggingface.co/t/distilbertmodel-to-sequence-classification/30220>\
**Category:** 🤗Transformers\
**Created:** [January 23, 2023, 1:00pm UTC](https://discuss.huggingface.co/t/distilbertmodel-to-sequence-classification/30220 "2023-01-23T13:00:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![re90](https://avatars.discourse-cdn.com/v4/letter/r/b9e5f3/32.png) [@re90](https://discuss.huggingface.co/u/re90)\
**Post date:** [January 23, 2023, 1:00pm UTC](https://discuss.huggingface.co/t/distilbertmodel-to-sequence-classification/30220/1 "2023-01-23T13:00:57Z")

</div>

Hi,

I’m trying to create a `DistilBertModel ` model for sequence classification, such that `max_position_embeddings=1024` (otherwise I would have used `DistilBertForSequenceClassification` which is defult to `max_position_embeddings=512` )

I define the model in the following way:

```auto
 configuration = DistilBertConfig(max_position_embeddings=1024)
 model = DistilBertModel(configuration)

```

When forwarding an input to the model in the following way:

```auto
output = model(ids, attention_mask = mask, return_dict=False)[0]

```

such that `ids.shape = (batch_size, 1024)` and `mask.shape = (batch_size, 1024)` the shape of the output is `(batch_size, 1024, 768)` .

My question is: What is the best practice to convert this output into a probability vector over the number of labels such the modified output shape would be `(batch_size, num_labels)`?

I thought of a few options including flattening the current output + an additional FC layer, but I’m not sure this is the best practice.

Thank you in advance 🙂