# GPT2Tokenizer.decode maps unicode sequences to the same string '�'

**URL:** <https://discuss.huggingface.co/t/gpt2tokenizer-decode-maps-unicode-sequences-to-the-same-string/32453>\
**Category:** 🤗Tokenizers\
**Created:** [February 24, 2023, 4:32pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-decode-maps-unicode-sequences-to-the-same-string/32453 "2023-02-24T16:32:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cbelem](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@Cbelem](https://discuss.huggingface.co/u/Cbelem)\
**Post date:** [February 24, 2023, 4:32pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-decode-maps-unicode-sequences-to-the-same-string/32453/1 "2023-02-24T16:32:23Z")

</div>

Hello everyone,

I have a naive question about tokenizers, particularly GPT2 Tokenizer. I have this encoded a text sentence, and I’ve obtained the token: 29826, which in [GPT2Tokenizer Vocabulary](https://huggingface.co/gpt2/raw/main/vocab.json) corresponds to the Unicode sequence “\u00e6\u0143”.

For some reason, I needed to convert 29826 back to its token, i.e., into text, so I used the following code snippet:

```python
from transformers import GPT2Tokenizer, AutoModelForCausalLM
import torch
import torch.nn.functional as F

tokenizer = GPT2Tokenizer.from_pretrained( "EleutherAI/gpt-neo-125M")
model = AutoModelForCausalLM.from_pretrained( "EleutherAI/gpt-neo-125M")
tokenizer.pad_token_id = tokenizer.eos_token_id

```

I found myself comparing the logits distribution for token\_id 29826 using the code below:

```python
# Version 1. No need to encode because we already have access to token_id
token_ids_1 = torch.tensor([[29826]])
logits_1 = model.forward(token_ids_1).logits.squeeze().detach().numpy()

# Version 2. Need to get input_ids
token = tokenizer.decode([29826])
token_ids_2 = tokenizer(token, return_tensors="pt", add_special_tokens=False).input_ids
logits_2 = model.forward(token_ids_2).logits.squeeze().detach().numpy()

## Visualize the logits distribution
import matplotlib.pyplot as plt
import seaborn as sns

sns.histplot(logits_1, label="Logits 1")
sns.histplot(logits_2, label="Logits 2")
plt.legend()

```

Surprisingly, the logit distributions were different (see attached picture). It appears that having this `tokenizer.decode` conversion step is actually creating the string ‘�’, which is being decoded to the index 4210 (a different unicode sequence `" \u00ef\u00bf\u00bd"`).

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/19a5555208d81e58b508d90cf1106a35c22bed68.png)

In fact, both expressions `tokenizer.decode([4210])` and `tokenizer.decode([29826])` get decoded to the same ‘�’ character instead of their actual unique expression. But I was hoping that they would decode to their actual unicode string.

**Is there any way I can deal with this? Is this expected? I’ve tried tweaking the string to decode to its actual unicode string but failed.**

Environment:

- `transformers` version: 4.26.1
- `tokenizers` version: 0.13.2
- Platform: Linux-5.4.0-113-generic-x86\_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface\_hub version: 0.12.0
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

I executed the code snippets on a Jupyter Notebook (jupyterlab 3.6.1).

---

<div class="post-metadata">

**Author:** ![Cbelem](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@Cbelem](https://discuss.huggingface.co/u/Cbelem)\
**Post date:** [February 24, 2023, 8:46pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-decode-maps-unicode-sequences-to-the-same-string/32453/2 "2023-02-24T20:46:59Z")

</div>

I guess the more appropriate way would be to use: `tokenizer.convert_ids_to_tokens([4210])`

---

<div class="post-metadata">

**Author:** ![Cbelem](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@Cbelem](https://discuss.huggingface.co/u/Cbelem)\
**Post date:** [February 24, 2023, 9:57pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-decode-maps-unicode-sequences-to-the-same-string/32453/3 "2023-02-24T21:57:46Z")

</div>

On the use of `tokenizer.convert_ids_tokens`, I’ve found that. the two expressions below lead to two different strings:

```auto
ts = tokenizer.encode("Hello!\n I can't do this anymore")
# [15496, 0, 198, 314, 460, 470, 466, 428, 7471]

```

and

```auto
tokenizer.convert_ids_to_tokens(ts)
# ['Hello', '!', 'Ċ', 'ĠI', 'Ġcan', "'t", 'Ġdo', 'Ġthis', 'Ġanymore']

```

How should I interpret the difference between the two?

---

<div class="post-metadata">

**Author:** ![Cbelem](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@Cbelem](https://discuss.huggingface.co/u/Cbelem)\
**Post date:** [March 15, 2023, 7:36pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-decode-maps-unicode-sequences-to-the-same-string/32453/4 "2023-03-15T19:36:38Z")

</div>

Based on [https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475:](https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475:)

> In GPT2 and Roberta tokenizers, the space before a word is part of a word, i.e. `"Hello how are you puppetter"` will be tokenized in `["Hello", "Ġhow", "Ġare", "Ġyou", "Ġpuppet", "ter"]`. You can notice the spaces included in the words a `Ġ` here. Spaces are converted in a special character (the `Ġ` ) in the tokenizer prior to BPE splitting mostly to avoid digesting spaces since the standard BPE algorithm used spaces in its process (this can seem a bit hacky but was in the original GPT2 tokenizer implementation by OpenAI).