Pure Python PreTrainedTokenizer is Broken

[daskol]

System Info

transformers v4.40.2
tokenizers v0.19.1

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

>>> from transformers import GPT2Tokenizer
>>> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
...
TypeError: unhashable type: 'AddedToken'

Expected behavior

No exception is raised.

[ArthurZucker]

Sorry but I cannot reproduce, could you make sure you are running this on 4.40?