[`LlamaTokenizerFast`] Refactor default llama #28881

ArthurZucker · 2024-02-06T03:43:58Z

What does this PR do?

from transformers import LlamaTokenizerFast, AddedToken
tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", legacy=False, from_slow=True)
tokenizer.add_tokens([AddedToken("<REPR_END>", rstrip=True, lstrip=True)], special_tokens=False)
tokenizer.tokenize("<REPR_END>inform<s>. Hey.       .")
['<REPR_END>', 'in', 'form', '<s>', '.', '▁Hey', '.', '▁▁▁▁▁▁', '▁.']
tokenizer.tokenize("inform<s>. Hey.       .")
['in', 'form', '<s>', '.', '▁Hey', '.', '▁▁▁▁▁▁', '▁.']

This requires huggingface/tokenizers#1476 to be merged.
Finally FIXES ALL THE remaining tokenizer issues!

Fixes #29617 as well (will update Gemma one)
fixes #28577
fixes #29617
fixes #29626
fixes #29694
fixes #29868
fixes #29872
fixes #30416
enabled by huggingface/tokenizers#1476

HuggingFaceDocBuilderDev · 2024-02-06T04:02:31Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

src/transformers/convert_slow_tokenizer.py

…tor-default-llama

ArthurZucker · 2024-04-18T09:42:43Z

Coming next release!

…tor-default-llama

LysandreJik

Sounds great to me

src/transformers/models/llama/tokenization_llama_fast.py

src/transformers/convert_slow_tokenizer.py

ArthurZucker · 2024-04-23T11:56:57Z

src/transformers/convert_slow_tokenizer.py

@@ -1329,7 +1329,7 @@ def tokenizer(self, proto):
                "You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
            )
        user_defined_symbols = [
-            AddedToken(token, normalized=False, special=False) for token in proto.trainer_spec.user_defined_symbols
+            AddedToken(token, normalized=True, special=False) for token in proto.trainer_spec.user_defined_symbols


This is a must? Unless with only the split this works.

ArthurZucker added 2 commits February 6, 2024 12:27

push legacy to fast as well

689c9aa

super strange

7a3ddf4

ArthurZucker mentioned this pull request Feb 6, 2024

Normalizer "replace" is quadratic in sequence length (impacts Llama 2 tokenizer) huggingface/tokenizers#1449

Closed

ArthurZucker commented Feb 20, 2024

View reviewed changes

src/transformers/convert_slow_tokenizer.py Show resolved Hide resolved

Update src/transformers/convert_slow_tokenizer.py

d754f3b

ArthurZucker mentioned this pull request Mar 7, 2024

Mistral Tokenizer.decode() add a space when use_fast=True #29452

Closed

huggingface deleted a comment from github-actions bot Mar 22, 2024

ArthurZucker added 5 commits March 22, 2024 13:44

make sure we are BC

Loading
Loading status checks…

e1ef2d3

fix Llama test

Loading
Loading status checks…

0897c61

nit

Loading
Loading status checks…

b188318

revert

f7e8b06

more test

d939358

ArthurZucker marked this pull request as ready for review March 22, 2024 09:07

ArthurZucker added 2 commits March 22, 2024 18:10

Merge branch 'main' of github.com:huggingface/transformers into refac…

5c48f71

…tor-default-llama

style

ce042c6

This was referenced Mar 25, 2024

Inconsistent behavior between tokenizer and fast tokenizer #28577

Closed

LlamaTokenizerFast wrong word_id references based on batch encoding #29617

Closed

Inconsistency behaviours between LlamaTokenizer and LlamaTokenizerFast when lstrip=True #29626

Closed

update

bbd26b0

small update w.r.t tokenizers

84d406c

huggingface deleted a comment from github-actions bot Apr 18, 2024

ArthurZucker added 3 commits April 22, 2024 15:50

Merge branch 'main' of github.com:huggingface/transformers into refac…

038869d

…tor-default-llama

nit

6fbb0ac

don't split

6bd696e

ArthurZucker mentioned this pull request Apr 23, 2024

Llama: Generating text token by token removes whitespaces #22710

Closed

4 tasks

ArthurZucker added 2 commits April 23, 2024 09:25

lol

436335d

add a test for add_prefix_space=False

afa51b1

ArthurZucker requested a review from LysandreJik April 23, 2024 07:49

LysandreJik approved these changes Apr 23, 2024

View reviewed changes

ArthurZucker added 2 commits April 23, 2024 10:06

fix gemma tokenizer as well

312cc1d

update

feeec97

vasqu mentioned this pull request Apr 23, 2024

Gemma's Tokenizer fails to split on spaces #30416

Closed

4 tasks

ArthurZucker added 3 commits April 23, 2024 13:25

fix gemma

e7953f5

nicer failures

ca25a81

fixup

7bfe577

pcuenca reviewed Apr 23, 2024

View reviewed changes

src/transformers/models/llama/tokenization_llama_fast.py Outdated Show resolved Hide resolved

ArthurZucker commented Apr 23, 2024

View reviewed changes

ArthurZucker added 6 commits April 23, 2024 15:20

update

05cd744

fix the example for legacy = False

9c92741

use huggyllama/llama-7b for the PR doctest

187efad

nit

9eac9c7

use from_slow

4c40705

fix llama

e775b7d

ArthurZucker merged commit e34da3e into main Apr 23, 2024
22 checks passed

ArthurZucker deleted the refactor-default-llama branch April 23, 2024 21:13

This was referenced Jun 5, 2024

[BUG] Fast tokenizer does not deal with AddedTokens properly(no problem in Transformers python tokenizer impl.) huggingface/tokenizers#1544

Closed

Unable to load t5-small tokenizer saved with latest packages in older versions #31139

Closed

Alienmaster mentioned this pull request Jun 15, 2024

GemmaTokenizerFast word_ids() returns only zeros #31437

Closed

4 tasks

hamishivi mentioned this pull request Jul 23, 2024

Bump transformers to 4.43.4 allenai/open-instruct#202

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[`LlamaTokenizerFast`] Refactor default llama #28881

[`LlamaTokenizerFast`] Refactor default llama #28881

ArthurZucker commented Feb 6, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Feb 6, 2024

ArthurZucker commented Apr 18, 2024

LysandreJik left a comment

ArthurZucker Apr 23, 2024

[LlamaTokenizerFast] Refactor default llama #28881

[LlamaTokenizerFast] Refactor default llama #28881

Conversation

ArthurZucker commented Feb 6, 2024 • edited Loading

What does this PR do?

HuggingFaceDocBuilderDev commented Feb 6, 2024

ArthurZucker commented Apr 18, 2024

LysandreJik left a comment

Choose a reason for hiding this comment

ArthurZucker Apr 23, 2024

Choose a reason for hiding this comment

[`LlamaTokenizerFast`] Refactor default llama #28881

[`LlamaTokenizerFast`] Refactor default llama #28881

ArthurZucker commented Feb 6, 2024 •

edited

Loading