Visualize Google Cloud Vertex AI large language models and their tokenization

With this application, you can better understand how large language models tokenize your text.

Enter your text, select a model, and see how it is tokenized!

Models currently supported

• textembedding-gecko
• textembedding-gecko-multilingual
• text-bison
• text-unicorn
• chat-bison
• code-gecko
• code-bison
• codechat-bison

Try it out!

You can try this application online, and select one of the available PaLM-based models, and see how your text is tokenized.