Files

.devops
.github
ci
cmake
common
docs
examples
ggml-cuda
ggml-sycl
gguf-py
grammars
kompute
kompute-shaders
media
models
- .editorconfig
- ggml-vocab-aquila.gguf
- ggml-vocab-baichuan.gguf
- ggml-vocab-bert-bge.gguf
- ggml-vocab-bert-bge.gguf.inp
- ggml-vocab-bert-bge.gguf.out
- ggml-vocab-command-r.gguf
- ggml-vocab-command-r.gguf.inp
- ggml-vocab-command-r.gguf.out
- ggml-vocab-deepseek-coder.gguf
- ggml-vocab-deepseek-coder.gguf.inp
- ggml-vocab-deepseek-coder.gguf.out
- ggml-vocab-deepseek-llm.gguf
- ggml-vocab-deepseek-llm.gguf.inp
- ggml-vocab-deepseek-llm.gguf.out
- ggml-vocab-falcon.gguf
- ggml-vocab-falcon.gguf.inp
- ggml-vocab-falcon.gguf.out
- ggml-vocab-gpt-2.gguf
- ggml-vocab-gpt-2.gguf.inp
- ggml-vocab-gpt-2.gguf.out
- ggml-vocab-gpt-neox.gguf
- ggml-vocab-gpt2.gguf
- ggml-vocab-llama-bpe.gguf
- ggml-vocab-llama-bpe.gguf.inp
- ggml-vocab-llama-bpe.gguf.out
- ggml-vocab-llama-spm.gguf
- ggml-vocab-llama-spm.gguf.inp
- ggml-vocab-llama-spm.gguf.out
- ggml-vocab-mpt.gguf
- ggml-vocab-mpt.gguf.inp
- ggml-vocab-mpt.gguf.out
- ggml-vocab-phi-3.gguf
- ggml-vocab-phi-3.gguf.inp
- ggml-vocab-phi-3.gguf.out
- ggml-vocab-qwen2.gguf
- ggml-vocab-qwen2.gguf.inp
- ggml-vocab-qwen2.gguf.out
- ggml-vocab-refact.gguf
- ggml-vocab-refact.gguf.inp
- ggml-vocab-refact.gguf.out
- ggml-vocab-stablelm.gguf
- ggml-vocab-starcoder.gguf
- ggml-vocab-starcoder.gguf.inp
- ggml-vocab-starcoder.gguf.out
pocs
prompts
requirements
scripts
spm-headers
tests
vulkan-shaders
.clang-tidy
.dockerignore
.ecrc
.editorconfig
.flake8
.gitignore
.gitmodules
.pre-commit-config.yaml
AUTHORS
CMakeLists.txt
CMakePresets.json
CONTRIBUTING.md
LICENSE
Makefile
Package.swift
README-sycl.md
README.md
SECURITY.md
convert-hf-to-gguf-update.py
convert-hf-to-gguf.py
convert-llama-ggml-to-gguf.py
flake.lock
flake.nix
ggml-alloc.c
ggml-alloc.h
ggml-backend-impl.h
ggml-backend.c
ggml-backend.h
ggml-blas.cpp
ggml-blas.h
ggml-common.h
ggml-cuda.cu
ggml-cuda.h
ggml-impl.h
ggml-kompute.cpp
ggml-kompute.h
ggml-metal.h
ggml-metal.m
ggml-metal.metal
ggml-quants.c
ggml-quants.h
ggml-rpc.cpp
ggml-rpc.h
ggml-sycl.cpp
ggml-sycl.h
ggml-vulkan-shaders.hpp
ggml-vulkan.cpp
ggml-vulkan.h
ggml.c
ggml.h
ggml_vk_generate_shaders.py
llama.cpp
llama.h
mypy.ini
pyrightconfig.json
requirements.txt
sgemm.cpp
sgemm.h
unicode-data.cpp
unicode-data.h
unicode.cpp
unicode.h

models

jaime-m-p

Per token attributes (ggml-org#7685 )

Jun 4, 2024

3b38d48 · Jun 4, 2024

History

This branch is 25 commits ahead of, 2054 commits behind ggml-org/llama.cpp:master.

Name	Name	Last commit message	Last commit date
parent directory ..
.editorconfig	.editorconfig	gguf : new file format with flexible meta data (beta) (ggml-org#2398 )	Aug 21, 2023
ggml-vocab-aquila.gguf	ggml-vocab-aquila.gguf	Work on the BPE tokenizer (ggml-org#3252 )	Oct 3, 2023
ggml-vocab-baichuan.gguf	ggml-vocab-baichuan.gguf	Add more tokenizer tests (ggml-org#3742 )	Oct 24, 2023
ggml-vocab-bert-bge.gguf	ggml-vocab-bert-bge.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-bert-bge.gguf.inp	ggml-vocab-bert-bge.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-bert-bge.gguf.out	ggml-vocab-bert-bge.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-command-r.gguf	ggml-vocab-command-r.gguf	command-r : add BPE pre-tokenization (ggml-org#7063 )	May 5, 2024
ggml-vocab-command-r.gguf.inp	ggml-vocab-command-r.gguf.inp	command-r : add BPE pre-tokenization (ggml-org#7063 )	May 5, 2024
ggml-vocab-command-r.gguf.out	ggml-vocab-command-r.gguf.out	command-r : add BPE pre-tokenization (ggml-org#7063 )	May 5, 2024
ggml-vocab-deepseek-coder.gguf	ggml-vocab-deepseek-coder.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-deepseek-coder.gguf.inp	ggml-vocab-deepseek-coder.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-deepseek-coder.gguf.out	ggml-vocab-deepseek-coder.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-deepseek-llm.gguf	ggml-vocab-deepseek-llm.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-deepseek-llm.gguf.inp	ggml-vocab-deepseek-llm.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-deepseek-llm.gguf.out	ggml-vocab-deepseek-llm.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-falcon.gguf	ggml-vocab-falcon.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-falcon.gguf.inp	ggml-vocab-falcon.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-falcon.gguf.out	ggml-vocab-falcon.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-gpt-2.gguf	ggml-vocab-gpt-2.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-gpt-2.gguf.inp	ggml-vocab-gpt-2.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-gpt-2.gguf.out	ggml-vocab-gpt-2.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-gpt-neox.gguf	ggml-vocab-gpt-neox.gguf	Add more tokenizer tests (ggml-org#3742 )	Oct 24, 2023
ggml-vocab-gpt2.gguf	ggml-vocab-gpt2.gguf	gpt2 : Add gpt2 architecture integration (ggml-org#4555 )	Dec 28, 2023
ggml-vocab-llama-bpe.gguf	ggml-vocab-llama-bpe.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-llama-bpe.gguf.inp	ggml-vocab-llama-bpe.gguf.inp	llama : lookup word in vocab before doing BPE merges (ggml-org#7193 )	May 11, 2024
ggml-vocab-llama-bpe.gguf.out	ggml-vocab-llama-bpe.gguf.out	llama : lookup word in vocab before doing BPE merges (ggml-org#7193 )	May 11, 2024
ggml-vocab-llama-spm.gguf	ggml-vocab-llama-spm.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-llama-spm.gguf.inp	ggml-vocab-llama-spm.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-llama-spm.gguf.out	ggml-vocab-llama-spm.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-mpt.gguf	ggml-vocab-mpt.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-mpt.gguf.inp	ggml-vocab-mpt.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-mpt.gguf.out	ggml-vocab-mpt.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-phi-3.gguf	ggml-vocab-phi-3.gguf	Per token attributes (ggml-org#7685 )	Jun 4, 2024
ggml-vocab-phi-3.gguf.inp	ggml-vocab-phi-3.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-phi-3.gguf.out	ggml-vocab-phi-3.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-qwen2.gguf	ggml-vocab-qwen2.gguf	llama : add BPE pre-tokenization for Qwen2 (ggml-org#7114 )	May 8, 2024
ggml-vocab-qwen2.gguf.inp	ggml-vocab-qwen2.gguf.inp	llama : add BPE pre-tokenization for Qwen2 (ggml-org#7114 )	May 8, 2024
ggml-vocab-qwen2.gguf.out	ggml-vocab-qwen2.gguf.out	llama : add BPE pre-tokenization for Qwen2 (ggml-org#7114 )	May 8, 2024
ggml-vocab-refact.gguf	ggml-vocab-refact.gguf	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-refact.gguf.inp	ggml-vocab-refact.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-refact.gguf.out	ggml-vocab-refact.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-stablelm.gguf	ggml-vocab-stablelm.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-starcoder.gguf	ggml-vocab-starcoder.gguf	llama : fix BPE pre-tokenization (ggml-org#6920 )	Apr 29, 2024
ggml-vocab-starcoder.gguf.inp	ggml-vocab-starcoder.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024
ggml-vocab-starcoder.gguf.out	ggml-vocab-starcoder.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036 )	May 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Files

models

models

Files

models

Directory actions

More options

Directory actions

More options

Latest commit

History

models

Folders and files

parent directory