Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add [python-sentencepiece] #20832

Open
1 of 6 tasks
Kreijstal opened this issue May 8, 2024 · 0 comments · May be fixed by #20841
Open
1 of 6 tasks

Add [python-sentencepiece] #20832

Kreijstal opened this issue May 8, 2024 · 0 comments · May be fixed by #20841

Comments

@Kreijstal
Copy link
Contributor

Kreijstal commented May 8, 2024

Package name

python-senetencepiece and sentencepiece

Brief description of package

Python sentencepiece is a google project which is basically a tokenizer for LLM if I understand correctly?
they can be either distributed as two different packages or just one, the reason to request here is that it can't be build with a mere pip install, since of course people assume once again windows==msvc

URL for package's homepage

No response

Provide a basic test case to validate the package's functionality.

import sentencepiece in python,

MINGW environments where you need the package

  • MINGW64
  • MINGW32
  • UCRT64
  • CLANG64
  • CLANG32
  • CLANGARM64

Are you willing to submit a PR?

No response

@Kreijstal Kreijstal linked a pull request May 8, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant