Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle \r gracefully as chunk separator in completion events #537

Open
khimaros opened this issue Feb 14, 2024 · 1 comment
Open

handle \r gracefully as chunk separator in completion events #537

khimaros opened this issue Feb 14, 2024 · 1 comment

Comments

@khimaros
Copy link

currently llama-cpp-python provides an OpenAI compatible API, but it joins multiple chunks into a single completion event with \r\r instead of \n, which causes the following exception:

VM190:3 Uncaught SyntaxError: Unexpected non-whitespace character after JSON at position 223
    at JSON.parse (<anonymous>)
    at eval (eval at <anonymous> (index-3e43d292.js:294:74), <anonymous>:1:6)
    at index-3e43d292.js:294:74
    at Array.map (<anonymous>)
    at L6e (index-3e43d292.js:293:20)
    at handleSubmit (index-3e43d292.js:305:824)

i am also following up with llama-cpp-python to see if i can upstream a fix, but it seems worthwhile for BetterChatGPT to be resilient to this sort of input.

abetlen added a commit to abetlen/llama-cpp-python that referenced this issue Feb 15, 2024
this fixes compatibility with some OpenAI clients, including BetterChatGPT (ztjhz/BetterChatGPT#537).

Co-authored-by: Andrei <abetlen@gmail.com>
@khimaros
Copy link
Author

khimaros commented Mar 5, 2024

happy to share that llama-cpp-python >= 0.2.38 uses newline delimited events. it would still be nice to be more resilient to other delimiters.

note however that #538 still requires a workaround in llama-cpp-python to disable the ping events, so it does not reliably work out of the box.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant