max_new_tokens as a parameter to answer_question() and default=256 instead of 512 #86
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
max_new_tokens is added as a parameter to answer_question() and default is set to 256 instead of 512.
Reasons:
Also, for avoiding the repetition loops, can some sort of an interrupt be added, as a callback like in GPT4All and a constraint when capturing repetition of tokens, unusual sequences like these dots etc.: repetition_penalty etc.? Also: a streaming mode? (I had a glance on the code, I may try to figure it out.)
Sample repetition loop:
Z:\LMSYS-Free-GPT4-Claude-15-4-2024-2024-04-15_18-33-46.mp4_snapshot_05.28.266.png
The image is a screenshot of a website called "LMSys Chatbot Arena". The website is predominantly white with blue and orange accents. The main focus is a banner at the top of the screen, which is white with red and blue text. The text reads "Free GPT4, claude, llama, code llama, mixral-of-experts, command-r-plus..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
The picture itself had "..." in the text.