Uses temperature 0.0 for Phi-3.

[gugarosa]

Sorry for sending a bunch of PRs on the same matter 😄

We decided to set temperature=0.0 to have a more controlled environment and attempt to fix the issue for real. One thing that we were thinking and I don't know the answer is the following:

When the truncation happen, it is done in the left or right side of the context? Is there a way to set it? I looked on the https://github.com/huggingface/chat-ui/blob/main/src/lib/server/models.ts file, but it could be somewhere else.

Thanks and best regards,
Gustavo.

[gary149]

Ok to merge (@nsarrazin is off)

When the truncation happen, it is done in the left or right side of the context? Is there a way to set it?

I think it's left side and no it cannot be set in the config.

[gugarosa]

Thanks for reviewing and the inputs on truncation @gary149!

[mishig25]

@gugarosa I've merged but reverted the changes afterwards #1152

The reason being that, I get this error from our api-inference backend:

Error: Input validation error: temperature must be strictly positive

Would something smaller like temperature: 0.1 that is not 0.0 work ?