expose max_decode_len and eos_token_id in decoding #328
Conversation
| @@ -157,6 +158,7 @@ def beam_search_decode( | |||
| input_batch: a dict with a minimum of the following entries: | |||
| prefix: Prompt IDs representing a Tensor of shape [batch, max_sequence_length]. | |||
| num_decodes: the number of beams to decode. | |||
| eos_token_id: The end of sentence token id. If not set, will use cfg.eos_token_id. | |||
There was a problem hiding this comment.
Any reason not to just set cfg.eos_token_id directly?
There was a problem hiding this comment.
setting cfg.eos_token_id might not be suitable for inference side? especially if we have different requests to run against the same instance with different eos token id.
There was a problem hiding this comment.
Is the motivation to customize when to stop decoding? If so, have you considered adding stop_decoding_condition similar to sample_decode?
| @@ -157,6 +158,7 @@ def beam_search_decode( | |||
| input_batch: a dict with a minimum of the following entries: | |||
| prefix: Prompt IDs representing a Tensor of shape [batch, max_sequence_length]. | |||
| num_decodes: the number of beams to decode. | |||
| eos_token_id: The end of sentence token id. If not set, will use cfg.eos_token_id. | |||
There was a problem hiding this comment.
Is the motivation to customize when to stop decoding? If so, have you considered adding stop_decoding_condition similar to sample_decode?
sg. let's discuss in an internal PR firstly. |
Expose
It looks like both have been added unit test in decoding_test.py. This change is to expose the parameter to causal_lm and decoder module.