Spaces:
Running
Running
Fix error when force_tokens includes multi-word sequence to preserve
#2
by
cornzz
- opened
Right now an error occurs when force_tokens includes a sequence which contains spaces and is contained in the prompt.
This is caused by word, label = line.split(label_sep)
, where line
is for example The answer is 1
(where "The answer is" is the sequence in force_tokens
and 1 is the corresponding label).
The error is thrown because line.split(label_sep)
returns ['The', 'answer', 'is', '1']
, which is too many arguments to be unpacked into word, label
.
The fix is to split only at the first occurence of label_sep
from the right.
qianhuiwu
changed pull request status to
merged
Thanks. Merge the fix for string split.