Skip to content
GitLab
Projects
Groups
Snippets
Help
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Open sidebar
lizhiping
text-generation-inference
Commits
a4fd6905
Commit
a4fd6905
authored
2 years ago
by
OlivierDehaene
Browse files
Options
Download
Email Patches
Plain Diff
fmt
parent
83e442ca
feat/better_tokens
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
12 additions
and
10 deletions
+12
-10
router/src/infer.rs
router/src/infer.rs
+12
-10
No files found.
router/src/infer.rs
View file @
a4fd6905
...
...
@@ -154,22 +154,24 @@ impl Infer {
// Validation
// logprobs, ids and texts must have the same lengths
if
tokens
.logprobs
.len
()
!=
tokens_length
||
tokens
.texts
.len
()
!=
tokens_length
{
return
Err
(
InferError
::
GenerationError
(
format!
(
"Prefill tokens do not have the correct lengths"
)))
if
tokens
.logprobs
.len
()
!=
tokens_length
||
tokens
.texts
.len
()
!=
tokens_length
{
return
Err
(
InferError
::
GenerationError
(
"Prefill tokens do not have the correct lengths"
.to_string
(),
));
}
result_prefill
=
Vec
::
with_capacity
(
tokens_length
);
// Create Token objects
// We do that here instead of in the Python code as Rust for loops are faster
for
((
id
,
logprob
),
text
)
in
tokens
.ids
.into_iter
()
.zip
(
tokens
.logprobs
.into_iter
()
)
.zip
(
tokens
.texts
.into_iter
())
{
result_prefill
.push
(
PrefillToken
{
id
,
text
,
logprob
,
});
for
((
id
,
logprob
),
text
)
in
tokens
.ids
.into_iter
()
.zip
(
tokens
.logprobs
.into_iter
())
.zip
(
tokens
.texts
.into_iter
())
{
result_prefill
.push
(
PrefillToken
{
id
,
text
,
logprob
});
}
}
// Push last token
...
...
This diff is collapsed.
Click to expand it.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment