Adds tokenization task for embeddings #365
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
flaviabeo
requested review from
alex-jw-brooks,
gkumbhat,
evaline-ju,
gabe-l-hart,
tharapalanivel and
Ssukriti
as code owners
gkumbhat
previously requested changes
Jun 27, 2024
| TokenizationResults | ||
| The token count | ||
| """ | ||
| result = self.model.tokenizer.encode_plus(text, return_offsets_mapping=True) |
There was a problem hiding this comment.
encode_plus is deprecated: https://huggingface.co/docs/transformers/v4.41.3/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode_plus
Can we use __call__ instead?
evaline-ju
reviewed
Jul 24, 2024
flaviabeo
force-pushed
the
tokenize
branch
4 times, most recently
from
5fd8f44 to
0822f3f
Compare
evaline-ju
reviewed
Jul 24, 2024
markstur
suggested changes
Jul 24, 2024
| ] | ||
| tokens = [Token(start=i[0], end=i[1], text=text[i[0] : i[1]]) for i in mapping] | ||
|
|
||
| return TokenizationResults(token_count=len(result.input_ids), results=tokens) |
There was a problem hiding this comment.
This needs index zero because we wrapped the text in a list for _get_tokenized().
Suggested change
| return TokenizationResults(token_count=len(result.input_ids), results=tokens) | |
| return TokenizationResults(token_count=len(result.input_ids[0), results=tokens) |
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Evaline Ju <69598118+evaline-ju@users.noreply.github.com> Signed-off-by: Flávia Béo <119421251+flaviabeo@users.noreply.github.com>
evaline-ju
dismissed
gkumbhat’s stale review
Dismissing on maintainer availability, the requested change appears addressed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements the
TokenizeTaskfor theEmbeddingsmodule. Works for all the models of type SentenceTransformer.cc: @mynhardtburger @markstur