Skip to content

Commit

Permalink
Bugfix batch_emb, Change Iteration logic due to error in import
Browse files Browse the repository at this point in the history
  • Loading branch information
LMorlok committed Jul 15, 2024
1 parent 0ff8c98 commit 53ad241
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions src/embed_text_package/embed_text.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ def get_embeddings(sentence_batches: list, model, tokenizer):
# Maybe tokenizer class is "Autotokenizer" (from transformer)

emb_batches = []
batch_emb = []

for batch in sentence_batches:
for sentence in batch:
for batch_i in range(len(sentence_batches)):
batch_emb = []
for sen_j in range(len(sentence_batches[batch_i])):
# 1) Get Tokens of sentence
sentence_tokens = tokenizer(sentence)["input_ids"]
sentence_tokens = tokenizer(sentence_batches[sen_j])["input_ids"]

# 2) Get Embeddings (hiddenstate of last input)
# Generate model inputs on same device as model
Expand Down

0 comments on commit 53ad241

Please sign in to comment.