answer contains <pad> leads to error in the example

[deangeckt]

when running the example:

nlp("42 is the answer to life, universe and everything.")

---

ValueError Traceback (most recent call last)
in
----> 1 nlp("42 is the answer to life, universe and everything.")

~\question_generation\pipelines.py in call(self, inputs)
58 qg_examples = self._prepare_inputs_for_qg_from_answers_prepend(inputs, answers)
59 else:
---> 60 qg_examples = self._prepare_inputs_for_qg_from_answers_hl(sents, answers)
61
62 qg_inputs = [example['source_text'] for example in qg_examples]

~\question_generation\pipelines.py in _prepare_inputs_for_qg_from_answers_hl(self, sents, answers)
140 answer_text = answer_text.strip()
141
--> 142 ans_start_idx = sent.index(answer_text)
143
144 sent = f"{sent[:ans_start_idx]} {answer_text} {sent[ans_start_idx + len(answer_text): ]}"

ValueError: substring not found

in _extract_answers() ,
when debugging i saw the "pad" in the dec (in the answer)

dec = [self.ans_tokenizer.decode(ids, skip_special_tokens=False) for ids in outs]

could be fixed when using skip_special_tokens to TRUE?

[deangeckt]

also in _prepare_inputs_for_qg_from_answers_hl()
i'd add this check to avoid further exceptions

if answer_text not in sent:
continue

[cdhx]

same issue
this is intermediate result during execute

input: 42 is the answer to life, the universe and everything.
sents, answers: ['42 is the answer to life, the universe and everything.'] [['<pad> 42']]
answer: [['<pad> 42']]

[YiLing28]

in pipelines.py, line 90, set skip_special_tokens=True
dec = [self.ans_tokenizer.decode(ids, skip_special_tokens=True) for ids in outs]
This may solve the problem.