Why is llama-cli doing string comparison to check antiprompt?

[SpaceCowboy850]

In llama-cli main.cpp, we have the antiprompts that we store as multiple tokens (antiprompt_ids). However, when it comes time to check it, we do a string comparison over the last 32 decoded tokens against params.antiprompt (vector of strings). But then, right after that we check antiprompt_ids (tokenized version), but ONLY check if the entry is a single token length.

This seems unnecessarily complex.

Wouldn't it be easier to just do a tokenized check on the length of each antiprompt_ids and avoid the string check and get rid of the "only check the token if the antiprompt is a single token" special casing?

In short, I'm trying to understand if there is reasoning behind this complexity.

[ggerganov]

Likely the logic can be simplified and PRs are welcome. Just keep in mind that tokenization is tricky - for example hello and hello in most cases will tokenize in 2 different tokens, so antiprompt checks most likely have to remain in text space rather than token space.

[SpaceCowboy850]

Thank you. I understand tokenization can be tricky, which is why I wanted to ask before attempting to simplify it. I'll look at it a bit.