Tokenizer tests and TokenizeLine updates

[paul1r]

What this PR does / why we need it:
The thrust of this PR is to ensure we have tests for each major function of the Bloom Tokenizer. In addition, there was some cleanup, in that constants are used to set some common parameters.

Lastly, the TokenizeLine() call was updated to correctly tokenize a line when a "skip tokenizer" is utilized.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
  • If the change is worth mentioning in the release notes, add add-to-release-notes label
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[poyzannur]

LGTM with a minor question on the use of the two methods you added.
Am i correct to assume TokenizeLineWithChunkPrefix should be used when writing blooms, whereas TokenizeLine should be used to tokenize search queries before querying the blooms?

[paul1r]

LGTM with a minor question on the use of the two methods you added. Am i correct to assume TokenizeLineWithChunkPrefix should be used when writing blooms, whereas TokenizeLine should be used to tokenize search queries before querying the blooms?

PopulateSeriesWithBloom would be used when writing blooms. TokenizeLine is just to get a quick set of tokens, for the quick sniff test if we need to dig into a chunk or not. The TokenizeLineWithChunkPrefix is to validate those tokens exist for a specific chunk. Should be more obvious once we start wiring this together

Realized this is for reads, not writes.

[owen-d]

There's no need for two functions here -- you can just use something like the following and apply it to any tokenizer (chunk_prefix_tokenizer or regular)

func SearchesForTokenizerAndLine(t Tokenizer, line string) (res [][]Token) {
  for i := 0; i < t.Skip()+1; i++ {
    res = append(res, t.Tokens(line[i:])) // this needs to account for runes vs bytes, but you get the idea 
  }
  return
}

[owen-d]

Two things:

• This actually needs to ensure the length is >= min + skip.
• len(str) doesn't return the number of runes, but the number of bytes. We need to account for runes since that's how we index. See this for more detail.

[owen-d]

This is only true if all of the skip offsets return at least one token. Otherwise, the length of the result will be less than the number of skips+1.

[paul1r]

yep, I've updated the doc accordingly, good catch

[owen-d]

This unnecessarily iterates all runes in the line, including offsets beyond skip+1.

[paul1r]

ack, added a break clause