Text tokenizer paragraph support

[JacobDaneh]

Hi,
I have a customized TTS engine that performs better with paragraphs than sentences.
In swift-toolkit, PublicationSpeechSynthesizer class we can set paragraph for unit parameter of text tokenizer, but unfortunately kotlin-toolkit does not support paragraph. Just word and sentence.

How could I add paragraph as text tokenizer unit?

Thank you for your help

[mickael-menu]

Unfortunately the native ICU BreakIterator we use on Android doesn't support paragraph splitting:

kotlin-toolkit/readium/shared/src/main/java/org/readium/r2/shared/util/tokenizer/TextTokenizer.kt

Line 57 in f87dd95

TextUnit.Paragraph -> throw IllegalArgumentException("IcuTextTokenizer does not handle TextContentUnit.Paragraph")

You could try implementing your own version of a text tokenizer to split by paragraph. Also, if you don't use a text tokenizer at all (or an empty one that doesn't split the input), you will get something that is similar to paragraph tokens, because the HTML content iterator is returning block elements by default.

Let us know if you find a satisfying solution.