Python: v0.2.0 - Hugging Face Tokenizer support #22
benbrandt
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
What's New
HuggingFaceTextSplitter
, which allows for using Hugging Face'stokenizers
package to count chunks by tokens with a tokenizer of your choice.Breaking Changes
trim_chunks
now defaults toTrue
instead ofFalse
. For most use cases, this is the desired behavior, especially with chunk ranges.Full Changelog: python-v0.1.4...python-v0.2.0
This discussion was created from the release Python: v0.2.0 - Hugging Face Tokenizer support.
Beta Was this translation helpful? Give feedback.
All reactions