feat: Add chunk overlap setting (#160)

* feat: Add chunk overlap setting

Allows for overlapping chunks. Will still use the semantic range sections to determine a good splitting point for the overlap as well.

* Make sure there aren't chunks emitted whose entire content was already emitted

* Consolidate snapshot test code

* Add overlap snapshot tests

* Update changelog and python bindings

.github/workflows/python.yml

@@ -26,6 +26,33 @@ permissions:
   contents: read
 
 jobs:
+  min_supported_version:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.8"
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          working-directory: bindings/python
+      - name: Build wheels
+        uses: PyO3/maturin-action@v1
+        with:
+          target: x86_64
+          args: --release --out dist
+          sccache: "true"
+          manylinux: auto
+          working-directory: bindings/python
+      - name: pytest
+        shell: bash
+        run: |
+          set -e
+          pip install --no-index --find-links dist --force-reinstall semantic-text-splitter
+          pip install pytest tokenizers
+          pytest
+
   linux:
     runs-on: ubuntu-latest
     strategy:

CHANGELOG.md

@@ -1,5 +1,31 @@
 # Changelog
 
+## v0.12.2
+
+### What's New
+
+**Support for chunk overlapping:** Several of you have been waiting on this for awhile now, and I am happy to say that chunk overlapping is now available in a way that still stays true to the spirit of finding good semantic break points.
+
+When a new chunk is emitted, if chunk overlapping is enabled, the splitter will look back at the semantic sections of the current level and pull in as many as possible that fit within the overlap window. **This does mean that none can be taken**, which is often the case when close to a higher semantic level boundary.
+
+When it will almost always produce an overlap is when the current semantic level couldn't be fit into a single chunk, and it provides overlapping sections since we may not have found a good break point in the middle of the section. Which seems to be the main motivation for using chunk overlapping in the first place.
+
+#### Rust Usage
+
+```rust
+let chunk_config = ChunkConfig::new(256)
+    // .with_sizer(sizer) // Optional tokenizer or other chunk sizer impl
+    .with_overlap(64)
+    .expect("Overlap must be less than desired chunk capacity");
+let splitter = TextSplitter::new(chunk_config); // Or MarkdownSplitter
+```
+
+#### Python Usage
+
+```python
+splitter = TextSplitter(256, overlap=64) # or any of the class methods to use a tokenizer
+```
+
 ## v0.12.1
 
 ### What's New

Cargo.toml

@@ -2,7 +2,7 @@
 members = ["bindings/*"]
 
 [workspace.package]
-version = "0.12.1"
+version = "0.12.2"
 authors = ["Ben Brandt <benjamin.j.brandt@gmail.com>"]
 edition = "2021"
 description = "Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python."