Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_text_splitters.markdown import ExperimentalMarkdownSyntaxTextSplitter, MarkdownHeaderTextSplitter
import os


splitter = ExperimentalMarkdownSyntaxTextSplitter(
    headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
], strip_headers=False, return_each_line=False)



for file in sorted(os.listdir("testdata")):
    print(file)
    with open(f"testdata/{file}", "r") as f:
        text = f.read()

    splits = splitter.split_text(text)

    for split in splits:
        print(split.metadata)
        print(split.page_content)
        print('-'*80)

    print('='*80)
    print()

Files

Files.zip

Error Message and Stack Trace (if applicable)

Output

sample1.md
{'Header 1': 'Header 1 from file 1'}
# Header 1 from file 1

Content 1 from file 1


--------------------------------------------------------------------------------
{'Header 1': 'Header 1 from file 1', 'Header 2': 'Header 2 from file 1'}
## Header 2 from file 1

Content 2 file file 1


More stuff in file 1

* list1.1
    * list 2.1
    * list 2.2
* list1.2
--------------------------------------------------------------------------------
================================================================================

sample2.md
{'Header 1': 'Header 1 from file 1'}
# Header 1 from file 1

Content 1 from file 1


--------------------------------------------------------------------------------
{'Header 1': 'Header 1 from file 1', 'Header 2': 'Header 2 from file 1'}
## Header 2 from file 1

Content 2 file file 1


More stuff in file 1

* list1.1
    * list 2.1
    * list 2.2
* list1.2
--------------------------------------------------------------------------------
{'Header 1': 'Header 1 from file 2'}
# Header 1 from file 2

Content 1 from file 2


--------------------------------------------------------------------------------
{'Header 1': 'Header 1 from file 2', 'Header 2': 'Header 2 from file 2'}
## Header 2 from file 2

Content 2 file file 2


More stuff in file 2

1. list1.1
    1. list 2.1
    2. list 2.2
1. list1.2
--------------------------------------------------------------------------------
================================================================================

Description

I was testing out the ExperimentalMarkdownSyntaxTextSplitter class due to issues with whitespacing in the MarkdownHeaderTextSplitter. I noticed that the class was mixing up text between subsequent split_text calls.

I do not believe this is intended. Please find the attached zip to reproduce the issue. Happy to help fix the issue.

Let me know if there are stable alternatives to achieve splitting by markdown headers in the mean time.

System Info

python -m langchain_core.sys_info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6031
Python Version: 3.12.4 (main, Jun 6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information

langchain_core: 0.2.38
langchain: 0.2.16
langchain_community: 0.2.16
langsmith: 0.1.114
langchain_text_splitters: 0.2.4

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.5
async-timeout: Installed. No version info available.
dataclasses-json: 0.6.7
httpx: 0.27.2
jsonpatch: 1.33
numpy: 1.26.4
orjson: 3.10.7
packaging: 24.1
pydantic: 2.8.2
PyYAML: 6.0.2
requests: 2.32.3
SQLAlchemy: 2.0.34
tenacity: 8.5.0
typing-extensions: 4.12.2