__test_iterative_text_splitter() — langchain Function Reference

Architecture documentation for the __test_iterative_text_splitter() function in test_text_splitters.py from the langchain codebase.

Function python LangChainCore ApiManagement

Entity Profile

LangChainCore→ ApiManagement→ __test_iterative_text_splitter() — langchain Function Reference

Dependency Diagram

graph TD
  7343eabd_8b1a_71a8_98c6_33a601915e10["__test_iterative_text_splitter()"]
  6d6b8ad4_1cfe_fbb0_e58e_76a50487c135["test_text_splitters.py"]
  7343eabd_8b1a_71a8_98c6_33a601915e10 -->|defined in| 6d6b8ad4_1cfe_fbb0_e58e_76a50487c135
  style 7343eabd_8b1a_71a8_98c6_33a601915e10 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/tests/unit_tests/test_text_splitters.py lines 401–416

def __test_iterative_text_splitter(
    *, chunk_size: int, keep_separator: bool
) -> list[str]:
    chunk_size += 1 if keep_separator else 0

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=0,
        separators=["X", "Y"],
        keep_separator=keep_separator,
    )
    text = "....5X..3Y...4X....5Y..."
    output = splitter.split_text(text)
    for chunk in output:
        assert len(chunk) <= chunk_size, f"Chunk is larger than {chunk_size}"
    return output