test_token_text_splitter() — langchain Function Reference

Architecture documentation for the test_token_text_splitter() function in test_text_splitter.py from the langchain codebase.

Function python LangChainCore Runnables

Entity Profile

LangChainCore→ Runnables→ test_token_text_splitter() — langchain Function Reference

Dependency Diagram

graph TD
  d26d12bd_a098_f7b6_ccf8_e35ad6575d42["test_token_text_splitter()"]
  d35bbf8f_3f92_b567_0710_bd1ead1e275e["test_text_splitter.py"]
  d26d12bd_a098_f7b6_ccf8_e35ad6575d42 -->|defined in| d35bbf8f_3f92_b567_0710_bd1ead1e275e
  style d26d12bd_a098_f7b6_ccf8_e35ad6575d42 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/tests/integration_tests/test_text_splitter.py lines 34–39

def test_token_text_splitter() -> None:
    """Test no overlap."""
    splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=0)
    output = splitter.split_text("abcdef" * 5)  # 10 token string
    expected_output = ["abcdefabcdefabc", "defabcdefabcdef"]
    assert output == expected_output