test_split_text_on_tokens() — langchain Function Reference

Architecture documentation for the test_split_text_on_tokens() function in test_text_splitters.py from the langchain codebase.

Function python LangChainCore Runnables

Entity Profile

LangChainCore→ Runnables→ test_split_text_on_tokens() — langchain Function Reference

Dependency Diagram

graph TD
  13a5b67f_7961_ba50_62ca_14cf1cd04feb["test_split_text_on_tokens()"]
  6d6b8ad4_1cfe_fbb0_e58e_76a50487c135["test_text_splitters.py"]
  13a5b67f_7961_ba50_62ca_14cf1cd04feb -->|defined in| 6d6b8ad4_1cfe_fbb0_e58e_76a50487c135
  style 13a5b67f_7961_ba50_62ca_14cf1cd04feb fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/tests/unit_tests/test_text_splitters.py lines 2923–2935

def test_split_text_on_tokens() -> None:
    """Test splitting by tokens per chunk."""
    text = "foo bar baz 123"

    tokenizer = Tokenizer(
        chunk_overlap=3,
        tokens_per_chunk=7,
        decode=(lambda it: "".join(chr(i) for i in it)),
        encode=(lambda it: [ord(c) for c in it]),
    )
    output = split_text_on_tokens(text=text, tokenizer=tokenizer)
    expected_output = ["foo bar", "bar baz", "baz 123"]
    assert output == expected_output