test_decode_returns_no_chunks() — langchain Function Reference

Architecture documentation for the test_decode_returns_no_chunks() function in test_text_splitters.py from the langchain codebase.

Function python LangChainCore MessageInterface

Entity Profile

LangChainCore→ MessageInterface→ test_decode_returns_no_chunks() — langchain Function Reference

Dependency Diagram

graph TD
  d205e4c7_7210_a0e2_4a24_413de6cc26f4["test_decode_returns_no_chunks()"]
  6d6b8ad4_1cfe_fbb0_e58e_76a50487c135["test_text_splitters.py"]
  d205e4c7_7210_a0e2_4a24_413de6cc26f4 -->|defined in| 6d6b8ad4_1cfe_fbb0_e58e_76a50487c135
  style d205e4c7_7210_a0e2_4a24_413de6cc26f4 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/tests/unit_tests/test_text_splitters.py lines 2938–2950

def test_decode_returns_no_chunks() -> None:
    """Test that when decode returns only empty strings, output is empty, not ['']."""
    text = "foo bar baz 123"

    tokenizer = Tokenizer(
        chunk_overlap=3,
        tokens_per_chunk=7,
        decode=(lambda _: ""),
        encode=(lambda it: [ord(c) for c in it]),
    )
    output = split_text_on_tokens(text=text, tokenizer=tokenizer)
    expected_output: list[Any] = []
    assert output == expected_output