langchain Architecture Docs
Home / Function/ test_sentence_transformers_multiple_tokens() — langchain Function Reference

test_sentence_transformers_multiple_tokens() — langchain Function Reference

Architecture documentation for the test_sentence_transformers_multiple_tokens() function in test_text_splitter.py from the langchain codebase.

Function python LangChainCore ApiManagement

Entity Profile

Dependency Diagram

graph TD
  926e90b3_3879_fd5f_839d_783e1e1a71c2["test_sentence_transformers_multiple_tokens()"]
  d35bbf8f_3f92_b567_0710_bd1ead1e275e["test_text_splitter.py"]
  926e90b3_3879_fd5f_839d_783e1e1a71c2 -->|defined in| d35bbf8f_3f92_b567_0710_bd1ead1e275e
  style 926e90b3_3879_fd5f_839d_783e1e1a71c2 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/tests/integration_tests/test_text_splitter.py lines 85–114

def test_sentence_transformers_multiple_tokens() -> None:
    splitter = SentenceTransformersTokenTextSplitter(chunk_overlap=0)
    text = "Lorem "

    text_token_count_including_start_and_stop_tokens = splitter.count_tokens(text=text)
    count_start_and_end_tokens = 2
    token_multiplier = (
        count_start_and_end_tokens
        + (splitter.maximum_tokens_per_chunk - count_start_and_end_tokens)
        // (
            text_token_count_including_start_and_stop_tokens
            - count_start_and_end_tokens
        )
        + 1
    )

    # `text_to_split` does not fit in a single chunk
    text_to_embed = text * token_multiplier

    text_chunks = splitter.split_text(text=text_to_embed)

    expected_number_of_chunks = 2

    assert expected_number_of_chunks == len(text_chunks)
    actual = splitter.count_tokens(text=text_chunks[1]) - count_start_and_end_tokens
    expected = (
        token_multiplier * (text_token_count_including_start_and_stop_tokens - 2)
        - splitter.maximum_tokens_per_chunk
    )
    assert expected == actual

Domain

Subdomains

Defined In

Source

Frequently Asked Questions

What does test_sentence_transformers_multiple_tokens() do?
test_sentence_transformers_multiple_tokens() is a function in the langchain codebase, defined in libs/text-splitters/tests/integration_tests/test_text_splitter.py.
Where is test_sentence_transformers_multiple_tokens() defined?
test_sentence_transformers_multiple_tokens() is defined in libs/text-splitters/tests/integration_tests/test_text_splitter.py at line 85.

Analyze Your Own Codebase

Get architecture documentation, dependency graphs, and domain analysis for your codebase in minutes.

Try Supermodel Free