langchain Architecture Docs
Home / Function/ test_sentence_transformers_count_tokens() — langchain Function Reference

test_sentence_transformers_count_tokens() — langchain Function Reference

Architecture documentation for the test_sentence_transformers_count_tokens() function in test_text_splitter.py from the langchain codebase.

Function python LangChainCore Runnables

Entity Profile

Dependency Diagram

graph TD
  3fba328e_0255_79c1_aec9_b77eab0e3c85["test_sentence_transformers_count_tokens()"]
  d35bbf8f_3f92_b567_0710_bd1ead1e275e["test_text_splitter.py"]
  3fba328e_0255_79c1_aec9_b77eab0e3c85 -->|defined in| d35bbf8f_3f92_b567_0710_bd1ead1e275e
  style 3fba328e_0255_79c1_aec9_b77eab0e3c85 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/tests/integration_tests/test_text_splitter.py lines 58–70

def test_sentence_transformers_count_tokens() -> None:
    splitter = SentenceTransformersTokenTextSplitter(
        model_name="sentence-transformers/paraphrase-albert-small-v2"
    )
    text = "Lorem ipsum"

    token_count = splitter.count_tokens(text=text)

    expected_start_stop_token_count = 2
    expected_text_token_count = 5
    expected_token_count = expected_start_stop_token_count + expected_text_token_count

    assert expected_token_count == token_count

Domain

Subdomains

Defined In

Source

Frequently Asked Questions

What does test_sentence_transformers_count_tokens() do?
test_sentence_transformers_count_tokens() is a function in the langchain codebase, defined in libs/text-splitters/tests/integration_tests/test_text_splitter.py.
Where is test_sentence_transformers_count_tokens() defined?
test_sentence_transformers_count_tokens() is defined in libs/text-splitters/tests/integration_tests/test_text_splitter.py at line 58.

Analyze Your Own Codebase

Get architecture documentation, dependency graphs, and domain analysis for your codebase in minutes.

Try Supermodel Free