test_custom_token_counting() — langchain Function Reference

Architecture documentation for the test_custom_token_counting() function in test_base.py from the langchain codebase.

Function python LangChainCore LanguageModelBase

Entity Profile

LangChainCore→ LanguageModelBase→ test_custom_token_counting() — langchain Function Reference

Dependency Diagram

graph TD
  2554ca02_22a3_9cac_5c61_1b11caf8d492["test_custom_token_counting()"]
  6395ab08_3ae5_72d1_1c6a_5b74f14238f7["test_base.py"]
  2554ca02_22a3_9cac_5c61_1b11caf8d492 -->|defined in| 6395ab08_3ae5_72d1_1c6a_5b74f14238f7
  style 2554ca02_22a3_9cac_5c61_1b11caf8d492 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/partners/openai/tests/unit_tests/llms/test_base.py lines 70–75

def test_custom_token_counting() -> None:
    def token_encoder(text: str) -> list[int]:
        return [1, 2, 3]

    llm = OpenAI(custom_get_token_ids=token_encoder)
    assert llm.get_token_ids("foo") == [1, 2, 3]