test_custom_token_counting() — langchain Function Reference

Architecture documentation for the test_custom_token_counting() function in test_base.py from the langchain codebase.

Function python LangChainCore LanguageModelBase

Entity Profile

LangChainCore→ LanguageModelBase→ test_custom_token_counting() — langchain Function Reference

Dependency Diagram

graph TD
  5046358d_56cb_f1e7_fb7b_09e320863a57["test_custom_token_counting()"]
  48232d20_f8c1_b597_14fa_7dc407e9bfe5["test_base.py"]
  5046358d_56cb_f1e7_fb7b_09e320863a57 -->|defined in| 48232d20_f8c1_b597_14fa_7dc407e9bfe5
  style 5046358d_56cb_f1e7_fb7b_09e320863a57 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/partners/openai/tests/unit_tests/chat_models/test_base.py lines 766–771

def test_custom_token_counting() -> None:
    def token_encoder(text: str) -> list[int]:
        return [1, 2, 3]

    llm = ChatOpenAI(custom_get_token_ids=token_encoder)
    assert llm.get_token_ids("foo") == [1, 2, 3]