test_custom_token_counting() — langchain Function Reference

Architecture documentation for the test_custom_token_counting() function in test_chat_models.py from the langchain codebase.

Function python LangChainCore LanguageModelBase

Entity Profile

LangChainCore→ LanguageModelBase→ test_custom_token_counting() — langchain Function Reference

Dependency Diagram

graph TD
  a08c7186_0a28_0c71_d003_5e276c6ebd6e["test_custom_token_counting()"]
  aaa4b344_20ed_7fa3_4067_f8f05affc01f["test_chat_models.py"]
  a08c7186_0a28_0c71_d003_5e276c6ebd6e -->|defined in| aaa4b344_20ed_7fa3_4067_f8f05affc01f
  style a08c7186_0a28_0c71_d003_5e276c6ebd6e fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/partners/mistralai/tests/unit_tests/test_chat_models.py lines 293–298

def test_custom_token_counting() -> None:
    def token_encoder(text: str) -> list[int]:
        return [1, 2, 3]

    llm = ChatMistralAI(custom_get_token_ids=token_encoder)
    assert llm.get_token_ids("foo") == [1, 2, 3]