dummy_token_counter() — langchain Function Reference

Architecture documentation for the dummy_token_counter() function in test_utils.py from the langchain codebase.

Function python LangChainCore MessageInterface called by 1

Entity Profile

LangChainCore→ MessageInterface→ dummy_token_counter() — langchain Function Reference

Dependency Diagram

graph TD
  341444f0_9d9a_88de_9a1e_da80115b274c["dummy_token_counter()"]
  03f6a5ae_d57a_eb66_626a_b9e082b763ea["test_utils.py"]
  341444f0_9d9a_88de_9a1e_da80115b274c -->|defined in| 03f6a5ae_d57a_eb66_626a_b9e082b763ea
  daeb82e7_d1da_efa6_b2c9_ba1828bc0d34["get_num_tokens_from_messages()"]
  daeb82e7_d1da_efa6_b2c9_ba1828bc0d34 -->|calls| 341444f0_9d9a_88de_9a1e_da80115b274c
  style 341444f0_9d9a_88de_9a1e_da80115b274c fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/core/tests/unit_tests/messages/test_utils.py lines 544–565

def dummy_token_counter(messages: list[BaseMessage]) -> int:
    # treat each message like it adds 3 default tokens at the beginning
    # of the message and at the end of the message. 3 + 4 + 3 = 10 tokens
    # per message.

    default_content_len = 4
    default_msg_prefix_len = 3
    default_msg_suffix_len = 3

    count = 0
    for msg in messages:
        if isinstance(msg.content, str):
            count += (
                default_msg_prefix_len + default_content_len + default_msg_suffix_len
            )
        if isinstance(msg.content, list):
            count += (
                default_msg_prefix_len
                + len(msg.content) * default_content_len
                + default_msg_suffix_len
            )
    return count