langchain Architecture Docs
Home / File/ test_token_counts.py — langchain Source File

test_token_counts.py — langchain Source File

Architecture documentation for test_token_counts.py, a python file in the langchain codebase. 2 imports, 0 dependents.

File python CoreAbstractions MessageSchema 2 imports 2 functions

Entity Profile

Dependency Diagram

graph LR
  5054c4b7_a1bf_c6cd_0643_bf82402c47c6["test_token_counts.py"]
  120e2591_3e15_b895_72b6_cb26195e40a6["pytest"]
  5054c4b7_a1bf_c6cd_0643_bf82402c47c6 --> 120e2591_3e15_b895_72b6_cb26195e40a6
  0b28cff6_d823_1571_d2bb_ec61508cc89c["langchain_openai"]
  5054c4b7_a1bf_c6cd_0643_bf82402c47c6 --> 0b28cff6_d823_1571_d2bb_ec61508cc89c
  style 5054c4b7_a1bf_c6cd_0643_bf82402c47c6 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

import pytest

from langchain_openai import ChatOpenAI, OpenAI

_EXPECTED_NUM_TOKENS = {
    "ada": 17,
    "babbage": 17,
    "curie": 17,
    "davinci": 17,
    "gpt-4": 12,
    "gpt-4-32k": 12,
    "gpt-3.5-turbo": 12,
    "o1": 11,
    "o3": 11,
    "gpt-4o": 11,
}

_MODELS = models = ["ada", "babbage", "curie", "davinci"]
_CHAT_MODELS = ["gpt-4", "gpt-4-32k", "gpt-3.5-turbo", "o1", "o3", "gpt-4o"]


@pytest.mark.xfail(reason="Old models require different tiktoken cached file")
@pytest.mark.parametrize("model", _MODELS)
def test_openai_get_num_tokens(model: str) -> None:
    """Test get_tokens."""
    llm = OpenAI(model=model)
    assert llm.get_num_tokens("表情符号是\n🦜🔗") == _EXPECTED_NUM_TOKENS[model]


@pytest.mark.parametrize("model", _CHAT_MODELS)
def test_chat_openai_get_num_tokens(model: str) -> None:
    """Test get_tokens."""
    llm = ChatOpenAI(model=model)
    assert llm.get_num_tokens("表情符号是\n🦜🔗") == _EXPECTED_NUM_TOKENS[model]

Domain

Subdomains

Functions

Dependencies

  • langchain_openai
  • pytest

Source

Frequently Asked Questions

What does test_token_counts.py do?
test_token_counts.py is a source file in the langchain codebase, written in python. It belongs to the CoreAbstractions domain, MessageSchema subdomain.
What functions are defined in test_token_counts.py?
test_token_counts.py defines 2 function(s): test_chat_openai_get_num_tokens, test_openai_get_num_tokens.
What does test_token_counts.py depend on?
test_token_counts.py imports 2 module(s): langchain_openai, pytest.
Where is test_token_counts.py in the architecture?
test_token_counts.py is located at libs/partners/openai/tests/unit_tests/test_token_counts.py (domain: CoreAbstractions, subdomain: MessageSchema, directory: libs/partners/openai/tests/unit_tests).

Analyze Your Own Codebase

Get architecture documentation, dependency graphs, and domain analysis for your codebase in minutes.

Try Supermodel Free