get_token_ids() — langchain Function Reference

Architecture documentation for the get_token_ids() function in base.py from the langchain codebase.

Function python LangChainCore LanguageModelBase

Entity Profile

LangChainCore→ LanguageModelBase→ get_token_ids() — langchain Function Reference

Dependency Diagram

graph TD
  18bbe714_f107_862c_59f3_a33245557810["get_token_ids()"]
  6bee45b2_b649_e251_1fdc_dcf49f8bb331["BaseOpenAI"]
  18bbe714_f107_862c_59f3_a33245557810 -->|defined in| 6bee45b2_b649_e251_1fdc_dcf49f8bb331
  style 18bbe714_f107_862c_59f3_a33245557810 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/partners/openai/langchain_openai/llms/base.py lines 637–655

    def get_token_ids(self, text: str) -> list[int]:
        """Get the token IDs using the tiktoken package."""
        if self.custom_get_token_ids is not None:
            return self.custom_get_token_ids(text)
        # tiktoken NOT supported for Python < 3.8
        if sys.version_info[1] < 8:
            return super().get_num_tokens(text)

        model_name = self.tiktoken_model_name or self.model_name
        try:
            enc = tiktoken.encoding_for_model(model_name)
        except KeyError:
            enc = tiktoken.get_encoding("cl100k_base")

        return enc.encode(
            text,
            allowed_special=self.allowed_special,
            disallowed_special=self.disallowed_special,
        )