get_token_ids() — langchain Function Reference

Architecture documentation for the get_token_ids() function in base.py from the langchain codebase.

Function python LangChainCore LanguageModelBase calls 1

Entity Profile

LangChainCore→ LanguageModelBase→ get_token_ids() — langchain Function Reference

Dependency Diagram

graph TD
  4844666f_f624_b0a3_473d_64deaf5a46b1["get_token_ids()"]
  2a683305_667b_3567_cab9_9f77e29d4afa["BaseChatOpenAI"]
  4844666f_f624_b0a3_473d_64deaf5a46b1 -->|defined in| 2a683305_667b_3567_cab9_9f77e29d4afa
  927425a4_5f3b_5196_e538_3d7cee86141b["_get_encoding_model()"]
  4844666f_f624_b0a3_473d_64deaf5a46b1 -->|calls| 927425a4_5f3b_5196_e538_3d7cee86141b
  style 4844666f_f624_b0a3_473d_64deaf5a46b1 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/partners/openai/langchain_openai/chat_models/base.py lines 1757–1765

    def get_token_ids(self, text: str) -> list[int]:
        """Get the tokens present in the text with tiktoken package."""
        if self.custom_get_token_ids is not None:
            return self.custom_get_token_ids(text)
        # tiktoken NOT supported for Python 3.7 or below
        if sys.version_info[1] <= 7:
            return super().get_token_ids(text)
        _, encoding_model = self._get_encoding_model()
        return encoding_model.encode(text)