get_token_ids() — langchain Function Reference

Architecture documentation for the get_token_ids() function in base.py from the langchain codebase.

Function python LangChainCore LanguageModelBase calls 1 called by 1

Entity Profile

LangChainCore→ LanguageModelBase→ get_token_ids() — langchain Function Reference

Dependency Diagram

graph TD
  4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8["get_token_ids()"]
  f5a29046_740d_1358_3615_8cc689b26ca9["BaseLanguageModel"]
  4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8 -->|defined in| f5a29046_740d_1358_3615_8cc689b26ca9
  4b493e08_cb6e_df06_b3b7_c129ab4710fd["get_num_tokens()"]
  4b493e08_cb6e_df06_b3b7_c129ab4710fd -->|calls| 4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8
  1f22d152_ac38_09f3_bb10_6c02757c305a["_get_token_ids_default_method()"]
  4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8 -->|calls| 1f22d152_ac38_09f3_bb10_6c02757c305a
  style 4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/core/langchain_core/language_models/base.py lines 307–319

    def get_token_ids(self, text: str) -> list[int]:
        """Return the ordered IDs of the tokens in a text.

        Args:
            text: The string input to tokenize.

        Returns:
            A list of IDs corresponding to the tokens in the text, in order they occur
                in the text.
        """
        if self.custom_get_token_ids is not None:
            return self.custom_get_token_ids(text)
        return _get_token_ids_default_method(text)