get_num_tokens() — langchain Function Reference

Architecture documentation for the get_num_tokens() function in base.py from the langchain codebase.

Function python LangChainCore LanguageModelBase calls 1 called by 1

Entity Profile

LangChainCore→ LanguageModelBase→ get_num_tokens() — langchain Function Reference

Dependency Diagram

graph TD
  4b493e08_cb6e_df06_b3b7_c129ab4710fd["get_num_tokens()"]
  f5a29046_740d_1358_3615_8cc689b26ca9["BaseLanguageModel"]
  4b493e08_cb6e_df06_b3b7_c129ab4710fd -->|defined in| f5a29046_740d_1358_3615_8cc689b26ca9
  20140000_1d28_439f_e76d_4451abf1068a["get_num_tokens_from_messages()"]
  20140000_1d28_439f_e76d_4451abf1068a -->|calls| 4b493e08_cb6e_df06_b3b7_c129ab4710fd
  4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8["get_token_ids()"]
  4b493e08_cb6e_df06_b3b7_c129ab4710fd -->|calls| 4bf94ed3_6d3e_2b7c_07b9_290dd3f7edf8
  style 4b493e08_cb6e_df06_b3b7_c129ab4710fd fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/core/langchain_core/language_models/base.py lines 321–336

    def get_num_tokens(self, text: str) -> int:
        """Get the number of tokens present in the text.

        Useful for checking if an input fits in a model's context window.

        This should be overridden by model-specific implementations to provide accurate
        token counts via model-specific tokenizers.

        Args:
            text: The string input to tokenize.

        Returns:
            The integer number of tokens in the text.

        """
        return len(self.get_token_ids(text))