count_tokens() — langchain Function Reference

Architecture documentation for the count_tokens() function in sentence_transformers.py from the langchain codebase.

Function python DocumentProcessing TextSplitters calls 2 called by 1

Entity Profile

DocumentProcessing→ TextSplitters→ count_tokens() — langchain Function Reference

Dependency Diagram

graph TD
  0d40cab5_841b_edff_d310_8ac26d084015["count_tokens()"]
  059dfb7c_30ac_164c_5a3e_708a02d51601["SentenceTransformersTokenTextSplitter"]
  0d40cab5_841b_edff_d310_8ac26d084015 -->|defined in| 059dfb7c_30ac_164c_5a3e_708a02d51601
  0d40cab5_841b_edff_d310_8ac26d084015["count_tokens()"]
  0d40cab5_841b_edff_d310_8ac26d084015 -->|calls| 0d40cab5_841b_edff_d310_8ac26d084015
  0172994e_4917_2bec_356d_ac072e832565["_encode()"]
  0d40cab5_841b_edff_d310_8ac26d084015 -->|calls| 0172994e_4917_2bec_356d_ac072e832565
  0d40cab5_841b_edff_d310_8ac26d084015["count_tokens()"]
  0d40cab5_841b_edff_d310_8ac26d084015 -->|calls| 0d40cab5_841b_edff_d310_8ac26d084015
  style 0d40cab5_841b_edff_d310_8ac26d084015 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/langchain_text_splitters/sentence_transformers.py lines 101–113

    def count_tokens(self, *, text: str) -> int:
        """Counts the number of tokens in the given text.

        This method encodes the input text using a private `_encode` method and
        calculates the total number of tokens in the encoded result.

        Args:
            text: The input text for which the token count is calculated.

        Returns:
            The number of tokens in the encoded text.
        """
        return len(self._encode(text))