_encode() — langchain Function Reference

Architecture documentation for the _encode() function in sentence_transformers.py from the langchain codebase.

Function python DocumentProcessing TextSplitters calls 1 called by 3

Entity Profile

DocumentProcessing→ TextSplitters→ _encode() — langchain Function Reference

Dependency Diagram

graph TD
  0172994e_4917_2bec_356d_ac072e832565["_encode()"]
  059dfb7c_30ac_164c_5a3e_708a02d51601["SentenceTransformersTokenTextSplitter"]
  0172994e_4917_2bec_356d_ac072e832565 -->|defined in| 059dfb7c_30ac_164c_5a3e_708a02d51601
  a7a0dc6a_7652_b658_2bb9_d850d67979ca["split_text()"]
  a7a0dc6a_7652_b658_2bb9_d850d67979ca -->|calls| 0172994e_4917_2bec_356d_ac072e832565
  0d40cab5_841b_edff_d310_8ac26d084015["count_tokens()"]
  0d40cab5_841b_edff_d310_8ac26d084015 -->|calls| 0172994e_4917_2bec_356d_ac072e832565
  0172994e_4917_2bec_356d_ac072e832565["_encode()"]
  0172994e_4917_2bec_356d_ac072e832565 -->|calls| 0172994e_4917_2bec_356d_ac072e832565
  0172994e_4917_2bec_356d_ac072e832565["_encode()"]
  0172994e_4917_2bec_356d_ac072e832565 -->|calls| 0172994e_4917_2bec_356d_ac072e832565
  style 0172994e_4917_2bec_356d_ac072e832565 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/langchain_text_splitters/sentence_transformers.py lines 117–123

    def _encode(self, text: str) -> list[int]:
        token_ids_with_start_and_end_token_ids = self.tokenizer.encode(
            text,
            max_length=self._max_length_equal_32_bit_integer,
            truncation="do_not_truncate",
        )
        return cast("list[int]", token_ids_with_start_and_end_token_ids)