init() — langchain Function Reference

Architecture documentation for the __init__() function in sentence_transformers.py from the langchain codebase.

Function python DataProcessing TextSplitters calls 1

Entity Profile

DataProcessing→ TextSplitters→ __init__() — langchain Function Reference

Dependency Diagram

graph TD
  967804df_0a20_11c7_b284_11c972adc069["__init__()"]
  e67e8512_91e2_e59f_208a_86a9aef296a7["SentenceTransformersTokenTextSplitter"]
  967804df_0a20_11c7_b284_11c972adc069 -->|defined in| e67e8512_91e2_e59f_208a_86a9aef296a7
  ebb3da73_ad87_a7f9_f2f7_2d72a26cbf90["_initialize_chunk_configuration()"]
  967804df_0a20_11c7_b284_11c972adc069 -->|calls| ebb3da73_ad87_a7f9_f2f7_2d72a26cbf90
  style 967804df_0a20_11c7_b284_11c972adc069 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/langchain_text_splitters/sentence_transformers.py lines 23–55

    def __init__(
        self,
        chunk_overlap: int = 50,
        model_name: str = "sentence-transformers/all-mpnet-base-v2",
        tokens_per_chunk: int | None = None,
        **kwargs: Any,
    ) -> None:
        """Create a new `TextSplitter`.

        Args:
            chunk_overlap: The number of tokens to overlap between chunks.
            model_name: The name of the sentence transformer model to use.
            tokens_per_chunk: The number of tokens per chunk.

                If `None`, uses the maximum tokens allowed by the model.

        Raises:
            ImportError: If the `sentence_transformers` package is not installed.
        """
        super().__init__(**kwargs, chunk_overlap=chunk_overlap)

        if not _HAS_SENTENCE_TRANSFORMERS:
            msg = (
                "Could not import sentence_transformers python package. "
                "This is needed in order to use SentenceTransformersTokenTextSplitter. "
                "Please install it with `pip install sentence-transformers`."
            )
            raise ImportError(msg)

        self.model_name = model_name
        self._model = SentenceTransformer(self.model_name)
        self.tokenizer = self._model.tokenizer
        self._initialize_chunk_configuration(tokens_per_chunk=tokens_per_chunk)