init() — langchain Function Reference

Architecture documentation for the __init__() function in base.py from the langchain codebase.

Function python DocumentProcessing TextSplitters

Entity Profile

DocumentProcessing→ TextSplitters→ __init__() — langchain Function Reference

Dependency Diagram

graph TD
  5ec91302_38b3_134a_1be8_f6a620a79035["__init__()"]
  fee5f91c_52d7_4d25_94a2_c45ac6b35d65["TokenTextSplitter"]
  5ec91302_38b3_134a_1be8_f6a620a79035 -->|defined in| fee5f91c_52d7_4d25_94a2_c45ac6b35d65
  style 5ec91302_38b3_134a_1be8_f6a620a79035 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/langchain_text_splitters/base.py lines 301–337

    def __init__(
        self,
        encoding_name: str = "gpt2",
        model_name: str | None = None,
        allowed_special: Literal["all"] | AbstractSet[str] = set(),
        disallowed_special: Literal["all"] | Collection[str] = "all",
        **kwargs: Any,
    ) -> None:
        """Create a new `TextSplitter`.

        Args:
            encoding_name: The name of the tiktoken encoding to use.
            model_name: The name of the model to use.

                If provided, this will override the `encoding_name`.
            allowed_special: Special tokens that are allowed during encoding.
            disallowed_special: Special tokens that are disallowed during encoding.

        Raises:
            ImportError: If the tiktoken package is not installed.
        """
        super().__init__(**kwargs)
        if not _HAS_TIKTOKEN:
            msg = (
                "Could not import tiktoken python package. "
                "This is needed in order to for TokenTextSplitter. "
                "Please install it with `pip install tiktoken`."
            )
            raise ImportError(msg)

        if model_name is not None:
            enc = tiktoken.encoding_for_model(model_name)
        else:
            enc = tiktoken.get_encoding(encoding_name)
        self._tokenizer = enc
        self._allowed_special = allowed_special
        self._disallowed_special = disallowed_special