NLTKTextSplitter Class — langchain Architecture

Architecture documentation for the NLTKTextSplitter class in nltk.py from the langchain codebase.

Class python DocumentProcessing

Entity Profile

DocumentProcessing→ NLTKTextSplitter Class — langchain Architecture

Dependency Diagram

graph TD
  213eacdd_0034_0c08_d507_c27d0748affc["NLTKTextSplitter"]
  c86e37d5_f962_cc1e_9821_b665e1359ae8["TextSplitter"]
  213eacdd_0034_0c08_d507_c27d0748affc -->|extends| c86e37d5_f962_cc1e_9821_b665e1359ae8
  0a45c4a1_846f_03df_b842_eb6b566c6404["nltk.py"]
  213eacdd_0034_0c08_d507_c27d0748affc -->|defined in| 0a45c4a1_846f_03df_b842_eb6b566c6404
  91006521_b301_e1cb_c4a5_d534eba20af9["__init__()"]
  213eacdd_0034_0c08_d507_c27d0748affc -->|method| 91006521_b301_e1cb_c4a5_d534eba20af9
  18590a0b_5de9_0196_2d21_608d79b9ef70["split_text()"]
  213eacdd_0034_0c08_d507_c27d0748affc -->|method| 18590a0b_5de9_0196_2d21_608d79b9ef70

Relationship Graph

Source Code

libs/text-splitters/langchain_text_splitters/nltk.py lines 19–72

class NLTKTextSplitter(TextSplitter):
    """Splitting text using NLTK package."""

    def __init__(
        self,
        separator: str = "\n\n",
        language: str = "english",
        *,
        use_span_tokenize: bool = False,
        **kwargs: Any,
    ) -> None:
        """Initialize the NLTK splitter.

        Args:
            separator: The separator to use when combining splits.
            language: The language to use.
            use_span_tokenize: Whether to use `span_tokenize` instead of
                `sent_tokenize`.

        Raises:
            ImportError: If NLTK is not installed.
            ValueError: If `use_span_tokenize` is `True` and separator is not `''`.
        """
        super().__init__(**kwargs)
        self._separator = separator
        self._language = language
        self._use_span_tokenize = use_span_tokenize
        if self._use_span_tokenize and self._separator:
            msg = "When use_span_tokenize is True, separator should be ''"
            raise ValueError(msg)
        if not _HAS_NLTK:
            msg = "NLTK is not installed, please install it with `pip install nltk`."
            raise ImportError(msg)
        if self._use_span_tokenize:
            self._tokenizer = nltk.tokenize._get_punkt_tokenizer(self._language)  # noqa: SLF001
        else:
            self._tokenizer = nltk.tokenize.sent_tokenize

    @override
    def split_text(self, text: str) -> list[str]:
        # First we naively split the large input into a bunch of smaller ones.
        if self._use_span_tokenize:
            spans = list(self._tokenizer.span_tokenize(text))
            splits = []
            for i, (start, end) in enumerate(spans):
                if i > 0:
                    prev_end = spans[i - 1][1]
                    sentence = text[prev_end:start] + text[start:end]
                else:
                    sentence = text[start:end]
                splits.append(sentence)
        else:
            splits = self._tokenizer(text, language=self._language)
        return self._merge_splits(splits, self._separator)