get_separators_for_language() — langchain Function Reference

Architecture documentation for the get_separators_for_language() function in character.py from the langchain codebase.

Function python DocumentProcessing TextSplitters called by 1

Entity Profile

DocumentProcessing→ TextSplitters→ get_separators_for_language() — langchain Function Reference

Dependency Diagram

graph TD
  38b02c9d_fd32_4960_73f8_de1c2c0d0827["get_separators_for_language()"]
  22d8d30b_9b36_1532_bb1c_4c9aa03a4bb8["RecursiveCharacterTextSplitter"]
  38b02c9d_fd32_4960_73f8_de1c2c0d0827 -->|defined in| 22d8d30b_9b36_1532_bb1c_4c9aa03a4bb8
  72753d58_c67a_9a23_1006_2861c5bf1d1a["from_language()"]
  72753d58_c67a_9a23_1006_2861c5bf1d1a -->|calls| 38b02c9d_fd32_4960_73f8_de1c2c0d0827
  style 38b02c9d_fd32_4960_73f8_de1c2c0d0827 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/text-splitters/langchain_text_splitters/character.py lines 179–803

    def get_separators_for_language(language: Language) -> list[str]:
        """Retrieve a list of separators specific to the given language.

        Args:
            language: The language for which to get the separators.

        Returns:
            A list of separators appropriate for the specified language.

        Raises:
            ValueError: If the language is not implemented or supported.
        """
        if language in {Language.C, Language.CPP}:
            return [
                # Split along class definitions
                "\nclass ",
                # Split along function definitions
                "\nvoid ",
                "\nint ",
                "\nfloat ",
                "\ndouble ",
                # Split along control flow statements
                "\nif ",
                "\nfor ",
                "\nwhile ",
                "\nswitch ",
                "\ncase ",
                # Split by the normal type of lines
                "\n\n",
                "\n",
                " ",
                "",
            ]
        if language == Language.GO:
            return [
                # Split along function definitions
                "\nfunc ",
                "\nvar ",
                "\nconst ",
                "\ntype ",
                # Split along control flow statements
                "\nif ",
                "\nfor ",
                "\nswitch ",
                "\ncase ",
                # Split by the normal type of lines
                "\n\n",
                "\n",
                " ",
                "",
            ]
        if language == Language.JAVA:
            return [
                # Split along class definitions
                "\nclass ",
                # Split along method definitions
                "\npublic ",
                "\nprotected ",
                "\nprivate ",
                "\nstatic ",
                # Split along control flow statements
                "\nif ",
                "\nfor ",
                "\nwhile ",
                "\nswitch ",
                "\ncase ",
                # Split by the normal type of lines
                "\n\n",
                "\n",
                " ",
                "",
            ]
        if language == Language.KOTLIN:
            return [
                # Split along class definitions
                "\nclass ",
                # Split along method definitions
                "\npublic ",
                "\nprotected ",
                "\nprivate ",
                "\ninternal ",