split_list_of_docs() — langchain Function Reference

Architecture documentation for the split_list_of_docs() function in reduce.py from the langchain codebase.

Function python AgentOrchestration ClassicChains called by 2

Entity Profile

AgentOrchestration→ ClassicChains→ split_list_of_docs() — langchain Function Reference

Dependency Diagram

graph TD
  c98dc7a1_c3dc_2557_0a0f_d34b8e2c7323["split_list_of_docs()"]
  c47d011a_bcd5_947b_be68_ac5c423fea9d["reduce.py"]
  c98dc7a1_c3dc_2557_0a0f_d34b8e2c7323 -->|defined in| c47d011a_bcd5_947b_be68_ac5c423fea9d
  5ec5e85d_0e34_733e_07b9_d13ee52ef47f["_collapse()"]
  5ec5e85d_0e34_733e_07b9_d13ee52ef47f -->|calls| c98dc7a1_c3dc_2557_0a0f_d34b8e2c7323
  4961cdaa_eb66_8549_74f5_80aa6ef431de["_acollapse()"]
  4961cdaa_eb66_8549_74f5_80aa6ef431de -->|calls| c98dc7a1_c3dc_2557_0a0f_d34b8e2c7323
  style c98dc7a1_c3dc_2557_0a0f_d34b8e2c7323 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/langchain_classic/chains/combine_documents/reduce.py lines 30–64

def split_list_of_docs(
    docs: list[Document],
    length_func: Callable,
    token_max: int,
    **kwargs: Any,
) -> list[list[Document]]:
    """Split `Document` objects to subsets that each meet a cumulative len. constraint.

    Args:
        docs: The full list of `Document` objects.
        length_func: Function for computing the cumulative length of a set of `Document`
            objects.
        token_max: The maximum cumulative length of any subset of `Document` objects.
        **kwargs: Arbitrary additional keyword params to pass to each call of the
            `length_func`.

    Returns:
        A `list[list[Document]]`.
    """
    new_result_doc_list = []
    _sub_result_docs = []
    for doc in docs:
        _sub_result_docs.append(doc)
        _num_tokens = length_func(_sub_result_docs, **kwargs)
        if _num_tokens > token_max:
            if len(_sub_result_docs) == 1:
                msg = (
                    "A single document was longer than the context length,"
                    " we cannot handle this."
                )
                raise ValueError(msg)
            new_result_doc_list.append(_sub_result_docs[:-1])
            _sub_result_docs = _sub_result_docs[-1:]
    new_result_doc_list.append(_sub_result_docs)
    return new_result_doc_list