_deduplicate_in_order() — langchain Function Reference

Architecture documentation for the _deduplicate_in_order() function in api.py from the langchain codebase.

Function python CoreAbstractions Serialization called by 2

Entity Profile

CoreAbstractions→ Serialization→ _deduplicate_in_order() — langchain Function Reference

Dependency Diagram

graph TD
  5aadbe30_58aa_8bc1_2fcd_e8fc84b92311["_deduplicate_in_order()"]
  203188c0_72d6_6932_bc21_edf25c4c00ef["api.py"]
  5aadbe30_58aa_8bc1_2fcd_e8fc84b92311 -->|defined in| 203188c0_72d6_6932_bc21_edf25c4c00ef
  5721a97d_0581_0694_e3e6_0ae44f2b3fb0["index()"]
  5721a97d_0581_0694_e3e6_0ae44f2b3fb0 -->|calls| 5aadbe30_58aa_8bc1_2fcd_e8fc84b92311
  02b67c59_d093_f33d_633c_d77332eb191e["aindex()"]
  02b67c59_d093_f33d_633c_d77332eb191e -->|calls| 5aadbe30_58aa_8bc1_2fcd_e8fc84b92311
  style 5aadbe30_58aa_8bc1_2fcd_e8fc84b92311 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/core/langchain_core/indexing/api.py lines 133–144

def _deduplicate_in_order(
    hashed_documents: Iterable[Document],
) -> Iterator[Document]:
    """Deduplicate a list of hashed documents while preserving order."""
    seen: set[str] = set()

    for hashed_doc in hashed_documents:
        if hashed_doc.id not in seen:
            # At this stage, the id is guaranteed to be a string.
            # Avoiding unnecessary run time checks.
            seen.add(cast("str", hashed_doc.id))
            yield hashed_doc