test_deduplication() — langchain Function Reference

Architecture documentation for the test_deduplication() function in test_indexing.py from the langchain codebase.

Function python LangChainCore ApiManagement

Entity Profile

LangChainCore→ ApiManagement→ test_deduplication() — langchain Function Reference

Dependency Diagram

graph TD
  62ad5369_6233_852b_904a_2202bb93b76b["test_deduplication()"]
  9bc4e8b6_769a_ae11_3fc1_309cb678c248["test_indexing.py"]
  62ad5369_6233_852b_904a_2202bb93b76b -->|defined in| 9bc4e8b6_769a_ae11_3fc1_309cb678c248
  style 62ad5369_6233_852b_904a_2202bb93b76b fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/tests/unit_tests/indexes/test_indexing.py lines 1174–1196

def test_deduplication(
    record_manager: SQLRecordManager,
    vector_store: VectorStore,
) -> None:
    """Check edge case when loader returns no new docs."""
    docs = [
        Document(
            page_content="This is a test document.",
            metadata={"source": "1"},
        ),
        Document(
            page_content="This is a test document.",
            metadata={"source": "1"},
        ),
    ]

    # Should result in only a single document being added
    assert index(docs, record_manager, vector_store, cleanup="full") == {
        "num_added": 1,
        "num_deleted": 0,
        "num_skipped": 1,
        "num_updated": 0,
    }