test_deduplication() — langchain Function Reference

Architecture documentation for the test_deduplication() function in test_indexing.py from the langchain codebase.

Function python CoreAbstractions Serialization

Entity Profile

CoreAbstractions→ Serialization→ test_deduplication() — langchain Function Reference

Dependency Diagram

graph TD
  dd3964a1_ef1f_eb48_040d_190b2883429c["test_deduplication()"]
  576ad89d_c8dc_eddf_9cd2_c8ae0e7c9978["test_indexing.py"]
  dd3964a1_ef1f_eb48_040d_190b2883429c -->|defined in| 576ad89d_c8dc_eddf_9cd2_c8ae0e7c9978
  style dd3964a1_ef1f_eb48_040d_190b2883429c fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/core/tests/unit_tests/indexing/test_indexing.py lines 1997–2024

def test_deduplication(
    record_manager: InMemoryRecordManager, vector_store: VectorStore
) -> None:
    """Check edge case when loader returns no new docs."""
    docs = [
        Document(
            page_content="This is a test document.",
            metadata={"source": "1"},
        ),
        Document(
            page_content="This is a test document.",
            metadata={"source": "1"},
        ),
    ]

    # Should result in only a single document being added
    assert index(
        docs,
        record_manager,
        vector_store,
        cleanup="full",
        key_encoder="sha256",
    ) == {
        "num_added": 1,
        "num_deleted": 0,
        "num_skipped": 1,
        "num_updated": 0,
    }