_generate_rest_batches() — langchain Function Reference

Architecture documentation for the _generate_rest_batches() function in vectorstores.py from the langchain codebase.

Function python CoreAbstractions Serialization calls 2 called by 1

Entity Profile

CoreAbstractions→ Serialization→ _generate_rest_batches() — langchain Function Reference

Dependency Diagram

graph TD
  1e94f47d_25cd_1c20_8165_13a044a957c8["_generate_rest_batches()"]
  bf62db79_4217_463c_798f_6f8528ed0d6e["Qdrant"]
  1e94f47d_25cd_1c20_8165_13a044a957c8 -->|defined in| bf62db79_4217_463c_798f_6f8528ed0d6e
  4da39fb6_0b3c_26e3_93cd_edfd42d30436["add_texts()"]
  4da39fb6_0b3c_26e3_93cd_edfd42d30436 -->|calls| 1e94f47d_25cd_1c20_8165_13a044a957c8
  2201d155_e4c2_dcd1_f107_11fdffb4b7bf["_embed_texts()"]
  1e94f47d_25cd_1c20_8165_13a044a957c8 -->|calls| 2201d155_e4c2_dcd1_f107_11fdffb4b7bf
  818b1334_70e9_80aa_1f55_1c72ba7d484c["_build_payloads()"]
  1e94f47d_25cd_1c20_8165_13a044a957c8 -->|calls| 818b1334_70e9_80aa_1f55_1c72ba7d484c
  style 1e94f47d_25cd_1c20_8165_13a044a957c8 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/partners/qdrant/langchain_qdrant/vectorstores.py lines 2197–2236

    def _generate_rest_batches(
        self,
        texts: Iterable[str],
        metadatas: list[dict] | None = None,
        ids: Sequence[str] | None = None,
        batch_size: int = 64,
    ) -> Generator[tuple[list[str], list[models.PointStruct]], None, None]:
        texts_iterator = iter(texts)
        metadatas_iterator = iter(metadatas or [])
        ids_iterator = iter(ids or [uuid.uuid4().hex for _ in iter(texts)])
        while batch_texts := list(islice(texts_iterator, batch_size)):
            # Take the corresponding metadata and id for each text in a batch
            batch_metadatas = list(islice(metadatas_iterator, batch_size)) or None
            batch_ids = list(islice(ids_iterator, batch_size))

            # Generate the embeddings for all the texts in a batch
            batch_embeddings = self._embed_texts(batch_texts)

            points = [
                models.PointStruct(
                    id=point_id,
                    vector=vector  # type: ignore[arg-type]
                    if self.vector_name is None
                    else {self.vector_name: vector},
                    payload=payload,
                )
                for point_id, vector, payload in zip(
                    batch_ids,
                    batch_embeddings,
                    self._build_payloads(
                        batch_texts,
                        batch_metadatas,
                        self.content_payload_key,
                        self.metadata_payload_key,
                    ),
                    strict=False,
                )
            ]

            yield batch_ids, points