langchain Architecture Docs
Home / File/ evaluation.py — langchain Source File

evaluation.py — langchain Source File

Architecture documentation for evaluation.py, a python file in the langchain codebase. 15 imports, 0 dependents.

File python CoreAbstractions Serialization 15 imports 2 functions 1 classes

Entity Profile

Dependency Diagram

graph LR
  7d4044f3_fdda_24d6_56ef_c4801aa638c4["evaluation.py"]
  2a7f66a7_8738_3d47_375b_70fcaa6ac169["logging"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 2a7f66a7_8738_3d47_375b_70fcaa6ac169
  242d0b7d_a8ef_b66d_169b_c791b32a9cc9["threading"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 242d0b7d_a8ef_b66d_169b_c791b32a9cc9
  7610c6f7_1851_b8bd_58e0_076478b94a9c["weakref"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 7610c6f7_1851_b8bd_58e0_076478b94a9c
  082415a8_067b_221f_7984_07d87009267d["concurrent.futures"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 082415a8_067b_221f_7984_07d87009267d
  8e2034b7_ceb8_963f_29fc_2ea6b50ef9b3["typing"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 8e2034b7_ceb8_963f_29fc_2ea6b50ef9b3
  8dfa0cac_d802_3ccd_f710_43a5e70da3a5["uuid"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 8dfa0cac_d802_3ccd_f710_43a5e70da3a5
  023156c8_e306_6129_d953_9f1dac71e6fd["langsmith"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 023156c8_e306_6129_d953_9f1dac71e6fd
  a094adde_9a9c_1316_c7fc_d5846bad792c["langsmith.evaluation.evaluator"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> a094adde_9a9c_1316_c7fc_d5846bad792c
  2b6e6b96_8f94_4170_1803_67a415a5e13a["langchain_core.tracers"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 2b6e6b96_8f94_4170_1803_67a415a5e13a
  707209ca_c3d6_f9c6_7ec7_2b2df92c9aa3["langchain_core.tracers._compat"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 707209ca_c3d6_f9c6_7ec7_2b2df92c9aa3
  59d7001f_fb28_1819_31fc_7fb0380a8b32["langchain_core.tracers.base"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 59d7001f_fb28_1819_31fc_7fb0380a8b32
  0dd80fba_b72d_91a8_6113_3bb8b0a7af05["langchain_core.tracers.context"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> 0dd80fba_b72d_91a8_6113_3bb8b0a7af05
  ec9bee24_c773_b0b6_adc6_6ee890d32c05["langchain_core.tracers.langchain"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> ec9bee24_c773_b0b6_adc6_6ee890d32c05
  cfe2bde5_180e_e3b0_df2b_55b3ebaca8e7["collections.abc"]
  7d4044f3_fdda_24d6_56ef_c4801aa638c4 --> cfe2bde5_180e_e3b0_df2b_55b3ebaca8e7
  style 7d4044f3_fdda_24d6_56ef_c4801aa638c4 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

"""A tracer that runs evaluators over completed runs."""

from __future__ import annotations

import logging
import threading
import weakref
from concurrent.futures import Future, ThreadPoolExecutor, wait
from typing import TYPE_CHECKING, Any, cast
from uuid import UUID

import langsmith
from langsmith.evaluation.evaluator import EvaluationResult, EvaluationResults

from langchain_core.tracers import langchain as langchain_tracer
from langchain_core.tracers._compat import run_copy
from langchain_core.tracers.base import BaseTracer
from langchain_core.tracers.context import tracing_v2_enabled
from langchain_core.tracers.langchain import _get_executor

if TYPE_CHECKING:
    from collections.abc import Sequence

    from langchain_core.tracers.schemas import Run

logger = logging.getLogger(__name__)

_TRACERS: weakref.WeakSet[EvaluatorCallbackHandler] = weakref.WeakSet()


def wait_for_all_evaluators() -> None:
    """Wait for all tracers to finish."""
    for tracer in list(_TRACERS):
        if tracer is not None:
            tracer.wait_for_futures()


class EvaluatorCallbackHandler(BaseTracer):
    """Tracer that runs a run evaluator whenever a run is persisted.

    Attributes:
        client: The LangSmith client instance used for evaluating the runs.
    """

    name: str = "evaluator_callback_handler"

    example_id: UUID | None = None
    """The example ID associated with the runs."""

    client: langsmith.Client
    """The LangSmith client instance used for evaluating the runs."""

    evaluators: Sequence[langsmith.RunEvaluator] = ()
    """The sequence of run evaluators to be executed."""

    executor: ThreadPoolExecutor | None = None
    """The thread pool executor used for running the evaluators."""

    futures: weakref.WeakSet[Future] = weakref.WeakSet()
    """The set of futures representing the running evaluators."""
// ... (167 more lines)

Domain

Subdomains

Functions

Classes

Dependencies

  • collections.abc
  • concurrent.futures
  • langchain_core.tracers
  • langchain_core.tracers._compat
  • langchain_core.tracers.base
  • langchain_core.tracers.context
  • langchain_core.tracers.langchain
  • langchain_core.tracers.schemas
  • langsmith
  • langsmith.evaluation.evaluator
  • logging
  • threading
  • typing
  • uuid
  • weakref

Source

Frequently Asked Questions

What does evaluation.py do?
evaluation.py is a source file in the langchain codebase, written in python. It belongs to the CoreAbstractions domain, Serialization subdomain.
What functions are defined in evaluation.py?
evaluation.py defines 2 function(s): collections, wait_for_all_evaluators.
What does evaluation.py depend on?
evaluation.py imports 15 module(s): collections.abc, concurrent.futures, langchain_core.tracers, langchain_core.tracers._compat, langchain_core.tracers.base, langchain_core.tracers.context, langchain_core.tracers.langchain, langchain_core.tracers.schemas, and 7 more.
Where is evaluation.py in the architecture?
evaluation.py is located at libs/core/langchain_core/tracers/evaluation.py (domain: CoreAbstractions, subdomain: Serialization, directory: libs/core/langchain_core/tracers).

Analyze Your Own Codebase

Get architecture documentation, dependency graphs, and domain analysis for your codebase in minutes.

Try Supermodel Free