langchain Architecture Docs
Home / File/ string_run_evaluator.py — langchain Source File

string_run_evaluator.py — langchain Source File

Architecture documentation for string_run_evaluator.py, a python file in the langchain codebase. 15 imports, 0 dependents.

File python LangChainCore Runnables 15 imports 1 functions 6 classes

Entity Profile

Dependency Diagram

graph LR
  2f0b23f2_7760_d68c_7feb_721c5231c4ec["string_run_evaluator.py"]
  e27da29f_a1f7_49f3_84d5_6be4cb4125c8["logging"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> e27da29f_a1f7_49f3_84d5_6be4cb4125c8
  02f66451_d2a9_e7c3_9765_c3a7594721ad["uuid"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> 02f66451_d2a9_e7c3_9765_c3a7594721ad
  50e20440_a135_6be3_a5a5_67791be5a2a6["abc"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> 50e20440_a135_6be3_a5a5_67791be5a2a6
  feec1ec4_6917_867b_d228_b134d0ff8099["typing"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> feec1ec4_6917_867b_d228_b134d0ff8099
  e61aa479_9dc0_09a0_8864_cbf23b8b506c["langchain_core.callbacks.manager"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> e61aa479_9dc0_09a0_8864_cbf23b8b506c
  f4426504_dae2_8ae3_99a7_67cc4d1997ba["langchain_core.load.dump"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> f4426504_dae2_8ae3_99a7_67cc4d1997ba
  679bd51a_077f_a2c5_f968_44da37489a0d["langchain_core.load.load"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> 679bd51a_077f_a2c5_f968_44da37489a0d
  e082ede2_868f_7149_8977_a2af7e5aeb38["langchain_core.load.serializable"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> e082ede2_868f_7149_8977_a2af7e5aeb38
  9444498b_8066_55c7_b3a2_1d90c4162a32["langchain_core.messages"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> 9444498b_8066_55c7_b3a2_1d90c4162a32
  b8aff3f8_6287_d5fe_af3e_1ecda79d9656["langsmith"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> b8aff3f8_6287_d5fe_af3e_1ecda79d9656
  fda1273a_102f_d6f6_7c3e_a3cece2788bd["langsmith.schemas"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> fda1273a_102f_d6f6_7c3e_a3cece2788bd
  f85fae70_1011_eaec_151c_4083140ae9e5["typing_extensions"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> f85fae70_1011_eaec_151c_4083140ae9e5
  9a0fc770_8c3f_14bc_3c7d_37852927778e["langchain_classic.chains.base"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> 9a0fc770_8c3f_14bc_3c7d_37852927778e
  37291248_07a6_a05c_821a_71cc0592429f["langchain_classic.evaluation.schema"]
  2f0b23f2_7760_d68c_7feb_721c5231c4ec --> 37291248_07a6_a05c_821a_71cc0592429f
  style 2f0b23f2_7760_d68c_7feb_721c5231c4ec fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

"""Run evaluator wrapper for string evaluators."""

from __future__ import annotations

import logging
import uuid
from abc import abstractmethod
from typing import Any, cast

from langchain_core.callbacks.manager import (
    AsyncCallbackManagerForChainRun,
    CallbackManagerForChainRun,
)
from langchain_core.load.dump import dumpd
from langchain_core.load.load import load
from langchain_core.load.serializable import Serializable
from langchain_core.messages import BaseMessage, get_buffer_string, messages_from_dict
from langsmith import EvaluationResult, RunEvaluator
from langsmith.schemas import DataType, Example, Run
from typing_extensions import override

from langchain_classic.chains.base import Chain
from langchain_classic.evaluation.schema import StringEvaluator
from langchain_classic.schema import RUN_KEY

_logger = logging.getLogger(__name__)


def _get_messages_from_run_dict(messages: list[dict]) -> list[BaseMessage]:
    if not messages:
        return []
    first_message = messages[0]
    if "lc" in first_message:
        return [load(dumpd(message)) for message in messages]
    return messages_from_dict(messages)


class StringRunMapper(Serializable):
    """Extract items to evaluate from the run object."""

    @property
    def output_keys(self) -> list[str]:
        """The keys to extract from the run."""
        return ["prediction", "input"]

    @abstractmethod
    def map(self, run: Run) -> dict[str, str]:
        """Maps the Run to a dictionary."""

    def __call__(self, run: Run) -> dict[str, str]:
        """Maps the Run to a dictionary."""
        if not run.outputs:
            msg = f"Run {run.id} has no outputs to evaluate."
            raise ValueError(msg)
        return self.map(run)


class LLMStringRunMapper(StringRunMapper):
    """Extract items to evaluate from the run object."""

// ... (418 more lines)

Domain

Subdomains

Functions

Classes

Dependencies

  • abc
  • langchain_classic.chains.base
  • langchain_classic.evaluation.schema
  • langchain_classic.schema
  • langchain_core.callbacks.manager
  • langchain_core.load.dump
  • langchain_core.load.load
  • langchain_core.load.serializable
  • langchain_core.messages
  • langsmith
  • langsmith.schemas
  • logging
  • typing
  • typing_extensions
  • uuid

Source

Frequently Asked Questions

What does string_run_evaluator.py do?
string_run_evaluator.py is a source file in the langchain codebase, written in python. It belongs to the LangChainCore domain, Runnables subdomain.
What functions are defined in string_run_evaluator.py?
string_run_evaluator.py defines 1 function(s): _get_messages_from_run_dict.
What does string_run_evaluator.py depend on?
string_run_evaluator.py imports 15 module(s): abc, langchain_classic.chains.base, langchain_classic.evaluation.schema, langchain_classic.schema, langchain_core.callbacks.manager, langchain_core.load.dump, langchain_core.load.load, langchain_core.load.serializable, and 7 more.
Where is string_run_evaluator.py in the architecture?
string_run_evaluator.py is located at libs/langchain/langchain_classic/smith/evaluation/string_run_evaluator.py (domain: LangChainCore, subdomain: Runnables, directory: libs/langchain/langchain_classic/smith/evaluation).

Analyze Your Own Codebase

Get architecture documentation, dependency graphs, and domain analysis for your codebase in minutes.

Try Supermodel Free