StringEvaluator Class — langchain Architecture

Architecture documentation for the StringEvaluator class in schema.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b["StringEvaluator"]
  c3049278_87a9_b830_293b_a3dcf7ed9dd3["_EvalArgsMixin"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|extends| c3049278_87a9_b830_293b_a3dcf7ed9dd3
  b8a2957b_df2e_04bc_f892_0752a91e1a55["schema.py"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|defined in| b8a2957b_df2e_04bc_f892_0752a91e1a55
  1902cfda_f719_4ec9_5970_5030237ac035["evaluation_name()"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|method| 1902cfda_f719_4ec9_5970_5030237ac035
  a707a026_9e7c_9d89_5137_ef8b21f72279["requires_reference()"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|method| a707a026_9e7c_9d89_5137_ef8b21f72279
  1e6583e3_4fcb_5943_1229_d0bcd6903c51["_evaluate_strings()"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|method| 1e6583e3_4fcb_5943_1229_d0bcd6903c51
  18f54728_8c17_5647_a4f6_df52b9f1d794["_aevaluate_strings()"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|method| 18f54728_8c17_5647_a4f6_df52b9f1d794
  89b45341_33d2_f0e2_5126_1ae243434a47["evaluate_strings()"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|method| 89b45341_33d2_f0e2_5126_1ae243434a47
  7005c138_072e_7038_918d_35c947f2e4ac["aevaluate_strings()"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b -->|method| 7005c138_072e_7038_918d_35c947f2e4ac

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/schema.py lines 136–262

class StringEvaluator(_EvalArgsMixin, ABC):
    """String evaluator interface.

    Grade, tag, or otherwise evaluate predictions relative to their inputs
    and/or reference labels.
    """

    @property
    def evaluation_name(self) -> str:
        """The name of the evaluation."""
        return self.__class__.__name__

    @property
    def requires_reference(self) -> bool:
        """Whether this evaluator requires a reference label."""
        return False

    @abstractmethod
    def _evaluate_strings(
        self,
        *,
        prediction: str | Any,
        reference: str | Any | None = None,
        input: str | Any | None = None,  # noqa: A002
        **kwargs: Any,
    ) -> dict:
        """Evaluate Chain or LLM output, based on optional input and label.

        Args:
            prediction: The LLM or chain prediction to evaluate.
            reference: The reference label to evaluate against.
            input: The input to consider during evaluation.
            **kwargs: Additional keyword arguments, including callbacks, tags, etc.

        Returns:
            The evaluation results containing the score or value.
            It is recommended that the dictionary contain the following keys:
                 - score: the score of the evaluation, if applicable.
                 - value: the string value of the evaluation, if applicable.
                 - reasoning: the reasoning for the evaluation, if applicable.
        """

    async def _aevaluate_strings(
        self,
        *,
        prediction: str | Any,
        reference: str | Any | None = None,
        input: str | Any | None = None,  # noqa: A002
        **kwargs: Any,
    ) -> dict:
        """Asynchronously evaluate Chain or LLM output, based on optional input and label.

        Args:
            prediction: The LLM or chain prediction to evaluate.
            reference: The reference label to evaluate against.
            input: The input to consider during evaluation.
            **kwargs: Additional keyword arguments, including callbacks, tags, etc.

        Returns:
            The evaluation results containing the score or value.
            It is recommended that the dictionary contain the following keys:
                 - score: the score of the evaluation, if applicable.
                 - value: the string value of the evaluation, if applicable.
                 - reasoning: the reasoning for the evaluation, if applicable.
        """  # noqa: E501
        return await run_in_executor(
            None,
            self._evaluate_strings,
            prediction=prediction,
            reference=reference,
            input=input,
            **kwargs,
        )

    def evaluate_strings(
        self,
        *,
        prediction: str,
        reference: str | None = None,
        input: str | None = None,  # noqa: A002
        **kwargs: Any,