AgentTrajectoryEvaluator Class — langchain Architecture

Architecture documentation for the AgentTrajectoryEvaluator class in schema.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  942e26b5_86b7_44c9_5c6d_e12c17d223b8["AgentTrajectoryEvaluator"]
  c3049278_87a9_b830_293b_a3dcf7ed9dd3["_EvalArgsMixin"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|extends| c3049278_87a9_b830_293b_a3dcf7ed9dd3
  b8a2957b_df2e_04bc_f892_0752a91e1a55["schema.py"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|defined in| b8a2957b_df2e_04bc_f892_0752a91e1a55
  f575f00a_7419_ba10_c6f9_9136902c46a5["requires_input()"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|method| f575f00a_7419_ba10_c6f9_9136902c46a5
  dead35f2_5d91_cb2b_4630_f4d666053668["_evaluate_agent_trajectory()"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|method| dead35f2_5d91_cb2b_4630_f4d666053668
  a6be454e_d689_fe5e_3358_4c8b8a8a2fa6["_aevaluate_agent_trajectory()"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|method| a6be454e_d689_fe5e_3358_4c8b8a8a2fa6
  f9083201_751d_03a7_715e_48b36d32dabf["evaluate_agent_trajectory()"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|method| f9083201_751d_03a7_715e_48b36d32dabf
  b659b7f6_5fb9_1be0_d430_ab613252fe80["aevaluate_agent_trajectory()"]
  942e26b5_86b7_44c9_5c6d_e12c17d223b8 -->|method| b659b7f6_5fb9_1be0_d430_ab613252fe80

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/schema.py lines 383–507

class AgentTrajectoryEvaluator(_EvalArgsMixin, ABC):
    """Interface for evaluating agent trajectories."""

    @property
    def requires_input(self) -> bool:
        """Whether this evaluator requires an input string."""
        return True

    @abstractmethod
    def _evaluate_agent_trajectory(
        self,
        *,
        prediction: str,
        agent_trajectory: Sequence[tuple[AgentAction, str]],
        input: str,  # noqa: A002
        reference: str | None = None,
        **kwargs: Any,
    ) -> dict:
        """Evaluate a trajectory.

        Args:
            prediction: The final predicted response.
            agent_trajectory:
                The intermediate steps forming the agent trajectory.
            input: The input to the agent.
            reference: The reference answer.
            **kwargs: Additional keyword arguments.

        Returns:
            The evaluation result.
        """

    async def _aevaluate_agent_trajectory(
        self,
        *,
        prediction: str,
        agent_trajectory: Sequence[tuple[AgentAction, str]],
        input: str,  # noqa: A002
        reference: str | None = None,
        **kwargs: Any,
    ) -> dict:
        """Asynchronously evaluate a trajectory.

        Args:
            prediction: The final predicted response.
            agent_trajectory:
                The intermediate steps forming the agent trajectory.
            input: The input to the agent.
            reference: The reference answer.
            **kwargs: Additional keyword arguments.

        Returns:
            The evaluation result.
        """
        return await run_in_executor(
            None,
            self._evaluate_agent_trajectory,
            prediction=prediction,
            agent_trajectory=agent_trajectory,
            reference=reference,
            input=input,
            **kwargs,
        )

    def evaluate_agent_trajectory(
        self,
        *,
        prediction: str,
        agent_trajectory: Sequence[tuple[AgentAction, str]],
        input: str,  # noqa: A002
        reference: str | None = None,
        **kwargs: Any,
    ) -> dict:
        """Evaluate a trajectory.

        Args:
            prediction: The final predicted response.
            agent_trajectory:
                The intermediate steps forming the agent trajectory.
            input: The input to the agent.
            reference: The reference answer.