LabeledPairwiseStringEvalChain Class — langchain Architecture

Architecture documentation for the LabeledPairwiseStringEvalChain class in eval_chain.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  fa921808_fa38_53ce_81bc_ca579d9b8656["LabeledPairwiseStringEvalChain"]
  6997d03c_6524_f97b_7017_b2f56540bc07["PairwiseStringEvalChain"]
  fa921808_fa38_53ce_81bc_ca579d9b8656 -->|extends| 6997d03c_6524_f97b_7017_b2f56540bc07
  7db782f0_522e_d3f7_666f_da5a1e80af4a["eval_chain.py"]
  fa921808_fa38_53ce_81bc_ca579d9b8656 -->|defined in| 7db782f0_522e_d3f7_666f_da5a1e80af4a
  86fa64c4_797a_aabb_977a_ebffb5c3b3b9["requires_reference()"]
  fa921808_fa38_53ce_81bc_ca579d9b8656 -->|method| 86fa64c4_797a_aabb_977a_ebffb5c3b3b9
  69672bbd_1b9c_053c_d740_7c250335ad18["from_llm()"]
  fa921808_fa38_53ce_81bc_ca579d9b8656 -->|method| 69672bbd_1b9c_053c_d740_7c250335ad18

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/comparison/eval_chain.py lines 411–474

class LabeledPairwiseStringEvalChain(PairwiseStringEvalChain):
    """Labeled Pairwise String Evaluation Chain.

    A chain for comparing two outputs, such as the outputs
    of two models, prompts, or outputs of a single model on similar inputs,
    with labeled preferences.

    Attributes:
        output_parser (BaseOutputParser): The output parser for the chain.

    """

    @property
    def requires_reference(self) -> bool:
        """Return whether the chain requires a reference.

        Returns:
            `True` if the chain requires a reference, `False` otherwise.

        """
        return True

    @classmethod
    def from_llm(
        cls,
        llm: BaseLanguageModel,
        *,
        prompt: PromptTemplate | None = None,
        criteria: CRITERIA_TYPE | str | None = None,
        **kwargs: Any,
    ) -> PairwiseStringEvalChain:
        """Initialize the LabeledPairwiseStringEvalChain from an LLM.

        Args:
            llm: The LLM to use.
            prompt: The prompt to use.
            criteria: The criteria to use.
            **kwargs: Additional keyword arguments.

        Returns:
            The initialized `LabeledPairwiseStringEvalChain`.

        Raises:
            ValueError: If the input variables are not as expected.

        """
        expected_input_vars = {
            "prediction",
            "prediction_b",
            "input",
            "reference",
            "criteria",
        }
        prompt_ = prompt or COMPARISON_TEMPLATE_WITH_REFERENCE
        if expected_input_vars != set(prompt_.input_variables):
            msg = (
                f"Input variables should be {expected_input_vars}, "
                f"but got {prompt_.input_variables}"
            )
            raise ValueError(msg)
        criteria_ = resolve_pairwise_criteria(criteria)
        criteria_str = "\n".join(f"{k}: {v}" for k, v in criteria_.items())
        criteria_str = CRITERIA_INSTRUCTIONS + criteria_str if criteria_str else ""
        return cls(llm=llm, prompt=prompt_.partial(criteria=criteria_str), **kwargs)