LabeledScoreStringEvalChain Class — langchain Architecture

Architecture documentation for the LabeledScoreStringEvalChain class in eval_chain.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  df005def_3ccc_8e5a_d8f1_4757105de3a2["LabeledScoreStringEvalChain"]
  a58ec485_afe5_c733_6e70_92d4365f961c["ScoreStringEvalChain"]
  df005def_3ccc_8e5a_d8f1_4757105de3a2 -->|extends| a58ec485_afe5_c733_6e70_92d4365f961c
  66e0c4fe_40d5_039b_24a9_26eb6155c2cc["eval_chain.py"]
  df005def_3ccc_8e5a_d8f1_4757105de3a2 -->|defined in| 66e0c4fe_40d5_039b_24a9_26eb6155c2cc
  323d29cb_9da1_dca1_8e57_f675bf2fe201["requires_reference()"]
  df005def_3ccc_8e5a_d8f1_4757105de3a2 -->|method| 323d29cb_9da1_dca1_8e57_f675bf2fe201
  3987e2f0_a8de_74dd_9ceb_d96cade3aefa["from_llm()"]
  df005def_3ccc_8e5a_d8f1_4757105de3a2 -->|method| 3987e2f0_a8de_74dd_9ceb_d96cade3aefa

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/scoring/eval_chain.py lines 414–484

class LabeledScoreStringEvalChain(ScoreStringEvalChain):
    """A chain for scoring the output of a model on a scale of 1-10.

    Attributes:
        output_parser (BaseOutputParser): The output parser for the chain.

    """

    @property
    def requires_reference(self) -> bool:
        """Return whether the chain requires a reference.

        Returns:
            `True` if the chain requires a reference, `False` otherwise.

        """
        return True

    @classmethod
    def from_llm(
        cls,
        llm: BaseLanguageModel,
        *,
        prompt: PromptTemplate | None = None,
        criteria: CRITERIA_TYPE | str | None = None,
        normalize_by: float | None = None,
        **kwargs: Any,
    ) -> LabeledScoreStringEvalChain:
        """Initialize the LabeledScoreStringEvalChain from an LLM.

        Args:
            llm: The LLM to use.
            prompt: The prompt to use.
            criteria: The criteria to use.
            normalize_by: The value to normalize the score by.
            **kwargs: Additional keyword arguments.

        Returns:
            The initialized LabeledScoreStringEvalChain.

        Raises:
            ValueError: If the input variables are not as expected.

        """
        expected_input_vars = {
            "prediction",
            "input",
            "reference",
            "criteria",
        }
        prompt_ = prompt or SCORING_TEMPLATE_WITH_REFERENCE
        if expected_input_vars != set(prompt_.input_variables):
            msg = (
                f"Input variables should be {expected_input_vars}, "
                f"but got {prompt_.input_variables}"
            )
            raise ValueError(msg)
        criteria_ = resolve_criteria(criteria)
        criteria_str = "\n".join(f"{k}: {v}" for k, v in criteria_.items()).strip()
        criteria_str = (
            CRITERIA_INSTRUCTIONS + f"{criteria_str}\n"
            if criteria_str
            else DEFAULT_CRITERIA
        )
        return cls(
            llm=llm,
            prompt=prompt_.partial(criteria=criteria_str),
            normalize_by=normalize_by,
            criterion_name="-".join(criteria_),
            **kwargs,
        )