resolve_criteria() — langchain Function Reference

Architecture documentation for the resolve_criteria() function in eval_chain.py from the langchain codebase.

Function python LangChainCore ApiManagement calls 1 called by 1

Entity Profile

LangChainCore→ ApiManagement→ resolve_criteria() — langchain Function Reference

Dependency Diagram

graph TD
  3aef6314_cf9d_eed1_2ed3_6139ee744a4c["resolve_criteria()"]
  8e14b60d_5880_d030_e540_4a171da7ff70["eval_chain.py"]
  3aef6314_cf9d_eed1_2ed3_6139ee744a4c -->|defined in| 8e14b60d_5880_d030_e540_4a171da7ff70
  a9d899da_d5d0_329a_f04c_073673c2d5f9["resolve_criteria()"]
  a9d899da_d5d0_329a_f04c_073673c2d5f9 -->|calls| 3aef6314_cf9d_eed1_2ed3_6139ee744a4c
  a9d899da_d5d0_329a_f04c_073673c2d5f9["resolve_criteria()"]
  3aef6314_cf9d_eed1_2ed3_6139ee744a4c -->|calls| a9d899da_d5d0_329a_f04c_073673c2d5f9
  style 3aef6314_cf9d_eed1_2ed3_6139ee744a4c fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/criteria/eval_chain.py lines 116–159

def resolve_criteria(
    criteria: CRITERIA_TYPE | str | None,
) -> dict[str, str]:
    """Resolve the criteria to evaluate.

    Parameters
    ----------
    criteria : CRITERIA_TYPE
        The criteria to evaluate the runs against. It can be:
            -  a mapping of a criterion name to its description
            -  a single criterion name present in one of the default criteria
            -  a single `ConstitutionalPrinciple` instance

    Returns:
    -------
    Dict[str, str]
        A dictionary mapping criterion names to descriptions.

    Examples:
    --------
    >>> criterion = "relevance"
    >>> CriteriaEvalChain.resolve_criteria(criteria)
    {'relevance': 'Is the submission referring to a real quote from the text?'}
    """
    if criteria is None:
        return {
            "helpfulness": _SUPPORTED_CRITERIA[Criteria.HELPFULNESS],
        }
    if isinstance(criteria, Criteria):
        criteria_ = {criteria.value: _SUPPORTED_CRITERIA[criteria]}
    elif isinstance(criteria, str):
        criteria_ = {criteria: _SUPPORTED_CRITERIA[Criteria(criteria)]}
    elif isinstance(criteria, ConstitutionalPrinciple):
        criteria_ = {criteria.name: criteria.critique_request}
    else:
        if not criteria:
            msg = (
                "Criteria cannot be empty. "
                "Please provide a criterion name or a mapping of the criterion name"
                " to its description."
            )
            raise ValueError(msg)
        criteria_ = dict(criteria)
    return criteria_