resolve_criteria() — langchain Function Reference

Architecture documentation for the resolve_criteria() function in eval_chain.py from the langchain codebase.

Function python LangChainCore MessageInterface called by 2

Entity Profile

LangChainCore→ MessageInterface→ resolve_criteria() — langchain Function Reference

Dependency Diagram

graph TD
  351b62b7_d2ea_9d3f_cc78_2712a988a68f["resolve_criteria()"]
  66e0c4fe_40d5_039b_24a9_26eb6155c2cc["eval_chain.py"]
  351b62b7_d2ea_9d3f_cc78_2712a988a68f -->|defined in| 66e0c4fe_40d5_039b_24a9_26eb6155c2cc
  3c6dc9b9_bf18_3520_a593_9147d40d2f2a["from_llm()"]
  3c6dc9b9_bf18_3520_a593_9147d40d2f2a -->|calls| 351b62b7_d2ea_9d3f_cc78_2712a988a68f
  3987e2f0_a8de_74dd_9ceb_d96cade3aefa["from_llm()"]
  3987e2f0_a8de_74dd_9ceb_d96cade3aefa -->|calls| 351b62b7_d2ea_9d3f_cc78_2712a988a68f
  style 351b62b7_d2ea_9d3f_cc78_2712a988a68f fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/scoring/eval_chain.py lines 53–97

def resolve_criteria(
    criteria: CRITERIA_TYPE | str | list[CRITERIA_TYPE] | None,
) -> dict:
    """Resolve the criteria for the pairwise evaluator.

    Args:
        criteria: The criteria to use.

    Returns:
        The resolved criteria.

    """
    if criteria is None:
        _default_criteria = [
            Criteria.HELPFULNESS,
            Criteria.RELEVANCE,
            Criteria.CORRECTNESS,
            Criteria.DEPTH,
        ]
        return {k.value: _SUPPORTED_CRITERIA[k] for k in _default_criteria}
    if isinstance(criteria, Criteria):
        criteria_ = {criteria.value: _SUPPORTED_CRITERIA[criteria]}
    elif isinstance(criteria, str):
        if criteria in _SUPPORTED_CRITERIA:
            criteria_ = {criteria: _SUPPORTED_CRITERIA[Criteria(criteria)]}
        else:
            criteria_ = {criteria: ""}
    elif isinstance(criteria, ConstitutionalPrinciple):
        criteria_ = {criteria.name: criteria.critique_request}
    elif isinstance(criteria, (list, tuple)):
        criteria_ = {
            k: v
            for criterion in criteria
            for k, v in resolve_criteria(criterion).items()
        }
    else:
        if not criteria:
            msg = (
                "Criteria cannot be empty. "
                "Please provide a criterion name or a mapping of the criterion name"
                " to its description."
            )
            raise ValueError(msg)
        criteria_ = dict(criteria)
    return criteria_