resolve_pairwise_criteria() — langchain Function Reference

Architecture documentation for the resolve_pairwise_criteria() function in eval_chain.py from the langchain codebase.

Function python LangChainCore MessageInterface called by 2

Entity Profile

LangChainCore→ MessageInterface→ resolve_pairwise_criteria() — langchain Function Reference

Dependency Diagram

graph TD
  4995eef9_93cd_17ab_915e_ed4ae1cf159f["resolve_pairwise_criteria()"]
  7db782f0_522e_d3f7_666f_da5a1e80af4a["eval_chain.py"]
  4995eef9_93cd_17ab_915e_ed4ae1cf159f -->|defined in| 7db782f0_522e_d3f7_666f_da5a1e80af4a
  7f35a36c_35ea_7f92_9706_72edf0eab7e3["from_llm()"]
  7f35a36c_35ea_7f92_9706_72edf0eab7e3 -->|calls| 4995eef9_93cd_17ab_915e_ed4ae1cf159f
  69672bbd_1b9c_053c_d740_7c250335ad18["from_llm()"]
  69672bbd_1b9c_053c_d740_7c250335ad18 -->|calls| 4995eef9_93cd_17ab_915e_ed4ae1cf159f
  style 4995eef9_93cd_17ab_915e_ed4ae1cf159f fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/comparison/eval_chain.py lines 52–96

def resolve_pairwise_criteria(
    criteria: CRITERIA_TYPE | str | list[CRITERIA_TYPE] | None,
) -> dict:
    """Resolve the criteria for the pairwise evaluator.

    Args:
        criteria: The criteria to use.

    Returns:
        The resolved criteria.

    """
    if criteria is None:
        _default_criteria = [
            Criteria.HELPFULNESS,
            Criteria.RELEVANCE,
            Criteria.CORRECTNESS,
            Criteria.DEPTH,
        ]
        return {k.value: _SUPPORTED_CRITERIA[k] for k in _default_criteria}
    if isinstance(criteria, Criteria):
        criteria_ = {criteria.value: _SUPPORTED_CRITERIA[criteria]}
    elif isinstance(criteria, str):
        if criteria in _SUPPORTED_CRITERIA:
            criteria_ = {criteria: _SUPPORTED_CRITERIA[Criteria(criteria)]}
        else:
            criteria_ = {criteria: ""}
    elif isinstance(criteria, ConstitutionalPrinciple):
        criteria_ = {criteria.name: criteria.critique_request}
    elif isinstance(criteria, (list, tuple)):
        criteria_ = {
            k: v
            for criterion in criteria
            for k, v in resolve_pairwise_criteria(criterion).items()
        }
    else:
        if not criteria:
            msg = (
                "Criteria cannot be empty. "
                "Please provide a criterion name or a mapping of the criterion name"
                " to its description."
            )
            raise ValueError(msg)
        criteria_ = dict(criteria)
    return criteria_