JsonEqualityEvaluator Class — langchain Architecture

Architecture documentation for the JsonEqualityEvaluator class in base.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6["JsonEqualityEvaluator"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b["StringEvaluator"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|extends| 3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b
  14f33a52_8365_e913_8362_3cf2caf1fd39["base.py"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|defined in| 14f33a52_8365_e913_8362_3cf2caf1fd39
  eee64e9b_0751_4bc1_a172_d25aeeca7946["__init__()"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|method| eee64e9b_0751_4bc1_a172_d25aeeca7946
  5986a97d_7506_e9c3_1fd7_c863a591ee3d["requires_input()"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|method| 5986a97d_7506_e9c3_1fd7_c863a591ee3d
  b502a8c1_ee3b_e6ff_3fee_08dae1b86d0e["requires_reference()"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|method| b502a8c1_ee3b_e6ff_3fee_08dae1b86d0e
  fe816892_665d_b9ac_a584_dcce03c8f098["evaluation_name()"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|method| fe816892_665d_b9ac_a584_dcce03c8f098
  7793dc6c_9c60_a7a0_ca86_d11624937249["_parse_json()"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|method| 7793dc6c_9c60_a7a0_ca86_d11624937249
  4021fe41_65d3_6fa7_5d31_3efed7bb0c71["_evaluate_strings()"]
  909e1aca_7c35_c9d5_2f9b_81064d86d9e6 -->|method| 4021fe41_65d3_6fa7_5d31_3efed7bb0c71

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/parsing/base.py lines 91–181

class JsonEqualityEvaluator(StringEvaluator):
    """Json Equality Evaluator.

    Evaluate whether the prediction is equal to the reference after
    parsing both as JSON.

    This evaluator checks if the prediction, after parsing as JSON, is equal
        to the reference,
    which is also parsed as JSON. It does not require an input string.

    Attributes:
        requires_input: Whether this evaluator requires an
            input string. Always False.
        requires_reference: Whether this evaluator requires
            a reference string. Always True.
        evaluation_name: The name of the evaluation metric.
            Always "parsed_equality".

    Examples:
        >>> evaluator = JsonEqualityEvaluator()
        >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 1}')
        {'score': True}
        >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 2}')
        {'score': False}

        >>> evaluator = JsonEqualityEvaluator(operator=lambda x, y: x["a"] == y["a"])
        >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 1}')
        {'score': True}
        >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 2}')
        {'score': False}

    """

    def __init__(self, operator: Callable | None = None, **_: Any) -> None:
        """Initialize the JsonEqualityEvaluator.

        Args:
            operator: A custom operator to compare the parsed JSON objects.
                Defaults to equality (`eq`).
        """
        super().__init__()
        self.operator = operator or eq

    @property
    @override
    def requires_input(self) -> bool:
        return False

    @property
    @override
    def requires_reference(self) -> bool:
        return True

    @property
    @override
    def evaluation_name(self) -> str:
        return "json_equality"

    def _parse_json(
        self,
        string: Any,
    ) -> dict | list | None | float | bool | int | str:
        if isinstance(string, str):
            return parse_json_markdown(string)
        return string

    @override
    def _evaluate_strings(
        self,
        prediction: str,
        reference: str | None = None,
        **kwargs: Any,
    ) -> dict:
        """Evaluate the prediction string.

        Args:
            prediction: The prediction string to evaluate.
            reference: The reference string to compare against.
            **kwargs: Additional keyword arguments (not used).

        Returns: