JsonValidityEvaluator Class — langchain Architecture

Architecture documentation for the JsonValidityEvaluator class in base.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  ca6449c4_5552_5059_7a40_009b5af57841["JsonValidityEvaluator"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b["StringEvaluator"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|extends| 3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b
  14f33a52_8365_e913_8362_3cf2caf1fd39["base.py"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|defined in| 14f33a52_8365_e913_8362_3cf2caf1fd39
  52fb50b2_1dfa_3a23_caa6_9b7df50da52f["__init__()"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|method| 52fb50b2_1dfa_3a23_caa6_9b7df50da52f
  cfa95561_249d_b0c2_689e_3ae1d68591d9["requires_input()"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|method| cfa95561_249d_b0c2_689e_3ae1d68591d9
  22743080_f365_e08c_31f1_29610286adca["requires_reference()"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|method| 22743080_f365_e08c_31f1_29610286adca
  7889cc16_b142_ee32_920e_dd2b82627370["evaluation_name()"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|method| 7889cc16_b142_ee32_920e_dd2b82627370
  5524730f_9608_c667_bf87_2f343457eefb["_evaluate_strings()"]
  ca6449c4_5552_5059_7a40_009b5af57841 -->|method| 5524730f_9608_c667_bf87_2f343457eefb

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/parsing/base.py lines 17–88

class JsonValidityEvaluator(StringEvaluator):
    """Evaluate whether the prediction is valid JSON.

    This evaluator checks if the prediction is a valid JSON string. It does not
        require any input or reference.

    Attributes:
        requires_input: Whether this evaluator requires an input
            string. Always False.
        requires_reference: Whether this evaluator requires a
            reference string. Always False.
        evaluation_name: The name of the evaluation metric.
            Always "json".

    Examples:
        >>> evaluator = JsonValidityEvaluator()
        >>> prediction = '{"name": "John", "age": 30, "city": "New York"}'
        >>> evaluator.evaluate(prediction)
        {'score': 1}

        >>> prediction = '{"name": "John", "age": 30, "city": "New York",}'
        >>> evaluator.evaluate(prediction)
        {'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes'}
    """

    def __init__(self, **_: Any) -> None:
        """Initialize the JsonValidityEvaluator."""
        super().__init__()

    @property
    @override
    def requires_input(self) -> bool:
        return False

    @property
    @override
    def requires_reference(self) -> bool:
        return False

    @property
    @override
    def evaluation_name(self) -> str:
        return "json_validity"

    @override
    def _evaluate_strings(
        self,
        prediction: str,
        **kwargs: Any,
    ) -> dict:
        """Evaluate the prediction string.

        Args:
            prediction: The prediction string to evaluate.
            **kwargs: Additional keyword arguments (not used).

        Returns:
            `dict` containing the evaluation score. The score is `1` if
                the prediction is valid JSON, and `0` otherwise.

                If the prediction is not valid JSON, the dictionary also contains
                a `reasoning` field with the error message.

        """
        try:
            parse_json_markdown(prediction, parser=json.loads)
        except json.JSONDecodeError as e:
            return {"score": 0, "reasoning": str(e)}
        except Exception as e:
            _logger.exception("Passing JSON failed with unexpected error.")
            return {"score": 0, "reasoning": str(e)}
        return {"score": 1}