JsonSchemaEvaluator Class — langchain Architecture

Architecture documentation for the JsonSchemaEvaluator class in json_schema.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  9babfe39_52f2_aa6b_17f6_a55234538495["JsonSchemaEvaluator"]
  3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b["StringEvaluator"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|extends| 3bfa222e_74bd_6c80_db4a_a3d3a09b5e7b
  b4b3c835_bb1d_7469_e4cc_b53cd7b6be10["json_schema.py"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|defined in| b4b3c835_bb1d_7469_e4cc_b53cd7b6be10
  7ce85c09_f752_c118_9d82_5cf159fac4e7["__init__()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 7ce85c09_f752_c118_9d82_5cf159fac4e7
  79c30f43_efff_76cf_c7a6_1546525e0b3e["requires_input()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 79c30f43_efff_76cf_c7a6_1546525e0b3e
  7c5a8927_36e8_0ab9_35b0_1b147f817f8a["requires_reference()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 7c5a8927_36e8_0ab9_35b0_1b147f817f8a
  23ba017b_22bd_6b8c_5f44_1febdfd76a92["evaluation_name()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 23ba017b_22bd_6b8c_5f44_1febdfd76a92
  17e51faf_bcc2_97a0_f42a_9ca542b6bfaf["_parse_json()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 17e51faf_bcc2_97a0_f42a_9ca542b6bfaf
  8b7d5d13_c859_2d6b_d657_495032d846c5["_validate()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 8b7d5d13_c859_2d6b_d657_495032d846c5
  810f2cc6_5067_feaf_c5ea_268fbeb54aeb["_evaluate_strings()"]
  9babfe39_52f2_aa6b_17f6_a55234538495 -->|method| 810f2cc6_5067_feaf_c5ea_268fbeb54aeb

Relationship Graph

Source Code

libs/langchain/langchain_classic/evaluation/parsing/json_schema.py lines 9–97

class JsonSchemaEvaluator(StringEvaluator):
    """An evaluator that validates a JSON prediction against a JSON schema reference.

    This evaluator checks if a given JSON prediction conforms to the provided JSON schema.
    If the prediction is valid, the score is True (no errors). Otherwise, the score is False (error occurred).

    Attributes:
        requires_input: Whether the evaluator requires input.
        requires_reference: Whether the evaluator requires reference.
        evaluation_name: The name of the evaluation.

    Examples:
        evaluator = JsonSchemaEvaluator()
        result = evaluator.evaluate_strings(
            prediction='{"name": "John", "age": 30}',
            reference={
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer"}
                }
            }
        )
        assert result["score"] is not None

    """  # noqa: E501

    def __init__(self, **_: Any) -> None:
        """Initializes the JsonSchemaEvaluator.

        Raises:
            ImportError: If the jsonschema package is not installed.
        """
        super().__init__()
        try:
            import jsonschema  # noqa: F401
        except ImportError as e:
            msg = (
                "The JsonSchemaEvaluator requires the jsonschema package."
                " Please install it with `pip install jsonschema`."
            )
            raise ImportError(msg) from e

    @property
    def requires_input(self) -> bool:
        """Returns whether the evaluator requires input."""
        return False

    @property
    def requires_reference(self) -> bool:
        """Returns whether the evaluator requires reference."""
        return True

    @property
    def evaluation_name(self) -> str:
        """Returns the name of the evaluation."""
        return "json_schema_validation"

    def _parse_json(self, node: Any) -> dict | list | None | float | bool | int | str:
        if isinstance(node, str):
            return parse_json_markdown(node)
        if hasattr(node, "model_json_schema") and callable(node.model_json_schema):
            # Pydantic v2 model
            return node.model_json_schema()
        if hasattr(node, "schema") and callable(node.schema):
            # Pydantic v1 model
            return node.schema()
        return node

    def _validate(self, prediction: Any, schema: Any) -> dict:
        from jsonschema import ValidationError, validate

        try:
            validate(instance=prediction, schema=schema)
        except ValidationError as e:
            return {"score": False, "reasoning": repr(e)}
        return {"score": True}

    @override
    def _evaluate_strings(
        self,