ChainStringRunMapper Class — langchain Architecture

Architecture documentation for the ChainStringRunMapper class in string_run_evaluator.py from the langchain codebase.

Class python

Entity Profile

Dependency Diagram

graph TD
  3ac90b6f_675b_ff6a_04b3_989d183b1348["ChainStringRunMapper"]
  6513456f_be2b_21ef_06af_21023f648595["StringRunMapper"]
  3ac90b6f_675b_ff6a_04b3_989d183b1348 -->|extends| 6513456f_be2b_21ef_06af_21023f648595
  2f0b23f2_7760_d68c_7feb_721c5231c4ec["string_run_evaluator.py"]
  3ac90b6f_675b_ff6a_04b3_989d183b1348 -->|defined in| 2f0b23f2_7760_d68c_7feb_721c5231c4ec
  299b611e_fb70_d4ca_6dcc_c6031a8162d9["_get_key()"]
  3ac90b6f_675b_ff6a_04b3_989d183b1348 -->|method| 299b611e_fb70_d4ca_6dcc_c6031a8162d9
  b8c4bf91_87d7_ce69_1350_1bcdf1bccbbb["map()"]
  3ac90b6f_675b_ff6a_04b3_989d183b1348 -->|method| b8c4bf91_87d7_ce69_1350_1bcdf1bccbbb

Relationship Graph

Source Code

libs/langchain/langchain_classic/smith/evaluation/string_run_evaluator.py lines 156–211

class ChainStringRunMapper(StringRunMapper):
    """Extract items to evaluate from the run object from a chain."""

    input_key: str | None = None
    """The key from the model Run's inputs to use as the eval input.
    If not provided, will use the only input key or raise an
    error if there are multiple."""
    prediction_key: str | None = None
    """The key from the model Run's outputs to use as the eval prediction.
    If not provided, will use the only output key or raise an error
    if there are multiple."""

    def _get_key(self, source: dict, key: str | None, which: str) -> str:
        if key is not None:
            return source[key]
        if len(source) == 1:
            return next(iter(source.values()))
        msg = (
            f"Could not map run {which} with multiple keys: "
            f"{source}\nPlease manually specify a {which}_key"
        )
        raise ValueError(msg)

    def map(self, run: Run) -> dict[str, str]:
        """Maps the Run to a dictionary."""
        if not run.outputs:
            msg = (
                f"Run with ID {run.id} lacks outputs required for evaluation."
                " Ensure the Run has valid outputs."
            )
            raise ValueError(msg)
        if self.input_key is not None and self.input_key not in run.inputs:
            msg = (
                f"Run with ID {run.id} is missing the expected input key"
                f" '{self.input_key}'.\nAvailable input keys in this Run"
                f"  are: {run.inputs.keys()}.\nAdjust the evaluator's"
                f" input_key or ensure your input data includes key"
                f" '{self.input_key}'."
            )
            raise ValueError(msg)
        if self.prediction_key is not None and self.prediction_key not in run.outputs:
            available_keys = ", ".join(run.outputs.keys())
            msg = (
                f"Run with ID {run.id} doesn't have the expected prediction key"
                f" '{self.prediction_key}'. Available prediction keys in this Run are:"
                f" {available_keys}. Adjust the evaluator's prediction_key or"
                " ensure the Run object's outputs the expected key."
            )
            raise ValueError(msg)

        input_ = self._get_key(run.inputs, self.input_key, "input")
        prediction = self._get_key(run.outputs, self.prediction_key, "prediction")
        return {
            "input": input_,
            "prediction": prediction,
        }