trajectory_eval_prompt.py — langchain Source File

Source Code

"""Prompt for trajectory evaluation chain."""

from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain_core.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
)

EVAL_TEMPLATE = """An AI language model has been given access to the following set of tools to help answer a user's question.

The tools given to the AI model are:
[TOOL_DESCRIPTIONS]
{tool_descriptions}
[END_TOOL_DESCRIPTIONS]

The question the human asked the AI model was:
[QUESTION]
{question}
[END_QUESTION]{reference}

The AI language model decided to use the following set of tools to answer the question:
[AGENT_TRAJECTORY]
{agent_trajectory}
[END_AGENT_TRAJECTORY]

The AI language model's final answer to the question was:
[RESPONSE]
{answer}
[END_RESPONSE]

Let's to do a detailed evaluation of the AI language model's answer step by step.

We consider the following criteria before giving a score from 1 to 5:

i. Is the final answer helpful?
ii. Does the AI language use a logical sequence of tools to answer the question?
iii. Does the AI language model use the tools in a helpful way?
iv. Does the AI language model use too many steps to answer the question?
v. Are the appropriate tools used to answer the question?"""  # noqa: E501

EXAMPLE_INPUT = """An AI language model has been given access to the following set of tools to help answer a user's question.

The tools given to the AI model are:
[TOOL_DESCRIPTIONS]
Tool 1:
Name: Search
Description: useful for when you need to ask with search

Tool 2:
Name: Lookup
Description: useful for when you need to ask with lookup

Tool 3:
Name: Calculator
Description: useful for doing calculations

Tool 4:
Name: Search the Web (SerpAPI)
Description: useful for when you need to answer questions about current events
[END_TOOL_DESCRIPTIONS]
// ... (87 more lines)