_extract_tokens_and_log_probs() — langchain Function Reference

Architecture documentation for the _extract_tokens_and_log_probs() function in base.py from the langchain codebase.

Function python AgentOrchestration ClassicChains called by 1

Entity Profile

AgentOrchestration→ ClassicChains→ _extract_tokens_and_log_probs() — langchain Function Reference

Dependency Diagram

graph TD
  8245ca05_0a13_88d7_9345_6e14d983a746["_extract_tokens_and_log_probs()"]
  4c01b940_b202_29af_2ff5_d0dfa5e5e804["base.py"]
  8245ca05_0a13_88d7_9345_6e14d983a746 -->|defined in| 4c01b940_b202_29af_2ff5_d0dfa5e5e804
  193d32b1_2740_febf_5c97_eebfa1726864["_call()"]
  193d32b1_2740_febf_5c97_eebfa1726864 -->|calls| 8245ca05_0a13_88d7_9345_6e14d983a746
  style 8245ca05_0a13_88d7_9345_6e14d983a746 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/langchain_classic/chains/flare/base.py lines 31–38

def _extract_tokens_and_log_probs(response: AIMessage) -> tuple[list[str], list[float]]:
    """Extract tokens and log probabilities from chat model response."""
    tokens = []
    log_probs = []
    for token in response.response_metadata["logprobs"]["content"]:
        tokens.append(token["token"])
        log_probs.append(token["logprob"])
    return tokens, log_probs