test_schema.py — langchain Source File
Architecture documentation for test_schema.py, a python file in the langchain codebase. 1 imports, 0 dependents.
Entity Profile
Dependency Diagram
graph LR d8514062_c8d1_f182_fc4d_4ffc4dc0ee86["test_schema.py"] a2ee9dce_2c05_f65f_71c7_74ac415394d8["langchain_core.language_models.base"] d8514062_c8d1_f182_fc4d_4ffc4dc0ee86 --> a2ee9dce_2c05_f65f_71c7_74ac415394d8 style d8514062_c8d1_f182_fc4d_4ffc4dc0ee86 fill:#6366f1,stroke:#818cf8,color:#fff
Relationship Graph
Source Code
"""Test formatting functionality."""
from langchain_core.language_models.base import _get_token_ids_default_method
class TestTokenCountingWithGPT2Tokenizer:
def test_tokenization(self) -> None:
# Check that the tokenization is consistent with the GPT-2 tokenizer
assert _get_token_ids_default_method("This is a test") == [1212, 318, 257, 1332]
def test_empty_token(self) -> None:
assert len(_get_token_ids_default_method("")) == 0
def test_multiple_tokens(self) -> None:
assert len(_get_token_ids_default_method("a b c")) == 3
def test_special_tokens(self) -> None:
# test for consistency when the default tokenizer is changed
assert len(_get_token_ids_default_method("a:b_c d")) == 6
Domain
Subdomains
Dependencies
- langchain_core.language_models.base
Source
Frequently Asked Questions
What does test_schema.py do?
test_schema.py is a source file in the langchain codebase, written in python. It belongs to the CoreAbstractions domain, MessageSchema subdomain.
What does test_schema.py depend on?
test_schema.py imports 1 module(s): langchain_core.language_models.base.
Where is test_schema.py in the architecture?
test_schema.py is located at libs/langchain/tests/integration_tests/test_schema.py (domain: CoreAbstractions, subdomain: MessageSchema, directory: libs/langchain/tests/integration_tests).
Analyze Your Own Codebase
Get architecture documentation, dependency graphs, and domain analysis for your codebase in minutes.
Try Supermodel Free