test_special_tokens() — langchain Function Reference

Architecture documentation for the test_special_tokens() function in test_schema.py from the langchain codebase.

Function python LangChainCore ApiManagement

Entity Profile

LangChainCore→ ApiManagement→ test_special_tokens() — langchain Function Reference

Dependency Diagram

graph TD
  55cd1802_deec_8c12_a03c_f1bd8bfd4088["test_special_tokens()"]
  76ce3c98_8b2b_4359_3dd9_9cadd328e8f1["TestTokenCountingWithGPT2Tokenizer"]
  55cd1802_deec_8c12_a03c_f1bd8bfd4088 -->|defined in| 76ce3c98_8b2b_4359_3dd9_9cadd328e8f1
  style 55cd1802_deec_8c12_a03c_f1bd8bfd4088 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

libs/langchain/tests/integration_tests/test_schema.py lines 17–19

    def test_special_tokens(self) -> None:
        # test for consistency when the default tokenizer is changed
        assert len(_get_token_ids_default_method("a:b_c d")) == 6