longhaul.cpp/docs/token-cache.md

Tokenizer and tokenized-text cache

libllama maintains a persistent cache containing constructed tokenizer snapshots and text-to-token results. The cache is enabled by default with a shared 5 GiB logical entry budget and is available to direct libllama callers as well as the llama.cpp tools.

The default database is token-cache.sqlite3 below the platform llama.cpp cache directory. LLAMA_CACHE changes the base directory. llama.cpp tools also accept:

--token-cache-size N     cache size in MiB (default: 5120, 0 disables)
--token-cache-dir PATH   use a different SQLite database
--token-cache-clear      clear the database before loading a model

The matching environment variables are LLAMA_ARG_TOKEN_CACHE_SIZE, LLAMA_ARG_TOKEN_CACHE_DIR, and LLAMA_ARG_TOKEN_CACHE_CLEAR.

Cache entries are shared by processes running as the same OS user. Tokenized text keys are SHA-256 digests and the original text is not stored. Token IDs can usually be detokenized, however, so the database must still be treated as sensitive user data. The cache directory and files are created with owner-only permissions where supported.

Writes use a bounded asynchronous queue. Cache failures, unavailable storage, or corrupt entries are treated as misses and do not prevent model loading or tokenization. Applications that require all pending entries to be committed can call llama_token_cache_flush().

The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded journal files are not included.