longhaul.cpp/docs/token-cache.md

Tokenizer and tokenized-text cache

libllama maintains a persistent cache containing constructed tokenizer snapshots and text-to-token results. The cache is enabled by default with a shared 5 GiB logical entry budget and is available to direct libllama callers as well as the llama.cpp tools.

The default database is token-cache.sqlite3 below the platform llama.cpp cache directory. LLAMA_CACHE changes the base directory. llama.cpp tools also accept:

--token-cache-size N     cache size in MiB (default: 5120, 0 disables)
--token-cache-dir PATH   use a different SQLite database
--token-cache-clear      clear the database before loading a model

The matching environment variables are LLAMA_ARG_TOKEN_CACHE_SIZE, LLAMA_ARG_TOKEN_CACHE_DIR, and LLAMA_ARG_TOKEN_CACHE_CLEAR.

Cache entries are shared by processes running as the same OS user. Tokenized text keys are SHA-256 digests and the original text is not stored. Token IDs can usually be detokenized, however, so the database must still be treated as sensitive user data. The cache directory and files are created with owner-only permissions where supported.

Writes use a bounded asynchronous queue. Cache failures, unavailable storage, or corrupt entries are treated as misses and do not prevent model loading or tokenization. Applications that require all pending entries to be committed can call llama_token_cache_flush().

The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded journal files are not included.

Tokenizer longhaul

Qwen3.5 MoE and Laguna models can move their reverse-token and BPE-merge indexes into a persistent, disk-backed index while keeping a bounded lookup working set:

--tokenizer-longhaul
--tokenizer-longhaul-cache N

N is the application-owned lookup cache budget in MiB and is required when the mode is enabled. The mode is independent of model --longhaul, so the two can be used together. The finite tokenizer index builds in the background while model loading continues. If tokenizer work arrives first, it waits for atomic index publication; build or storage failures are reported instead of silently falling back to an unbounded tokenizer.

The persistent index uses the directory selected by --token-cache-dir and is subject to --token-cache-size. Tokenized-text results continue to be cached on demand and therefore do not have a finite completion point.