54 lines
2.4 KiB
Markdown
54 lines
2.4 KiB
Markdown
# Tokenizer and tokenized-text cache
|
|
|
|
libllama maintains a persistent cache containing constructed tokenizer snapshots
|
|
and text-to-token results. The cache is enabled by default with a shared 5 GiB
|
|
logical entry budget and is available to direct libllama callers as well as the
|
|
llama.cpp tools.
|
|
|
|
The default database is `token-cache.sqlite3` below the platform llama.cpp cache
|
|
directory. `LLAMA_CACHE` changes the base directory. llama.cpp tools also accept:
|
|
|
|
```text
|
|
--token-cache-size N cache size in MiB (default: 5120, 0 disables)
|
|
--token-cache-dir PATH use a different SQLite database
|
|
--token-cache-clear clear the database before loading a model
|
|
```
|
|
|
|
The matching environment variables are `LLAMA_ARG_TOKEN_CACHE_SIZE`,
|
|
`LLAMA_ARG_TOKEN_CACHE_DIR`, and `LLAMA_ARG_TOKEN_CACHE_CLEAR`.
|
|
|
|
Cache entries are shared by processes running as the same OS user. Tokenized
|
|
text keys are SHA-256 digests and the original text is not stored. Token IDs can
|
|
usually be detokenized, however, so the database must still be treated as
|
|
sensitive user data. The cache directory and files are created with owner-only
|
|
permissions where supported.
|
|
|
|
Writes use a bounded asynchronous queue. Cache failures, unavailable storage,
|
|
or corrupt entries are treated as misses and do not prevent model loading or
|
|
tokenization. Applications that require all pending entries to be committed can
|
|
call `llama_token_cache_flush()`.
|
|
|
|
The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded
|
|
journal files are not included.
|
|
|
|
## Tokenizer longhaul
|
|
|
|
Qwen3.5 MoE and Laguna models can move their reverse-token and BPE-merge indexes
|
|
into a persistent, disk-backed index while keeping a bounded lookup working set:
|
|
|
|
```text
|
|
--tokenizer-longhaul
|
|
--tokenizer-longhaul-cache N
|
|
```
|
|
|
|
`N` is the application-owned lookup cache budget in MiB and is required when
|
|
the mode is enabled. The mode is independent of model `--longhaul`, so the two
|
|
can be used together. The finite tokenizer index builds in the background while
|
|
model loading continues. If tokenizer work arrives first, it waits for atomic
|
|
index publication; build or storage failures are reported instead of silently
|
|
falling back to an unbounded tokenizer.
|
|
|
|
The persistent index uses the directory selected by `--token-cache-dir` and is
|
|
subject to `--token-cache-size`. Tokenized-text results continue to be cached
|
|
on demand and therefore do not have a finite completion point.
|