33 lines
1.5 KiB
Markdown
33 lines
1.5 KiB
Markdown
# Tokenizer and tokenized-text cache
|
|
|
|
libllama maintains a persistent cache containing constructed tokenizer snapshots
|
|
and text-to-token results. The cache is enabled by default with a shared 5 GiB
|
|
logical entry budget and is available to direct libllama callers as well as the
|
|
llama.cpp tools.
|
|
|
|
The default database is `token-cache.sqlite3` below the platform llama.cpp cache
|
|
directory. `LLAMA_CACHE` changes the base directory. llama.cpp tools also accept:
|
|
|
|
```text
|
|
--token-cache-size N cache size in MiB (default: 5120, 0 disables)
|
|
--token-cache-dir PATH use a different SQLite database
|
|
--token-cache-clear clear the database before loading a model
|
|
```
|
|
|
|
The matching environment variables are `LLAMA_ARG_TOKEN_CACHE_SIZE`,
|
|
`LLAMA_ARG_TOKEN_CACHE_DIR`, and `LLAMA_ARG_TOKEN_CACHE_CLEAR`.
|
|
|
|
Cache entries are shared by processes running as the same OS user. Tokenized
|
|
text keys are SHA-256 digests and the original text is not stored. Token IDs can
|
|
usually be detokenized, however, so the database must still be treated as
|
|
sensitive user data. The cache directory and files are created with owner-only
|
|
permissions where supported.
|
|
|
|
Writes use a bounded asynchronous queue. Cache failures, unavailable storage,
|
|
or corrupt entries are treated as misses and do not prevent model loading or
|
|
tokenization. Applications that require all pending entries to be committed can
|
|
call `llama_token_cache_flush()`.
|
|
|
|
The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded
|
|
journal files are not included.
|