stuff

docs/token-cache.md

@@ -0,0 +1,32 @@
+# Tokenizer and tokenized-text cache
+
+libllama maintains a persistent cache containing constructed tokenizer snapshots
+and text-to-token results. The cache is enabled by default with a shared 5 GiB
+logical entry budget and is available to direct libllama callers as well as the
+llama.cpp tools.
+
+The default database is `token-cache.sqlite3` below the platform llama.cpp cache
+directory. `LLAMA_CACHE` changes the base directory. llama.cpp tools also accept:
+
+```text
+--token-cache-size N     cache size in MiB (default: 5120, 0 disables)
+--token-cache-dir PATH   use a different SQLite database
+--token-cache-clear      clear the database before loading a model
+```
+
+The matching environment variables are `LLAMA_ARG_TOKEN_CACHE_SIZE`,
+`LLAMA_ARG_TOKEN_CACHE_DIR`, and `LLAMA_ARG_TOKEN_CACHE_CLEAR`.
+
+Cache entries are shared by processes running as the same OS user. Tokenized
+text keys are SHA-256 digests and the original text is not stored. Token IDs can
+usually be detokenized, however, so the database must still be treated as
+sensitive user data. The cache directory and files are created with owner-only
+permissions where supported.
+
+Writes use a bounded asynchronous queue. Cache failures, unavailable storage,
+or corrupt entries are treated as misses and do not prevent model loading or
+tokenization. Applications that require all pending entries to be committed can
+call `llama_token_cache_flush()`.
+
+The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded
+journal files are not included.