Add tokenizer longhaul cache
This commit is contained in:
@@ -30,3 +30,24 @@ call `llama_token_cache_flush()`.
|
||||
|
||||
The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded
|
||||
journal files are not included.
|
||||
|
||||
## Tokenizer longhaul
|
||||
|
||||
Qwen3.5 MoE and Laguna models can move their reverse-token and BPE-merge indexes
|
||||
into a persistent, disk-backed index while keeping a bounded lookup working set:
|
||||
|
||||
```text
|
||||
--tokenizer-longhaul
|
||||
--tokenizer-longhaul-cache N
|
||||
```
|
||||
|
||||
`N` is the application-owned lookup cache budget in MiB and is required when
|
||||
the mode is enabled. The mode is independent of model `--longhaul`, so the two
|
||||
can be used together. The finite tokenizer index builds in the background while
|
||||
model loading continues. If tokenizer work arrives first, it waits for atomic
|
||||
index publication; build or storage failures are reported instead of silently
|
||||
falling back to an unbounded tokenizer.
|
||||
|
||||
The persistent index uses the directory selected by `--token-cache-dir` and is
|
||||
subject to `--token-cache-size`. Tokenized-text results continue to be cached
|
||||
on demand and therefore do not have a finite completion point.
|
||||
|
||||
Reference in New Issue
Block a user