Add tokenizer longhaul cache

This commit is contained in:
Owen Qwen
2026-07-30 23:32:52 -05:00
parent a673afad85
commit e0665b1042
21 changed files with 1078 additions and 23 deletions
+21
View File
@@ -30,3 +30,24 @@ call `llama_token_cache_flush()`.
The 5 GiB limit accounts for logical entry payloads. SQLite metadata and bounded
journal files are not included.
## Tokenizer longhaul
Qwen3.5 MoE and Laguna models can move their reverse-token and BPE-merge indexes
into a persistent, disk-backed index while keeping a bounded lookup working set:
```text
--tokenizer-longhaul
--tokenizer-longhaul-cache N
```
`N` is the application-owned lookup cache budget in MiB and is required when
the mode is enabled. The mode is independent of model `--longhaul`, so the two
can be used together. The finite tokenizer index builds in the background while
model loading continues. If tokenizer work arrives first, it waits for atomic
index publication; build or storage failures are reported instead of silently
falling back to an unbounded tokenizer.
The persistent index uses the directory selected by `--token-cache-dir` and is
subject to `--token-cache-size`. Tokenized-text results continue to be cached
on demand and therefore do not have a finite completion point.