Cockpit
ai-toolbox-cockpit
cmd line
llama-server -m /models/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q4_K_XL.gguf
-c 126976
-ngl 999
--host 0.0.0.0
--port 8080
--load-mode none
-fa 1
-b 2048
-ub 2048
--jinja
--temp 1.0
--top-p 0.95
--top-k 64
--load-mode none
Controls how model weights are loaded into memory. The default is auto (memory-map the file, unless a device doesn't support it). none loads weights with no special handling — no mmap, no mlock, no DirectIO. Other options: mmap (memory-map — lazy loading, low RSS but possible page faults during inference), mlock (keep model resident in RAM, never swapped), dio (DirectIO). In practice none is similar to the old --no-mmap: slower startup, but avoids mmap-related issues.
-fa 1 (--flash-attn 1)
Enables Flash Attention. It fuses the attention computation (Q·Kᵀ, softmax, ·V) so it doesn't materialize the full attention matrix, saving VRAM and speeding up inference — especially with long contexts. 1/on forces it on, 0/off disables, auto (default) enables where supported. Some backends/layer configs fall back to off automatically.
-b 2048 (--batch-size)
Maximum logical batch size — the number of prompt tokens that can be processed per inference iteration. This is the prompt-processing (prefill) path; larger batches make prompt processing faster.
-ub 2048 (--ubatch-size)
Maximum physical batch size — the chunk size the backend actually executes. The logical batch (-b) is split into chunks of at most this size. Setting both to 2048 means the full logical batch runs in one physical chunk (fewer graph evals, best prefill throughput, more VRAM). If -ub were smaller (e.g. 512, the default), a 2048 prompt would be processed in 4 chunks.
--jinja
Tells the server to use the Jinja2 chat template embedded in the GGUF file instead of a built-in/hardcoded prompt format for the architecture. This is the recommended way to get exact, up-to-date chat formatting (system prompts, tool-calling format, thinking/reasoning tags, etc.) as defined by the model authors. It also enables proper tool-call handling for models whose template supports it.
--temp 1.0
Sampling temperature. Logits are divided by T before softmax: T=1.0 leaves the model's native probabilities unchanged (no sharpening/flattening). Lower (e.g. 0.1-0.3) = more deterministic/focused; higher (>1) = more random. 1.0 is a neutral, faithful-to-model setting.
--top-p 0.95
Nucleus sampling: only consider the smallest set of tokens whose cumulative probability reaches 95%, then renormalize and sample from that set. Cuts off the long tail of unlikely tokens. 0.95 is the llama.cpp default and a common general-purpose value.
--top-k 64
Hard cutoff: keep only the 64 highest-probability tokens before sampling (applied before top-p in the default sampler order). Prevents sampling from the bottom of the distribution regardless of how the probabilities are distributed. 64 is a common choice that balances coherence and diversity; 0 would disable it.
Together, the sampling settings (temp 1.0, top-p 0.95, top-k 64) form a fairly standard, slightly conservative decoding config — close to llama.cpp defaults, with the batch sizes tuned for fast prompt prefill on a beefy machine.