Optimal llama.cpp + qwen3.6 settings on Strix Halo

Recipes // Inference

Inference llama.cpp Strix Halo qwen3.6-27b

Optimal llama.cpp + qwen3.6 settings on Strix Halo

published: Jul 31, 2026

Optimal configuration for running Qwen3.6-27B UD-Q4_K_XL on a Strix Halo MiniPC with AMD Ryzen AI Max+ 395 (128 GiB LPDDR5x, Vulkan or HIP backend).

Requirements

Component	Description
Hardware	FEVM FA-EX1 – Ryzen AI Max+ 395, Radeon 8060S (gfx1151), 128 GiB LPDDR5x-8500 (~215 GB/s bandwidth, ~112 GiB GTT on Linux)
llama.cpp	b10064
Model	Qwen3.6-27B UD-Q4_K_XL.gguf (~17.5 GiB)
Model (MTP)	Qwen3.6-27B-MTP UD-Q4_K_XL – required for MTP speculative decoding
mmproj	mmproj-BF16.gguf (multimodal, optional)
Template	chat_template.jinja v21.3

Basic Launch (Vulkan)

Bash

#!/usr/bin/env bash

LLAMA_SERVER=/mnt/local/nix/servers/agents/llama/main/llama-server
MODEL=/path/to/Qwen3.6-27B-UD-Q4_K_XL.gguf
TEMPLATE=/path/to/chat_template.jinja

$LLAMA_SERVER \
    --host 0.0.0.0 \
    --port 1234 \
    -c 0 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.05 \
    --prio 2 \
    --cpu-strict 1 \
    --no-warmup \
    --no-mmap \
    --jinja \
    --chat-template-file ${TEMPLATE} \
    -ctk q8_0 \
    -ctv q8_0 \
    --cache-reuse 256 \
    --cache-ram 32768 \
    -np 1 \
    --metrics \
    --alias "qwen3.6-27b-q4_k_xl" \
    -m ${MODEL}

Parameters and Rationale

Parameter	Value	Why
`-c` (context)	`0`	Default context is 262144, but it’s better to limit it to 131072
`--batch-size`	`2048`	Optimal for prompt processing at 215 GB/s bandwidth
`--ubatch-size`	`512`	Physical micro-batch size. Reduces peak VRAM/RAM consumption during prefill
`--temp`	`0.7`	Standard creativity balance. Above 1.0 may cause hallucinations with Q4 quantization
`--top-p`	`0.8`	Cuts tokens with low cumulative probability, improves stability
`--top-k`	`20`	Limits selection to a fixed number of tokens
`--min-p`	`0.0`	Disabled – not needed when top-p/top-k are set correctly
`--prio 2 --cpu-strict 1`	–	Sets elevated process priority in the OS to prevent freezes, strict thread distribution
`--no-mmap`	–	Avoids mapping issues with large files
`-ctk q8_0 -ctv q8_0`	`q8_0`	Compresses KV cache to save memory on large contexts at the cost of slight accuracy and speed reduction
`--cache-reuse`	`256`	Only useful in multi-user chatbots (servers) for preserving system context (system prompt)
`-np 1`	`1`	Number of parallel slots for concurrent requests. For a single user, 1 is optimal

Multimodal Support (mmproj)

To work with images, add the --mmproj flag:

Bash

MODEL_MM=/path/to/mmproj-BF16.gguf

# ... add to the launch command:
    --mmproj ${MODEL_MM} \

Advanced Launch (HIP + MTP Speculative Decoding)

MTP (Multi-Token Prediction) uses the model’s built-in heads to predict multiple tokens simultaneously – without loading a separate draft model. On Strix Halo, this delivers ~21 tok/s vs ~12 tok/s without MTP at a 93–95% token acceptance rate.

Bash

#!/usr/bin/env bash

LLAMA_SERVER=/mnt/local/nix/servers/agents/llama/main/llama-server
MODEL=/path/to/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf
TEMPLATE=/path/to/chat_template.jinja
MODEL_MM=/path/to/mmproj-BF16.gguf

export ROCBLAS_USE_HIPBLASLT=1

$LLAMA_SERVER \
    --host 0.0.0.0 \
    --port 1234 \
    -c 0 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.0 \
    --prio 2 \
    --cpu-strict 1 \
    --no-warmup \
    --no-mmap \
    --jinja \
    --chat-template-file ${TEMPLATE} \
    -ctk q8_0 \
    -ctv q8_0 \
    --cache-reuse 256 \
    --cache-ram 32768 \
    -np 1 \
    --metrics \
    --alias "qwen3.6-27b-mtp-q4_k_xl-agent" \
    --spec-type draft-mtp \
    --spec-draft-n-max 3 \
    --spec-draft-p-min 0.75 \
    --mmproj ${MODEL_MM} \
    -m ${MODEL}

MTP Parameters

Parameter	Value	Why
`--spec-type draft-mtp`	`draft-mtp`	Uses the model’s built-in MTP heads as a drafter
`--spec-draft-n-max`	`3`	Up to 3 tokens per speculation; more – token acceptance rate drops
`--spec-draft-p-min`	`0.75`	Minimum probability for accepting a draft token

Building llama.cpp for HIP

Bash

cmake -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DGGML_HIP_NO_VMM=ON \
  ..

Performance

Benchmarks (agentic coding)

Config	Tool Calls	Hard Tasks	Polyglot	Decoding	Token Acceptance Rate
Q4_K_XL + MTP	99.3%	93.8%	65%	21 → 16 tok/s	93–95%
Q6_K + MTP	100%	68.8%	65%	21 → 16 tok/s	92–94%
Q4 without spec decode	–	–	–	~12 tok/s	–

Expected Performance (Vulkan, no MTP)

Metric	Value
Prompt processing (pp512)	~250–350 tok/s
Generation (tg128)	~10–12 tok/s
Theoretical maximum	`215 / 17.5 ≈ 12.3 tok/s`

The UD-Q4_K_XL model weighs ~17.5 GiB. With unlimited context, the KV cache grows as needed; with -ctk q8_0 -ctv q8_0, the cache takes up less space.

⚠️ Temperature and TDP

When running in HIP + MTP mode, the MPC consumes up to 130W, with CPU/GPU temperatures staying in the 85–92°C range. This is normal operating range for Strix Halo, but:

Ensure adequate cooling and airflow
Monitor temperature via sensors or rocm-smi

For Other Tasks

Code Generation (Higher Accuracy)

Bash

--temp 0.3 --top-p 0.5 --top-k 5 \
--repeat-penalty 1.1

Creative Text

Bash

--temp 1.0 --top-p 0.95 --top-k 40 \
--repeat-penalty 1.15

Known Issues

-ctk / -ctv defaults – with default values, prefill speed drops to ~70 tok/s on 80K context. Explicitly set -ctk q8_0 -ctv q8_0.
Chat template is required – use chat_template.jinja v21.3; the default template may cause loop breaks and tool call hangs.
Flash attention may not work with some GGUF models. Disable with --flash-attn off if you see output artifacts.
Vulkan backend on Linux requires recent AMD drivers (amdgpu + Mesa 24.1+).

tekBlueprint