Qwen3.8-27B on one DGX Spark: 38 tok/s with vLLM

Since 20 October 2026, Qwen3.8-27B runs on one of the four NVIDIA DGX Spark machines in our office, next to Nemotron 3.5 Bolt on the same box. Qwen3.8-27B is a dense model: every one of its 27 billion parameters is read for every generated token. On a machine whose bottleneck is memory bandwidth, that makes it slow by default.

We measured the same model under Ollama and under vLLM, each with and without speculative decoding, and under vLLM with two different draft models. With the same settings, vLLM is 1.5 to 1.8 times as fast as Ollama. The draft model makes an even bigger difference, up to a factor of three, and which draft model wins depends on how many requests arrive at once.

Results

Ollama ran on a second DGX Spark with identical hardware; vLLM on the machine that serves the model today. All numbers from 21 October 2026.

Three things stand out:

1. Both engines need a draft model to be fast. Ollama ships this model with its built-in MTP layer switched on by default, which we only noticed when we checked the server log.
2. With the same kind of draft model (MTP), vLLM is 1.8 times as fast as Ollama for a single request.
3. DFlash2 is the fastest choice for one request, MTP for several at once: with 4 or 8 concurrent requests, MTP beats DFlash2 by 27 % and 12 %.

We run DFlash2 today, because our load is low and mostly one request at a time. We log the load to decide whether to switch to MTP (see the end of this article).

How we measure

All numbers come from our own benchmark script, run directly on the machine against the OpenAI-compatible endpoint of vLLM or Ollama, not through our gateway. The script is published here: bench.py.

• Requests are streamed. Reasoning tokens and answer tokens are counted together.
• Output: up to 512 tokens per request, temperature 0, thinking enabled. At 512 tokens the model is still reasoning, so the numbers describe reasoning output.
• Input: ten short German prompts from our real workload (code, SQL, ops diagnostics, explanations). A single-request run uses the first prompt, a run with 4 or 8 concurrent requests uses the first four or eight.
• Single request: decode rate only, (completion tokens − 1) / (time of last chunk − time of first chunk).
• Concurrent runs: total completion tokens divided by wall time, prefill included.
• Each configuration ran at least three times; we report the median.
• To switch Ollama's draft model off, the OpenAI endpoint offers no option. For that variant we used Ollama's native API with draft_num_predict: 0 and the same counting rule. A control run with the draft model switched on gave the same result on both APIs (native API with the draft model on: 16.1, 15.7 and 14.9 tok/s, against 16.2 tok/s on the OpenAI endpoint; the native window had two outside requests).
• Idle check: for every run we counted requests from outside in the server logs. The vLLM runs had none. For Ollama, the second Spark serves other models through our gateway, so we logged outside requests per run: four of the five single-request runs with MTP had none, one had three (16.3 tok/s, no visible effect), and the other series had two to four per measurement window. For the value without draft model we repeated single runs until five had no outside request at all: 8.0 to 8.1 tok/s, the same median as before (runs with outside requests: 7.6 to 8.0). The GLiNER service on that machine received no requests; clef-flash stays loaded there and only received its regular keep-alive request.

Ollama against vLLM

With its default settings, Ollama processes one request at a time. The server log shows n_seq_max = 1. Four concurrent requests therefore wait in line, and the aggregate stays at single-stream speed. With OLLAMA_NUM_PARALLEL=4 in a separate Ollama instance, the server still loaded the model with n_seq_max = 1, presumably because MTP drafting allows only one slot; we have not checked that in the source. Our attempt to measure four concurrent requests on that instance was stopped by our watchdog when the production models on the same machine reloaded.

Without a draft model, vLLM is 1.5 times as fast (12.3 against 8.1 tok/s) and uses 79 % of the bandwidth, Ollama 47 %. With MTP, vLLM pulls clearly ahead, although Ollama proposes more tokens per step (4 against 3) and gets more of them accepted (3.19 against 2.62 tokens per step). Per step of the main model, Ollama manages 8.1 steps per second without and 5.1 with MTP, vLLM 12.3 and 10.9. Drafting costs Ollama far more time per step. That is our reading of the numbers; we have not profiled Ollama.

What the memory bandwidth allows

A DGX Spark moves about 273 GB/s. For every step, the main model has to read all its weights once, except the embedding table, from which only one row per token is read. We added up the tensor sizes in both files:

The NVFP4 checkpoint is larger per step because the projections of the linear-attention layers are stored in FP8, not in 4 bit. Without a draft model, every token is one step, so neither engine can exceed about 15 to 17 tok/s on this machine. Everything above that comes from speculative decoding: several tokens per step.

The vision encoder (0.9 GB) and the MTP layer are not part of this count. The vision encoder only runs for images, the MTP layer only drafts.

The draft model decides

Speculative decoding lets a small, fast model propose several tokens ahead. The main model checks all of them in one pass and keeps the ones it would have produced itself. Because one pass of the dense main model costs the same whether it checks one token or several, every accepted extra token is almost free.

(vLLM in both cases.) DFlash2 gets more tokens per step, so it wins when only one request runs. It also proposes eight tokens per request, and with several requests in a batch those proposals cost compute that the batch itself then lacks. MTP proposes fewer tokens and scales better. That is our reading of the numbers; we have not profiled it.

For comparison: two community reports with vLLM and MTP on one DGX Spark give 23 to 26 tok/s in chat use (see sources), close to our 28.6.

Both methods are designed not to change the output. The DFlash2 model card states that greedy output matches the target model exactly. We have not checked that ourselves.

Long inputs

Same three vLLM configurations, a public-domain German text from Project Gutenberg (Goethe, "Die Leiden des jungen Werther" and "Faust") cut to the target length, a request for a three-sentence summary, reasoning budget 64, up to 256 output tokens, random prefix per request so that no prefix cache could help:

DFlash2 at 8,000 and 32,000 tokens: median of three runs. All other cells: one run each.

Prefill runs at about 2,600 tokens per second at 8,000 tokens and drops to about 1,400 at 100,000. Long inputs also shrink the gain from the draft model: decode with DFlash2 falls from 38.0 tok/s with short prompts to about 20. Our guess is that attention over the long context takes a growing share of each step, and the draft model cannot shorten that part. We have not measured it.

Memory: what the budget means, and two traps

A DGX Spark has 128 GB of memory shared between CPU and GPU, of which CUDA sees 121.7 GiB. When it runs out, the machine freezes instead of returning an error. We start every new configuration the first time under a watchdog that kills the container within a second when available memory drops below 12,000 MiB. It fired once in this test.

--gpu-memory-utilization describes the peak during startup, not the steady footprint. The vLLM log of this build breaks it down:

The two shares add up to 101 GiB, but nvidia-smi showed 85.3 GiB for both processes in operation. About 12 GiB of the difference are the peak activations that only exist during startup. The remaining 3.6 GiB are probably memory outside PyTorch that nvidia-smi does not attribute to the process; we have not traced that in detail. On another machine with vLLM 0.28, budget and measurement agreed within 1 GiB. Whether vLLM changed the accounting between those versions, we have not found documented. In practice it means: a restart needs that peak again, so the free memory in operation is not all spare.

First trap: --gpu-memory-utilization does not cover the MTP layer. With MTP at 0.47, available memory fell from 17.6 to 10.2 GiB while vLLM allocated the KV cache, and the watchdog killed the container. With DFlash2 at the same 0.47, 22 GiB stayed available. We repeated the MTP run at 0.42; the lowest value was then 15.7 GiB. We had seen the same behaviour before with Qwen3.6-35B on another machine.

Second trap: two models on one machine. When both vLLM services started at the same time, each measured the other's growing memory as its own overhead while sizing the KV cache. Bolt calculated −52 GiB of available KV memory, Qwen3.8-27B 8.4 instead of 13.3 GiB, and both aborted. Our systemd units now wait before starting until the other service is either not running or fully loaded (script below).

The 262,144-token context also needs room: at a share of 0.35, vLLM refused to start, because one request at full length needs 11.9 GiB of KV cache and only 8.1 GiB was left. Today the model runs at 0.47, with 22.5 GiB of KV cache, room for about 1.9 requests at full length. Available on the machine with both models in operation on 21 October: 21.5 GiB.

The reasoning budget is not optional

Qwen3.8-27B reasons before it answers. We ran the same ten German prompts with max_tokens 2048 and temperature 0:

Without a budget, the reasoning consumed the whole token allowance in 8 of 10 cases. With 768, every prompt got an answer, but 5 of 10 were still cut off at max_tokens 2048. In production our gateway sends the budget of 768 and sets no max_tokens; the caller decides, and without a value vLLM allows output up to the remaining context. Language was detected with a simple function-word heuristic on the first 300 characters, so read that column as a trend. This vLLM build accepts the field without --reasoning-config; we tested that with a single request.

Our configuration

The image is eugr/spark-vllm:nightly-20261016, a vLLM development build for the DGX Spark (vLLM 0.32.0rc1.dev301, FlashInfer 0.7.2). The start command follows the community recipe qwen3.8-27b-nvfp4-dflash2 from the eugr/spark-vllm-docker repository, adapted for one machine and a shared memory budget.

vLLM itself has no authentication. Never expose it directly to the internet. In our setup, an inference gateway with API keys sits in front, and the vLLM port is only reachable from the internal network. The example binds to 127.0.0.1, which is right when the gateway runs on the same machine. If your gateway runs on another host, add the address of the internal interface (-p <internal IP>:8004:8004) and restrict access with a firewall. That is what we do.

docker run --name vllm-qwen27b --gpus all --ipc host --shm-size=16g --ulimit memlock=-1 \
  -p 127.0.0.1:8004:8004 \
  -v /srv/hf-cache:/root/.cache/huggingface -e HF_HUB_OFFLINE=1 \
  eugr/spark-vllm:nightly-20261016 \
  vllm serve nvidia/Qwen3.8-27B-NVFP4 --revision 482ca0f3832238542f8f5295dde86b5f22711d80 \
    --served-model-name qwen3.8-27b \
    --host 0.0.0.0 --port 8004 --tensor-parallel-size 1 --trust-remote-code \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.47 \
    --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 4096 \
    --enable-chunked-prefill --async-scheduling --enable-prefix-caching \
    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","revision":"50307d4c4cde6860d4eee73e2547cd786fe8e8a4","num_speculative_tokens":8,"draft_tensor_parallel_size":1}' \
    --load-format instanttensor \
    --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Notes:

• For MTP instead of DFlash2, replace the speculative config with '{"method":"mtp","num_speculative_tokens":3}' and lower the memory share (we use 0.42), because the MTP layer is not counted.
• Qwen3.8-27B has 48 linear-attention layers (Gated DeltaNet) among its 64. An open vLLM issue about crashes in exactly that path on the GB10 (#54173) therefore applies in principle, and we run with prefix caching. So far the service has run for about one day without a crash, with 46 real requests and a few hundred benchmark requests. That proves nothing yet: on another machine, crashes in a similar setup came five times in six days. Prefix caching pays off for conversations: a follow-up question on 32,000 tokens of context got its first token after 0.5 s instead of 14.5 s. The FP8 kernel bug on the GB10 that crashed another of our services (FlashInfer #3566) does not apply to this image: FlashInfer 0.7.2 removes cuDNN from the automatic FP8 choice on this chip, which our log confirms.
• Model and draft model are pinned to a revision, and HF_HUB_OFFLINE=1 prevents a silent download of newer weights at restart.
• No --rm: when a start fails, the container and its log stay available.
• --host 0.0.0.0 refers to the inside of the container. What is reachable from outside is decided by -p.

The wait condition that keeps two vLLM services from starting at the same time runs as ExecStartPre in the systemd unit of each service, with the other service as argument:

#!/bin/bash
# Wait until another vLLM service is either not running or fully loaded.
# Usage: wait-for-vllm.sh <container> <port> <systemd-unit> [max_seconds]
NAME="$1"; PORT="$2"; UNIT="$3"; MAX="${4:-1200}"; t=0
while [ "$t" -lt "$MAX" ]; do
  st=$(docker inspect -f '{{.State.Status}}' "$NAME" 2>/dev/null) || st=missing
  curl -sf -m 3 "http://127.0.0.1:$PORT/v1/models" >/dev/null && exit 0
  # Unit active but container not there yet: it is just starting, keep waiting.
  [ "$st" != running ] && ! systemctl is-active --quiet "$UNIT" && exit 0
  sleep 10; t=$((t+10))
done
# Timeout: do not start next to an instance that is still loading.
# With Restart=always, systemd tries again after RestartSec.
exit 1

On timeout the script returns an error on purpose. Starting anyway would bring back exactly the failure it is meant to prevent. With Restart=always systemd tries again. Our units allow 5 starts within 10 minutes (StartLimitIntervalSec=600, StartLimitBurst=5, the same as on our other machines); with 20 minutes of waiting per attempt, at most one attempt falls into that window, so the service is never locked out.

What we would do differently

1. Read the server log before comparing engines. We had compared Ollama with draft model against vLLM without one, because Ollama switches MTP on by default.
2. Measure the version without a draft model first. Without that baseline we would have credited the engine with a gain that comes from speculative decoding.
3. Test draft models under the load you expect, not only with one request. The winner for one request lost at four.
4. Plan memory for MTP separately, and leave room for the startup peak.
5. Never start two vLLM instances on one unified-memory machine at the same time.

Open question: DFlash2 or MTP

Since 21 October we record every five seconds how many requests the model is processing. If the share of time with four or more concurrent requests grows, we will switch to MTP; the layer is already in the checkpoint, so it costs one restart.

Sources

• Qwen3.8-27B base model (Apache 2.0): https://huggingface.co/Qwen/Qwen3.8-27B
• NVIDIA NVFP4 checkpoint: https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4
• DFlash2 draft model (Apache 2.0): https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2
• Community recipe qwen3.8-27b-nvfp4-dflash2: https://github.com/eugr/spark-vllm-docker
• vLLM issue #54173, crashes in the GDN path on GB10: https://github.com/vllm-project/vllm/issues/54173
• FlashInfer issue #3566 (closed) and the fix that removes cuDNN from the automatic FP8 choice on SM12x: https://github.com/flashinfer-ai/flashinfer/issues/3566, https://github.com/flashinfer-ai/flashinfer/pull/4165
• Community measurement, Qwen3.8-27B NVFP4 with vLLM 0.27.1 and MTP on one DGX Spark: https://gist.github.com/Mirrdhyn/d476d17723c9d68eac72f03ec9447f2b
• NVIDIA developer forum, Qwen3.8-27B NVFP4 on a single DGX Spark with vLLM 0.26.1 and MTP: https://forums.developer.nvidia.com/t/qwen3-8-27b-nvfp4-on-a-single-dgx-spark-up-to-1m-context-vllm-mtp-measurements/380244