The Better Model That Wouldn't Fit: Qwen 3.8 27B on a 32 GB Arc Pro B70

Oct 22, 2026

The previous post ended with a model I actually like running and a parenthetical about a 27B dense model being "a post of its own." This is that post. It ends differently: with the model parked.

The premise was simple. Qwen 3.6 35B A3B is fast because only a few billion parameters are active per token, and that same property is why it drifts on long agent runs, dense instructions, and code that has to execute. A 27B dense model is a better thinker. The question was whether one could live on a single 32 GB Intel Arc Pro B70 at a context length I would actually use.

It cannot. But "too heavy" turned out to be a sloppy phrase, because the weight is four separate budgets and only one of them is the weights.

What "too heavy" decomposes into

On a single card, a 27B model at long context spends four independent budgets:

  1. Weights. 17–20 GB at INT4/Q4, about 30.9 GB at the official FP8 — before a single byte of cache.
  2. KV cache. At 128K–220K this is not the remainder of the card; it is the card.
  3. The drafter. Speculative decoding is how you make a dense model feel fast, and a drafter is another resident model.
  4. Cold prefill. The one nobody budgets for, and the one that actually killed this.

The first three are capacity, and capacity fails loudly — the server refuses to start. The fourth is latency, and it fails quietly: the model loads, runs, and takes seven minutes to answer.

The measurement

One B70, vLLM 0.29, a W4A16 Qwen 3.8 27B target, FP8 KV, one active sequence, prefix caching off, one seeded random prompt per point, 128 output tokens, greedy. K is binary: 64K is 65,536 tokens.

Decode rate versus context for a 27B target with and without a DFlash2 drafter on one 32 GB Arc Pro B70. Target-only decode falls from 26.8 to 19.7 tok/s across 64K to 220K. The drafter more than doubles it at 64K but only gains 22% at 220K, and the BF16 drafter cannot be loaded past about 156.8K at all.

Context Configuration Cold TTFT Prefill TPOT Decode Accept. length
64K target only 65.4 s 1,002 tok/s 37.3 ms 26.8 tok/s –
64K + BF16 drafter 66.2 s 989 18.2 ms 54.9 3.94
128K target only 181.7 s 721 42.8 ms 23.3 –
128K + BF16 drafter 183.1 s 716 33.1 ms 30.2 2.76
180K target only 313.3 s 588 47.4 ms 21.1 –
180K + INT4 drafter 329.1 s 560 23.8 ms 42.1 4.71
220K target only 437.2 s 515 50.8 ms 19.7 –
220K + INT4 drafter 459.1 s 491 41.6 ms 24.1 3.07

Two things to read off this table.

Decode holds up surprisingly well. 26.8 → 19.7 tok/s across a 3.4× context increase. If you only ever looked at tokens per second, you would conclude the 27B "basically works."

Everything else collapses. Cold time-to-first-token goes from 65 seconds to 437. A cold 220K turn costs over seven minutes before the first token arrives, and prefill throughput degrades from 1,002 to 515 tok/s as the attention and cache pressure climb.

These were single seeded random prompts with repetitive output — capacity and performance probes, not chat quality, and no confidence intervals. They bound what is possible; they do not describe how the model feels.

The drafter runs out of card first

The BF16 drafter is what makes the 64K row look good, and it auto-fit to about 156.8K tokens and then simply could not load. The 180K and 220K points needed an experimental INT4 drafter run in eager mode. Even that was tight: at 98% GPU utilization it fit 224,000 tokens with a 4,096-token scheduler budget, which was not enough for a binary 220K request plus its output. Dropping the scheduler budget to 2,048 raised capacity to 242,816 tokens — and that cost is baked into the 220K row above.

Precision is not a knob here either. On XPU, an FP16 drafter overflows and produces zero draft acceptance; BF16 is mandatory for the high-acceptance route.

Speculative decoding is an amortization problem

This is the part I found genuinely interesting. A drafter buys you a lower per-token time, but it also buys you a higher time-to-first-token, because the target has to verify draft passes over an ever larger cache. The extra TTFT has to be paid back out of the tokens you generate:

Context 64K 128K 180K 220K
output tokens needed to pay back the extra TTFT 43 148 672 2,361

At 64K, speculation pays for itself in a short reply. At 220K you need to generate more than two thousand tokens before the drafter has earned its own overhead — and the acceptance rate that made it worthwhile is exactly the thing that varies most with prompt content. (The 180K INT4 point hit an acceptance length of 4.71, unusually high; context length alone is a dangerous routing threshold.)

The general lesson generalizes past this card: speculative decoding is not a speed setting, it is a bet on output length, and the bet gets worse as context grows. On a long-context agent workload — the workload I actually care about, which lives at 180K+ — the drafter is the first thing that stops making sense.

There was also no way to route around it. vLLM could vary speculative tokens by batch size but had no per-sequence context-length policy, and target-only and drafter servers could not be resident on the same 32 GB card, so switching profiles meant a restart.

The llama.cpp attempts

Before the vLLM sweep I tried the same model through llama.cpp's SYCL server, with DFlash2 drafters at several precisions, --parallel 1 and 2. It produced no benchmark table. What it produced was a context-init failure in the DFlash path, a port collision with the live service, and a lost device when running two sequences with SYCL graphs disabled. That is a record of what went wrong, not a measurement, and I am not going to dress it up as the latter.

Why I stopped

Three reasons, in order of how much they weighed.

The ceiling is below my workload. The agent conversations that motivated this whole line of work live at 179–194K tokens. A route that tops out near 156.8K with a drafter is disqualified before quality enters the picture.

Cold prefill is unfixable by configuration. 515 tok/s at 220K is a property of the model, the cache layout and the card. No flag recovers seven minutes.

I never actually measured the quality gap. This is the uncomfortable one. The entire 27B pursuit rested on "35B A3B drifts under stress testing" — a judgement I made from working around it, not from a benchmark. I never ran a task-level comparison of 27B against the model I use now. It is entirely possible I was chasing a gap I had not quantified.

So: parked, not disproved. What I actually established is much narrower than the title of this post implies — 27B does not fit on this card via vLLM with INT4 or FP8 weights and a DFlash2 drafter at 180K context or beyond — and other serving routes and weight formats remain untested. I would rather say that here than let the title overclaim.

Back to MoE

The port that 27B was supposed to take is running a mixture-of-experts model again: Qwen 3.8 Flash Next, IQ3_S, 262K context, on Strata. So the honest ending is that I went looking for a dense model to replace my MoE with, and came back.

What decided it was not a benchmark. It was watching where each architecture spends its budget on the same 32 GB of memory.

A dense 27B pays for every parameter on every token. A MoE pays for the experts the router actually picks, and buys the rest with bandwidth — which sounds like the worse problem until you notice it is the solvable one. Streaming experts from host memory is an engineering problem with an answer. Making 30 GB of dense weights share a card with a 200K-token KV cache is arithmetic.

Which is the practical lesson, and the reason I stopped measuring and wrote this up instead: the MoE's weakness is a bandwidth problem and the dense model's weakness is a capacity problem, and only one of those responds to good engineering. The bandwidth problem turned out to be plenty of a problem in its own right — it got a post of its own two days after I switched.

Two things I want to be precise about, because it would be easy to oversell this ending.

I did not go back to the model that disappointed me. Flash Next is a newer MoE generation than Qwen 3.6 35B A3B, and the drift I kept designing around is much less of an issue in it. "MoE again" is not the same as "the same MoE again."

And I still have no head-to-head quality measurement between the 27B and Flash Next. Everything above is an argument about fit — memory, context, prefill, drafter economics — not about which model thinks better. On this card, at this context length, fit was the binding constraint and it settled the question before quality got a vote. If the card changes, the question reopens.

What I would do before trying again

  • Compare candidate weight formats and serving routes on identical real coding and chat prompts at 2K, 32K and 128K, with cold and reused-prefix turns recorded separately.
  • Measure TTFT, tokens after first token, acceptance, peak VRAM and answer correctness in the same run.
  • Exercise reasoning parsers, tool-call parsers, and exact retrieval — the failure modes that motivated the switch.
  • Keep short greedy/counting results separate from sampled coding performance; they answer different questions.
  • And before any of it: define the quality bar the 27B has to clear, in writing, so "better model" stops being a vibe.

That last bullet is the one I skipped, and it is the reason this post ends with a parked model instead of a benchmark.