Cold cache, a fresh prompt each request, context growing 1k → 200k (400k for DeepSeek V4) — the big picture, not a coding-only lens. The two rates that matter per context: prefill (prompt processing tok/s) and generation (decode tok/s). Prefill is where devices diverge most — and it's what dominates real coding/agentic turns (big context, small reply).
One growing session: 6 turns, context 40k→50k, every turn sharing the previous one's prefix. Turn 1 misses the cache; turns 2–6 hit it. Top: the accumulated wall-clock across the whole session per device — the slope between turns is one cached turn, and the final value is the total time spent. Do the Macs catch up? (Apple numbers assume the same prefix caching.) Below, the same warm session as a heatmap (hosts × turns; greener = faster) so you can read the warm-up across every box at a glance.
Single user, cold cache, across the full context spread (1k→200k). Each bar is one device's request, split into prefill (reading the prompt, gray) and generation (streaming the answer, device color), sorted fastest first — shorter is better (green best / red worst). As the context grows, prefill takes over.
Scale from 1 to 6 concurrent streams (multi-user cold, 10k context — distinct contexts, so this isolates concurrency from caching). As more streams pile on: does aggregate throughput rise (batching), stay flat, or fall? And how long does the whole batch take?
Tokens per watt for prompt processing (PP) and generation (TG), broken into single-user vs 6 concurrent streams (all at 10k context; where a run's 10k power window is unusable the single-user panels fall back to that machine's 50k request and label the bar). The DGX Spark ×2 is adjusted +70W per node — its exporter only reads GPU power, so this is the whole-box estimate. M5 Ultra is a projection and has no measured power.
The "which hardware" answer, in the units you actually care about: minutes of your day, per machine, per model. Then the $5–8k decision: M5 Ultra or DGX Spark ×2, side by side.
DGX Spark ×2 is the measured 2-node cluster (~$8k); M5 Ultra is the projected Mac Studio (1.2 TB/s, 80-core GPU — $5,499 @ 96 GB, ~$12k @ 256 GB, ~$20k+ @ 512 GB). Four comparisons: decode speed, prompt ingestion, a real agentic turn (50k context, 500-token reply), and a 6-stream batch.
What was measured. Each run's warm phase is one growing corpus —
40k→50k in 2k-token turns, like a coding session that keeps appending. Session
wall-clock is just the sum of each turn's measured total_ms. The no-cache
line re-prices every turn at cold prefill cost for its context length (log–log
interpolated from the measured cold points) plus the same measured generation time — what
the session would cost if the server forgot the prefix every turn. Multi-user numbers sum
the concurrent batch's wall span per warm step.
Caveats. No think-time between turns, so every session number is a best case, and single runs mean individual cells wobble. Prefix caching is a server config choice, not a hardware limit — MLX re-processed every turn for Qwen3.6/Qwen3.8 (warm ≈ cold), while DeepSeek V4 on the same Mac cached fine and ran 2.6x faster. Qwen3.6/Qwen3.8 single-user numbers are averaged across each machine's runs; concurrent batches use the latest run only.
Model quants. Every model ran at the biggest quantization its machine could hold, in the same format everywhere it fit. Macs ran MLX, NVIDIA boxes ran NVFP4 (NVIDIA's 4-bit float) or FP8.