Tuning someone else's speedup

Generating a token on a laptop means dragging the entire model through memory. At 8 bits a 27B model is 28 GB, so the ceiling is arithmetic you can do on a napkin: bandwidth divided by model size, and no amount of idle GPU moves it.

Speculative decoding is the way around that. A small drafter proposes several tokens, the big model verifies the whole proposal in one pass, and that pass costs roughly what generating a single token would have. Accepted guesses come out free.

So when Inco AI published DFlash 2 claiming 2.7–3.4× at batch size 1 on Qwen3.8-27B — the model already sitting in my cache — the question was worth a day of measuring: does that hold on an Apple laptop?

Close enough to matter. 2.4× at 4-bit, 3.1× at 8-bit. But the number in their setup instructions was wrong for my machine by 10%, and my first benchmark reported no speedup at all while the server sat there claiming everything was fine. Those two things are the reason this is worth writing up, because neither is specific to this drafter.

Worth knowing what the drafter is, because it explains a lot of what follows: it is not a small language model. It reads the target’s hidden states at five fixed layer depths and predicts every position of the block in parallel — one pass, not one token at a time. DFlash 2 adds a two-tap convolution so each position sees its predecessor, and a selector that picks a coherent path through the candidates.

All measurements below are the same five prompts — two maths, two code, one prose — at 400 tokens, end-to-end throughput, on one machine.

The heavier model gains more

The 8-bit model gains more than the 4-bit one, which is worth a second look because the instinct runs the other way — the bigger thing should be harder to speed up.

It is the napkin arithmetic again. Speculation converts N sequential passes into one verification pass, so the win is bounded by how much of a pass is spent moving weights rather than computing. At 28 GB per token the 8-bit model is almost entirely bandwidth; at 15 GB there is proportionally more compute in the mix, and the verification pass over four positions is no longer nearly free.

Which also means the gap between the two quantizations narrows but does not close. On identical prompts, 4-bit runs 50.0 and 48.3 tok/s against 8-bit’s 29.4 and 28.8 — still 1.7×, the same ratio as before speculation. Both got faster; neither changed places.

Code and maths gain most (up to 2.6× at 4-bit), free-running prose least (2.0×). Predictable, once you think about what is easy to guess four tokens ahead.

The documented block size is not the fast one

The setup instructions say to set the runtime block size to 5. It is one number and each run takes two minutes, so I swept it:

Four, not five — and the documented value costs 10%. More striking is the right-hand side: a block of 12 gives up a third of the win.

This is the same bandwidth-versus-compute line drawn from the other direction. On a datacentre GPU the verification pass is nearly free compute, so longer blocks keep paying — propose more, accept more. On an M5 Pro it is not free. Verifying twelve positions costs real time, acceptance grows sublinearly, and past four the extra proposals stop covering their own cost.

There is nothing wrong with their default. It is simply a number tuned on hardware with a different compute-to-bandwidth ratio, and it arrives in the instructions looking like a constant rather than what it is.

One neighbouring setting turned out to matter as much, and is not a tuning question — it has a right answer. The drafter can be quantized when it loads. At the same block size:

Obvious in hindsight: the drafter runs on every draft pass, so its weights cross the memory bus as often as the target’s do. Left in bf16 it spends exactly the resource the whole scheme exists to save.

A benchmark that measured nothing

Before any of those numbers, my first real measurement was this:

Nothing. And a tidy explanation was right there for the taking — speculation needs spare compute to verify with, laptops have none to spare, the technique doesn’t transfer. Plausible, self-consistent, and I was close to writing it down.

It was in the log:

WARNING - DFlash start failed for mlx-community--Qwen3.8-27B-4bit:
          Received 23 parameters not in model:
INFO    - Successfully loaded mlx-community--Qwen3.8-27B-4bit as vlm (fallback from DFlash)

Twenty-three tensors. The drafter has five layers with a convolution before and after each attention and MLP sublayer — four per layer, twenty — plus the selector’s three. Exactly the parts version 2 adds to version 1. I had installed the older kernel, and it did not know what those weights were.

Then the server carried on serving. Correct answers, plausible throughput, no speculation whatsoever. From the client side that is indistinguishable from a working server that is merely slow — which is to say, indistinguishable from the baseline I was measuring against.

The general shape of this is worth keeping, because it has nothing to do with drafters. An optimization you cannot observe directly is one you cannot benchmark. If the only signal is the number you are trying to improve, then “no improvement” and “not running” produce identical evidence, and you will reach for whichever explanation you already believed.

So the rule I now follow: before trusting any before-and-after, find the independent signal that the mechanism engaged at all. Here it is one line — DFlashEngine loaded — and it is the only proof there is. My own CLI now refuses to start when the drafter is missing, rather than letting the server shrug and serve.

The claim I could not confirm

The part of the pitch I most wanted to check is that the output is provably unchanged — speculation as a pure speedup, with rejection sampling restoring the target’s exact distribution.

At temperature 0 this is cheap to test: same prompt, same server, compare bytes. Each configuration is perfectly deterministic with itself. They disagree with each other, and block size 4 disagrees with block size 5.

The divergence is not dramatic — 35 tokens in:

baseline: …Need produce final with code and explanation…
dflash:   …Need provide code and explanation…

Two near-equivalent words at what looks like a coin-flip. Both answers ran to completion and were correct.

My reading is floating point, not a broken drafter: verifying four positions in one batched pass rounds differently from decoding them one at a time, and at a near-tie that flips which token wins. Garbage would mean a broken drafter; a synonym means arithmetic. I did not go further — confirming that would mean reading out the logits at the divergence and showing the top two are within rounding of each other, which I have not done.

So, stated at the weight the evidence supports: on this stack, at one temperature, on one prompt, byte-identical output did not hold. Whether that matters depends entirely on what you are doing. For handing mechanical coding work to a local model, it does not. If you are relying on bit-exact reproducibility, test it on your own workload rather than taking either their word or mine.

The rest of the friction

Three smaller things, recorded because each cost real time and none of them is in any documentation.

The install pins a repository that does not exist. The tagged DFlash 2 release points its kernel dependency at a URL that 404s, so installing that tag fails outright at dependency resolution. The support had already landed upstream in the parent fork; the working recipe is to install the mainline for the dependency set, then layer the kernel and the tag’s own code over it. This is the one obstacle that failed loudly, which in hindsight made it the cheapest.

The server renames its own process. It is launched as omlx serve --base-path …, then calls setproctitle. By the time anything looks, the process is omlx-server and the command line is gone — flags, paths, everything. My stop command matched on the launch command, found nothing, fell back to a stale pid file, reported success and left 30 GB resident. I found an orphan from an earlier run still holding memory while congratulating myself on a clean shutdown. Anything that identifies processes by command line is one setproctitle away from silently finding zero.

The drafter is invisible to memory planning. My CLI prints a memory plan before starting anything, and speculation made it fiction twice over: the drafter is 3.76 GB resident for the life of the server, and DFlash keeps a prefix cache that, left alone, helps itself to 8 GiB on top of everything already accounted for. Both now go into the plan — the cache gets exactly the budget the planner had already set aside, 3.3 GB instead of 8, inside the plan instead of beside it.

Worth knowing on Apple silicon generally: the GPU’s recommended working set here is 37.4 GB of the 48 installed. A 31 GB model plus drafter plus caches fits, but with less room than the RAM figure suggests.

Was it worth it

A second Python environment, because the engine needs an older Python than the rest of my tooling. A drafter trained for one specific model architecture, of which very few exist. A log check before every measurement.

Against that: 9 tokens per second became 29. For handing mechanical work to a local model, that is the difference between starting a task and going to make coffee, and watching it finish.

Takeaways

1. Find the signal that says the mechanism is on. If the only evidence is the metric you are trying to move, "no gain" and "never ran" look identical — and you will pick the explanation you already believed.
2. Published defaults are tuned on published hardware. Block size 5 costs 10% here. The sweep that found 4 took ten minutes.
3. Bandwidth-bound is a spectrum, not a switch. It explains why the 8-bit model gains more, why longer blocks stop paying on a laptop, and why the drafter must be quantized — all the same argument.
4. Test the losslessness claim on your own workload. Temperature 0, same prompt, compare the bytes. Two minutes, and mine did not pass.