Every failure was plumbing
I spent a week handing real Go tasks to a 27B model running on a laptop. I expected to find the model's ceiling. I found nine infrastructure bugs instead — and then a harder question about whether the whole thing pays.
- Subject
- Local model, real work
- Hardware
- Apple silicon · 48 GB
- Model
- Qwen3.8-27B-8bit · MLX
- Faults located
- 9
- Attributable to the model
- 0
Going in, my assumption was the obvious one: 27B is small, it will fail at anything non-trivial, and my job is to locate the ceiling.
That assumption was wrong in an interesting way. Every failure I chased turned out to be infrastructure. Not once was it the model. Here is the list, because the list is the useful part — and after it, the part nobody writes up, which is whether any of this was worth doing.
The setup
An Apple-silicon Mac with 48 GB of unified memory, and one model:
MLX serves it behind an OpenAI-compatible endpoint on localhost. Every number below is that one model on that one machine; yours will differ, but the failure modes almost certainly won’t.
Around it sits a small Go CLI. The division of labour is the whole idea: a strong model writes the spec and the failing tests, the local model writes the implementation, and a machine gate decides whether it counts.
# once
localmodels download mlx-community/Qwen3.8-27B-8bit
localmodels serve -d mlx-community/Qwen3.8-27B-8bit
# per step
localmodels flow tests <task> <step> # model writes the failing tests
localmodels flow code <task> <step> # model writes the implementation
The gate is allowlist → gofmt → go build → go vet → go test. Nothing merges until it is green, and green is a machine’s opinion, not the model’s.
The crash that wasn’t about memory
Long generations died with:
RuntimeError: [metal::malloc] Resource limit (499000) exceeded.
I had two confident theories. Both were wrong, and both were killed by an experiment rather than by argument — which is the only reason this got solved at all.
The real cause: I was passing --prompt-cache-bytes as a memory guard. It is not a guard, it is a trim trigger. The server runs trim_to(limit − active) after every batch, where active is the KV of the in-flight request. During a long generation active grows, the difference collapses toward zero, and the server evicts the entire prompt cache over and over. That churn exhausts the device’s supply of Metal buffer handles.
The agent loop cost 10× more than the task
I started by driving the model through a coding agent. Per attempt it sent eleven requests averaging 22,468 tokens — because the agent’s system prompt and tool schemas ride along on every single call.
At 8 tokens per second, tokens are not a billing line, they are wall clock. And the spec already says which files change and what has to be in them; the gate already decides whether it worked. There is nothing left for an agent to plan.
The model was thinking, and I couldn’t see it
A file took 28 minutes and looked like a hung socket. It wasn’t hung. This is a reasoning model, and MLX puts the scratchpad in delta.reasoning, not delta.content. I was reading only content, so from where I sat, nothing was happening.
Which raised the better question: does the reasoning earn its cost on mechanical work? Same step, thinking on versus off — gate green both times, contract matched verbatim both times.
A green gate that meant nothing
One run printed PASS when one of its three files had never been written. The old file still compiled and still passed the tests — and the spec had no test for the new flag, so nothing noticed.
The fix has two parts. The runner now requires every file to actually be written. And the rule that matters more: every file in the allowlist needs a test that goes red without it.
I was paying for the file, not for the change
The model returned whole files. At 8 tokens per second, that means your wall clock is the size of the file, not the size of the edit. One 352-line file was retyped in full to change three lines: six minutes.
Switching to search-and-replace edit blocks, measured on the identical step:
It brought its own failure mode, and it is a nasty one: a line the model pulls into the SEARCH block for context and forgets to repeat in REPLACE is a line it silently deletes. That is exactly how one run lost an attempt — a cat := models.NewCatalog() vanished because my spec said “replace the whole block” without saying where the block ended.
Two guards earn their keep here. A SEARCH that matches zero times is an error — and a SEARCH that matches twice is also an error, rather than a coin flip.
A failed attempt poisoned the next one
With whole-file replies, a failed attempt was harmless — the next reply overwrote everything. With edits it was fatal. The failed attempt’s edits stayed in the tree, so the next prompt showed the model a file that was neither the base its spec described nor a working version. It re-derived the same edits and none of them matched. Three attempts in a row died on SEARCH not found.
A run locked the entire repository
The gate reverts anything dirty outside the allowlist. Correct for the model — and it also reverted my work in progress. For nine minutes at a stretch I couldn’t touch a single unrelated file while a run was going.
Two defaults nobody had set
Driving the model from an interactive agent looked hopeless. A single turn ran seven minutes and produced nothing. Two server defaults, neither of them mine, neither of them wrong on its own:
The model reasons unless told otherwise. Even a turn whose prompt was almost entirely a cache hit spent its whole output budget on scratchpad nobody reads.
MLX samples greedily. Its default is --temp 0.0, and greedy decoding on a short tool-use prompt walks into a repetition loop that runs until the output cap. My own batch client never hit this — it sends a temperature with every request. The agent sends none, so it inherited the default.
gofmt and go vet without being asked.A flag that never worked, behind a green gate
serve --context passed --max-kv-size to the server. The current MLX release removed that flag, so the server exited before it ever bound a port. The feature had never worked. Not once.
The gate stayed green the whole time — because nothing in the test suite launches a real server. It surfaced only when I finally tried to use the thing for something else.
What actually got faster
With the plumbing fixed, six consecutive real tasks went through the loop. The step log, verbatim:
| Step | Files | Outcome |
|---|---|---|
| Resolve the active model | 2 | green 1st · 280 lines · 5m05s |
| The same step, re-run with edits | 2 | green 2nd · 1,207 tokens · 2m25s |
| Extract a memory-pressure parser | 1 | green 1st · 169 lines · 2m49s |
| Context-limit check — tests and code by the model | 1 | tests 1m20s · code 1m18s |
Collapse seven if branches into one config table | 2 | green 1st · 3 edits · 3m09s |
| Mark a plan step done from the CLI | 2 | green 1st · 3 edits · 1m54s |
Typical step, once edits replaced whole files: two files, three or four edits, two to three minutes, green on the first attempt.
Handing it the tests
Letting the same model write both the tests and the code is the move that can quietly turn this loop into a model agreeing with itself. So nothing is taken on trust — four checks, all mechanical:
- the suite must be green before the new tests land, or “red” proves nothing
- each test file must turn it red on its own, with the other new files absent
- a rewritten test file may add tests but never drop the ones already there
- test files and implementation files may not overlap
A file that fails the red-alone check is thrown away, not argued with.
The tests came back genuinely good — eight cases, both boundaries, the exact error strings. But only because the spec named those strings verbatim. Write “return a sensible error” and the test will pin whatever the model invents, and your green gate stops meaning anything.
Was it worth it? The honest accounting
Everything above is about making the loop work. Whether the loop pays is a separate question, and for a long stretch the answer was no.
For a small step, the spec plus the red tests ran to roughly 130 lines of my writing to obtain about 45 lines of implementation. That is a loss on any reading. I could have typed the 45 lines in less time than it took to describe them precisely enough for something else to type them.
The crossover, on my numbers, sits somewhere around 200–300 lines of mechanical output: a refactor that touches several files the same way, a table-driven conversion, boilerplate that follows from a pattern already in the repo. Above that line the spec is cheaper than the typing. Below it, it isn’t.
Two things move the line in the loop’s favour, and neither is speed.
The spec is not overhead, it is the review artifact. I have to decide what the change is either way. Writing it down in a form precise enough for a model is the same work as thinking it through, minus the illusion that I already had.
The failure mode is legible. When a hand-written change is subtly wrong, you find out in production. When this loop is wrong, it is wrong in one of three places — the spec was vague, the test was weak, or the gate didn’t cover it — and all three are things you go fix once.
if blocks that satisfied every assertion in the spec and missed the entire point. The gate was green. Green was correct. The code was not what anyone wanted.The limit nobody automated away
On one step the generated tests pinned a new function perfectly — and said nothing about whether anything calls it. An implementation that defined the function and never wired it up would have sailed through every check, feature completely dead.
The model got the wiring right. I only know that because I read the diff.
A machine gate tells you the code compiles and the tests pass. It has no opinion on whether the code is meaningful. That part hasn’t moved, and I don’t think it’s about to.
Running it yourself
If you want to reproduce the setup rather than the mistakes, the two defaults from fault 08 are the ones to override when serving with mlx_lm directly:
mlx_lm.server --model mlx-community/Qwen3.8-27B-8bit \
--temp 0.6 --top-p 0.95 --top-k 20 \
--chat-template-args '{"enable_thinking": false}'
Both stay per-request overridable — a client that sends temperature or chat_template_kwargs wins over the server. Which is exactly why my batch client never noticed the problem the interactive agent hit head-on.
One more trap if you point a coding agent at it: check the agent’s config schema before you trust your own config file. Mine had maxTokens, tools and maxRAMGB in it — none of which exist in the schema, whose model entries are additionalProperties: false. It had been silently wrong for a week.
Takeaways
- Assume plumbing before you assume the model. Nine failures, nine infrastructure bugs, zero capability limits.
- Kill hypotheses with experiments, not arguments. Both of my confident theories about that Metal crash were wrong.
- Pay for the change, not the file. At local decode speeds, output tokens are the entire budget.
- A green gate is worth exactly as much as the test behind it. Ask what would still pass if the feature were deleted.
- Specify verbatim. Every ambiguity I left was resolved by the model in the simplest way available — and that was my bug, not its.
- Know where your crossover is. Handing off work smaller than the spec that describes it is a hobby, not a speedup. That's fine — just don't confuse the two.