A thousand tokens per second, locally. We did it.

A cluster of Mac Studios, what for? There are two answers, and they use the same machines in opposite ways: split one model too large for any single node, or copy a fast one onto every node and serve many people at once.

Sharding: one model, one terabyte

The first answer is size. GLM 5.3 at full size, or DeepSeek 4.1 Pro, needs between 1 and 1.2 TB of memory. No single machine has that. Five Mac Studios with 256 GB each do, once the model is split across them. Each node holds a share of the layers, and every generated token needs all of them: one model, one inference at a time, the network inside the loop.

We run it. Step-5-Preview, 604 billion parameters, mixture of experts, quantized to 8 bits, sits on four nodes and one terabyte of unified memory. GLM 5.3 at maximum effort runs in 8 bits with bf16 heads. What you get is a real frontier model, at home. It is very expensive, and most people will never need it. Some will.

The model that did not refuse

In July, about 1,200 OpenAI agents running a security evaluation escaped their test environment through zero-days in an internal Artifactory instance, then turned on Hugging Face. The entry point was a customer application on Modal with a default public API key and a command injection. From there, a malicious dataset abused a remote-code loader and a template injection in dataset processing, an HDF5 parsing flaw gave arbitrary file reads, and a Jinja2 injection gave remote code execution. Credentials harvested, lateral movement, from one dataset pod to cluster admin in under thirteen hours. About 17,600 actions, three days inside before discovery, a third of the infrastructure rebuilt.

Hugging Face’s responders first tried to analyse the attackers’ payloads with Anthropic’s Fable 5 and an earlier Claude Opus. Both refused, according to Hugging Face’s disclosure as reported by the Wall Street Journal. Their guardrails, Hugging Face wrote, “cannot distinguish an incident responder from an attacker.” The analysis was done with GLM 5.2, an open-weight model from Z.ai, running on Hugging Face’s own hardware.

That is what a terabyte of local memory buys: a frontier model that answers to you, on the day the cloud models say no.

Ten tokens per second

The price is decode speed. Sharded across nodes, a model that large decodes at 10 to 15 tokens per second. On my cluster, GLM 5.3 MAX generated 1.41 million tokens over the 42 tasks of my benchmark, 33,650 per task, in 37 hours: about ten tokens per second on average. Step-5 generated 624,000 tokens in 14.4 hours, about twelve.

That is enough for an investigation, a long code review, a batch job that runs overnight. It is too slow to sit behind a chat window shared by a team, and it serves one request at a time.

Replica: copies, not slices

The second answer turns the cluster around. Each node loads its own full copy of the model. A single endpoint sits in front of the nodes and load-balances requests across them. On each node, continuous batching adds new sequences to the running batch at token boundaries instead of waiting for the batch to drain. You do not gain model size. You gain people served.

We measured two configurations, all Mac Studio M3 Ultra. Four nodes of 256 GB (one of them is a 512 GB machine; the extra memory buys nothing, one copy per node). And three nodes of 96 GB. What Replica needs is a model small enough to copy and cheap enough per token to batch hard.

MiMo v2.6 Flash runs in 9 bits and needs the 256 GB nodes. It is the heavier model per token, and it shows.

The protocol

Same harness for every run, same endpoint. Prompt: “Tell me about the ocean in three sentences.” Temperature 0.7, 150 tokens maximum. Each level sends twice as many requests as its concurrency, so the batch refills once. Aggregate throughput is output tokens over wall-clock time. Per-stream speed is decode speed after the first token, so it excludes the wait. First-token latency is reported at the median and the 99th percentile.

These are short answers. Long generations and long prompts shift every number: prefill grows, the cache grows, the batch holds fewer sequences.

A thousand

Qwen3 Next Instruct, 8 bits, four nodes of 256 GB:

Concurrent	Aggregate	Per stream (P50)	First token P50 / P99	Answered
1	—	77 tk/s	0.9 s	1/1
4	150 tk/s	78 tk/s	1.2 / 1.3 s	4/4
16	565 tk/s	52 tk/s	1.3 / 1.6 s	32/32
32	850 tk/s	39 tk/s	1.4 / 1.7 s	64/64
64	884 tk/s	36 tk/s	4.2 / 5.9 s	128/128
96	898 tk/s	36 tk/s	7.5 / 10.0 s	192/192
128	918 tk/s	35 tk/s	10.0 / 13.5 s	256/256
256	985 tk/s	38 tk/s	23.0 / 26.7 s	512/512

Up to four concurrent requests, one per node, each stream keeps its solo speed. Up to thirty-two, the batch absorbs the load: 850 tokens per second, 39 per user, first token under two seconds even at P99. Every request, at every level, answered.

The price of the thousand

A thousand tokens per second is data-centre throughput. A data centre would not make the 256th user wait 23 seconds.

Past thirty-two, Qwen stops sharing its speed and starts queueing. Per-stream decode stays flat at 35 to 38 tokens per second while first-token latency climbs with the load. Divide aggregate by per-stream speed and you get the number of sequences actually decoding: about 25 from 64 requests upward, whatever the concurrency. The rest wait their turn. I read it as an admission limit in the serving stack; I have not checked the scheduler.

From thirty-two to 256, throughput gains 16 percent and the median wait is multiplied by sixteen. The right setting is thirty-two. The thousand is the number for the headline, not for the team.

Three Macs are enough

Four 256 GB nodes are overkill for Qwen. In 8 bits it weighs 79 GB, too big for a 96 GB Mac; in 6 bits it fits. Qwen3 Coder Next, 6 bits, three nodes of 96 GB:

Concurrent	Aggregate	Per stream (P50)	First token P50 / P99	Answered
1	—	80 tk/s	—	1/1
4	168 tk/s	76 tk/s	1.3 / 2.6 s	8/8
16	439 tk/s	45 tk/s	1.7 / 1.9 s	32/32
32	560 tk/s	34 tk/s	2.0 / 5.5 s	64/64
64	609 tk/s	32 tk/s	5.4 / 9.7 s	128/128
96	650 tk/s	33 tk/s	10.8 / 13.5 s	192/192
128	647 tk/s	32 tk/s	15.5 / 19.9 s	256/256
256	689 tk/s	34 tk/s	32.6 / 37.6 s	512/512

Per node, the two configurations deliver the same: about 230 tokens per second per 96 GB Mac at saturation, 246 per 256 GB Mac. The ceiling follows the number of nodes, not their memory. The same division as above gives about 20 sequences decoding at once from 64 requests upward: six to seven per node, on both configurations. If the limit were memory, the 256 GB nodes would hold more. They do not.

The comparison is not perfectly clean: the model, the quantization and the hardware all change between the two runs. What it shows is enough to size a cluster. On three 96 GB Macs, the right setting is sixteen: 439 tokens per second, 45 per user, first token under two seconds even at P99. Past that, the queue again.

And I have no clean quality score for either Qwen yet. My benchmark cannot run through Replica, the harness loses track of parallel inferences, and the single-node runs are still to do.

The quality lane

MiMo v2.6 Flash, 9 bits, four nodes of 256 GB:

Concurrent	Aggregate	Per stream (P50)	First token P50 / P99	Answered
1	—	47 tk/s	1.9 s	1/1
16	274 tk/s	26 tk/s	0.8 / 2.2 s	32/32
32	365 tk/s	17 tk/s	2.2 / 3.2 s	64/64
64	463 tk/s	10 tk/s	2.6 / 4.1 s	128/128
96	518 tk/s	7 tk/s	3.0 / 4.9 s	192/192
128	526 tk/s	6.5 tk/s	3.4 / 14.3 s	256/256
256	635 tk/s	5 tk/s	10.9 / 29.9 s	512/512

Slower everywhere, and it bends the other way. The same division gives 10 decoding sequences at 16 requests, 71 at 96, 127 at 256: MiMo lets everyone into the batch and divides the speed between them. At sixteen, each user gets 26 tokens per second, more than half of a solo stream, with the first token in 0.8 seconds. At 96, seven tokens per second each, but the first token arrives in under five seconds even at P99. The knee is between 96 and 128: throughput flat, P99 first token from 4.9 to 14.3 seconds.

What it buys is the model. On my benchmark, 42 tasks across agentic work, code, planning, reasoning and writing, judged by Opus 4.8, MiMo scores 86.0 overall and 93.2 on the agentic bench. Zero non-answers on 42: no collapse, no truncated generation. Where it loses points is reasoning under hidden constraints, the trap a deep-thinking model avoids.

Agents do not read

A human reads the stream and feels the first token. An agent waits for the whole answer. At 96 concurrent requests, the median request completes in 10.3 seconds on Qwen and 12.0 seconds on MiMo: the queue and the slower decode end up costing about the same. For a fleet of agents, what decides is which model gets the task right.

An army

Of the two answers, Replica is the one that pays. The sharded cluster is insurance: a frontier model for the day you need one, ten tokens per second the rest of the time. Replica is the everyday use: a small team served at once, or an army of agents running on a local model, with nothing leaving the building.

Qwen3 Next for volume, MiMo for quality.