35B MoE running at 20.5 tok/s on an iPhone, without skipping experts

[u/GasSmooth7439]

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts.

The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first.

Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves:

• 20.51 tok/s on iPhone 18 Pro Max
• 6.07× faster than the 3.38 tok/s baseline
• 26.7% smaller expert weights with only a 0.16-point drop across five benchmarks
• Around 3 GB peak memory in a low-memory configuration

The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni.

The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.

[u/Left-Bug-7295]

Is this technique suitable for desktop inference with GPU and larger models?

[u/FullstackSensei → u/Left-Bug-7295]

It's basically what strata does.

Apparently, expert IDs don't change much between layers in many (most?) MoE models. i.e: if layer X used expert 50, there's a good chance layer X+1 will also use it. So, strata will pre-load that expert for the next layer while the current layer is still executing.

It does speed things up, but don't expect miracles, especially with larger models. It works here because the model itself is also heavily quantized.

[u/zyxciss → u/FullstackSensei]

Last time I worked with Strata, it didn't prefetch experts; it used a hot cache and a CPU pool instead... blah blah. Well, it's amazing, though. It's 8× faster than llama.cpp: I get 8 tok/s, while llama.cpp barely manages 1–2 tok/s, and my fork hits 24–26 tok/s

[u/FullstackSensei → u/zyxciss]

If you have enough RAM, all experts are loaded. If you don't, it prefetches from disk.

[u/zyxciss → u/FullstackSensei]

and well that prefetch is weak with a hit rate that's unstable. My fork has got it stable and way FASTER

[u/FullstackSensei → u/zyxciss]

Good for you. Now we know what your comment is about

[u/jazir55 → u/zyxciss]

What fork

[u/Holiday_Point_603 → u/Left-Bug-7295]

I was just about to ask, if this works on our inference devices it could be huge to run large models with acceptable speeds

[u/No-Business5854 → u/Left-Bug-7295]

Would be nice, i have 6gb of vram but enough ram for the rest of experts

[u/KITTYCAT_5318008]

Which quant?

[u/KITTYCAT_5318008 → u/[deleted]]

Not size, quantisation. Like iq2_xxs or q5_k_xl.

[u/Manerfish → u/[deleted]]

that's parameter count, not quant

[u/zyxciss]

I have been working on the idea of predict which experts the next layer might need preload their weights 
https://www.reddit.com/r/LocalLLaMA/comments/1x1tclb/qwen_38_flash_nextgsqrcoiq2_xs_at_21_toks_on_just/

[u/Skyline34rGt → u/zyxciss]

I wonder can this help me run Qwen 3.8 Flash with at least Q4 (so >110GB) at Rtx3060 12Gb + 64Gb ddr4? with good speed?

[u/zyxciss → u/Skyline34rGt]

Well... well, it would get you well above what current engines can offer. The problem is the complexity of building a Lookahead Expert Scheduling (LES) system. Since I have little to no AI assistance, I write most of the stuff by hand in big 2026, apparently, because I’m broke and can’t afford a subscription

[u/jazir55 → u/zyxciss]

Since I have little to no AI assistance, I write most of the stuff by hand in big 2026, apparently, because I’m broke and can’t afford a subscription

I gotcha, here's a bunch of ways for free inference:

Get the KiloCode VS Code extension, they have a constantly rotating suite of free models from OpenRouter, lots of stealth releases and newly released models.

The Nvidia NIM API is free and they have most of the popular open source models (aside from Qwen). You can use the Nvidia NIM API in either KiloCode or OpenCode.

OpenCode has the models they offer for daily quota'd free use via the OpenCode CLI.

OpenCode Go is their subscription plan, it's $10/month and you get an insane number of requests if you use Mimo 2.6 or Muse Spark 1.3 contributor. You can be running agents all day with OpenCode Go, I task tons of subagents at a time.

OpenCode Go is optional if you want a lot of usage, otherwise between NIM/Kilo/Zen free models you'll be good to go.

[u/xrvz]

I wonder if Qwen 3.8 FN would run at ~10 t/s then on an iPhone 18 Pro.

[u/CTRL___ALT___DEL → u/xrvz]

10 t/s for 3.8 FN would be unusable. That model will spend a good 100k tokens implementing the most basic task. I routinely get massive 40k token single-turn thinking blocks. It feels slow at 100 tok/s - can't imagine just 10.

[u/StupidScaredSquirrel]

Surely the catch is that while decode is fast, prompt prefill os of abysmal and unusable speed? Does anybody know?

[u/Temporary-Mix8022]

I've got a Pixel. Guessing I'll be able to get 0.0002 tokens per second? 

That's insane. About 100x on the 0.0000002TPS I got on the Google dev kit where they still don't even have hardware accelerated AI on their OWN phone.

Sorry. Segway. Came here to want. Thanks for reading 

[u/Dry_Basket8424]

the prefetch idea is the interesting part to me, it's basically the same problem as offloading experts to system ram on a desktop, just flash instead of pcie. curious how hot the phone gets after a few minutes at 20 tok/s though, mine throttles hard on anything sustained

[u/powertodream]

which ios app

[u/thestillwind → u/powertodream]

Custom one probably

[u/Minute_Laugh8065]

This is the right bottleneck to attack. The same problem shows up on 12 GB desktop cards like the Intel Arc B580: MoE models that don't fit end up with experts in system RAM running on the CPU. Lookahead prefetching into VRAM over PCIe is basically the same trick one memory tier up. I work on llama.cpp's SYCL backend for Arc, and I'm curious how much of LES is device-agnostic. Is the expert predictor separate from the Metal/ARM kernels in llama.cpp-omni, or tied to them?