llama.cpp merged PR #29887 two weeks ago and I think it is the most interesting thing to land in local inference this year, so I went through the PR, the benchmarks and the review thread and wrote up how it actually works. Short version below, long version linked at the end.

The problem

Qwen3.8-Flash-Next at Q4_0 is 93.7 GiB, and 65.4 GiB of that is experts. It is 125B total but only 6B active: 512 experts per layer, the router picks 10 (plus 1 shared) per token. One expert is 3 x 2560 x 640 = 4.9M params = about 2.6 MiB at 4-bit. 10 experts x 48 layers = 480 experts per token = about 1.3 GB of weights touched per token.

The observation

Expert routing has locality of reference. Token N+1 mostly picks the same experts as token N. A handful of experts are hot, most are almost never touched. That is the same property your CPU cache, your OS page cache, your browser cache, every CDN and Redis allkeys-lru all rely on.

The fix

PR #29887 (by am17an, a port of Tether’s qvac-fabric work, merged by ggerganov on Oct 7) sets aside a slice of VRAM as an LRU cache of experts. Hit: compute from VRAM, zero bytes moved. Miss: copy that one 2.6 MiB expert over PCIe and evict the least recently used one. The log prints the hit rate at exit.

Numbers from the PR description, same model file, same machine:

The 5090 gains more for two reasons: PCIe 5.0 halves the cost of every miss, and 32 GB means a bigger cache and fewer misses.

How to use it

llama-server -m Qwen3.8-Flash-Next-Q4_0.gguf -fa on -c 32768 -b 2048 -ub 2048 -cmoe --moe-cache-mib 11000

The PR author’s rule of thumb is about 10% of your expert bytes (65.4 GiB -> 6544 MiB). Env var form is LLAMA_ARG_MOE_CACHE_MIB.

SGLang merged the same idea 12 days later as “paged experts” in PR #42557

The part I find most interesting is LRU , Locality of reference go back to 1960s.

And i find locality of reference principle highly underrated.

Full write-up with the memory math script, an expert cache simulator you can run, and the benchmark table: https://cloudmash.blog/posts/moe-expert-offload-lru-cache-consumer-gpu/

llama.cpp PR #29887, MoE expert cache: https://github.com/ggml-org/llama.cpp/pull/29887

SGLang PR #42557, paged experts: https://github.com/sgl-project/sglang/pull/42557

A couple of days ago I posted about having a few spare office computers my job threw away and I'm happy to report the motherboard jump worked flawlessly and I'm currently downloading windows 11 on one them after deleting all the old partitions. Thanks for all your help, it's been an awesome learning experience for me 🙏🏾

A quantum processor can do more than execute a fixed sequence and reveal its answer at the end. It can measure selected qubits while a circuit is running, use the result to choose an action and continue. Repeating that cycle across a large system is an important challenge for quantum hardware.

An experiment involving Rutgers University, IBM and collaborators has now used that capability to steer up to 100 qubits from scrambling toward an orderly state. Published in Nature Physics, the study tested how competing quantum operations produce a collective transition between chaotic and controlled behavior.

The work demonstrates coordination between quantum gates, measurements and feedback over thousands of operations. Those capabilities matter for future error-corrected machines. However, the experiment controlled a deliberately designed physical system; it did not demonstrate a computer correcting arbitrary mistakes during a useful calculation.

im a student new to the computing/IT sector, i find that my teachers are dismissive towards me and kind of just exclude me or mansplain anytime i try to contribute or ask questions in class.

ive come to this conclusion as i am the only woman in class, and they only do this to me but they are always happy to help the other students who are all male and treat them way differently.

ive taken matters into my own hands and i want to get ahead of the curriculum but im struggling with the practical side of database and programming.

so far i know how to create tables and assign a primary key but thats about it. i want to be able to create relationships and actually understand it. rather then just do things on autopilot.

ive been using the database design app on google play store since my laptop is chromebook but i cant seem to get a design view and input data into it.

any advice on good apps to use and practical tips on how to approach both database and programming/coding will be very helpful!!!

i dont want to get stuck in tutorial hell ....

Which one is the best from your perspective? For me, those options are somehow resistant in this AI context.

A quantum processor can perform beautifully when its qubits take turns, then stumble when several work together. That problem threatens efforts to build useful machines, which must coordinate many operations before fragile quantum information fades. A five-qubit silicon experiment now shows how to preserve much of that accuracy during simultaneous control.

Researchers at the RIKEN Center for Emergent Matter Science and collaborating institutions traced a major source of interference to the microwave signals controlling the qubits. They then developed a calibration method that compensates for unwanted changes in quantum phase.

Their results, published in Nature Physics, support a simpler wiring approach for larger silicon quantum processors.

It’s crazy to think that the entire modern AI revolution started with researchers just trying to mathematically mimic a single biological neuron. When you zoom out, the timeline of AI architectures looks exactly like an evolutionary tree.

We went from struggling to solve basic logic puzzles (shoutout to the XOR problem) to building early neural networks, realizing they forgot things too quickly, battling the vanishing gradient problem, inventing LSTMs to give models actual memory, and eventually landing on Transformers with attention, the architecture that essentially ate the world. Every major roadblock forced a mutation that led to a massive paradigm shift.

But looking at this evolutionary timeline brings up a massive question: If Transformers are just the current "apex predator" of AI architecture, what exactly is the next evolutionary leap?

Are we going to see a completely new paradigm that makes self attention look as primitive as a basic AND gate? How do we even begin to perceive or predict what lies beyond our current models when the jumps are this exponential?

I went down a massive rabbit hole mapping out this entire evolutionary timeline. I wanted to see the exact breadcrumbs that led us here, so I put together a deep dive covering the journey from the first perceptrons to the modern era, complete with the foundational paper citations and the historical tweets that marked these shifts in real-time.

If you want to look at the complete map of how we got here to help predict where we are going, I compiled the whole history here: https://cloudmash.blog/posts/evolution-of-ai-perceptron-to-text-to-video/

Would love to hear your theories on what does the post Transformer era actually look like ? Like what can we imagine , how this intelligence on our command can do for us , how enterprise and big players can use it , what can happen ?

Does anyone know which HSM vendors NVIDIA, Intel, Google, and Apple use to protect private keys for firmware signing and Secure Boot, and which commercial HSMs support FIPS 140-3, non-exportable keys, offline/air-gapped operation, and multi-party signing authorization?

I work as a security guard for this office lease building with a bunch of different companies in it. One of the companies threw out a bunch of computers, keyboards etc so I managed to grab 3 " dell optiplex 7460 all in one" computers, power cable, mouse and keyboard. Ideally I'd like to keep on for myself and sell the other 2 after getting into the computer itself but as you can imagine they're all still password protected from the old employees using them. Top of the screen upon booting says access to this system is for authorized personnel only. Has the company name. Asks for a user name and password. Any ideas on what I can do with these or how to maybe do a factory reset of some kind?

Hello! I’m currently working on a business project for school about an idea I have for creating a more advanced air filter for your computer. I’m trying to collect as much responses as possible by the end of the week . If you have a few minutes, I’d really appreciate it if you could take my survey and answer the questions honestly. Every response counts and It would help a lot on getting this business idea started as well as help my grade. Thanks!

https://docs.google.com/forms/d/e/1FAIpQLSc4wzvd00ikGmJ-nBM6uZFH6kBMV7l_J6gLDQC7dt7EIGrUfw/viewform?usp=dialog

Right now, the AI space feels entirely focused on massive datacenter clusters and renting H100s by the hour. But after spending way too much time looking at the actual footprint of these models, I realized that 90% of use cases are completely over engineered.

You don’t always need a multi GPU setup. The AI ecosystem is actually a massive spectrum.

I recently sat down and mapped out the exact tiers of AI models based on their size, the hardware needed to run them, and the point of diminishing returns.

Here are the two extremes and the sweet spot in the middle:

• The 100KB Extreme (TinyML) (Tensorflow Lite , sensor anamoly detection models): We are talking models that run on microcontrollers drawing single-digit milliwatts. They run on kilohertz processors using ultra-quantized integer math. You can run basic sensor anomaly detection or wake-word detection on a device powered by a coin cell battery.
• The Local Sweet Spot (4GB to 40GB) (Mistral 7B, Gemma 2 9B/27B, Qwen 2.5 14B/32B): This is where the magic happens for most devs right now. You can run highly capable 7B to 35B parameter models (like Llama 3 or Qwen) at 4-bit quantization on a standard Mac or a consumer GPU (like an RTX 3060 or 4090). It’s perfect for local RAG, coding assistance, and uncensored chat. VRAM is your only real bottleneck here.
• The 2.5TB Behemoths (Deepseek, Llama , Kimi k3): State of the art massive Mixture of Experts (MoE) routing. To even load these, you need dedicated power infrastructure and server racks of specialized accelerators drawing thousands of watts.

The missing piece: Figuring out the exact math for your hardware

The hardest part about building right now is looking at a model on Hugging Face and trying to calculate exactly how much VRAM you need, what quantization to use, and whether your CPU/GPU will choke on the context window.

So, I wrote a complete deep dive breaking down the math for all tiers of the AI spectrum.

If you want to see the architectural differences at each scale, and a cheat sheet for matching the right model size to your specific hardware, I put the full breakdown on my blog here:

https://cloudmash.blog/posts/ai-model-size-memory-hardware-guide/

Let me know what you guys think especially if you've found any ultra efficient small models/technique that punch above their weight on consumer hardware. And also I would love to hear whether quantization have resulted in major difference in quality , like if anyone have that kind of experience in that.

EDIT: Hello everyone, taking advice from your comments, I have added new model examples and kept old ones as they fit in the already assigned tiers, for reference , year of launch is added alongside the model.

Can storing games on an HDD for a long time without plugging it in cause damage?

I has want to make mechanical computer run base 3 coded software. My solution, see saws.

See saws connected to shafts in conjunction with other see saw and shafts in specific ways to enable logic gates running on 1,0, and -1

(-1 pictured above)

-1=far left side down

0= perfect equilibrium

1=right side down

What would the math behind converting binary logic gates to trinary like so be?

How would I calculate that?

I’ve started building a browser-based sandbox for experimenting with balanced ternary computing.

The reason is fairly simple: I want a place where I, and hopefully other people, can try different ternary designs without first deciding what the “right” architecture is.

The simulator uses:

-1, 0, +1

and lets you build small primitives, combine them into reusable components, nest those components several levels deep, test them independently and inspect how signals propagate.

I’m deliberately trying not to just recreate binary logic with three values. Instead, I want to explore whether things like three-way selectors, -1 / 0 / +1 comparison results, ternary control signals, MIN/MAX, different normalize/carry designs and different primitive sets lead to something more natural.

One of the goals is also to make it possible to compare multiple implementations of the same thing and eventually share designs with other people.

It’s still quite low-level, so it’s probably easiest to get into if you already have some familiarity with digital logic or computer architecture.

Everything runs locally in the browser. There is no backend, no account and no cloud storage. Your projects stay in your browser unless you export them.

The project is completely free and MIT licensed.

Live:
https://grunna.github.io/ternary/

Source:
https://github.com/grunna/ternary

It’s still early and experimental, which is also why feedback and alternative design ideas are very welcome.

Full disclosure, this is my podcast. I think this community would enjoy the stories, though.

Ted started out in tech support at Delphi Internet in the early '90s, back when teenagers were opening accounts with fake credit card numbers. He tracked one of them to a house in Pennsylvania and called it. The kid's father, a reverend, answered and promised to take him out to the woodshed.

Ted went on to run security for Boston investment firms. His favorite hire was Mr. Mojo, a guy he paid to physically break into his own offices. Mojo got into one building wearing a visitor sticker he'd pulled out of the smokers' trash. In Dublin he got through a locked door by blowing into a plastic bag.

Ted was also at DEF CON in 1999, where someone shut off the air conditioning during the opening ceremony. His theory is that the best hackers come from music or philosophy, not computer science.

It's onefjef episode 62, it's audio only, and it's on all the platforms. Here's the Spotify and Apple Podcasts links.