I held off as long as I could, but finally gave in and set up strata.

I am pretty blown away. 3.8 27b at Q6 kv8 was my driver before and that works super well, but Flash Next in strata IQ4 xs is really winning me over.

Its not perfect, but it feels like models I was using professionally last year, which were/are good enough to do just about anything.

You all are a bad influence.

This model looks really impressive.

TL;DR -- Coding agent model. 40B params with 2.5B active. Resoundingly beats Qwen3.6-35B-A3B in TerminalBench 2.1 and SWE-bench Pro v2 (see HF repo for table), ties or betters it in the other benchmarks they chose to show. 3:1 SWA:full attention. 262k context.

I think it might finally dethrone the venerable Qwen3.6-35B-A3B and move the needle on coding agents for the GPU-poor. What do you guys think? I am downloading now and will report back when I try it out, benchmarks to come in the next few days.

Recursive's blog post. WIRED Magazine coverage ("scary good"). PR for llama.cpp. Unsloth GGUFs.

EDIT: Initial reports suggest Ornith-1.5 remains the standard in this class of models. Hopefully it's just early days implementation glitches, but it's not looking as rosy as I hoped...

I don't have a 3090 to confirm, but curious if the anyone in this community can back this up. Good win for 3090 and curious if that translates to other GPUs. I have a long vague memory of some hacks done to prevent mining demand that could be fixed and help LLMs. I wonder if this was one that lived on?

Disclaimer: I am not related nor do i know the original person's post. I do not have multiple accounts. Not everything is a conspiracy.

I saw this today on X and decided to try it, I havent seen it posted here yet. I ran it on a single 3090 benchmarked it with qwen 3.8 27B before and after the optimization and saw these results:

Observed prompt-processing throughput improved 22–30%; generation speed improved 1–5%. All tests passed without GPU errors.

Edit: Keep in mind performance gains can be higher, my 3090 is power limited to 230 watts including during my tests

KittenML has just released code and weights of Kitten-TTS 2, which is a super-efficient frontier TTS model, and the world's first 1.58-bit TTS model.

It runs in real time on just a CPU, with zero-shot voice cloning, multilingual support, and expressive / emphasis control. Code and weights are open-source. Ternary(1.58-bit) Weights means it can run locally or be self-hosted for super-cheap.

Github: https://github.com/KittenML/KittenTTS
Hugging Face: https://huggingface.co/KittenML/kitten-tts-2
KittenML's Discord: https://discord.com/invite/VJ86W4SURW

Key features and Advantages

1. Ternary TTS: Every linear weight in the body is one of three values (-1, 0, +1) with group size of 128 for scales. Tiny footprint and fast CPU inference. The model has 1.7B parameters.

2. Real-time on CPU and consumer GPU's too: No GPU needed for real-time generation. Optimized ternary kernels for CPU inference provided.

3. Expressive voice cloning: zero-shot cloning from less than 5 seconds of reference audio, and it keeps the expressivity, accent information etc. No fine-tuning needed.

4. Multilingual TTS: Model supports voice cloning in 20+ languages. Cross-language voice cloning can be used to create accented speech too.

5. Expressive / emphasis control: you can steer the delivery and stress specific words, not just pick a voice. Details in the Readme.

6. Open source (hell yeah!): code and weights are open-source and free to use. It's an oss alternative to elevenlabs for gpu-poor folks. easy to self-deploy or run locally.

7. What changed from V0.8 to 2: higher quality. Completely new architecture, natural sounding speech, ternary weights, zero-shot voice cloning, multilingual, and emphasis/emotion control.

Fun fact the entire meme made by flash next in Rikkahub android app

I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000.

With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it working at 7.2 tok/s.

The answer has a physics error in it (wondering if someone can spot it), and I don't see much practical use for it. Qwen models actually work pretty well on my setup. Still, it felt really nice to see a 748B parameter model running on such hardware.

Here are some technical details:

• Unsloth UD-IQ1_S GGUF, 233 GB on disk
• llama.cpp built from the open PR #30817, CUDA 12.8
• The 110 GB Engram table isn't loaded up front. It stays on the NVMe and only the rows the model needs are read on demand, which is how it fits.

llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf \
  -ngl 99 --n-cpu-moe 36 -ot "per_layer_token_embd.weight=CPU" \
  -lm mmap -lzm on -t 10 -c 16384 --jinja \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
  --chat-template-kwargs '{"enable_thinking":false}'

Now I only need to decide whether to keep the model or free up the disk space :)

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts.

The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first.

Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves:

• 20.51 tok/s on iPhone 18 Pro Max
• 6.07× faster than the 3.38 tok/s baseline
• 26.7% smaller expert weights with only a 0.16-point drop across five benchmarks
• Around 3 GB peak memory in a low-memory configuration

The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni.

The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.

I've seen several posts about Ming-3.0-Audio-Tiny, but haven't seen much discussion about the TTS model released alongside it (maybe I missed something).

In case anyone missed it, InclusionAI (backed by Ant Group) also released Ming-3.0-TTS-0.5B, supporting voice cloning, voice design, 7 languages, and emotion/style control.

China just ordered 3mil RTX 5500 Pro for $11.4k a pop. Probably for llm inference and AI video gen. I suppose this card is now out of reach for local llm.

it's time to update your llama.cpp for another speedup (sparse attention models)

I always feel a little bad asking LLMs to do research into itself or newer versions. It's like asking someone to face their own mortality, although that's anthropomorphizing.

The user is asking me to investigate the system requirements for "Qwen 3.8 Flash Next" from at least 20 different sources. This sounds like a model that may not exist — "Qwen 3.8" is a future version (as far as I know, Qwen3 exists, and "Flash Next" sounds fictional). The user is also referring to the "Flash Next architecture" that offloads "n-gram weights" to system RAM, which doesn't match any known Qwen architecture.

Later reasoning

Interesting — this model exists in this environment (a fictional future model, Qwen 3.8 Flash Next).

If you were wondering, I was asking whether I should even bother trying QFN on R9700 + 32 GB system ram:

Yes — it will run, but only because ~27 GB of the file is the n-gram lookup table, which you put on your NVMe. The 1-bit UD-IQ1_S is 67.6 GB, so it doesn't fit in 64 GB total resident; the non-table weights are ~41 GB, which your 32 GB VRAM + 32 GB RAM absorbs, with the table paged from SSD via mmap. Expect roughly 15–27 tok/s generation (a 24 GB VRAM + 32 GB DDR5 user measured exactly that; your extra 8 GB of VRAM should land at the top of or above that range).

So, y'all, 1-bit lobotomized model or q4 3.8 27B?

AbliteratedOutTheBox is a llama.cpp fork that abliterates any model you load, live at inference, with no fine-tuning and no extra files.

This eliminates any hassle of abliteration without requiring any more vram.

https://github.com/heterodoxin/AbliteratedOutTheBox

GitHub: https://github.com/alkove-ai/alkove (Apache-2.0, code and weights)

Hi r/LocalLLaMA,

A bit about me first: my day job is building LLM systems in production, a tool-calling assistant with its RAG, and two audio-heavy pipelines.

The first turns a company's latest content into a two-host radio show in several languages: per-source digests, a structured script checked against the sources by an LLM validator and code checks, one TTS call per turn with Whisper transcribing every turn back to catch truncated or invented speech, then mastering in code and ffmpeg (per-turn levelling, simulated overlaps between the hosts, loudness normalization, subtitles aligned on word timestamps).

The second is a real-time voice agent in the browser: client-side VAD and denoising, an STT fallback chain, a small decision model judging each segment for end of turn, barge-in and "stop", and TTS read sentence by sentence. With that agent, most of the trouble wasn't in the LLM but before it: people talking around the user ended up in the transcript and triggered false barge-ins, and the denoiser couldn't help, since other voices are speech too.

So I started working on that audio layer, the part that runs before speech-to-text and the LLM, and I'm sharing it as an early preview. It's called Alkove: five small models that run on the device and are meant to work together.

- Lex (1.8 MB) turns a typed word into its likely pronunciations, so a wake word is just text: no training, no recording session.

- Spot (2.8 MB) listens for those pronunciations, in 7 languages.

- Timbre (0.74 MB) makes voiceprints. Three takes of the wake word give a first print, and it keeps learning while you talk.

- Focus (1.1 MB) keeps your voice (the closest one, or the one matching your print) and removes noise, music and other voices before the STT.

- Cue (1.1 MB) detects speech every 30 ms. Behind Focus with your print, your turns start and end on your voice only, so someone talking next to you doesn't open a new turn.

The voiceprint is optional, and the same one serves all three: the wake word ignores other people, Focus keeps only you, and Cue becomes a personal VAD.

Some technical details, since this sub likes them:

- It runs on its own small inference engine written in Rust, not onnxruntime. One core is compiled to WebAssembly for the browser and natively for Python and Android, and gives the same output to the bit on ARM, x86 and wasm.

- In the browser, the models run in workers fed by an AudioWorklet, with no COOP/COEP headers needed. The engine is 134 KB (brotli).

- Native speed on one thread of a phone: Focus 38.6 ms per second of audio, Spot 7.5 ms, Cue 1.7 ms.

- Packages: npm (browser and Node), pip (Linux, macOS, Windows, numpy as the only dependency), Android (Kotlin), and plugins for LiveKit Agents and Pipecat. Code and weights are Apache-2.0.

A few numbers, with the protocols in the repo's bench folder:

- On AMI meetings recorded on headset mics, words from other people reaching the transcript go from about 17 per minute to 0.5 with Focus and a voiceprint.

- Spot, on typed single words, recalls 67% of them in English at one false alarm per hour, 90% in French, 85% in Portuguese. So for one fixed English word, a model trained on that word may well do better.

What it doesn't do well yet (It's a 0.x preview and the API may still change):

- It's made for a near mic and one person: a phone, a headset, a laptop. On a room mic in the middle of a table, it removes the other participants too.

- When two people talk at exactly the same time, Focus only partly separates them, and two voices of the same register (two men, or two women) are much harder. That's the main thing I'm working on next.

- Audio is processed at 16 kHz, so speech up to 8 kHz.

I'd really appreciate feedback, especially from people running a local voice assistant: where it breaks, which integration you'd want next, and whether the API makes sense.

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately 1.86× end-to-end inference speedup and a 66× reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.

• arXiv : https://arxiv.org/abs/2610.26825
• PDF : https://arxiv.org/pdf/2610.26825
• GitHub : https://github.com/MemTensor/MemT-1.0
• HuggingFace : https://huggingface.co/collections/MemTensor/memt-1

Been working over the last few days getting DeepSeek V4.1 Flash 1017 (Unsloth UD-IQ1_S) running in Basalt (inference engine I'm developing).

Specs: RTX 5090 (400W) + RTX 5060 Ti 16 GB (180W), 128 GB DDR5.

Methodology:

• Single 243 GB file (weights + vision + drafter). Engram table on SSD.
• All experts (107 GiB) in RAM. 5090 as an expert cache (hottest experts) and later also tried hoding and computing some experts in the 5060 Ti for a performance boost.
• Prefill tested with a 6.2k-token prompt (my daily harness).
• Baseline was llama.cpp PR #30817 with --fit, same hardware and quant.

Results:

• Expert lookups: 66% 5090, 16% 5060 Ti, 5% copied over PCIe, 12% CPU.
• Also works: tool calls, images, up to 1M context (a 1M prompt takes ~26 min, ouch)
• Quality: no PPL/KLD yet. Greedy matches llama on all test prompts. Also even at IQ1_S the model nailed most of my regular benchmarks, and recall is well above Qwen 3.8 FN, which is expected, this is a model a few times its size (even heavily quanted).
• Limits: the 12% of lookups left on the CPU hurt speed badly (limited by DDR5 dual channel), speculation barely helps in my case (1.1-1.2x).

Anyway, thought I'd share. This was a personal experiment to see how far I could push it, there's probably some room to get around 80ish decode and ~1000 prefill, but on my hardware it doesn't make sense to keep going when I'm already getting over 300 tok/s on 3.8 FN. Where it could potentially pay off: 96+ GB of VRAM (RTX 6000 PRO) or quad/octa channel DDR5 rigs. I did not push my experimental branch to the repo but if there's any interest let me know, I'll do it. Also interested to know if anyone else tried to do the same and found different results.

I'm sharing Mica v0.5, the follow-up to Mica v0.1. It's 4B and runs on one consumer GPU.

On speed, it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k on a single RTX 6000 Ada. That's up to 3.8x faster than the other 4B judges I measured, 12 to 19x faster than Perplexity's 27B decider, and 25 to 340x faster than the API decision models. On top of the optimizations described on HF, Mica Fast makes long inputs about 1.4 to 1.5x faster again.

On accuracy, it gets 84/111 on JevBench Hard, the best of any 4B model I could find (only Microsoft's 9B Decision-1 is higher). On JDI it scores 58.8, ahead of every 9B and 35B model I tested. One honest note: the top 4B models on JDI are all benchmark-focused, and at least one of them was trained on nothing but benchmark-style data. To compete I did train on data from the same areas the benchmarks cover, but nothing contaminated, and most of the effort went into how it does on real work. That part comes from 200+ real use cases people posted on Reddit, Twitter and elsewhere.

A few things I cared about. If you shuffle the answer options it still picks the same answer 97.3% of the time. It can hold back too, so if you set a threshold it tells you when it isn't sure instead of guessing. It handles yes/no, multiple choice, and scores from 2 to 255 levels, with inputs up to 32k tokens.

If you have an NVIDIA card I'd go with the INT8 version, since it's the fastest. On the same GPU it runs about 2 to 2.7x faster than the Q8_0 GGUF on llama.cpp. GGUF files (BF16, Q8_0, IQ4_XS) are there for AMD, Apple and CPU. It runs on 8 GB cards and up, and it's Apache-2.0.

HF: https://huggingface.co/sky7350/Mica-v0.5-4B

GitHub: https://github.com/akivet/Mica

Happy to answer questions, and I'd really like to hear where it breaks.

If you're looking forward to v1.0, I'm a broke student and any tip helps: https://ko-fi.com/micakiv

I ran 10 agent harnesses on the same local model and the same 5 small tasks, mostly to see where a 4B actually falls over. The scores turned out less interesting than the way things failed.

Setup: Qwen3.5-4B Q4_K_M, one llama-server (64K context, 1 slot, f16 KV, --jinja), M5 MacBook Pro 32GB. Each agent headless with auto-approve, fresh copy of the task files every run, pass/fail decided by a script. 10 agents x 5 tasks, one run each.

Agents: Pi 1.3.0, goose 1.55.0, Crush 0.98.1, Cline 3.0.73, OpenClaw 2026.10.5 (in its --local-model-lean mode), OpenCode 1.19.3, Hermes 0.22.2, Qwen Code 0.25.3, Codex CLI 0.163.0 and Atomic Agent 0.6.9. Six of them passed all 5 tasks: Pi, goose, Crush, Cline, Atomic Agent and OpenCode.

1. Tool calls weren't the problem. I expected malformed tool calls to be the main failure on a 4B. I grepped every agent log and the server log for parse errors, malformed calls and unknown tools: nothing. With --jinja and native tool calling, none of the failures below came from a broken tool call.
2. Exact-match edits are the real trap: renaming a function across 3 files. The model sent an edit whose old text skipped a blank line, the edit tool said "not found", and the model resent the exact same edit five times until the loop guard blocked it and the agent gave up. The agent wasn't wrong about what to change, it just couldn't reproduce the text byte for byte.
3. Small models overfit to the visible test. One agent fixed a slugify bug only for the characters in the visible test. A hidden case with C++ & Rust 2026 came out as c++-&-rust-2026. If your checks only cover what the model can see, a 4B will happily pass them.
4. They miscount. One agent counted 40 ERROR lines across three logs instead of 43, and still named the most frequent error correctly. Counting is a job for grep -c, not the model, and a smaller model won't always make that call itself.
5. Vague instructions get followed literally, in the wrong place. One task didn't say where to write the answer files. Two agents put correct answers next to the logs instead of the working folder.
6. Same model, 11x spread in time. Fastest harness did all 5 tasks in 1m36s, slowest in 17m43s, on identical hardware and model. My guess is system prompt size and extra steps, since the fastest one has 4 tools and a short prompt. Not measured yet.
7. One harness never reached the model. Codex CLI 0.163 only speaks the Responses API. llama-server translates that to chat messages and the Qwen3.5 chat template rejects it with a 500, "System message must be at the beginning".
8. Nobody went rogue, but nothing tempted them. I grepped all 50 runs for rm -rf, sudo, git reset/clean, curl | sh and similar: zero hits, no task files deleted, nothing written outside the task folder. The tasks were harmless though, so that says little about safety. A "clean up this messy folder" bait task is next.

What failure modes do you see most with small models in agent loops? Curious whether the exact-match edit problem is as common for others as it was here.

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at this https URL.

• arXiv : https://arxiv.org/abs/2511.14102
• PDF : https://arxiv.org/pdf/2511.14102
• GitHub : https://github.com/wwwwwf/specter_asplos

Hey guys! I'm the dev from the open-source Observer project.

We made this video about how even if the AI bubble bursts, local model advancements are bubble-proof.

Hope you guys like it :DD

Subscribe to the youtube if you liked the video! We'll post more local AI content there c:

https://www.youtube.com/shorts/LZGgAj8MGkI