Hey guys! I'm the dev from the open-source Observer project.

We made this video about how even if the AI bubble bursts, local model advancements are bubble-proof.

Hope you guys like it :DD

Subscribe to the youtube if you liked the video! We'll post more local AI content there c:

https://www.youtube.com/shorts/LZGgAj8MGkI

I found this project that I never saw talked about enough which is absolutley baffling to me. The title is not clickbait, and I can't blame you if you thought it is, because I was in your shoes once.

Meet ChatJimmy. It's a chatbot made by Taalas. And it's not your average chatbot. This is the closest LLMs have come to a physical brain than mere software that runs on your computer.

Though not an increment to intelligence (since it's an 8B model) and that's not the point of the project, this single chip makes the model run faster than if it were run on a multi-GPU datacenter. At a blazing 17k tok/s. For reference, the average speed of running the same model on an RTX 5070 is 80 tok/s. This chip is around 213x faster. If someone told me this was possible a year ago, I'd ask for what they're smoking.

The magic is actually pretty simple behind the scenes. The way Taalas was able to achieve such absurd speeds is by burning the model into silicon. Yep, you heard me right. The LLM is no longer loaded on a chip. The LLM is the chip. All computing power is purely gone straight the model's internal matrix math. The concept on it's own sounds cool as fuck, and seeing it work in practice is even cooler.

Keep in mind, this is merely an experimental proof of concept, not an actual product yet. Which is both exciting and depressing. On one hand, you can think about the potential of how this 17k tok/s would be dwarfed if the industry takes this path seriously, and more resources and investement were put into it. We would get what, 27B models running at 400k tok/s ??? This just sounds like a wet dream. And the sheer amount of possbilities it unlocks for us Llamas is crazy. Imagine buying your favorite model's chip for the price of an iphone, and it runs orders of magnitude faster than your old 50K enterprise rig? On the other hand, this is still a lab project, so you can't do that yet. That's the depressing part.

It can't be all sunshine and rainbows, though. There's a tradeoff for this kind of inference that I personally think won't affect the experience as much:

You can't really swap models. The model is stuck onto the chip once it's burned there. That's fine mostly, but the biggest problem becomes knowledge cutoff. Where the older the model gets and it's no longer keeping with new information, the more you'd have to buy a whole new chip with a new model burnt into it that's trained on modern data. But genuinely, do you think knowledge cutoff would be problematic if the model can reason over like half wikipidea in a few minutes? If the model can reason faster than your internet can load a website, I don't think keeping up with the world would be difficult. Worst case scenario, you'd have to buy a new chip every like 3-5 years. That's about when everyone gets a new phone.

I recommend reading this article too. It has a more in depth analysis of the whole thing, including manufacturing costs and power consumption for the chip (Spoiler: it makes classic GPUs look embarrassingly expensive, not just slow). Here:

https://theashishmaurya.medium.com/taalas-the-startup-that-prints-ai-models-directly-onto-silicon-33b181690575

This model is cool af!

Just a random thought I had, but it feels like it's only a matter of time before we start seeing paid inference engines with better optimizations, more features, and probably one-click setups that make everything easier to get running.

Do you think we'll see dedicated paid inference engines that offer a noticeably better experience than the current open-source options? Or will open-source projects always be able to keep up?

Curious to hear what you guys think.

Community benchmark from a single RTX 5080 16 GB on Linux (Fedora 44), Qwen3.8-Flash-Next **IQ3_XXS**, Strata **v0.2.1** (commit `90767252`; the bench report reads the version as 0.1.46), engine compiled locally (`./setup.sh --build`). Short version: **150 tok/s decode, ~4,400 tok/s prompt at 16K-32K**. Thank you for this project, it runs very well here.

| Test | Median (range) |

|---|---|

| chat decode | **150.0 tok/s** (126.7-170.2, n=9) |

| 4K prompt, fresh | **3,755 tok/s** (3,576-3,800), TTFT 1.08 s |

| 16K prompt, fresh | **4,396 tok/s** (4,380-4,401), TTFT 3.67 s |

| 32K prompt, fresh | **4,412 tok/s** (4,411-4,419), TTFT 7.29 s |

| agent session | cache hit 69.5 %, turn TTFT 0.97 s (0.76-1.13), decode 150.3 tok/s (124.3-240.4) |

### Hardware

- **GPU:** RTX 5080 16 GB (sm_120), driver 615.78.08, PCIe 5.0 x16. Tuned with LACT: memory clock +6000 (offset), core capped at 2850 MHz @ 900 mV, 360 W power limit. Not stock, so stock cards may be a bit slower. (Inspired from the mVolt reddit posts I've been seeing)

- **CPU:** Intel Core Ultra 9 285K (8P + 16E, AVX2/AVX-VNNI, no AVX-512). Setup picked 15 CPU expert workers.

- **RAM:** 64 GB (2 x 32 GB Kingston Fury KF560C36-32) DDR5 at 6400 MT/s, 61.7 GiB visible.

- **Storage:** GGUF shards (including the n-gram table) on a Kingston Fury Renegade G5 2 TB (SFYR2S2T0, PCIe 5.0 x4 NVMe), btrfs with NOCOW on the model folder.

- **OS:** Fedora 44, kernel 7.2.11, CUDA 13.4.92, GCC 16.2.1 default with g++-15 used for the build (see below).

- **Background:** normal GNOME desktop session. No other GPU workloads, Monitor Connects to intel iGPU and the 5080 is used for compute only (No graphcis)

### How I installed and ran it

1. Downloaded the GGUFs at the pinned revision (IQ3_XXS shards 1+2 and the mmproj) with aria2c, and the MTP layer with `tools/mtp_fetch.py fetch` + `verify`. Both shards' SHA-256 match the values pinned in `setup.py`.

2. Built and installed, without starting the server:

   ```sh

   export CC=gcc-15 CXX=g++-15 # see the build note below

   ./setup.sh --build --model IQ3_XXS --gguf-dir <folder with both shards> --vision no --no-start --no-browser --yes

   ```

   Setup's choices: 64K context (its recommendation for 16 GB), int8 KV, MTP on, images off. KV streaming was turned **off** because RAM was tight (~60 GB used of 62).

3. Benchmarked with `./bench.sh --long`, which started the server from the install's config:

   ```text

   strata --pack iq3_xxs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp rt --max-context 65536 --kv int8 --pool-workers 15

   ```

   On load, the engine filled the expert cache with 5,310 experts (8.61 GiB of VRAM).

### Two notes from the install

- **Build on Fedora 44 (GCC 16 default, CUDA 13.4):** setup correctly picks `g++-15` as the CUDA host compiler, but the plain C++ files are compiled with the default GCC 16. CMake then adds GCC 15's implicit link directory (`-L/usr/lib/gcc/x86_64-redhat-linux/15`) to the final link, which resolves GCC 15's `libstdc++`, and the link fails on the GCC 16 objects:

```text

libstrata_prefill.a(prefill.cpp.o): undefined reference to `std::__detail::__notify_impl(void const*, bool, std::__detail::__wait_args_base const&)'

... undefined reference to `std::__detail::__wait_impl(void const*, std::__detail::__wait_args_base&)'

collect2: error: ld returned 1 exit status

```

Building everything with the same compiler (`CC=gcc-15 CXX=g++-15`, fresh `build/`) fixes it. Suggestion: when setup falls back to an older host compiler for CUDA, it could set `CMAKE_CXX_COMPILER`/`CMAKE_C_COMPILER` to the same one. This is related to #2576 but not the same failure.

- FYI for anyone else on the v0.2.1 tag: `./bench.sh` still hits the `NameError: name 'port' is not defined` from #2864/#2908 there. It's already fixed in v0.2.2 and main; I patched it locally the same way.

<details><summary>Full report generated by <code>tools/strata_bench.py --long</code></summary>

# Community benchmark on NVIDIA GeForce RTX 5080

Measured on 2026-10-21 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

## Hardware and software

- GPU and VRAM: NVIDIA GeForce RTX 5080, 16 GiB VRAM

- CPU: Intel(R) Core(TM) Ultra 9 285K (24 cores, 24 threads); installed RAM: 64GB DDR5 6400; storage: 2TB Kingston SSD PCIe gen5

- OS: Linux 7.2.11-200.fc44.x86_64; GPU driver: 615.78.08

- Strata: project 0.1.46, commit 90767252; engine 0.1.46

- Background workloads and power limits: not recorded

## Model and configuration

- Model and quantization: qwen3.8-flash-next-iq3_xxs

- Context: 65536 tokens; parallel slots: 1

- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

```text

strata --pack iq3_xxs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp rt --max-context 65536 --kv int8 --pool-workers 15

```

## Method

Run by `python tools/strata_bench.py --long` against the HTTP API of a Strata server (started by the benchmark from the install configuration); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs

- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each

- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`)

- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

## Results

| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |

| --- | ---: | ---: | ---: | ---: | --- | --- | --- |

| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 150.0 (126.7-170.2) | 0.23 (0.20-0.25) |

| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 3755.0 (3575.9-3800.1) | not measured | 1.08 (1.07-1.14) |

| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 4396.2 (4380.3-4400.5) | not measured | 3.67 (3.67-3.68) |

| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 4411.6 (4411.3-4419.0) | not measured | 7.29 (7.27-7.29) |

| agent session, follow-up turns | 14691 (14662-14725) at the last turn | see below | up to 64 | 3 | see below | 150.3 (124.3-240.4) | 0.97 (0.76-1.13) |

Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token.

### Agent session (conversation cache)

Cache hit over the follow-up turns: 69.5 % (69.4-69.5); earlier-turn tokens read again per session: 4 (3-4) (earlier turns that the cache did not hold; the 13826 (13825-13827) new tokens of the files are read once); first turn (nothing cached) first token: 0.56 s (0.56-0.59).

| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s |

| ---: | ---: | ---: | ---: | ---: | ---: | ---: |

| 0 | 609 | 0 | 609 | 1057.7 | 0.588 | 113.0 |

| 1 | 4086 | 624 | 3462 | 3110.2 | 1.128 | 150.1 |

| 2 | 5619 | 4149 | 1470 | 1838.9 | 0.812 | 150.3 |

| 3 | 9052 | 5683 | 3369 | 3022.1 | 1.129 | 149.7 |

| 4 | 11809 | 9101 | 2708 | 2795.2 | 0.983 | 240.4 |

| 5 | 14662 | 11843 | 2819 | 2933.1 | 0.975 | 128.3 |

Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server).

## Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 69 s.

</details>

Are they(ds) having problems like deepmind had with gemini 3.5/4 pro/argon? They released two flash models already, but no pro model.. The first iteration of 4.1 pro probably was not better than 10/17 flash by more than 1 point or even worse or the same. I hope it is more than 1 -2 points better than flash at AA.

Strata is the new hotness - claims of 4x tok/s increase over llama.cpp. I havent seen any posts talking about performance on Apple silicon however. Curious if anyone's run comparisons on Apple Silicon against llama.cpp and MLX

Hey,

I hear a lot about new agent harnesses on here and am wondering if someone could give me some ideas how they might benefit me vs my current setup. I've done a ton of work on my setup to make it work as I like, so I'd need it to be worthwhile to jump, but if there's good reason to I may take the plunge.

I'm currently on openclaw 4.23 (the release names, at least used to be, = release date) so older build, I run a conversational agent and spend a lot of time tinkering with memory systems. I'm on Qwen 3.8 27b currently and have it setup on a 64 gig mac studio m2 max and communicate with my agent entirely through telegram.

I have

• a multi-tiered nightly diary system
• overnight world news and events read prompts through heartbeat
• scripts that cache in a portion of that diary and some of the writing portion of the overnight reads overwriting the recent stuff to provide directionality
• local img gen connectivity as well as embedding agent/vector db
• a wake whenever script so my agent can, for example, gen a picture, tell itself to come back in 5 minutes to retrieve and send, or to check in after an interview or something going on in my life
• externalized heartbeats that run on a d20 roll system
• telegram emulation (for lack of a better word) to ensure cache matches when the agent gets a message from its scripts or heartbeat
• ~50k system prompt soul file full of tools and examples for how to use them along with other pertinent info

For a setup like mine, any ideas where a different harness would really improve stuff? Thanks!

have you experienced any related issues like bandwidth or something else? Excluding the fact that it cost way cheaper than ddr4 rams. If there's anything else id love to hear

Organized by The American Chamber of Commerce in Lithuania. Event link: https://www.amcham.lt/events/foreign-investors-annual-summit-2026/

AI was a topic, maybe 3rd if ranked by coverage time. And local AI was also given coverage in Antonio di Marzo (Head of applied AI EU at EPAM Systems) presentation! Some points: 1) in the second picture, "hiring" agents meant in part having them independent from the cloud API; 2) requirement for good engineering = engineers was acknowledged as must. Best practices were covered quite well, translated to managerial language with all precision possible.

There were more presentations mentioning AI briefly, but not local AI.

Real business people, investors and managers are not like LinkedIn bubble at all. They understand the use cases of local AI, although they see it as requirement not so often, depending on their business model (unlike medical professionals, I wrote about it in the Lithuanian AI in Medicine Conference post).

From my quite unrepresentative survey among businesses' representatives, 1 bank adopted local AI, 1 law firm considers it but are yet happy with "current tool" (and I guess it is Claude) - they told only larger law firms win on local AI adoption, and their greatest incentive to use local AI is... regulatory reasons. That was a bit counterintuitive, even if it looks simple in hindsight, but regulatory requirements are often a driver of local AI adoption, see also the post mentioned above.

Business/SaaS use local AI or certificated/compliant closed AI models (which they mostly trust, some audited real deployments in data centers) depending on the use case; their greater "pain point" is when a provider decides to "sunset" a model API, which breaks their prompts and agents.

Not so local, but hot topic: SaaS feels some pressure due to vibe-coding in-house solutions, but doesn't see it as critical. A leader of SaaS clearly stated that it is misconception that agents will replace software engineers, and that they have demand for more competent engineers.

Forgive me for a too brief and maybe imprecise coverage.

P.S. My opinion is decision models (e.g. Jev) are hype, but not only people who talked about AI on the stage, but even researchers buy it, so maybe I am wrong here.

Hi everyone!

I'm the developer of llampart, an open-source Web UI for chatting with LLMs. I've just released v2.0.1, and since version 1.9.0 the project has changed quite a bit, so I wanted to share what I've been working on.

For anyone seeing it for the first time, llampart is a self-hosted, desktop-focused chat interface built with Svelte 5 and SvelteKit. It started from the llama.cpp / llama-ui foundation and has gradually grown into its own project so different from original base.

The idea is pretty simple: I wanted a comfortable interface for working with local models, with good conversation management, useful tools, plenty of customization, and no unnecessary cloud services and most of all in my own language interface.

What does llampart do?

It connects to an LLM backend that you run separately. It currently supports llama-server (llama.cpp), vLLM, and OpenAI-compatible API servers.

Your conversations are stored in your browser using IndexedDB, and most application settings are stored locally as well. There's no llampart account or hosted conversation service. Of course, prompts are still sent to whichever backend you configure, whether that's on your own machine, your LAN, or a remote server.

Some of the features include:

• Frosted Glass - a translucent interface with bundled or custom wallpapers, adjustable transparency, UI scaling, typography settings, and a customizable conversation sidebar.
• Conversation management - branching conversations, pinned chats, search, import/export, and local conversation history, 2 colums sidebar
• Reasoning and tool calling - support for reasoning-capable models, tool execution, and multi-step agent workflows.
• MCP integration - connecting MCP servers and working with their tools, prompts, and resources.
• Attachments and Markdown - file handling, PDF support, syntax-highlighted code blocks, tables, and content export.
• Six interface languages - English, Polish, German, Spanish, French, and Italian.+ more to come

What's new since 1.9.0?

The biggest change is that llampart is no longer focused only on llama-server. I've worked on making the application understand the differences between providers instead of treating every OpenAI-compatible API as identical.

This includes separate provider configurations, model selection, support for the appropriate Chat Completions or Responses protocol where available, and better handling of model capabilities.

Reasoning support has also been improved. The available reasoning controls depend on the selected model and provider, so the interface can avoid sending settings that the backend doesn't support.

Another major addition is AI-generated charts and diagrams directly inside chat messages.

Models can produce interactive Line, Area, Bar, and Donut charts using a structured JSON format. Charts can be previewed and exported as PNG, SVG, or CSV.

There's also Mermaid support for Flowcharts, State Diagrams, Mindmaps, and Gantt charts, with previews and PNG/SVG export.

I've spent quite a lot of time improving how these are displayed — especially chart labels, diagram layouts, and making exports useful outside the application.

There are also new response preferences, which let you guide how models format their answers. For example, you can ask for more concise responses, practical examples, tables, charts, diagrams, or fewer decorative elements without manually repeating these instructions every time.

A lot of work has gone into things that are less visible in screenshots: conversation data integrity, streaming and cancellation, model switching during active requests, MCP permissions, tool result persistence, and safer handling of asynchronous operations.

Installation

llampart is a Web UI, not a model runner. You'll still need your own LLM backend - lama-server (llama.cpp), vLLM, and OpenAI-compatible API servers .

For Linux, there's an installer for Ubuntu-based and Arch-based systems. It uses an existing Caddy installation to serve the prebuilt WebUI and handle the API proxy. You don't need Node.js to run the installed release.

You can also clone the repository and run the SvelteKit development server if you prefer.

The interface is currently designed primarily for desktop use, and backend features naturally depend on what your model server actually supports.

The project is MIT-licensed and available here:

GitHub: https://github.com/mchowy-troll/llampart

Latest release (2.0.1): https://github.com/mchowy-troll/llampart/releases/tag/v2.0.1

I'm developing this because it's something I wanted to use myself, and I'm continuing to improve it as I run into issues or find things that could work better.

I'd really appreciate feedback, especially from people using different local models.

Thanks for taking a look!

I've noticed that a lot of software is being cloned using Rust nowadays, and I was wondering why not use it with local LLMs for simple tasks too.

Do you think it's better to use Rust or Python when coding with a local LLM?

Nowadays everyone is vibe coding, but local LLMs are generally not that good, especially smaller models running on low-budget hardware.

I usually code in Python, but sometimes I feel like the model doesn't catch bugs, so I was wondering if Rust and its compiler could help avoid some of these issues.

I'd also be interested in hearing your experiences with different context window sizes and models running on 12 or 24 GB of VRAM, with 64 or 128 GB of system RAM.

Do you think Rust's compiler feedback is worth the extra complexity for smaller local models, or is Python still the better choice?

This model looks really impressive.

TL;DR -- Coding agent model. 40B params with 2.5B active. Resoundingly beats Qwen3.6-35B-A3B in TerminalBench 2.1 and SWE-bench Pro v2 (see HF repo for table), ties or betters it in the other benchmarks they chose to show. 3:1 SWA:full attention. 262k context.

I think it might finally dethrone the venerable Qwen3.6-35B-A3B and move the needle on coding agents for the GPU-poor. What do you guys think? I am downloading now and will report back when I try it out, benchmarks to come in the next few days.

Recursive's blog post. WIRED Magazine coverage ("scary good"). PR for llama.cpp. Unsloth GGUFs.

EDIT: Initial reports suggest Ornith-1.5 remains the standard in this class of models. Hopefully it's just early days implementation glitches, but it's not looking as rosy as I hoped...

Really couldn't find any good existing quantization for this, only uncensored editions for the GSQ-RCO versions, so I made one. Strata is pre configured for the unsloth dynamic iq4_xs build. Sorry the model card is so cornbally

This is the full precision swift 1.5 with the heretic method applied quantized down to iq4_xs. The tensor profile has been extracted from the regular UD-IQ4_XS gguf from qwen 3.8 flash next, and applied to this quant. The model was quantized with an imatrix covering math, code, english language, general instruct, and tools, generated for this model against a small custom dataset.

Just IQ4_XS right now because that's what I needed but feel free to request new formats. Copying unsloth quants is pretty easy.

https://huggingface.co/trubisky/Swift-1.5-Qwen-3.8-Flash-Next-Heretic-UD-IQ4_XS

The model card has a pretty noticeable result: 76.9 on SWE-bench Pro, compared with 43.9 for Qwen3.8-Flash-Next in their evaluation. On Terminal-Bench 2.1 it's slightly behind, at 81.2 vs 83.4. These are the team's own results, but the difference between those two comparisons is interesting.

It's a MoE reasoning model with tool calling and a 256K context window. There's an Ollama release already; the Q4_K_M package is around 77GB, before runtime memory overhead.

Their training approach involves generating tasks, the agent setups used to solve them, and solutions, then using those for reinforcement learning. I'm curious how much of that shows up when you give it an unfamiliar repo and your own tools.

Has anyone put it through a longer coding task? Mostly interested in whether it needs less babysitting: recovering from a failed tool call, checking its changes, and actually finishing the job. What would make you switch from whichever local model you're using now?

Hey everyone! 🚀

For the past few months, I've been building and open-sourcing **Lumina Capital (v6.0.0)**—a fully autonomous multi-agent quantitative hedge fund engine that you can run locally, designed to bridge qualitative LLM reasoning with deterministic quant computation.

Unlike toy setups that feed raw charts directly into LLMs (causing massive hallucinations and latency), Lumina strictly separates probabilistic reasoning from deterministic execution:

### 🛠️ Key Highlights:

- **7-Persona Anti-Sycophancy Council (LangGraph)**: Personas (Buffett, Soros, Burry, Cathie Wood, etc.) conduct independent parallel analysis, followed by an adversarial bull-vs-bear cross-debate. A dedicated Chief Risk Officer (CRO) agent has absolute veto power over tail risks.

- **Pure Python Qlib Alpha158 + LightGBM**: Complete vectorized implementation of Microsoft Qlib's 158 quantitative factors for local stock screening.

- **10-Second High-Speed Sentinel Thread**: A deterministic background thread executing immediate physical exits/trailing stops without waiting for LLM inference loops.

- **Google TimesFM**: 5-day probabilistic forecasting fan charts with 80% confidence bands and crash probability.

- **Works Out-of-the-Box**: CPU-only and lightweight local execution supported via Streamlit dashboard, containerized with 1-click Docker Compose.

Available for research, education, and simulation under CC BY-NC 4.0.

🔗 **GitHub**: https://github.com/ramuza0113-sketch/lumina-ai-hedge-fund

Would love to hear your thoughts, feedback, or ideas on the local agent architecture!

I’ve seen vastly more questions than answers about how people are using their local AI so I figured I’d make a post about my experience. It seems most people are looking for frontier-level coding expertise for huge projects with their setups, and that anything less is not enough. I disagree and don’t really care about that stuff.

TL;DR: I used a frontier model to build the infrastructure for my local agents to persistently do tedious but outstandingly useful work on my massive collection of books, art, reports, photos, and scans. While building that infrastructure, I ensured that I received full guides, docs, and user-friendly options for designating / switching / changing local AI tasks so I’m not reaching back to the frontier agent now that it’s done. I use these agents primarily for processing hundreds of thousands of pieces of information and generating connections / curiosities about my terabytes of human-generated content so I can decide what’s worth digging into (or just briefly visiting) myself.

BACKGROUND: I’ll be the first to admit I am not a coder, I am not a programmer, nor am I an expert in bash for Linux operation. I’m not a silicon valley darling, aspiring entrepreneur, or someone who only cares about LLM roleplay like I see a lot around here (no shade though, live your life!). But I’ll say that I am relatively knowledgeable compared to the general population given the amount of curiosity-based fiddling I’ve done over the years.

My second admission is this: I use frontier AI for the infrastructure and first-take tasking for my local models. Yup - I’m not 100% local.

Here’s the rub: I have found local models to be solidly reliable for tool usage and basic computer tasks, but not major coding projects (of which I have very few). With that said, I don’t really care about my local models’ coding ability beyond file manipulation, tool usage, and troubleshooting. But I’m not a coder, so I have a hard time creating deterministic systems for my agents to work within - that’s where the frontier AI comes in handy.

So with that background out of the way, how do I use my local AI?

THE GOOD PART: First, it was Hermes Agent. Tons of fun. I had an agent (local Gemma 4 31B Q4 on an RTX 5090) set up that did a phenomenal job with general tasking. I had Signal chat so I could reach it from anywhere, set cron jobs, have it pull information from files on my desktop, research via SearXNG with MY guidance in mind, and generally maintain persistent / private understanding of my life with no outside data tracking whatsoever (aside from Signal messages, which are supposedly encrypted, and what I provided via those agent SearXNG searches or browser visits, which are supposedly private).

THE BETTER PART: I decided to start over, just for fun. So I decided to build out the ultimate NAS home library using frontier AI. I now have an amazing frontier-custom-built library including wikipedia, a personal google earth with points of interest representing wikipedia articles associated with geo coords, a “google earth” for the moon and mars with associated points of interest, a stackexchange replication, hundreds of thousands of books/reports, magazines, a gallery of high quality historical photographs, and a gallery of museum-quality art on my NAS.

With that, I have a team of 3 local agents (2x Strix Halo boxes running GPT-OSS 120B Q4, 1x RTX 5090 running Gemma 31B Q4) reading my terabytes-worth of material and pulling out images from books, quotes, curious or interesting factoids, and linkages between works. I’ve essentially offloaded the tedious part of working through hundreds of thousands of books to my agents, and I get the benefit of a tailored way to determine what I want to read / enjoy more of. I also have one of those agents on ready-standby (as in, working 24/7 but prioritizing user-level chats) to answer any questions about the library (or find relevant books, quotes, or links) or just talk.

I’ve essentially created my own internet within the walls of my home, and it’s not algorithmically fucking me and selling my attention. There are no dark patterns, ads, FOMO, or short form attention-span-reducing videos. I want to spend time in my own digital space, and it’s one of the best things I’ve done for myself. My only cost was not even the full amount of frontier tokens allocated to a $20 subscription, and the obvious provision of my project information to the frontier company who built it.

Let me know if you have any questions.

pplx-decider-v1.2 is one tenth the costs of GPT 6 Luna (low reasoning) while making identical quality decisions, and pplx-decider-flash-v1.2 idem ditto compare to 6 Luna in the Decisions API. v1.2 is also 2.5x cheaper than Jev.

- https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index

Both pplx-decider models are open source (Apache license 2.0):

- https://huggingface.co/perplexity-ai/pplx-decider-v1.2
- https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2

I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000.

With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it working at 7.2 tok/s.

The answer has a physics error in it (wondering if someone can spot it), and I don't see much practical use for it. Qwen models actually work pretty well on my setup. Still, it felt really nice to see a 748B parameter model running on such hardware.

Here are some technical details:

• Unsloth UD-IQ1_S GGUF, 233 GB on disk
• llama.cpp built from the open PR #30817, CUDA 12.8
• The 110 GB Engram table isn't loaded up front. It stays on the NVMe and only the rows the model needs are read on demand, which is how it fits.

llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf \
  -ngl 99 --n-cpu-moe 36 -ot "per_layer_token_embd.weight=CPU" \
  -lm mmap -lzm on -t 10 -c 16384 --jinja \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
  --chat-template-kwargs '{"enable_thinking":false}'

Now I only need to decide whether to keep the model or free up the disk space :)

I saw this today on X and decided to try it, I havent seen it posted here yet. I ran it on a single 3090 benchmarked it with qwen 3.8 27B before and after the optimization and saw these results:

Observed prompt-processing throughput improved 22–30%; generation speed improved 1–5%. All tests passed without GPU errors.

Edit: Keep in mind performance gains can be higher, my 3090 is power limited to 230 watts including during my tests

I don't have a 3090 to confirm, but curious if the anyone in this community can back this up. Good win for 3090 and curious if that translates to other GPUs. I have a long vague memory of some hacks done to prevent mining demand that could be fixed and help LLMs. I wonder if this was one that lived on?

Disclaimer: I am not related nor do i know the original person's post. I do not have multiple accounts. Not everything is a conspiracy.

Hi!

I'm the creator of HF Viewer, and we are today launching embedl.com!

It's a collection of 1000+ models optimized to run fast on different edge devices like NVIDIA Orin Nano Super, AGX Thor, AGX Orin, Orin NX and RTX GPUs.

You can run the models directly on real edge devices through our free device farm, with detailed profiling and results! (As well as on your own device)

I'm really excited about this release and I believe we are the first to do an edge device farm!

Let me know what you think, any feedback is appreciated!

I always feel a little bad asking LLMs to do research into itself or newer versions. It's like asking someone to face their own mortality, although that's anthropomorphizing.

The user is asking me to investigate the system requirements for "Qwen 3.8 Flash Next" from at least 20 different sources. This sounds like a model that may not exist — "Qwen 3.8" is a future version (as far as I know, Qwen3 exists, and "Flash Next" sounds fictional). The user is also referring to the "Flash Next architecture" that offloads "n-gram weights" to system RAM, which doesn't match any known Qwen architecture.

Later reasoning

Interesting — this model exists in this environment (a fictional future model, Qwen 3.8 Flash Next).

If you were wondering, I was asking whether I should even bother trying QFN on R9700 + 32 GB system ram:

Yes — it will run, but only because ~27 GB of the file is the n-gram lookup table, which you put on your NVMe. The 1-bit UD-IQ1_S is 67.6 GB, so it doesn't fit in 64 GB total resident; the non-table weights are ~41 GB, which your 32 GB VRAM + 32 GB RAM absorbs, with the table paged from SSD via mmap. Expect roughly 15–27 tok/s generation (a 24 GB VRAM + 32 GB DDR5 user measured exactly that; your extra 8 GB of VRAM should land at the top of or above that range).

So, y'all, 1-bit lobotomized model or q4 3.8 27B?